Executive summary: Replication Games are a promising concept for scaling social-science replication, but in the author’s experience they underperformed due to fixable organizational and incentive issues—especially weak team dynamics, poor matching, and lack of follow-through.
Key points:
The author argues that replication is a resource-efficient way to test influential findings and that structured programs like the Institute for Replication are a “fruitful way forward.”
The Replication Games format—short-term, team-based hackathons with limited future interaction—exacerbates free-rider problems and weak accountability.
In the author’s first game, poor team matching (e.g., unfamiliar software and lack of coordination) led them to drop out and produce no replication.
In the second game, uneven preparation and an assignment error (duplicate paper across teams) made much of the team’s work redundant.
The author reports that other participants expressed similar frustrations, suggesting these issues may not be isolated.
The author suggests stronger incentives for follow-through (e.g., selective future participation) and better coordination, while noting risks like incentivizing adversarial replication practices.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: Replication Games are a promising concept for scaling social-science replication, but in the author’s experience they underperformed due to fixable organizational and incentive issues—especially weak team dynamics, poor matching, and lack of follow-through.
Key points:
The author argues that replication is a resource-efficient way to test influential findings and that structured programs like the Institute for Replication are a “fruitful way forward.”
The Replication Games format—short-term, team-based hackathons with limited future interaction—exacerbates free-rider problems and weak accountability.
In the author’s first game, poor team matching (e.g., unfamiliar software and lack of coordination) led them to drop out and produce no replication.
In the second game, uneven preparation and an assignment error (duplicate paper across teams) made much of the team’s work redundant.
The author reports that other participants expressed similar frustrations, suggesting these issues may not be isolated.
The author suggests stronger incentives for follow-through (e.g., selective future participation) and better coordination, while noting risks like incentivizing adversarial replication practices.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.