Many thanks for the review! I’ll keep your perspectives in mind over the next few months as we ramp up our next round of Replication Games.
Here to confirm much of what was said is true. Always happy to talk about how to improve the process.
Completely agree with the challenges of group dynamics. It is hard to know ex ante how teams will interact. Right now we try and keep people who program in the same language, have similar research interests, and are at different stages of their career (graduate student, postdoc, professors, etc.). We are at the whim of who signs up and do the best with the volunteers.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment?
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help?
Less important but clarifying:
Papers are not screened for influence; they are screened for data, code and likelihood to reproduce as indicated from a README (the selection is mainly on journals who provide a codebase and allow reproduction)
I4R can handle food, building and practicalities (it just depends on the event but happens more often at conferences)
I4R tries to give at least a year off before returning to the same city.
The event participated in was joint with another institute which lead to some complications (I only booked my flight 5 days before the event not really having expected to go).
Many thanks for the review! I’ll keep your perspectives in mind over the next few months as we ramp up our next round of Replication Games.
Here to confirm much of what was said is true. Always happy to talk about how to improve the process.
Completely agree with the challenges of group dynamics. It is hard to know ex ante how teams will interact. Right now we try and keep people who program in the same language, have similar research interests, and are at different stages of their career (graduate student, postdoc, professors, etc.). We are at the whim of who signs up and do the best with the volunteers.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment?
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help?
Less important but clarifying:
Papers are not screened for influence; they are screened for data, code and likelihood to reproduce as indicated from a README (the selection is mainly on journals who provide a codebase and allow reproduction)
I4R can handle food, building and practicalities (it just depends on the event but happens more often at conferences)
I4R tries to give at least a year off before returning to the same city.
The event participated in was joint with another institute which lead to some complications (I only booked my flight 5 days before the event not really having expected to go).
Happy Replicating!