Many thanks for the review! I’ll keep your perspectives in mind over the next few months as we ramp up our next round of Replication Games.
Here to confirm much of what was said is true. Always happy to talk about how to improve the process.
Completely agree with the challenges of group dynamics. It is hard to know ex ante how teams will interact. Right now we try and keep people who program in the same language, have similar research interests, and are at different stages of their career (graduate student, postdoc, professors, etc.). We are at the whim of who signs up and do the best with the volunteers.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment?
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help?
Less important but clarifying:
Papers are not screened for influence; they are screened for data, code and likelihood to reproduce as indicated from a README (the selection is mainly on journals who provide a codebase and allow reproduction)
I4R can handle food, building and practicalities (it just depends on the event but happens more often at conferences)
I4R tries to give at least a year off before returning to the same city.
The event participated in was joint with another institute which lead to some complications (I only booked my flight 5 days before the event not really having expected to go).
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment? It will certainly give participants a better flavour of commitment. I think that this will lead to more early dropouts, so I have no idea whether this will be a sensible policy. I also suspect most will deem it unnecessary and overly paternalistic. Teams that take it seriously will meet up themselves.
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help? Yes, definitely! One week is pretty short, but I think it would create a deadline (you have to have something baked together and prepared) that was lacking.
Also: I think that two attempts at replication games is not a good sample size (especially if the success and fun is not normally distributed), so I will attend another one. See you around!
Many thanks for the review! I’ll keep your perspectives in mind over the next few months as we ramp up our next round of Replication Games.
Here to confirm much of what was said is true. Always happy to talk about how to improve the process.
Completely agree with the challenges of group dynamics. It is hard to know ex ante how teams will interact. Right now we try and keep people who program in the same language, have similar research interests, and are at different stages of their career (graduate student, postdoc, professors, etc.). We are at the whim of who signs up and do the best with the volunteers.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment?
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help?
Less important but clarifying:
Papers are not screened for influence; they are screened for data, code and likelihood to reproduce as indicated from a README (the selection is mainly on journals who provide a codebase and allow reproduction)
I4R can handle food, building and practicalities (it just depends on the event but happens more often at conferences)
I4R tries to give at least a year off before returning to the same city.
The event participated in was joint with another institute which lead to some complications (I only booked my flight 5 days before the event not really having expected to go).
Happy Replicating!
Hi Derek,
thanks for your kind reply.
Do you think setting a day to talk with teams one-on-one before the event would help mitigate some of the randomness associated with team formation and get a better flavour of commitment? It will certainly give participants a better flavour of commitment. I think that this will lead to more early dropouts, so I have no idea whether this will be a sensible policy. I also suspect most will deem it unnecessary and overly paternalistic. Teams that take it seriously will meet up themselves.
One thing I’m piloting to improve follow up is setting a day to talk with teams about report progression, say a week after an event. Do you think that would help? Yes, definitely! One week is pretty short, but I think it would create a deadline (you have to have something baked together and prepared) that was lacking.
Also: I think that two attempts at replication games is not a good sample size (especially if the success and fun is not normally distributed), so I will attend another one. See you around!