As a small note, we might get more precise estimates of the effects of a program by predicting magnitudes rather than whether something will replicate (which is what we’re doing with the Social Science Prediction Platform). That said, I think a lot of work needs to be done before we can have trust in predictions, and there will always be a gap between how comfortable we are extrapolating to other things we could study vs. “unquantifiable” interventions.
(There’s an analogy to external validity here, where you can do more if you can assume the study you predict is drawn from the same set as those you have studied, or the same set if weighted in some way. You could in principle make an ordering of how feasible something is to be studied, and regress your ability to predict on that, but that would be incredibly noisy and not practical as things stand, and past some threshold you don’t observe studies anymore and have little to say without making strong assumptions about generalizing past that threshold.)
I’d note that the problem with predicting magnitudes is simply that it’s harder to do than predicting a binary “will it replicate,” though both are obviously valuable.
As a small note, we might get more precise estimates of the effects of a program by predicting magnitudes rather than whether something will replicate (which is what we’re doing with the Social Science Prediction Platform). That said, I think a lot of work needs to be done before we can have trust in predictions, and there will always be a gap between how comfortable we are extrapolating to other things we could study vs. “unquantifiable” interventions.
(There’s an analogy to external validity here, where you can do more if you can assume the study you predict is drawn from the same set as those you have studied, or the same set if weighted in some way. You could in principle make an ordering of how feasible something is to be studied, and regress your ability to predict on that, but that would be incredibly noisy and not practical as things stand, and past some threshold you don’t observe studies anymore and have little to say without making strong assumptions about generalizing past that threshold.)
Agreed on all points!
I’d note that the problem with predicting magnitudes is simply that it’s harder to do than predicting a binary “will it replicate,” though both are obviously valuable.