Thanks for posting it here and for your work at OWID!
Do you have any thoughts on how to scale RCTs to larger, messier projects? By now, the EA community has more resources at their hands and the results for small RCTs might not scale to larger interventions. Have you thought of ways in which RCTs could still be leveraged for large-scale interventions or are they just too hard to make work, e.g. on the policy level?
I wish I could answer this better, but I don’t know enough to have a good answer to how to scale policy RCTs, especially since they’re quite different from clinical RCTs (they often can’t administer the treatment in a standardised way, there’s usually no way to blind participants to what they’re receiving, they usually don’t track/measure participants as regularly, etc.) Though those are also factors that make them messier in larger projects.
But I think there are often situations where they can be leveraged for large-scale interventions. A good recent example is this experiment on street lighting and its effect in reducing crime. There are some features of the policy make it easier to study at scale. Crime data exists at the right scale (you don’t need to track individual participants to find out about crime rates), streetlighting is easy to standardise, you can measure the effects at the level of neighbourhood clusters rather than at the level of individuals. So maybe that’s a good way of thinking about how to scale up RCTs—to find treatments and outcomes that are easier to implement and measure at a large scale.
Is there a paper by him you would recommend reading on the topic? I’ve seen this one, which I agree with in parts – with good theory and evidence from other research on which policies work, there’s less need for RCTs, but I think there’s a role for both to answer different questions.
most of his blogs for centre for global development are relevant. His recent paper on “randomizing development: method or madness?” contains most of his main arguments. He also has a blog called Lantrant where he frequently criticises the use of RCTs in economics. In my view, almost all of his critiques are correct.
Thanks for posting it here and for your work at OWID!
Do you have any thoughts on how to scale RCTs to larger, messier projects? By now, the EA community has more resources at their hands and the results for small RCTs might not scale to larger interventions.
Have you thought of ways in which RCTs could still be leveraged for large-scale interventions or are they just too hard to make work, e.g. on the policy level?
Hey Marius, thank you!
I wish I could answer this better, but I don’t know enough to have a good answer to how to scale policy RCTs, especially since they’re quite different from clinical RCTs (they often can’t administer the treatment in a standardised way, there’s usually no way to blind participants to what they’re receiving, they usually don’t track/measure participants as regularly, etc.) Though those are also factors that make them messier in larger projects.
I’ve read this blog post by Michael Clemens, which I found was a useful summary of two books on the topic: https://cgdev.org/blog/scaling-programs-effectively-two-new-books-potential-pitfalls-and-tools-avoid-them
But I think there are often situations where they can be leveraged for large-scale interventions. A good recent example is this experiment on street lighting and its effect in reducing crime. There are some features of the policy make it easier to study at scale. Crime data exists at the right scale (you don’t need to track individual participants to find out about crime rates), streetlighting is easy to standardise, you can measure the effects at the level of neighbourhood clusters rather than at the level of individuals. So maybe that’s a good way of thinking about how to scale up RCTs—to find treatments and outcomes that are easier to implement and measure at a large scale.
I enjoyed your post a lot. Lant Pritchett is a prominent critic of using RCTs for large scale social interventions—he might be worth reading.
Thank you very much!
Is there a paper by him you would recommend reading on the topic? I’ve seen this one, which I agree with in parts – with good theory and evidence from other research on which policies work, there’s less need for RCTs, but I think there’s a role for both to answer different questions.
most of his blogs for centre for global development are relevant. His recent paper on “randomizing development: method or madness?” contains most of his main arguments. He also has a blog called Lantrant where he frequently criticises the use of RCTs in economics. In my view, almost all of his critiques are correct.