Thanks for writing this—I think it’s accessible, informative, and interesting, which is difficult to pull off when writing about research methods!
I think it’s telling that all the examples of the effectiveness of RCTs in this article come from clinical trials. However, you don’t limit yourself to this domain in the headline or summary of the article (e.g. “How would we know about the effects of a new idea, treatment or policy?”).
Our World in Data is often used by people (including myself) to gather development data. So I think it could be worth adding a caveat that many of the strengths you discuss in the article don’t apply to RCTs conducted on social programs or policies. For example, it’s difficult or impossible to have double-blinding or a placebo group; it’s difficult to randomize effectively due to spillover effects; it’s harder to get a large sample size when you’re studying effects on villages or countries; and generalization is far more difficult (while a drug that works for a Brazilian is likely to work for an Indonesian, but a policy that works in Brazil is unlikely to have the same effect in Indonesia).
I completely agree with you on the differences between clinical RCTs and development/public policy RCTs.
Part of the reason for that is that it was originally meant to be a longer piece, with some policy RCT examples, how clustering works, etc. but it was already fairly long, and those were harder to explain concisely. And secondly simply because I have a background in health/medicine, which meant it was easy to draw examples from the field.
Hopefully I signposted this a little by saying that the procedures I mention are those found in medicine / clinical RCTs, but from your comment I think it was probably not enough. I’ll think about this and clarify or add some caveats to the article that make it clearer. Thank you!
Thanks for writing this—I think it’s accessible, informative, and interesting, which is difficult to pull off when writing about research methods!
I think it’s telling that all the examples of the effectiveness of RCTs in this article come from clinical trials. However, you don’t limit yourself to this domain in the headline or summary of the article (e.g. “How would we know about the effects of a new idea, treatment or policy?”).
Our World in Data is often used by people (including myself) to gather development data. So I think it could be worth adding a caveat that many of the strengths you discuss in the article don’t apply to RCTs conducted on social programs or policies. For example, it’s difficult or impossible to have double-blinding or a placebo group; it’s difficult to randomize effectively due to spillover effects; it’s harder to get a large sample size when you’re studying effects on villages or countries; and generalization is far more difficult (while a drug that works for a Brazilian is likely to work for an Indonesian, but a policy that works in Brazil is unlikely to have the same effect in Indonesia).
Hey Stephen, thanks very much!
I completely agree with you on the differences between clinical RCTs and development/public policy RCTs.
Part of the reason for that is that it was originally meant to be a longer piece, with some policy RCT examples, how clustering works, etc. but it was already fairly long, and those were harder to explain concisely. And secondly simply because I have a background in health/medicine, which meant it was easy to draw examples from the field.
Hopefully I signposted this a little by saying that the procedures I mention are those found in medicine / clinical RCTs, but from your comment I think it was probably not enough. I’ll think about this and clarify or add some caveats to the article that make it clearer. Thank you!