Thanks so much Rory and for the links to your earlier post and the USAID stuff!
I think your criticism is a good criticism of RCTs in general, but it seems to me more a criticism comment about RCT design then being a clear argument against comparing with cash transfers. RCTs on development NEED longer term outcome measurement, and surely need at a minimum 2 data points at 2 different times after the study. And of course the most important data point is after many months or even many years as you talked about in your article.
I’m not at all sure about the ethical side either . Medical RCTs compare a new trial treatment against the most up-to-date treatment—not so much because we worry about “tricking” a patient like you say (there are still plenty of RCTs with sugar placebo pills which is deemed ethically OK), we are still OK with a kind of ‘deception’. What we AREN’T OK with is doing a trial where we give the control arm nothing at all, when we know there is a better option than nothing for the medical condition. And I’d argue that cash is usually a better option than nothing for many development conditions.
That’s a great and sobering point about the counterfactual potentially being inaction if cash transfers won the day. Why should the counterfactual be inaction though? I would hope as development people we are good enough that if Cash was equivalent or better than intervention X, this wouldn’t lead us not to inaction but instead to give more cash instead. Maybe I’m naive and idealistic though, and maybe you’re right that there is actually a practical advantage in seeing a positive impact of intervention X, even if it is worse than a cash transfer. I don’t think that should be the case though.
That’s the whole question really—should we spend our millions on RCTs asking “What is the impact of X”, or “Is X better than cash”. What we really want to know, the practical question which underlies the research question is is “Should we be implementing this intervention at scale”. I’d argue that to answer that, the question vs. Cash is the one that matters more.
Thanks so much for your reply, I can see you’ve thought about this far more than me and I loved your original post—weird that searches on the forum didn’t bring it up, maybe they should employ google search on the site haha.
Thanks so much Rory and for the links to your earlier post and the USAID stuff!
I think your criticism is a good criticism of RCTs in general, but it seems to me more a criticism comment about RCT design then being a clear argument against comparing with cash transfers. RCTs on development NEED longer term outcome measurement, and surely need at a minimum 2 data points at 2 different times after the study. And of course the most important data point is after many months or even many years as you talked about in your article.
I’m not at all sure about the ethical side either . Medical RCTs compare a new trial treatment against the most up-to-date treatment—not so much because we worry about “tricking” a patient like you say (there are still plenty of RCTs with sugar placebo pills which is deemed ethically OK), we are still OK with a kind of ‘deception’. What we AREN’T OK with is doing a trial where we give the control arm nothing at all, when we know there is a better option than nothing for the medical condition. And I’d argue that cash is usually a better option than nothing for many development conditions.
That’s a great and sobering point about the counterfactual potentially being inaction if cash transfers won the day. Why should the counterfactual be inaction though? I would hope as development people we are good enough that if Cash was equivalent or better than intervention X, this wouldn’t lead us not to inaction but instead to give more cash instead. Maybe I’m naive and idealistic though, and maybe you’re right that there is actually a practical advantage in seeing a positive impact of intervention X, even if it is worse than a cash transfer. I don’t think that should be the case though.
That’s the whole question really—should we spend our millions on RCTs asking “What is the impact of X”, or “Is X better than cash”. What we really want to know, the practical question which underlies the research question is is “Should we be implementing this intervention at scale”. I’d argue that to answer that, the question vs. Cash is the one that matters more.
Thanks so much for your reply, I can see you’ve thought about this far more than me and I loved your original post—weird that searches on the forum didn’t bring it up, maybe they should employ google search on the site haha.