The idea makes a lot of sense, but my guess is that the circumstance where the cost is driven by the intervention itself isn’t that common: In the context of charities, we’re thinking about applying RCTs to test whether an intervention works. Generally the intervention is happening anyway. The cost of RCTs then doesn’t come from applying the intervention to the treatment group—it comes from establishing the experimental conditions where you have a randomised group of participants and the ability to collect data on them.
Hey Aidan—that’s a good point. I think it will probably apply to different extents for different cases, but probably not to all cases. Some scenarios I can imagine:
A charity uses its own funds to run an RCT of a program it already runs at scale:
In this case, you are right that treatment is happening “anyway” and in a sense the $ saved in having a smaller treatment group will just end up being spent on more “treatment”, just not in the RCT.
Even in this case I think the charity would prefer to fund its intervention in a non-RCT context: providing an intervention in an RCT context is inherently costlier than doing it under more normal circumstances, for example if you are delivering assets, your trucks have to drive past control villages to get to treatment ones, increasing delivery costs.
That’s pretty small though, I agree that otherwise the intervention is basically “already happening” and the effective savings are smaller than implied in my post
That said, if the charity has good reason to think their intervention works and so spending more on treatment is “good”, the value of the RCT in the first place seems lower to me
2) A charity uses its own funds to run an RCT of a trial program it doesn’t operate at scale:
In this case, the charity is running the RCT because it isn’t sure the intervention is a good one
Reducing the RCT treatment group frees up funds for the charity to spend on the programs that it does know work, with overall higher EV
3) A donor wants to fund RCTs to generate more evidence:
The donor is funding the RCT because they aren’t sure the intervention works
Keeping RCT costs lower means they can fund more RCTs, or more proven interventions
4) A charity applies for donor funds for an RCT of a new program:
In this case, the cheaper study is more likely to get funded, so the larger control/smaller treatment is a better option for the charity
Overall, I think cases 2/3/4 benefit from the cheaper study. Scenario 1 seems more like what you have in mind and is a good point, I just think there will be enough scenarios where the cheaper trial is useful, and in those cases the charity might consider this treatment/control optimisation.
Thanks Rory - I think your general idea is good, and in some cases could be a good option!
I could be wrong, but from my experience working in the development world these 4 scenarios aren’t really how RCTs generally happen. Usually there will be a partnership with a RCT running NGO (like IPA) or a university department (J-PAL at MIT) where the partner organisation pays for and organise everything.
Sometimes scenario 4 could happen as part of a grant application
This doesn’t change the existence of a budget constraint, though. The partner organization, especially a grant funder like JPAL/IPA, will grant you a certain amount of their resources to use. I don’t see why you wouldn’t want to optimize the use of their resources.
100% the original post stands, in any scenario we would want to optimise use of resources. I don’t think JPAL/IPA is generally a funder though—they do the research themselves so they are the ones to convince ;).
Ah, that’s helpful data. My experience in RCTs mostly comes from One Acre Fund, where we ran lots of RCTs internally on experimental programs, or just A/B tests, but that might not be very typical!
Would be super interested to see the results of some of these RCTs / AB tests. Were any of them published apart from the Lime SMS study? We’re looking for great examples of learning orgs that do this and some studies from 1AF would be a great motivator/example.
The idea makes a lot of sense, but my guess is that the circumstance where the cost is driven by the intervention itself isn’t that common: In the context of charities, we’re thinking about applying RCTs to test whether an intervention works. Generally the intervention is happening anyway. The cost of RCTs then doesn’t come from applying the intervention to the treatment group—it comes from establishing the experimental conditions where you have a randomised group of participants and the ability to collect data on them.
Hey Aidan—that’s a good point. I think it will probably apply to different extents for different cases, but probably not to all cases. Some scenarios I can imagine:
A charity uses its own funds to run an RCT of a program it already runs at scale:
In this case, you are right that treatment is happening “anyway” and in a sense the $ saved in having a smaller treatment group will just end up being spent on more “treatment”, just not in the RCT.
Even in this case I think the charity would prefer to fund its intervention in a non-RCT context: providing an intervention in an RCT context is inherently costlier than doing it under more normal circumstances, for example if you are delivering assets, your trucks have to drive past control villages to get to treatment ones, increasing delivery costs.
That’s pretty small though, I agree that otherwise the intervention is basically “already happening” and the effective savings are smaller than implied in my post
That said, if the charity has good reason to think their intervention works and so spending more on treatment is “good”, the value of the RCT in the first place seems lower to me
2) A charity uses its own funds to run an RCT of a trial program it doesn’t operate at scale:
In this case, the charity is running the RCT because it isn’t sure the intervention is a good one
Reducing the RCT treatment group frees up funds for the charity to spend on the programs that it does know work, with overall higher EV
3) A donor wants to fund RCTs to generate more evidence:
The donor is funding the RCT because they aren’t sure the intervention works
Keeping RCT costs lower means they can fund more RCTs, or more proven interventions
4) A charity applies for donor funds for an RCT of a new program:
In this case, the cheaper study is more likely to get funded, so the larger control/smaller treatment is a better option for the charity
Overall, I think cases 2/3/4 benefit from the cheaper study. Scenario 1 seems more like what you have in mind and is a good point, I just think there will be enough scenarios where the cheaper trial is useful, and in those cases the charity might consider this treatment/control optimisation.
Thanks Rory - I think your general idea is good, and in some cases could be a good option!
I could be wrong, but from my experience working in the development world these 4 scenarios aren’t really how RCTs generally happen. Usually there will be a partnership with a RCT running NGO (like IPA) or a university department (J-PAL at MIT) where the partner organisation pays for and organise everything.
Sometimes scenario 4 could happen as part of a grant application
This doesn’t change the existence of a budget constraint, though. The partner organization, especially a grant funder like JPAL/IPA, will grant you a certain amount of their resources to use. I don’t see why you wouldn’t want to optimize the use of their resources.
100% the original post stands, in any scenario we would want to optimise use of resources. I don’t think JPAL/IPA is generally a funder though—they do the research themselves so they are the ones to convince ;).
Ah, that’s helpful data. My experience in RCTs mostly comes from One Acre Fund, where we ran lots of RCTs internally on experimental programs, or just A/B tests, but that might not be very typical!
Would be super interested to see the results of some of these RCTs / AB tests. Were any of them published apart from the Lime SMS study? We’re looking for great examples of learning orgs that do this and some studies from 1AF would be a great motivator/example.