I assumed more people were aware of this. I’m using it in a trial we’re about to start. But as others have said, in many trials the treatment is not particularly more costly. But probably a factor in detailed interventions in poverty and health in poor countries. Have you looked into how many studies in development economics and GH&D with costly interventions do this?
As a quick data point I just checked the 6 RCTs GiveDirectly list on their website. I figure cash is pretty expensive so it’s the kind of intervention where this makes sense.
It looks like most cash studies, certainly with just 1 treatment arm, aren’t optimising for cost:
Study
Control
Treatment
The short-term impact of unconditional cash transfers to the poor: experimental evidence from Kenya
432
503
BENCHMARKING A CHILD NUTRITION PROGRAM AGAINST CASH: EVIDENCE FROM RWANDA
74 villages
74 villages (nutrition program) 100 (cash)
Cash crop: evaluating large cash transfers to coffee farming communities in Uganda
1894
1894
Using Household Grants to Benchmark the Cost Effectiveness of a USAID Workforce Readiness Program
488
485 NGO program 762 cash 203 cash + NGO
General equilibrium effects of cash transfers: experimental evidence from Kenya
325 villages
328 villages
Effects of a Universal Basic Income during the pandemic
100 villages
44 longterm UBI 80 shortterm UBI 71 lump sum
Suggests either 1) there’s some value in sharing this idea more or 2) there’s a good reason these economists aren’t making this adjustment. Someone on Twitter suggested “problems caused by unbalanced samples and heteroskedasticity” but that was beyond my poor epidemiologist’s understanding and they didn’t clarify further.
Unbalanced samples are not a problem per se. You can run into a problem of representation/generalization for the smaller sample but this argument is independent of balancing and only has to do with small sample sizes.
@david_reinstein made an excellent point about heteroscedasticity / variance. To factor this into your original post: You want to optimize the cost-effectiveness of the precision of your group-level difference score. This is achieved by minimizing the standard errors (SE) of the group-level estimates of each sample, which are just the standard deviations (SD) divided by the square root of the respective observations. So your term would expand to: Control-to-treat-ratio = sqrt(treatment_cost/control_cost) * control_SD/treatment_SD. The problem, in practice, is that you usually know the costs a priori but not the SDs. If variances are not equal, however, I would agree with @david_reinstein that the treatment group will more likely show greater variance on your outcome variable (if control group has more variance, I would rather reconsider the choice of the outcome variable).
If you want to read more about the concept of precision and its relation to statistical power (also cf. the paper that @Karthik Tadepalli cited), we just put together a preprint here that is supposed to double as a teaching ressource: https://doi.org/10.31234/osf.io/m8c4k (introduction and discussion will suffice since the middle part focusses on biological/neuroscientific measurements that have vastly different properties than, e.g., questionnaire data). Here is the glossary that is mentioned in the paper: https://osf.io/2wjc4 And here is the associated Twitter post with some digest about the most important insights: https://twitter.com/bioDGPs_DGPA/status/1616014732254756865
I assumed more people were aware of this. I’m using it in a trial we’re about to start. But as others have said, in many trials the treatment is not particularly more costly. But probably a factor in detailed interventions in poverty and health in poor countries. Have you looked into how many studies in development economics and GH&D with costly interventions do this?
As a quick data point I just checked the 6 RCTs GiveDirectly list on their website. I figure cash is pretty expensive so it’s the kind of intervention where this makes sense.
It looks like most cash studies, certainly with just 1 treatment arm, aren’t optimising for cost:
AGAINST CASH: EVIDENCE FROM RWANDA
100 (cash)
farming communities in Uganda
USAID Workforce Readiness Program
762 cash
203 cash + NGO
experimental evidence from Kenya
80 shortterm UBI
71 lump sum
Suggests either 1) there’s some value in sharing this idea more or 2) there’s a good reason these economists aren’t making this adjustment. Someone on Twitter suggested “problems caused by unbalanced samples and heteroskedasticity” but that was beyond my poor epidemiologist’s understanding and they didn’t clarify further.
The “problems caused by unbalanced samples” doesn’t seem coherent to me; I’m not sure what they are talking about.
If the underlying variance is different between the treatment and the control group:
That might justify a larger sample for the group with larger variance
But I would expect the expected variance to tend to be larger for the treatment group in many/most relevant cases
Overall, there will still tend to be some efficiency advantage of having more of the less-costly group, generally the control group
Unbalanced samples are not a problem per se. You can run into a problem of representation/generalization for the smaller sample but this argument is independent of balancing and only has to do with small sample sizes.
@david_reinstein made an excellent point about heteroscedasticity / variance. To factor this into your original post: You want to optimize the cost-effectiveness of the precision of your group-level difference score. This is achieved by minimizing the standard errors (SE) of the group-level estimates of each sample, which are just the standard deviations (SD) divided by the square root of the respective observations. So your term would expand to:
Control-to-treat-ratio = sqrt(treatment_cost/control_cost) * control_SD/treatment_SD.
The problem, in practice, is that you usually know the costs a priori but not the SDs. If variances are not equal, however, I would agree with @david_reinstein that the treatment group will more likely show greater variance on your outcome variable (if control group has more variance, I would rather reconsider the choice of the outcome variable).
If you want to read more about the concept of precision and its relation to statistical power (also cf. the paper that @Karthik Tadepalli cited), we just put together a preprint here that is supposed to double as a teaching ressource: https://doi.org/10.31234/osf.io/m8c4k (introduction and discussion will suffice since the middle part focusses on biological/neuroscientific measurements that have vastly different properties than, e.g., questionnaire data).
Here is the glossary that is mentioned in the paper: https://osf.io/2wjc4
And here is the associated Twitter post with some digest about the most important insights: https://twitter.com/bioDGPs_DGPA/status/1616014732254756865