Make RCTs cheaper: smaller treatment, bigger control groups

Epistemic status: I think this is a statistical “fact” but I feel a bit cautious since so few people seem to take advantage of it

Summary

It may not always be optimal for cost or statistical power to have equal-sized treatment/​control groups in a study. When your intervention is quite expensive relative to data collection, you can maximise statistical power or save costs by using a larger control group and smaller treatment group. The optimal ratio of treatment sample to control sample is just the square root of the cost per treatment participant divided by the square root of the cost per control participant.

Why larger control groups seem better

Studies generally have equal numbers of treatment and control participants. This makes intuitive sense: a study with 500 treatment and 500 control will be more powerful than a study with 499 treatment and 501 control, for example. This is due to the diminishing power returns to increasing your sample size: the extra person removed from one arm hurts your power more than the extra person added to the other arm increases it.

But what if your intervention is expensive relative to data collection? Perhaps you are studying a $720 cash transfer and it costs $80 to complete each survey, for a total cost of $800 per treatment participant ($720 + $80) and $80 per control. Now, for the same cost as 500 treatment and 500 control, you could have 499 treatment and 510 control, or 450 treatment and 1000 control: up to a point, the loss in precision from the smaller treatment is more than offset by the 10x larger increase in your control group, resulting in a more powerful study overall. In other words: when your treatment is expensive, it is generally more powerful to have a larger control group, because it’s just so much cheaper to add control participants.

How much larger? The exact ratio of treatment:control that optimises statistical power is surprisingly simple, it’s just the ratio of the square roots of the costs of adding to each arm i.e. sqrt(control_cost) : sqrt(treatment_cost) (See Appendix for justification). For example, if adding an extra treatment participant costs 16x more than adding a control participant, you should optimally have sqrt(16/​1) = 4x as many control as treatment.


Quantifying the benefits

With this approach, you either get free extra power for the same money or save money without losing power. For example, let’s look at the hypothetical cash transfer study above with treatment participants costing $800 and control participants $80. The optimal ratio of control to treatment is then sqrt(800/​80) = 3.2 :1, resulting in either:

Saving money without losing power: the study is currently powered to measure an effect of 0.175 SD and, with 500 treatment and control, costs $440,000. With a 3.2 : 1 ratio (*types furiously in Stata*) you could achieve the same power with a sample of 337 treatment and 1079 control, which would cost $356,000: saving you a cool $84k without any loss of statistical power.

Getting extra power for the same budget: alternatively, if you still want to spend the full $440k, you could then afford 416 treatment and 1,331 control, cutting your detectable effect from 0.175 SD to 0.155 SD at no extra cost.

Caveats

Ethics: there may be ethical reasons for not wanting a larger control group, for example in a medical trial where you would be denying potentially life-saving treatments to sick patients. Even outside of medicine, control participants’ time is important and you may wish to avoid “wasting” it on participating in your study (although you could use some of the savings to compensate control participants, if that won’t mess with your study).

Necessarily limited samples: obviously if there is a practical limit to increasing your control group size, such as only being able to operate in a limited geography, this may not be an option.

Natural skepticism? This isn’t a common technique, you might just trust that the market for ideas is efficient and if this really was a thing you would have heard about it from somewhere else by now. It kind of blows my mind that this isn’t done more often, which both makes me want to tell people about it and be skeptical. We used this approach for a pretty large RCT I worked on in Tanzania, and no one complained.

Conclusion

If you treatment is quite expensive relative to data collection costs, consider using a larger control group in the ratio of sqrt(treatment_cost/​control_cost) and enjoy that spare money or additional statistical power.

Appendix

I am not claiming to have discovered this myself. I first read this equation in Running Randomized Evaluations and was able to derive the same result myself here.

I believe this holds for cluster RCTs, just remember that the increased control sample here would come in the form of additional control clusters, rather than larger clusters.

If you are doing power calculations in Stata and want to factor in different treatment/​control group sizes, you just add ratio(X) to the sampsi command, where “X” is the treatment/​control ratio. For a cluster RCT using clustersampsi you… need to do something involving harmonic means, I forget exactly, but poke me on the Forum and I’ll happily dig through some old code.