Would also love this. I think a useful contrast will be A/B testing in big tech firms. My amateur understanding is big tech firms can and should run hundreds of “RCTs” because:
No need to acquire subjects.
Minimal disruption to business since you only need to siphon off a minuscule portion of your user base.
Tech experiments can finish in days while field experiments need at least a few weeks and sometimes years.
If we assume treatments are heavy tailed, then a big tech firm running hundreds of A/B tests is more likely to learn of a weird trick that grows the business when compared to a NGO who may only get one shot.
Yes, exactly. The marginal cost of an A/B test in tech is incredibly low, while for NGOs an RCT represents a significant portion of their budget and operational capacity.
This difference in costs explains why tech can use A/B tests for iterative learning, trying hundreds of small variations, while NGOs need to be much more selective about what to test.
And despite A/B testing being nearly free, most decisions at big tech firms aren’t driven by experimental evidence.
Would also love this. I think a useful contrast will be A/B testing in big tech firms. My amateur understanding is big tech firms can and should run hundreds of “RCTs” because:
No need to acquire subjects.
Minimal disruption to business since you only need to siphon off a minuscule portion of your user base.
Tech experiments can finish in days while field experiments need at least a few weeks and sometimes years.
If we assume treatments are heavy tailed, then a big tech firm running hundreds of A/B tests is more likely to learn of a weird trick that grows the business when compared to a NGO who may only get one shot.
Yes, exactly. The marginal cost of an A/B test in tech is incredibly low, while for NGOs an RCT represents a significant portion of their budget and operational capacity.
This difference in costs explains why tech can use A/B tests for iterative learning, trying hundreds of small variations, while NGOs need to be much more selective about what to test.
And despite A/B testing being nearly free, most decisions at big tech firms aren’t driven by experimental evidence.