I would welcome a blog post about RCTs, and if you decide to write one, I hope you consider the perspective below.
As far as I can tell ~0% of nonprofits are interested in rigorously studying their programs in any way, RCTs or otherwise, and I can’t help but suspect that this is largely because mostly when we do run RCTs we find that these cherished programs have ~no effect. It’s not at all surprising to me that most charities that conduct RCTs feel pressured to do so by donors; but on the other hand basically all charity activities ultimately flow from donor preferences, because donors are the ones with most of the power.
Living Goods is one interesting example, where they ran an RCT because a donor demanded it, got an unexpected (positive) result, and basically pivoted the whole charity based on that. I view that as a success story.
I am certainly not claiming that RCTs are appropriate for all kinds of programs, or some kind of silver bullet. It’s more like, if you ask charities “would you like more or less accountability for results”, the answer is almost always going to be, “less, thanks”.
This is a great point. There’s an important distinction though between evaluating new programs led by early-stage NGOs (like those coming from Charity Entrepreneurship) versus established programs directing millions in funding. I think RCTs make sense for the latter group.
As far as I can tell ~0% of nonprofits are interested in rigorously studying their programs in any way, RCTs or otherwise
There’s also a difference between the typical NGOs and EA-founded ones. In my experience, EA founders actively want to rigorously evaluate their programs, they don’t want to work for ineffective interventions.
I would welcome a blog post about RCTs, and if you decide to write one, I hope you consider the perspective below.
As far as I can tell ~0% of nonprofits are interested in rigorously studying their programs in any way, RCTs or otherwise, and I can’t help but suspect that this is largely because mostly when we do run RCTs we find that these cherished programs have ~no effect. It’s not at all surprising to me that most charities that conduct RCTs feel pressured to do so by donors; but on the other hand basically all charity activities ultimately flow from donor preferences, because donors are the ones with most of the power.
Living Goods is one interesting example, where they ran an RCT because a donor demanded it, got an unexpected (positive) result, and basically pivoted the whole charity based on that. I view that as a success story.
I am certainly not claiming that RCTs are appropriate for all kinds of programs, or some kind of silver bullet. It’s more like, if you ask charities “would you like more or less accountability for results”, the answer is almost always going to be, “less, thanks”.
This is a great point. There’s an important distinction though between evaluating new programs led by early-stage NGOs (like those coming from Charity Entrepreneurship) versus established programs directing millions in funding. I think RCTs make sense for the latter group.
There’s also a difference between the typical NGOs and EA-founded ones. In my experience, EA founders actively want to rigorously evaluate their programs, they don’t want to work for ineffective interventions.