Thank you for writing this Mitra, it’s always valuable to hear critiques of current approaches in the EA community. As Peter noted above, your experiences and views would be greatly valued by the community.
I will attempt to respond to some of these questions, but note that my responses may not reflect the views of everyone in the community, and I may miss some crucial points.
Are you effective enough to notice that you could be 10x more effective if instead of selling wells to villages, you focused resources of finding and supporting local entrepreneurs to build their own businesses doing so ?
I think many EAs share this concern. You may be interested in this post, which is a critque of the current “EA” approach to global poverty.
Is your altruism effective enough to notice who is building them for $2200?
In the global poverty space, GiveWell aims to find charities who produce the most bang for buck. Of course, they may get this wrong sometimes in practice. But in theory, if someone achieves the same benefit for less cost, GiveWell would prioritise this opportunity.
Have you measured the return on impact well enough to know that $100 is the cost of the measurement, that it mostly collects meaningless numbers, and you could dig 5% more wells if you eliminated that?
I’d be interested to hear why you think measurement numbers are meaningless? Take the case of malaria control—if some areas are much more malarious than others, it seems important to spend some money to know which areas to focus on.
Have you noticed that as many as half the wells, or solar panels or toilets sit there broken waiting for the next donor? Are you measuring the right thing, would you notice if those $2000 wells fail in 5 years while $2500 wells might fail in 10 ?
GiveWell’s cost-effectiveness analyses look at costs per treated person, so they try to account for situations where some of the treatments/materials are not used. They also account for the length of time a treatment lasts, which may resolve the second question.
Are you innovative enough to figure out that if you got the village to invest $1000, not only could you support twice as many villages but the village would be more likely to maintain it and use it, if they had skin in the game. Are you flexible enough to create a social enterprise rather than a charity, and to fund its overheads rather than expecting it to make a profit?
If you’re interested, this interview with Karen Levy of Evidence Action has sections on both “participatoriness” and “sustainability”. The issue may be too complex to cover here, but it would be valuable to understand the crux of your disagreement.
Are you innovative enough to invest in someone developing a cheaper machine, or one that reduces the cost even further, or are you demanding certainty and measurability too much to consider them anything that pays off at scale, but in the long term.
This is not precisely what you described, but Charity Entrepeneuship aims to find innovative solutions to challenging problems, such as the Lead Elimination Exposure Project, which advocates for lead paint regulation to reduce health and economic damage in the long term.
Hopefully this didn’t come across as dismissive, but I think it’s worth giving due credit to GiveWell and other members of the EA community.
I should have numbered the points, would make it easier to reference !
I agree with the critique in that post you linked—RCTs will rarely show you what intervention will be catalytic and change things in the long run, it will systematically under-prioritize high-risk, and ignore approaches that require refining to get to scale. This is why I don’t pay much attention to GiveWell, though it might catch the first kind of saving (a cheaper well) but GiveWell’s methodology—like those of most other charity evaluators—is designed to ignore most of the other questions that determine effectiveness. For example their RCT won’t be back in 10 years to see if the wells are still working.
The rest of that post you refer to is about economic arguments (direct versus indirect poverty elimination) - from what I’ve seen it is probably only partially correct (and I haven’t had time to read the whole article) , GDP is lousy measure of “happiness” and the GDP/capita measure also ignores HOW that wealth is spread, inequality not only means that a rich country can have a lot of very poor (e.g. in the US) but inequality is itself a significant cause of unhappiness. Reasonable people could hold reasoned positions on different sides of that argument so I don’t want to dive too deep.
In terms of impact measurement, I argue that it mostly collects meaningless numbers based on experience in the field, the measurement is typically designed to gather the numbers the donors want, not the numbers that actually reflect an increase in effectiveness. More importantly collecting and reporting data is a significant cost, often as much as 5-10% of the total budget, and that means 5-10% of the budget is diverted from creating impact to measuring it. Any effective organization is gathering data, but is gathering data that help it determine whether what its doing is effective, so for example it might just take a small time-bound sample—enough to know it should change something, and then measure something different, this is never enough to provide statistics to “impact measurement” hungry donors. Of course, even after asking many times, including publicly at conferences, I have yet to see Impact Measurement Consultants measure their own Effectiveness i.e. the amount of increased impact they generate compared to the decrease in impact from diversion of resources to its measurement.
In terms of “participatoriness”, Karen may be correct in some contexts, about services that people have a “Right” to, but there are many counter examples, and in particular a “right” to a service, doesn’t mean you have any likelihood of receiving it, which is why philanthropy or development is needed. For example I did a (unscientific) survey looking at health clinics starting about an hour outside Lusaka. Every clinic had some kind of solar system, more than half were non-operational (which goes back to my earlier point about design), more importantly , the clinics weren’t doing anything to fix them, they were waiting for someone else to come along and give them another one. If they’d spent even a small amount of money to pay for that system then they’d have looked for help repairing it, or for example in figuring out that one always-on porch light was using about a third of the entire systems capacity.
In terms of Social Enterprises—my point is not principally about what you call “sustainability”, its about leverage, if you are smart enough to fund the creation of a business, that can, for example supply clean water at a price people can afford, then your donation is essentially unbounded because an entity that makes 1c per intervention is infinitely scalable while one that costs 1c per intervention only scales as a function of the donor dollars. Such Social Enterprises need investment to get them to the scale where the economics work—they rarely return enough to satisfy a Venture Capitalist, and without investment they will never get to the point where they look good in impact/spend, but if you measure future impact against the investment to reach scale then I bet many SEs are far more Effective than almost any charity.
On the last point, Its good to see something like Charity Entrepreneurship—but I’d take it one step further, a lot of challenges in global poverty are that a solution exists in the west, but is far too expensive to universally deploy in a poorer country—this is especially true for medical devices. Developing cost effective solutions is very often possible—I’ve seen many in my work—but I’m asking whether EA would be Effective enough to see that investment (in the R&D to put together the solution) as more effective than continuing to supply the more expensive western solution to a very few recipients.
Thank you for writing this Mitra, it’s always valuable to hear critiques of current approaches in the EA community. As Peter noted above, your experiences and views would be greatly valued by the community.
I will attempt to respond to some of these questions, but note that my responses may not reflect the views of everyone in the community, and I may miss some crucial points.
Are you effective enough to notice that you could be 10x more effective if instead of selling wells to villages, you focused resources of finding and supporting local entrepreneurs to build their own businesses doing so ?
I think many EAs share this concern. You may be interested in this post, which is a critque of the current “EA” approach to global poverty.
Is your altruism effective enough to notice who is building them for $2200?
In the global poverty space, GiveWell aims to find charities who produce the most bang for buck. Of course, they may get this wrong sometimes in practice. But in theory, if someone achieves the same benefit for less cost, GiveWell would prioritise this opportunity.
Have you measured the return on impact well enough to know that $100 is the cost of the measurement, that it mostly collects meaningless numbers, and you could dig 5% more wells if you eliminated that?
I’d be interested to hear why you think measurement numbers are meaningless? Take the case of malaria control—if some areas are much more malarious than others, it seems important to spend some money to know which areas to focus on.
Have you noticed that as many as half the wells, or solar panels or toilets sit there broken waiting for the next donor? Are you measuring the right thing, would you notice if those $2000 wells fail in 5 years while $2500 wells might fail in 10 ?
GiveWell’s cost-effectiveness analyses look at costs per treated person, so they try to account for situations where some of the treatments/materials are not used. They also account for the length of time a treatment lasts, which may resolve the second question.
Are you innovative enough to figure out that if you got the village to invest $1000, not only could you support twice as many villages but the village would be more likely to maintain it and use it, if they had skin in the game. Are you flexible enough to create a social enterprise rather than a charity, and to fund its overheads rather than expecting it to make a profit?
If you’re interested, this interview with Karen Levy of Evidence Action has sections on both “participatoriness” and “sustainability”. The issue may be too complex to cover here, but it would be valuable to understand the crux of your disagreement.
Are you innovative enough to invest in someone developing a cheaper machine, or one that reduces the cost even further, or are you demanding certainty and measurability too much to consider them anything that pays off at scale, but in the long term.
This is not precisely what you described, but Charity Entrepeneuship aims to find innovative solutions to challenging problems, such as the Lead Elimination Exposure Project, which advocates for lead paint regulation to reduce health and economic damage in the long term.
Hopefully this didn’t come across as dismissive, but I think it’s worth giving due credit to GiveWell and other members of the EA community.
All the best,
Lucas
Thanks Lucas
I should have numbered the points, would make it easier to reference !
I agree with the critique in that post you linked—RCTs will rarely show you what intervention will be catalytic and change things in the long run, it will systematically under-prioritize high-risk, and ignore approaches that require refining to get to scale. This is why I don’t pay much attention to GiveWell, though it might catch the first kind of saving (a cheaper well) but GiveWell’s methodology—like those of most other charity evaluators—is designed to ignore most of the other questions that determine effectiveness. For example their RCT won’t be back in 10 years to see if the wells are still working.
The rest of that post you refer to is about economic arguments (direct versus indirect poverty elimination) - from what I’ve seen it is probably only partially correct (and I haven’t had time to read the whole article) , GDP is lousy measure of “happiness” and the GDP/capita measure also ignores HOW that wealth is spread, inequality not only means that a rich country can have a lot of very poor (e.g. in the US) but inequality is itself a significant cause of unhappiness. Reasonable people could hold reasoned positions on different sides of that argument so I don’t want to dive too deep.
In terms of impact measurement, I argue that it mostly collects meaningless numbers based on experience in the field, the measurement is typically designed to gather the numbers the donors want, not the numbers that actually reflect an increase in effectiveness. More importantly collecting and reporting data is a significant cost, often as much as 5-10% of the total budget, and that means 5-10% of the budget is diverted from creating impact to measuring it. Any effective organization is gathering data, but is gathering data that help it determine whether what its doing is effective, so for example it might just take a small time-bound sample—enough to know it should change something, and then measure something different, this is never enough to provide statistics to “impact measurement” hungry donors. Of course, even after asking many times, including publicly at conferences, I have yet to see Impact Measurement Consultants measure their own Effectiveness i.e. the amount of increased impact they generate compared to the decrease in impact from diversion of resources to its measurement.
In terms of “participatoriness”, Karen may be correct in some contexts, about services that people have a “Right” to, but there are many counter examples, and in particular a “right” to a service, doesn’t mean you have any likelihood of receiving it, which is why philanthropy or development is needed. For example I did a (unscientific) survey looking at health clinics starting about an hour outside Lusaka. Every clinic had some kind of solar system, more than half were non-operational (which goes back to my earlier point about design), more importantly , the clinics weren’t doing anything to fix them, they were waiting for someone else to come along and give them another one. If they’d spent even a small amount of money to pay for that system then they’d have looked for help repairing it, or for example in figuring out that one always-on porch light was using about a third of the entire systems capacity.
In terms of Social Enterprises—my point is not principally about what you call “sustainability”, its about leverage, if you are smart enough to fund the creation of a business, that can, for example supply clean water at a price people can afford, then your donation is essentially unbounded because an entity that makes 1c per intervention is infinitely scalable while one that costs 1c per intervention only scales as a function of the donor dollars. Such Social Enterprises need investment to get them to the scale where the economics work—they rarely return enough to satisfy a Venture Capitalist, and without investment they will never get to the point where they look good in impact/spend, but if you measure future impact against the investment to reach scale then I bet many SEs are far more Effective than almost any charity.
On the last point, Its good to see something like Charity Entrepreneurship—but I’d take it one step further, a lot of challenges in global poverty are that a solution exists in the west, but is far too expensive to universally deploy in a poorer country—this is especially true for medical devices. Developing cost effective solutions is very often possible—I’ve seen many in my work—but I’m asking whether EA would be Effective enough to see that investment (in the R&D to put together the solution) as more effective than continuing to supply the more expensive western solution to a very few recipients.
Mitra