I have many years of experience with the Energy Efficiency policy analysis community where there is a very strong emphasis on impact evaluation. I also have a little experience seeing impact evaluations implemented in the international development community.
I see four reasons why impact evaluation is not as prevalent in EA as other professional communities:
(1) Impact evaluation can get very bureaucratized and lead to high administrative overhead expenses and organizational constraints which can make programs more costly to implement and less efficient in their operations;
(2) Impact evaluation if it is to be quantitative needs clear metrics to measure performance, and this requires consensus on goals and theories of change. While EAs have an open mind, they also have a fairly large diversity of opinions on the details of which specific near-term goals are highest priority. This diversity of opinion, makes it difficult to make a singular theory of change and makes it difficult to select clear performance metrics for many projects (because of the lack of theory of change).
(3) Much of EA is focused on providing a low-overhead service to charitable donors so that their donations can have the most cost-effective impact. Since donors also have a diversity of opinions with respect to specific objectives and goals, the focus is on cost-effectiveness calculations and transparency using a variety of metrics. Thus impact is viewed from the perspective of a diversity of donors who may have different impact goals. Thus in practice impact is often evaluated in terms of “funds directed” which is the response that donors have in terms of the EA movement meeting their image of good cost-effectiveness.
(4) Maximizing marginal cost-effectiveness may be better and more efficient than doing impact evaluations: If there is a supply of “world improving opportunities” and a demand of “wanting to improve the world”, then maximizing the marginal cost-effectiveness of “wanting to improve the world” purchases (i.e. EA donations) should maximize the net surplus value produced from “world improving activities” in general given constrained resourced available to those wanting to pay for a better world. This just the law of supply and demand applied to people donating to make a better world. Thus the EA approach of focusing on marginal cost-effectiveness may be better and more economically efficient than performing lots of impact evaluations of specific EA activities.
In short, the “impact evaluation” of the EA movement is in terms of maximizing marginal cost-effectiveness, maximizing the number of people involved and maximizing the funds directed to the causes that meet maximum marginal cost-effectiveness criteria. This may not require a special impact evaluation effort with some specific, consensus “theory of change,” but it may simply require a bit of movement statistics collection and compilation to track the cost effectiveness, the amount of money directed, and the number of people participating in the EA movement.
On 2) I think this shouldn’t be a barrier, since the first step in impact evaluation is establishing a theory of change, and any individual organisation of a given maturity should have a clear picture of this, and consensus on their goals. (I’m not advocating for a singular ToC across the whole EA movement, just for individual organisations)
On 3) I agree this is a factor for evaluations for funders. But I think organisations should carry out internal impact evaluation according to their theory of change, so I think should still be able to have clear and established goals. They can say to funders “Our Theory of Change is X, and along those lines our impact has been Y”. Funders might also carry out their own evaluations in terms of the goals they most care about.
On 4), I agree there’s a trade-off between spending on direct activities and impact evaluation. Of course the value of impact evaluation is that it may improve the effectiveness of the direct activities. So I don’t think the existence of the trade-off is in itself a reason that impact evaluation shouldn’t be prevalent.
I have many years of experience with the Energy Efficiency policy analysis community where there is a very strong emphasis on impact evaluation. I also have a little experience seeing impact evaluations implemented in the international development community.
I see four reasons why impact evaluation is not as prevalent in EA as other professional communities:
(1) Impact evaluation can get very bureaucratized and lead to high administrative overhead expenses and organizational constraints which can make programs more costly to implement and less efficient in their operations;
(2) Impact evaluation if it is to be quantitative needs clear metrics to measure performance, and this requires consensus on goals and theories of change. While EAs have an open mind, they also have a fairly large diversity of opinions on the details of which specific near-term goals are highest priority. This diversity of opinion, makes it difficult to make a singular theory of change and makes it difficult to select clear performance metrics for many projects (because of the lack of theory of change).
(3) Much of EA is focused on providing a low-overhead service to charitable donors so that their donations can have the most cost-effective impact. Since donors also have a diversity of opinions with respect to specific objectives and goals, the focus is on cost-effectiveness calculations and transparency using a variety of metrics. Thus impact is viewed from the perspective of a diversity of donors who may have different impact goals. Thus in practice impact is often evaluated in terms of “funds directed” which is the response that donors have in terms of the EA movement meeting their image of good cost-effectiveness.
(4) Maximizing marginal cost-effectiveness may be better and more efficient than doing impact evaluations: If there is a supply of “world improving opportunities” and a demand of “wanting to improve the world”, then maximizing the marginal cost-effectiveness of “wanting to improve the world” purchases (i.e. EA donations) should maximize the net surplus value produced from “world improving activities” in general given constrained resourced available to those wanting to pay for a better world. This just the law of supply and demand applied to people donating to make a better world. Thus the EA approach of focusing on marginal cost-effectiveness may be better and more economically efficient than performing lots of impact evaluations of specific EA activities.
In short, the “impact evaluation” of the EA movement is in terms of maximizing marginal cost-effectiveness, maximizing the number of people involved and maximizing the funds directed to the causes that meet maximum marginal cost-effectiveness criteria. This may not require a special impact evaluation effort with some specific, consensus “theory of change,” but it may simply require a bit of movement statistics collection and compilation to track the cost effectiveness, the amount of money directed, and the number of people participating in the EA movement.
Thanks a lot Robert!
On 1) great point, I agree.
On 2) I think this shouldn’t be a barrier, since the first step in impact evaluation is establishing a theory of change, and any individual organisation of a given maturity should have a clear picture of this, and consensus on their goals. (I’m not advocating for a singular ToC across the whole EA movement, just for individual organisations)
On 3) I agree this is a factor for evaluations for funders. But I think organisations should carry out internal impact evaluation according to their theory of change, so I think should still be able to have clear and established goals. They can say to funders “Our Theory of Change is X, and along those lines our impact has been Y”. Funders might also carry out their own evaluations in terms of the goals they most care about.
On 4), I agree there’s a trade-off between spending on direct activities and impact evaluation. Of course the value of impact evaluation is that it may improve the effectiveness of the direct activities. So I don’t think the existence of the trade-off is in itself a reason that impact evaluation shouldn’t be prevalent.