The Giving What We Can research team is excited to share the results of our 2024 round of evaluations of charity evaluators and grantmakers!
In this round, we completed three evaluations that will inform our donation recommendations for the 2024 giving season. As with our 2023 round, there are substantial limitations to these evaluations, but we nevertheless think that they are a significant improvement to a landscape in which there were no independent evaluations of evaluators’ work.
In this post, we share the key takeaways from each of our 2024 evaluations and link to the full reports. We also include an update explaining our decision to remove The Humane League from our list of recommended programs. Our website has now been updated to reflect the new fund and charity recommendations that came out of these evaluations.
Founders Pledge Global Health and Development Fund (FP GHDF)
Based on our evaluation, we have decided not to currently include FP GHDF in our list of recommended charities and funds and do not plan to allocate a portion of GWWC’s Global Health and Wellbeing Fund budget to the FP GHDF at this point. However, we still think FP GHDF is worth considering for impact-focused donors and we will continue to host the program on the GWWC donation platform.
Some takeaways that inform this decision include:
Our belief that FP GHDF evaluations do not robustly demonstrate that the opportunities they fund exceed their stated bar of 10x GiveDirectly in expectation, due to the presence of errors and insufficiently justified subjective inputs we found in some of their BOTECs that could cause the estimated cost-effectiveness to fall below this threshold.
Our uncertainty about the future quality of evaluations following the recent departure of the senior researcher responsible for managing the fund and our belief that there are currently insufficient peer review processes, such as red-teaming, in place to reassure us that the fund quality will remain at least consistent following this transition.
Taken together, these issues left us unable to justifiably conclude that the FP GHDF is currently competitive with GiveWell’s recommendations.
While our decision is not to recommend FP GHDF at this time, we would like to emphasise that we did not conclude that the marginal cost-effectiveness of the GHDF is unambiguously not competitive with GiveWell’s recommended charities and funds — in fact, we think the GHDF might be competitive with GiveWell now or in the near future. Instead, our finding is that we can’t currently justifiably conclude that the fund is competitive with GiveWell in expectation, based on the evidence we have seen in our evaluation and the uncertainty surrounding staffing changes.
We also highlight positive findings from our evaluation. For example, we think that FP GHDF’s strategy of funding promising early-stage organisations plausibly has significant positive externalities, and the fact that FP GHDF has made several grants to organisations before GiveWell indicates they have some track record for identifying these promising funding opportunities.
For more information, please see our 2024 evaluation report for FP GHDF.
Animal welfare
Animal Charity Evaluators’ Movement Grants (ACE MG)
The key finding informing this decision is that ACE MG’s recent marginal grants and overall grant decision-making look to be of sufficiently high quality to be competitive with our current top recommendation in animal welfare: EA Funds’ Animal Welfare Fund (EA AWF) — by our evaluation of EA AWF in 2023. This is in large part due to our view that ACE MG has improved meaningfully since our previous evaluation, when we concluded that ACE MG performed slightly less strongly on several proxies for the marginal cost-effectiveness of its grants. In particular, we noted improvements in the quality of ACE MG’s:
Marginal grantmaking
Weighting of geographical neglectedness
Approach to grant selection and sizing
We did find what we think to be room for improvement in ACE MG’s approach to more clearly integrating scope, particularly when comparing between applicants implementing different types of interventions. However, we don’t think this room for improvement affects the ACE MG’s position as being — to our knowledge — among the best places to recommend to donors.
We do not believe that our investigation identified ACE MG to be justifiably better or worse than EA AWF in terms of marginal cost-effectiveness — based on our 2023 evaluation of EA AWF — and so we continue to recommend EA AWF as a donation option alongside ACE MG, and to allocate half of our Effective Animal Advocacy Fund to EA AWF.
For more information, please see our 2024 evaluation report for ACE MG.
Animal Charity Evaluators’ Charity Evaluation Program
Some takeaways that informed our decision include:
While ACE has made improvements to its approach for capturing scope and cost-effectiveness since our 2023 evaluation, we are not yet convinced that these play a sufficient role in ACE’s ultimate charity recommendation decisions for us to be confident that ACE reliably makes scope sensitive recommendations.
We believe that ACE’s Charity Evaluation Program’s current approach does not sufficiently emphasise marginal cost-effectiveness as the most decision-relevant factor in evaluation decisions. For example, ACE primarily evaluates on the basis of organisations’ existing programs, rather than explicitly focusing their evaluation on those programs that are most likely to be funded on the margin. This is in direct contrast to ACE’s Movement Grants, which explicitly evaluates programs that ACE would be influencing funding to on the margin.
Additionally, while our decision is not to rely on ACE’s Charity Evaluation program, we would like to emphasise that we are not making the case that the marginal cost-effectiveness of (any particular) ACE Recommended Charity is clearly not competitive with ACE MG or the EA AWF. In fact, we think it is plausible that some of these charities are competitive with our recommended funds, and may be great options to consider for donors with particular worldviews and who have time to delve deeper into ACE’s materials. Instead, our conclusion is that we can’t justifiably conclude that the charity evaluation program consistently identifies charities competitive with the funding opportunities supported by our current recommendations (ACE MG and EA AWF), based on the evidence we have seen in our evaluation.
For more information, please see our 2024 evaluation report for ACE’s Charity Evaluation Program.
Additional recommendation updates
The Humane League’s corporate campaigns program
As part of our review of our recommended programs for the 2024 Giving Season, we have made the decision to remove The Humane League’s (THL) corporate campaigns program from our list of recommended programs. Our reason for this is that we believe we can no longer justifiably recommend THL’s program alongside our existing recommendations in animal welfare.
While our decision was to no longer recommend THL’s program, we want to be clear that our decision does not reflect any negative update on THL’s work. Instead, this is reflective of evidence being out of date, us maintaining the scope of the evaluating evaluators project, and keeping to our principles of usefulness, justifiability and transparency.
Specifically, this conclusion is predominantly based on the following:
We generally base our charity and fund recommendations on our evaluating evaluators project, for reasons explained here. This year, the evaluators and grantmakers we’ll rely on – based on our evaluations – will be EA Funds’ Animal Welfare Fund and ACE’s Movement Grants, and neither of these currently recommends THL as a charity (as neither makes charity recommendations).
In THL’s case last year, as explained in our report, we went beyond the scope of our usual project by using evidence provided by three further evaluators (Founders Pledge, Rethink Priorities and Open Philanthropy) to supplement the recommendation of an evaluator we had evaluated but decided not to rely on at that point (ACE) in order to ultimately recommend THL’s program. This was based on our overarching principles of usefulness, transparency and justifiability, via our judgement that providing a competitive alternative to the EA AWF – if we transparently and justifiably could – would be valuable.
However, because we expect the approach of grantmakers/evaluators to change more slowly than the work/programs of an individual charity, we are not comfortable relying on the information we used last year without undertaking a reinvestigation,[1] which we don’t expect to do (see below)
Given we now have two recommendations in animal welfare, we consider it somewhat less useful to look further into THL (or any other individual program in animal welfare) and we don’t think we can currently justify going beyond the scope of our evaluating evaluators project in the same way as we could last year.
THL’s corporate campaigns remain a supported program on our platform for donors who wish to continue supporting their work, and though we can no longer justifiably recommend them, we still think they are an option worth considering for impact-focused donors with particular worldviews and time to engage with the evidence supporting their corporate campaigns program.
Conclusion
We have created a webpage for those interested in learning more about our 2024 iteration of evaluating evaluators.
In the future, we plan to continue to evaluate evaluators – extending the list beyond the seven we’ve covered across our first two rounds, improving our methodology, and reinvestigating evaluators we’ve previously looked into to see how their approach/work has or has not changed.
Note that we do maintain our recommendation of EA AWF, as (like we stated above) we expect the quality of evaluations of a grantmaker – and by extension, the quality of the programs it funds when provided with an extra dollar – to change more slowly than the program funded by providing an extra dollar to any individual charity.
GWWC’s 2024 evaluations of evaluators
Introduction
The Giving What We Can research team is excited to share the results of our 2024 round of evaluations of charity evaluators and grantmakers!
In this round, we completed three evaluations that will inform our donation recommendations for the 2024 giving season. As with our 2023 round, there are substantial limitations to these evaluations, but we nevertheless think that they are a significant improvement to a landscape in which there were no independent evaluations of evaluators’ work.
In this post, we share the key takeaways from each of our 2024 evaluations and link to the full reports. We also include an update explaining our decision to remove The Humane League from our list of recommended programs. Our website has now been updated to reflect the new fund and charity recommendations that came out of these evaluations.
Please also see our website for more context on why and how we evaluate evaluators.
We look forward to your questions and comments!
Key takeaways from each of our 2024 evaluations
The three evaluators that we evaluated in our 2024 round of the evaluating evaluators project were:
Founders Pledge Global Health and Development Fund
Animal Charity Evaluators’ Movement Grants
Animal Charity Evaluators’ Charity Evaluation Program
Global health and wellbeing
Founders Pledge Global Health and Development Fund (FP GHDF)
Based on our evaluation, we have decided not to currently include FP GHDF in our list of recommended charities and funds and do not plan to allocate a portion of GWWC’s Global Health and Wellbeing Fund budget to the FP GHDF at this point. However, we still think FP GHDF is worth considering for impact-focused donors and we will continue to host the program on the GWWC donation platform.
Some takeaways that inform this decision include:
Our belief that FP GHDF evaluations do not robustly demonstrate that the opportunities they fund exceed their stated bar of 10x GiveDirectly in expectation, due to the presence of errors and insufficiently justified subjective inputs we found in some of their BOTECs that could cause the estimated cost-effectiveness to fall below this threshold.
Our uncertainty about the future quality of evaluations following the recent departure of the senior researcher responsible for managing the fund and our belief that there are currently insufficient peer review processes, such as red-teaming, in place to reassure us that the fund quality will remain at least consistent following this transition.
Taken together, these issues left us unable to justifiably conclude that the FP GHDF is currently competitive with GiveWell’s recommendations.
While our decision is not to recommend FP GHDF at this time, we would like to emphasise that we did not conclude that the marginal cost-effectiveness of the GHDF is unambiguously not competitive with GiveWell’s recommended charities and funds — in fact, we think the GHDF might be competitive with GiveWell now or in the near future. Instead, our finding is that we can’t currently justifiably conclude that the fund is competitive with GiveWell in expectation, based on the evidence we have seen in our evaluation and the uncertainty surrounding staffing changes.
We also highlight positive findings from our evaluation. For example, we think that FP GHDF’s strategy of funding promising early-stage organisations plausibly has significant positive externalities, and the fact that FP GHDF has made several grants to organisations before GiveWell indicates they have some track record for identifying these promising funding opportunities.
For more information, please see our 2024 evaluation report for FP GHDF.
Animal welfare
Animal Charity Evaluators’ Movement Grants (ACE MG)
Based on our evaluation, we have decided to include ACE MG in our list of recommended charities and funds and we expect to allocate half of GWWC’s Effective Animal Advocacy Fund to ACE MG.
The key finding informing this decision is that ACE MG’s recent marginal grants and overall grant decision-making look to be of sufficiently high quality to be competitive with our current top recommendation in animal welfare: EA Funds’ Animal Welfare Fund (EA AWF) — by our evaluation of EA AWF in 2023. This is in large part due to our view that ACE MG has improved meaningfully since our previous evaluation, when we concluded that ACE MG performed slightly less strongly on several proxies for the marginal cost-effectiveness of its grants. In particular, we noted improvements in the quality of ACE MG’s:
Marginal grantmaking
Weighting of geographical neglectedness
Approach to grant selection and sizing
We did find what we think to be room for improvement in ACE MG’s approach to more clearly integrating scope, particularly when comparing between applicants implementing different types of interventions. However, we don’t think this room for improvement affects the ACE MG’s position as being — to our knowledge — among the best places to recommend to donors.
We do not believe that our investigation identified ACE MG to be justifiably better or worse than EA AWF in terms of marginal cost-effectiveness — based on our 2023 evaluation of EA AWF — and so we continue to recommend EA AWF as a donation option alongside ACE MG, and to allocate half of our Effective Animal Advocacy Fund to EA AWF.
For more information, please see our 2024 evaluation report for ACE MG.
Animal Charity Evaluators’ Charity Evaluation Program
Based on our evaluation, we have decided not to currently include ACE’s Recommended Charities or the Recommended Charity Fund (RCF) in our list of recommended charities and funds nor to allocate a portion of GWWC’s Effective Animal Advocacy Fund to these programs at this point in time. However, we still think ACE’s Recommended Charities are worth considering for impact-focused donors and we may consider inviting some of these programs to apply to become supported programs on the GWWC donation platform in early 2025.
Some takeaways that informed our decision include:
While ACE has made improvements to its approach for capturing scope and cost-effectiveness since our 2023 evaluation, we are not yet convinced that these play a sufficient role in ACE’s ultimate charity recommendation decisions for us to be confident that ACE reliably makes scope sensitive recommendations.
We believe that ACE’s Charity Evaluation Program’s current approach does not sufficiently emphasise marginal cost-effectiveness as the most decision-relevant factor in evaluation decisions. For example, ACE primarily evaluates on the basis of organisations’ existing programs, rather than explicitly focusing their evaluation on those programs that are most likely to be funded on the margin. This is in direct contrast to ACE’s Movement Grants, which explicitly evaluates programs that ACE would be influencing funding to on the margin.
Additionally, while our decision is not to rely on ACE’s Charity Evaluation program, we would like to emphasise that we are not making the case that the marginal cost-effectiveness of (any particular) ACE Recommended Charity is clearly not competitive with ACE MG or the EA AWF. In fact, we think it is plausible that some of these charities are competitive with our recommended funds, and may be great options to consider for donors with particular worldviews and who have time to delve deeper into ACE’s materials. Instead, our conclusion is that we can’t justifiably conclude that the charity evaluation program consistently identifies charities competitive with the funding opportunities supported by our current recommendations (ACE MG and EA AWF), based on the evidence we have seen in our evaluation.
For more information, please see our 2024 evaluation report for ACE’s Charity Evaluation Program.
Additional recommendation updates
The Humane League’s corporate campaigns program
As part of our review of our recommended programs for the 2024 Giving Season, we have made the decision to remove The Humane League’s (THL) corporate campaigns program from our list of recommended programs. Our reason for this is that we believe we can no longer justifiably recommend THL’s program alongside our existing recommendations in animal welfare.
While our decision was to no longer recommend THL’s program, we want to be clear that our decision does not reflect any negative update on THL’s work. Instead, this is reflective of evidence being out of date, us maintaining the scope of the evaluating evaluators project, and keeping to our principles of usefulness, justifiability and transparency.
Specifically, this conclusion is predominantly based on the following:
We generally base our charity and fund recommendations on our evaluating evaluators project, for reasons explained here. This year, the evaluators and grantmakers we’ll rely on – based on our evaluations – will be EA Funds’ Animal Welfare Fund and ACE’s Movement Grants, and neither of these currently recommends THL as a charity (as neither makes charity recommendations).
In THL’s case last year, as explained in our report, we went beyond the scope of our usual project by using evidence provided by three further evaluators (Founders Pledge, Rethink Priorities and Open Philanthropy) to supplement the recommendation of an evaluator we had evaluated but decided not to rely on at that point (ACE) in order to ultimately recommend THL’s program. This was based on our overarching principles of usefulness, transparency and justifiability, via our judgement that providing a competitive alternative to the EA AWF – if we transparently and justifiably could – would be valuable.
However, because we expect the approach of grantmakers/evaluators to change more slowly than the work/programs of an individual charity, we are not comfortable relying on the information we used last year without undertaking a reinvestigation,[1] which we don’t expect to do (see below)
Given we now have two recommendations in animal welfare, we consider it somewhat less useful to look further into THL (or any other individual program in animal welfare) and we don’t think we can currently justify going beyond the scope of our evaluating evaluators project in the same way as we could last year.
THL’s corporate campaigns remain a supported program on our platform for donors who wish to continue supporting their work, and though we can no longer justifiably recommend them, we still think they are an option worth considering for impact-focused donors with particular worldviews and time to engage with the evidence supporting their corporate campaigns program.
Conclusion
We have created a webpage for those interested in learning more about our 2024 iteration of evaluating evaluators.
In the future, we plan to continue to evaluate evaluators – extending the list beyond the seven we’ve covered across our first two rounds, improving our methodology, and reinvestigating evaluators we’ve previously looked into to see how their approach/work has or has not changed.
Note that we do maintain our recommendation of EA AWF, as (like we stated above) we expect the quality of evaluations of a grantmaker – and by extension, the quality of the programs it funds when provided with an extra dollar – to change more slowly than the program funded by providing an extra dollar to any individual charity.