This forks a shortform post from seven months ago. I wanted to update and signal boost.
Some things have changed in those seven months, but it seems nothing major … I am adjusting to make this current.
ImpactMatters acquired by CharityNavigator
ImpactMatters was founded in 2015 by Dean Karlan and Elijah Goldberg. They brought evidence-based impact ratings to a wider set of charities than GiveWell. Rather than focusing only on the very most effective charities they investigated impact and effectiveness across a wider range of charities willing to participate. (In some ways, this resembled SoGive). E.g., “in November 2019, ImpactMatters released over 1,000 ratings.”
I saw strong potential for Impact Matters to move an EA-adjacent impactfulness metric beyond a small list of Givewell and ACE charities, to move the conversation, get charities to compete on this basis, and ‘raise awareness’ (ugh, hate that expression). (I was not so happy about their rating much-less-impactful USA-based charities with international charities without making clear the distinction, but perhaps a necessary evil).
I thought this had great positive potential, for the same reasons I thought Impact Matters had potential… and even more ’bringing this into the mainstream.
However, I’m not so happy with the way things are presented
The Impact Ratings don’t convey a GiveWell-like ‘impact per dollar’ measure. See their methodology (incompletely explained) here and pasted in footnote[1]
In the presentation, the Impact ratings are a bit folded into and mixed up with the Encompass ratings. E.g., I couldn’t figure out how to sort or filter charities by their ‘Impact and Results Score’ itself. I could not easily locate any charities with imperfect scores in this category! (Can you find any?)
Impact Ratings are not prominent or mentioned when one is looking through most categories of charities (e.g., my mother was looking for charities her organization could support dealing with “Human trafficking, COVID-19, hunger, or the environment” and nothing about impact came up)
In some presentations on their page cause-categories with order-of-magnitude differences in impact are presented side-by-side, but only comparable with ’within-category ratings. Thus, a charity building wells in Africa might conceivable receive a much lower score, and thus appear to be much less effective, than a charity giving university scholarships to students in the USA.
Quoted from the above link: A nonprofit receives a total of 0 out of 100 points if, after a thorough search of its public materials, we were unable to find sufficient information to estimate the impact of a substantial portion of its programs.
A nonprofit receives a total of 50 out of 100 points if it has published sufficient information for us to estimate the impact of a substantial portion of its programs, but we found it was not cost-effective. We determine cost-effectiveness by comparing our estimate of the nonprofit’s impact to a benchmark for performance.
A nonprofit receives a total of 75 out of 100 points (a passing score) if it is cost-effective by our estimates.
A nonprofit receives a total of 100 out of 100 points if it is highly cost-effective by our estimates.
ImpactMatters was acquired by CharityNavigator; but it doesn’t seem to have been incorporated, presented, or used in a great way. (Update/boost)
Caveats:
This forks a shortform post from seven months ago. I wanted to update and signal boost.
Some things have changed in those seven months, but it seems nothing major … I am adjusting to make this current.
ImpactMatters acquired by CharityNavigator
ImpactMatters was founded in 2015 by Dean Karlan and Elijah Goldberg. They brought evidence-based impact ratings to a wider set of charities than GiveWell. Rather than focusing only on the very most effective charities they investigated impact and effectiveness across a wider range of charities willing to participate. (In some ways, this resembled SoGive). E.g., “in November 2019, ImpactMatters released over 1,000 ratings.”
I saw strong potential for Impact Matters to move an EA-adjacent impactfulness metric beyond a small list of Givewell and ACE charities, to move the conversation, get charities to compete on this basis, and ‘raise awareness’ (ugh, hate that expression). (I was not so happy about their rating much-less-impactful USA-based charities with international charities without making clear the distinction, but perhaps a necessary evil).
In late 2020 CharityNavigator acquired Impact Matters. They have added “Impact and Results Scores” for
100(update: 1000) or more charities and this is incorporated into their ‘Encompass Rating’ but not into their basic headline and most prominent and famous “stars system”, nor the “perfect scores” list.I thought this had great positive potential, for the same reasons I thought Impact Matters had potential… and even more ’bringing this into the mainstream.
However, I’m not so happy with the way things are presented
The Impact Ratings don’t convey a GiveWell-like ‘impact per dollar’ measure. See their methodology (incompletely explained) here and pasted in footnote[1]
In the presentation, the Impact ratings are a bit folded into and mixed up with the Encompass ratings. E.g., I couldn’t figure out how to sort or filter charities by their ‘Impact and Results Score’ itself. I could not easily locate any charities with imperfect scores in this category! (Can you find any?)
Impact Ratings are not prominent or mentioned when one is looking through most categories of charities (e.g., my mother was looking for charities her organization could support dealing with “Human trafficking, COVID-19, hunger, or the environment” and nothing about impact came up)
In some presentations on their page cause-categories with order-of-magnitude differences in impact are presented side-by-side, but only comparable with ’within-category ratings. Thus, a charity building wells in Africa might conceivable receive a much lower score, and thus appear to be much less effective, than a charity giving university scholarships to students in the USA.
They only have impact ratings for 15 (updated) charities working internationally (vs 320 ratings for charities that only work within regions in the USA, I believe), and no animal or otherwise EA-relevant, as far as I know.
Top-ten lists: Nine of ten charities on their top 10 super-sized charities with perfect impact scores apparently work only in the US. These are nearly all local/regional food banks. 8⁄10 (I believe) of the charities on their 10 Charities Worth Watching with Perfect Impact Scores appear to be US domestic only.
What do you think? Is this being used well? How could it be done better? How could we push them in the right direction?
Quoted from the above link: A nonprofit receives a total of 0 out of 100 points if, after a thorough search of its public materials, we were unable to find sufficient information to estimate the impact of a substantial portion of its programs. A nonprofit receives a total of 50 out of 100 points if it has published sufficient information for us to estimate the impact of a substantial portion of its programs, but we found it was not cost-effective. We determine cost-effectiveness by comparing our estimate of the nonprofit’s impact to a benchmark for performance. A nonprofit receives a total of 75 out of 100 points (a passing score) if it is cost-effective by our estimates. A nonprofit receives a total of 100 out of 100 points if it is highly cost-effective by our estimates.