ImpactMatters was acquired by CharityNavigator; but it doesn’t seem to have been incorporated, presented, or used in a great way. (Update/​boost)

[david_reinstein]

Caveats:

• This forks a shortform post from seven months ago. I wanted to update and signal boost.

• Some things have changed in those seven months, but it seems nothing major … I am adjusting to make this current.

ImpactMatters acquired by CharityNavigator

ImpactMatters was founded in 2015 by Dean Karlan and Elijah Goldberg. They brought evidence-based impact ratings to a wider set of charities than GiveWell. Rather than focusing only on the very most effective charities they investigated impact and effectiveness across a wider range of charities willing to participate. (In some ways, this resembled SoGive). E.g., “in November 2019, ImpactMatters released over 1,000 ratings.”

I saw strong potential for Impact Matters to move an EA-adjacent impactfulness metric beyond a small list of Givewell and ACE charities, to move the conversation, get charities to compete on this basis, and ‘raise awareness’ (ugh, hate that expression). (I was not so happy about their rating much-less-impactful USA-based charities with international charities without making clear the distinction, but perhaps a necessary evil).

In late 2020 CharityNavigator acquired Impact Matters. They have added “Impact and Results Scores” for 100 (update: 1000) or more charities and this is incorporated into their ‘Encompass Rating’ but not into their basic headline and most prominent and famous “stars system”, nor the “perfect scores” list.

I thought this had great positive potential, for the same reasons I thought Impact Matters had potential… and even more ’bringing this into the mainstream.

However, I’m not so happy with the way things are presented

1. The Impact Ratings don’t convey a GiveWell-like ‘impact per dollar’ measure. See their methodology (incompletely explained) here and pasted in footnote^[1]

2. In the presentation, the Impact ratings are a bit folded into and mixed up with the Encompass ratings. E.g., I couldn’t figure out how to sort or filter charities by their ‘Impact and Results Score’ itself. I could not easily locate any charities with imperfect scores in this category! (Can you find any?)

3. Impact Ratings are not prominent or mentioned when one is looking through most categories of charities (e.g., my mother was looking for charities her organization could support dealing with “Human trafficking, COVID-19, hunger, or the environment” and nothing about impact came up)

4. In some presentations on their page cause-categories with order-of-magnitude differences in impact are presented side-by-side, but only comparable with ’within-category ratings. Thus, a charity building wells in Africa might conceivable receive a much lower score, and thus appear to be much less effective, than a charity giving university scholarships to students in the USA.

5. They only have impact ratings for 15 (updated) charities working internationally (vs 320 ratings for charities that only work within regions in the USA, I believe), and no animal or otherwise EA-relevant, as far as I know.

6. Top-ten lists: Nine of ten charities on their top 10 super-sized charities with perfect impact scores apparently work only in the US. These are nearly all local/​regional food banks. ⁸⁄₁₀ (I believe) of the charities on their 10 Charities Worth Watching with Perfect Impact Scores appear to be US domestic only.

What do you think? Is this being used well? How could it be done better? How could we push them in the right direction?

---

1. ↩︎

   Quoted from the above link: A nonprofit receives a total of 0 out of 100 points if, after a thorough search of its public materials, we were unable to find sufficient information to estimate the impact of a substantial portion of its programs. A nonprofit receives a total of 50 out of 100 points if it has published sufficient information for us to estimate the impact of a substantial portion of its programs, but we found it was not cost-effective. We determine cost-effectiveness by comparing our estimate of the nonprofit’s impact to a benchmark for performance. A nonprofit receives a total of 75 out of 100 points (a passing score) if it is cost-effective by our estimates. A nonprofit receives a total of 100 out of 100 points if it is highly cost-effective by our estimates.

[david_reinstein]

Some positive developments here:

Charities Working in High-Impact Causes Highly rated charities working in unusually high-impact causes, identified by the effective altruism community and our analysts

[Dan Stein]

I’m not directly involved with this effort, but I do know that CN has a new person in charge of trying to make these impact ratings work, and they are trying to think very creatively about how to improve and add value. They need something they can do at scale and it’s a tough job. I know this because their team has been reaching out to various people in the space (myself included) to help with brainstorming. So just to say, they are definitely not burying this effort.

I think it’s still unclear if CN will be able to turn this effort into something compelling, but I think it’s worth waiting to see what their new team comes up with before passing final judgement.

[david_reinstein]

Thanks. It’s still moderately-early days. I just don’t like the direction it seems to be going in.

[brb243]

I think that intra-area comparisons may be ok, also due to acceptability by users, who may think traditionally about charity (for example, open Charity Navigator because they received an emotional appeal from a US-based charity), only more areas which relate to EA causes should be included and listed in a (strategic) combination with non-EA cause areas (more clicks) (maybe the impact units and top charity per impact unit should be listed alongside or so - $40,000 to train a guide dog for 1 person, $100 to restore eyesisght of 1 person, etc), and more areas which involve EA and non-EA charities should be included (e. g. providing meals to the hungry as opposed to meals in US homeless shelters—then, for example, Yemen Aid, which provides 6.4 meals/​$1, can compete with $2/​meal) - the ‘at the risk of famine’ and ‘food insecure’ distinction is lost by this grouping but this can be included in further description. This can be sold to Charity Navigator as greater total impact. Potentially some computing capacity of some persons in EA looking to volunteer, or in companies, such as Google, to improve the popularity of CN can come in a bundle? The risk is that Charity Navigator will lose funding influence share due to less captivating entertainment of users interested in charity assessment but ways for the opposite should be developed.

You should speak with Sanjay to speak with Elijah Goldberg or Tamsin Chen.

Podcasters can ask for a brief interview with Dean Karlan, for example to elaborate on ways of motivating charities, especially those in neglected cause and geographical areas with high potential, to develop solutions that support systemic change toward institutions that safeguard continuous improvement of living standards of an increasing number of individuals, and test these solutions affordably. [For example, should they present data in a specific format to Charity Navigator to get high rating, or emulate and improve the approaches of top charities in order to gain further funding?] - the last time I checked (asked the Charity Navigator ‘agent’), only US-based charities (including internationally operating ones who have only a foundation in the US) were evaluated, due to the use of tax statements, but the incorporation of international organizations’ evaluation was considered. Elijah was also thinking of developing a simple questionnaire to assess impact.

Also, Northwestern students can inquire regarding a class project similar to the Pro bono student impact audit opportunity shared by ImpactMatters in 2020 to develop a rapid evaluation framework within select causes. This may already exist.

In terms of a thorough but more affordable and impactful evaluation, I was/​am suggesting (linked in this post) that

• counterfactual beneficiaries’ outcomes are estimated by increasingly more robust methods (organizations may be interested in accurate predictions due to estimates’ records and later data validation)

• funders’ alternative investments are considered (e. g. personal investment vs. funding of a comparable program)

• in-program’s-absence caretakers’ alternative investments are considered (e. g. into necessities vs. luxuries vs. ‘independence’ capital development)

• pro bono impact increase consultancy is offered

  • at least for charities which can increase their cost-effectiveness with relative facility, such as by geographical expansion or animal welfare considerations

I was also thinking about measuring subjective wellbeing changes via the visual analog scale method as the end goal. This, if given also to individuals (hypothesized for non-humans) who would benefit indirectly from an externality or another change, could address the issue of an organization addressing multiple issues on priority needs basis and thus having difficulties to focus on one or several impact metrics and may detect systemic change better.

The key would be to get organizations to fill a form, such as by automatically making an appealing annual report for them, offering discounts on evaluation and fundraising consultancy (such as narrating competitiveness due to higher impact), or the Charity Navigator listing.

To conclude, the current rating works well, but important additionalities may be available.

[david_reinstein]

David Moss:

There was some discussion of the original acquisition here.

Historically, Charity Navigator has been extremely hostile to effective altruism, as you probably know, so perhaps this isn’t surprising

My response

Thank you, I had not seen Luke Freeman @givingwhatwecan’s earlier post

That 2013 opinion piece/​hit job is shocking. But that was 9 years ago or so.

I doubt CN would have acquired IM just to bury it; there might be some room for positive suasion here.

[david_reinstein]

Moving some comments from the Shortform...

Aaron Gertler wrote:

I spent a few minutes looking at the impact feature, and I… will also go with “not satisfied”.

From their review of Village Enterprise:

Impact & Results scores of livelihood support programs are based on income generated relative to cost. Programs receive an Impact & Results score of 100 if they increase income for a beneficiary by more than $1.50 for every $1 spent and a score of 75 if income increases by more than $0.85 for every $1 spent. If a nonprofit reports impact but doesn't meet the threshold for cost-effectiveness, it earns a score of 50.

My charitable interpretation is that the “$0.85” number is meant to represent one year’s income, and to imply a higher number over time (e.g. you have new skills or a new business that boosts your income for years to come).

But I also think it’s plausible that “$0.85” is meant to refer to the total increase, such that you could score “75″ by running a program that, in your own estimation, helps people less than just giving them money.

(The “lowest score is 50” element puzzled me at first, but this page clarifies that you score “0″ if CN can’t find enough information to estimate your impact in the first place.)

---

Still, this is much better than the original CN setup, and I hope this is an early beta version with many improvements on the way.

[david_reinstein]

I don’t see much in the way of improvement … does anyone else?