[Question] What is the state of the art in EA cost-effectiveness modelling?

[Froolow]

Hi all,

A friend linked me to the ‘red team’ contest, and since I undertake cost-effectiveness modelling professionally I thought that would be a useful place I could contribute, potentially.

I’m not an active member of the EA community, so I’d like to ensure I don’t straw man the state of the art here; would it be fair to say GiveWell’s evaluation of some of its most cost-effective charities represents what the EA community would consider to be a high-quality cost-effectiveness model? If not, what would the EA community consider to be a high-quality cost-effectiveness model in an EA context?

https://​​www.givewell.org/​​how-we-work/​​our-criteria/​​cost-effectiveness/​​cost-effectiveness-models

(To be clear, the model is great but I think there are a number of areas where it could be improved upon!)

Thanks so much,

Froolow

[Lorenzo Buonanno]

I would say that GiveWell’s cost-effectiveness analyses are considered excellent (here is a guide from 2019), but they should be taken in context.
From https://​​www.givewell.org/​​how-we-work/​​our-criteria/​​cost-effectiveness/​​cost-effectiveness-models “we consider our cost-effectiveness numbers to be extremely rough.”
″There are many limitations to cost-effectiveness estimates, and we do not assess charities only—or primarily—based on their estimated cost-effectiveness.”

And this old blog post: https://​​blog.givewell.org/​​2011/​​08/​​18/​​why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/​​

They definitely have a much smaller weight than what I first assumed: I initially thought all GiveWell did was to make cost-effectiveness analyses of dozens of charities and recommend the most cost-effective ones. It seems that’s completely wrong, and they rely a lot on other criteria and less quantitative or public information.

You might also be interested in the series: Concerns about AMF from GiveWell reading (especially Part 3 and Part 2)

And this FAQ from 2017 from GiveWell

[Froolow]

Thank you—really helpful additional information and very useful to have it confirmed that GiveWell are considered high quality models by the EA community. Really appreciate it.

[Vasco Grilo]

Hi Lorenzo,

They definitely have a much smaller weight than what I first assumed: I initially thought all GiveWell did was to make cost-effectiveness analyses of dozens of charities and recommend the most cost-effective ones. It seems that’s completely wrong, and they rely a lot on other criteria and less quantitative or public information.

I think GiveWell actually puts a major weight on their cost-effectiveness analyses. Elie Hassenfeld (co-founder and CEO of GiveWell) mentioned in the Clearer Thinking podcast that (emphasis mine):

ELIE: You know, there’s a lot of attention paid to the numbers, and certainly plenty of high profile institutions were behind the report. I think that, in my experience, GiveWell is one of the few institutions that’s, I don’t know, trying to make decisions based on cost-effectiveness analysis in doing that in a sort of, consistent and principled way. GiveWell cost- effectiveness estimates are not the only input into our decisions to fund malaria programs and deworming programs, there are some other factors, but they’re certainly 80% plus of the case. I think we’re relatively unique in that way. I don’t think there are other groups, certainly I can’t think of any ones as I’m sitting here now, that are using numbers in that same way. In some ways, I think that is why we have real value added in the world, because I don’t think that explicit cost-effectiveness estimates is the only way to give effectively, but it’s certainly a strategy that I think should be employed significantly. I’m glad that we can be the ones to come in and play it.

[Gavin]

I’d say Michael Dickens and Sam Nolan are probably peak performance.

This is a classic, devastating methodological piece too.

[Froolow]

Thank you—these are really helpful to help me understand. On the Sam Nolan piece especially quantifying uncertainty was one of the biggest critiques I had of the GiveWell model so I’m glad this has already been considered!

[NunoSempere]

this has already been considered!

This has been considered by Sam, but it’s still a valid criticism of GiveWell’s models as they are now.

[Stefan_Schubert]

I would guess that other orgs besides GiveWell also have cost-effectiveness models/​analyses.

[Gavin]

Couldn’t find any public OP analyses on a cursory look

[Stefan_Schubert]

I guess that if one wants to red team effective altruist cost-effectiveness analyses that inform, e.g. giving decisions, non-public analyses may be relevant.

[Gavin]

See also

https://​​forum.effectivealtruism.org/​​posts/​​kyJtzRHd6hLzfxshd/​​announcing-the-legal-priorities-project-writing-competition