CEAs are always very uncertain and fragile, and I generally wouldnât put too much stock on the precise multiple relative to GiveWell. For us, we only really care if itâs >GW, and we use >=10x as a nominal threshold to account for the fact that GiveWell generally spends a lot more time and effort discounting, relative to ourselves or Foudners Pledge or other researchers.
Overally, health policy interventions are uniquely cost-effective simply because of (a) large scale of impact (policies affect people across the whole country, or at least region or city); (b) low cost per capita (due to economies of scale), and this tends to outweigh the risk of failure (typically, weâre looking at 10% success rates, though better charities in promising countries are closer to 30-50% maybe). Still, itâs enormously risky, and remember that the median outcome is zero impact
Yeah agreed, thatâs why I mentioned the cluster thing as a point in favor of your reportâs analytical methodology, and brought up the VC vs index fund analogy, essentially to steer the focus away from pure EV-maxing
instead of looking only at EV, which is a fragile sequence thinking approach, you take a cluster thinking approach (which is better for making good decisions) and also consider things like the underlying theory of changeâs evidence quality (high in this case), expert views (checks out), and downsides (marginal), all of which the author analyses extensively.
I think overwhelmingly relying on explicit quantitative cost-effectiveness analyses is generally better. However, one should be mindful to account for considerations not covered, and that one can often cover more considerations by not modelling them formally.
GiveWell cost- effectiveness estimates are not the only input into our decisions to fund malaria programs and deworming programs, there are some other factors, but theyâre certainly 80% plus of the case.
There is more context from GiveWell here. The key parts are below.
The numerical cost-effectiveness estimate in the spreadsheet is nearly always the most important factor in our recommendations, but not the only factor. That is, we donât solely rely on our spreadsheet-based analysis of cost-effectiveness when making grants.
We donât have an institutional position on exactly how much of the decision comes down to the spreadsheet analysis (though Elieâs take of â80% plusâ definitely seems reasonable!) and it varies by grant, but many of the factors we consider outside our models (e.g. qualitative factors about an organization) are in the service of making impact-oriented decisions. See this post for more discussion.
For a small number of grants, the case for the grant relies heavily on factors other than expected impact of that grant per se. For example, we sometimes make exit grants in order to be a responsible funder and treat partner organizations considerately even if we think funding could be used more cost-effectively elsewhere.
Iâve read those comments awhile back and I donât think they support your view for relying overwhelmingly on explicit quantitative cost-effectiveness analyses. In particular the key parts I got out of Isabelâs comment werenât what you quoted but instead (emphasis mine not hers)
Cost-effectiveness is the primary driver of our grantmaking decisions. But, âoverall estimated cost-effectiveness of a grantâ isnât the same thing as âoutput of cost-effectiveness analysis spreadsheet.â (This blog post is old and not entirely reflective of our current approach, but it covers a similar topic.)
and
That is, we donât solely rely on our spreadsheet-based analysis of cost-effectiveness when making grants.
which is in direct contradistinction to your style as I understand it, and aligned with what Holden wrote earlier in that link you quoted (emphasis his this time)
While some people feel that GiveWell puts too much emphasis on the measurable and quantifiable, there are others who go further than we do in quantification, and justify their giving (or other) decisions based on fully explicit expected-value formulas. The latter group tends to critique us â or at least disagree with us â based on our preference for strong evidence over high apparent âexpected value,â and based on the heavy role of non-formalized intuition in our decisionmaking. This post is directed at the latter group.
We believe that people in this group are often making a fundamental mistake, one that we have long had intuitive objections to but have recently developed a more formal (though still fairly rough) critique of. The mistake (we believe) is estimating the âexpected valueâ of a donation (or other action) based solely on a fully explicit, quantified formula, many of whose inputs are guesses or very rough estimates. We believe that any estimate along these lines needs to be adjusted using a âBayesian priorâ; that this adjustment can rarely be made (reasonably) using an explicit, formal calculation; and that most attempts to do the latter, even when they seem to be making very conservative downward adjustments to the expected value of an opportunity, are not making nearly large enough downward adjustments to be consistent with the proper Bayesian approach.
This view of ours illustrates why â while we seek to ground our recommendations in relevant facts, calculations and quantifications to the extent possible â every recommendation we make incorporates many different forms of evidence and involves a strong dose of intuition. And we generally prefer to give where we have strong evidence that donations can do a lot of good rather than where we have weak evidence that donations can do far more good â a preference that I believe is inconsistent with the approach of giving based on explicit expected-value formulas (at least those that (a) have significant room for error (b) do not incorporate Bayesian adjustments, which are very rare in these analyses and very difficult to do both formally and reasonably).
Note that Iâm not saying CEAs donât matter, or that CEA-focused approaches are unreliable â Iâm a big believer in the measurability of things people often claim canât be measured, I think in principle EV-maxing is almost always correct but in practice it can be perilous and on the margin people should instead be working a bit more on how different moral conceptions cash out in different recommendations more systematically e.g. with RPâs work, if a CEA-based case canât be made for a grant I get very skeptical, I in fact also consider CEAs the main input into my thinking on these kinds of things, etc. I am simply wary of single-parameter optimisation taken to the limit in general (for anything, really, not just for donating), and I see your approach as being willing to go much further along that path than I do (and Iâm already further along that path than almost anyone I meet IRL).
But Iâve seen enough back-and-forth between people in the cluster and sequence camps to have the sense that nobody really ends up changing their mind substantively and I doubt this will happen here either, sorry, so I will respectfully bow out of the conversation.
I think thereâs something to be said that if the qualitative stuff looks bad (e.g. experts are negative, studies are bunch of non-randomized stuff susceptible to endogeneity concerns), you always have the option of just implementing aggressive discounts to the CEA, and the fact that it looks too good means the CEA is done poorly, not that the CEA-focused approach is unreliable.
Makes sense. I also like explicit quantification because it is more transparent, making it easier to examine assumptions, identify main uncertainties, and therefore improve estimates in the future. With approaches associated with cluster thinking, I think it is often unclear which assumptions are driving the decisions, or whether the decisions being made actually follow from the assumptions.
CEAs are always very uncertain and fragile, and I generally wouldnât put too much stock on the precise multiple relative to GiveWell. For us, we only really care if itâs >GW, and we use >=10x as a nominal threshold to account for the fact that GiveWell generally spends a lot more time and effort discounting, relative to ourselves or Foudners Pledge or other researchers.
Overally, health policy interventions are uniquely cost-effective simply because of (a) large scale of impact (policies affect people across the whole country, or at least region or city); (b) low cost per capita (due to economies of scale), and this tends to outweigh the risk of failure (typically, weâre looking at 10% success rates, though better charities in promising countries are closer to 30-50% maybe). Still, itâs enormously risky, and remember that the median outcome is zero impact
Yeah agreed, thatâs why I mentioned the cluster thing as a point in favor of your reportâs analytical methodology, and brought up the VC vs index fund analogy, essentially to steer the focus away from pure EV-maxing
Thanks for the discussion!
I think overwhelmingly relying on explicit quantitative cost-effectiveness analyses is generally better. However, one should be mindful to account for considerations not covered, and that one can often cover more considerations by not modelling them formally.
People often point to GiveWellâs posts on not taking expected value estimates literally, and sequence versus cluster thinking to moderate enthusiam about explicit quantitative cost-effectiveness analyses, but GiveWell overwhelmingly relies on these. Elie Hassenfeld, GiveWellâs CEO, mentioned on the Clearer Thinking podcast that:
There is more context from GiveWell here. The key parts are below.
Iâve read those comments awhile back and I donât think they support your view for relying overwhelmingly on explicit quantitative cost-effectiveness analyses. In particular the key parts I got out of Isabelâs comment werenât what you quoted but instead (emphasis mine not hers)
and
which is in direct contradistinction to your style as I understand it, and aligned with what Holden wrote earlier in that link you quoted (emphasis his this time)
Note that Iâm not saying CEAs donât matter, or that CEA-focused approaches are unreliable â Iâm a big believer in the measurability of things people often claim canât be measured, I think in principle EV-maxing is almost always correct but in practice it can be perilous and on the margin people should instead be working a bit more on how different moral conceptions cash out in different recommendations more systematically e.g. with RPâs work, if a CEA-based case canât be made for a grant I get very skeptical, I in fact also consider CEAs the main input into my thinking on these kinds of things, etc. I am simply wary of single-parameter optimisation taken to the limit in general (for anything, really, not just for donating), and I see your approach as being willing to go much further along that path than I do (and Iâm already further along that path than almost anyone I meet IRL).
But Iâve seen enough back-and-forth between people in the cluster and sequence camps to have the sense that nobody really ends up changing their mind substantively and I doubt this will happen here either, sorry, so I will respectfully bow out of the conversation.
Thanks for the context, Mo!
I think thereâs something to be said that if the qualitative stuff looks bad (e.g. experts are negative, studies are bunch of non-randomized stuff susceptible to endogeneity concerns), you always have the option of just implementing aggressive discounts to the CEA, and the fact that it looks too good means the CEA is done poorly, not that the CEA-focused approach is unreliable.
Makes sense. I also like explicit quantification because it is more transparent, making it easier to examine assumptions, identify main uncertainties, and therefore improve estimates in the future. With approaches associated with cluster thinking, I think it is often unclear which assumptions are driving the decisions, or whether the decisions being made actually follow from the assumptions.