Yeah agreed, that’s why I mentioned the cluster thing as a point in favor of your report’s analytical methodology, and brought up the VC vs index fund analogy, essentially to steer the focus away from pure EV-maxing
instead of looking only at EV, which is a fragile sequence thinking approach, you take a cluster thinking approach (which is better for making good decisions) and also consider things like the underlying theory of change’s evidence quality (high in this case), expert views (checks out), and downsides (marginal), all of which the author analyses extensively.
I think overwhelmingly relying on explicit quantitative cost-effectiveness analyses is generally better. However, one should be mindful to account for considerations not covered, and that one can often cover more considerations by not modelling them formally.
GiveWell cost- effectiveness estimates are not the only input into our decisions to fund malaria programs and deworming programs, there are some other factors, but they’re certainly 80% plus of the case.
There is more context from GiveWell here. The key parts are below.
The numerical cost-effectiveness estimate in the spreadsheet is nearly always the most important factor in our recommendations, but not the only factor. That is, we don’t solely rely on our spreadsheet-based analysis of cost-effectiveness when making grants.
We don’t have an institutional position on exactly how much of the decision comes down to the spreadsheet analysis (though Elie’s take of “80% plus” definitely seems reasonable!) and it varies by grant, but many of the factors we consider outside our models (e.g. qualitative factors about an organization) are in the service of making impact-oriented decisions. See this post for more discussion.
For a small number of grants, the case for the grant relies heavily on factors other than expected impact of that grant per se. For example, we sometimes make exit grants in order to be a responsible funder and treat partner organizations considerately even if we think funding could be used more cost-effectively elsewhere.
I’ve read those comments awhile back and I don’t think they support your view for relying overwhelmingly on explicit quantitative cost-effectiveness analyses. In particular the key parts I got out of Isabel’s comment weren’t what you quoted but instead (emphasis mine not hers)
Cost-effectiveness is the primary driver of our grantmaking decisions. But, “overall estimated cost-effectiveness of a grant” isn’t the same thing as “output of cost-effectiveness analysis spreadsheet.” (This blog post is old and not entirely reflective of our current approach, but it covers a similar topic.)
and
That is, we don’t solely rely on our spreadsheet-based analysis of cost-effectiveness when making grants.
which is in direct contradistinction to your style as I understand it, and aligned with what Holden wrote earlier in that link you quoted (emphasis his this time)
While some people feel that GiveWell puts too much emphasis on the measurable and quantifiable, there are others who go further than we do in quantification, and justify their giving (or other) decisions based on fully explicit expected-value formulas. The latter group tends to critique us – or at least disagree with us – based on our preference for strong evidence over high apparent “expected value,” and based on the heavy role of non-formalized intuition in our decisionmaking. This post is directed at the latter group.
We believe that people in this group are often making a fundamental mistake, one that we have long had intuitive objections to but have recently developed a more formal (though still fairly rough) critique of. The mistake (we believe) is estimating the “expected value” of a donation (or other action) based solely on a fully explicit, quantified formula, many of whose inputs are guesses or very rough estimates. We believe that any estimate along these lines needs to be adjusted using a “Bayesian prior”; that this adjustment can rarely be made (reasonably) using an explicit, formal calculation; and that most attempts to do the latter, even when they seem to be making very conservative downward adjustments to the expected value of an opportunity, are not making nearly large enough downward adjustments to be consistent with the proper Bayesian approach.
This view of ours illustrates why – while we seek to ground our recommendations in relevant facts, calculations and quantifications to the extent possible – every recommendation we make incorporates many different forms of evidence and involves a strong dose of intuition. And we generally prefer to give where we have strong evidence that donations can do a lot of good rather than where we have weak evidence that donations can do far more good – a preference that I believe is inconsistent with the approach of giving based on explicit expected-value formulas (at least those that (a) have significant room for error (b) do not incorporate Bayesian adjustments, which are very rare in these analyses and very difficult to do both formally and reasonably).
Note that I’m not saying CEAs don’t matter, or that CEA-focused approaches are unreliable — I’m a big believer in the measurability of things people often claim can’t be measured, I think in principle EV-maxing is almost always correct but in practice it can be perilous and on the margin people should instead be working a bit more on how different moral conceptions cash out in different recommendations more systematically e.g. with RP’s work, if a CEA-based case can’t be made for a grant I get very skeptical, I in fact also consider CEAs the main input into my thinking on these kinds of things, etc. I am simply wary of single-parameter optimisation taken to the limit in general (for anything, really, not just for donating), and I see your approach as being willing to go much further along that path than I do (and I’m already further along that path than almost anyone I meet IRL).
But I’ve seen enough back-and-forth between people in the cluster and sequence camps to have the sense that nobody really ends up changing their mind substantively and I doubt this will happen here either, sorry, so I will respectfully bow out of the conversation.
I think there’s something to be said that if the qualitative stuff looks bad (e.g. experts are negative, studies are bunch of non-randomized stuff susceptible to endogeneity concerns), you always have the option of just implementing aggressive discounts to the CEA, and the fact that it looks too good means the CEA is done poorly, not that the CEA-focused approach is unreliable.
Makes sense. I also like explicit quantification because it is more transparent, making it easier to examine assumptions, identify main uncertainties, and therefore improve estimates in the future. With approaches associated with cluster thinking, I think it is often unclear which assumptions are driving the decisions, or whether the decisions being made actually follow from the assumptions.
Yeah agreed, that’s why I mentioned the cluster thing as a point in favor of your report’s analytical methodology, and brought up the VC vs index fund analogy, essentially to steer the focus away from pure EV-maxing
Thanks for the discussion!
I think overwhelmingly relying on explicit quantitative cost-effectiveness analyses is generally better. However, one should be mindful to account for considerations not covered, and that one can often cover more considerations by not modelling them formally.
People often point to GiveWell’s posts on not taking expected value estimates literally, and sequence versus cluster thinking to moderate enthusiam about explicit quantitative cost-effectiveness analyses, but GiveWell overwhelmingly relies on these. Elie Hassenfeld, GiveWell’s CEO, mentioned on the Clearer Thinking podcast that:
There is more context from GiveWell here. The key parts are below.
I’ve read those comments awhile back and I don’t think they support your view for relying overwhelmingly on explicit quantitative cost-effectiveness analyses. In particular the key parts I got out of Isabel’s comment weren’t what you quoted but instead (emphasis mine not hers)
and
which is in direct contradistinction to your style as I understand it, and aligned with what Holden wrote earlier in that link you quoted (emphasis his this time)
Note that I’m not saying CEAs don’t matter, or that CEA-focused approaches are unreliable — I’m a big believer in the measurability of things people often claim can’t be measured, I think in principle EV-maxing is almost always correct but in practice it can be perilous and on the margin people should instead be working a bit more on how different moral conceptions cash out in different recommendations more systematically e.g. with RP’s work, if a CEA-based case can’t be made for a grant I get very skeptical, I in fact also consider CEAs the main input into my thinking on these kinds of things, etc. I am simply wary of single-parameter optimisation taken to the limit in general (for anything, really, not just for donating), and I see your approach as being willing to go much further along that path than I do (and I’m already further along that path than almost anyone I meet IRL).
But I’ve seen enough back-and-forth between people in the cluster and sequence camps to have the sense that nobody really ends up changing their mind substantively and I doubt this will happen here either, sorry, so I will respectfully bow out of the conversation.
Thanks for the context, Mo!
I think there’s something to be said that if the qualitative stuff looks bad (e.g. experts are negative, studies are bunch of non-randomized stuff susceptible to endogeneity concerns), you always have the option of just implementing aggressive discounts to the CEA, and the fact that it looks too good means the CEA is done poorly, not that the CEA-focused approach is unreliable.
Makes sense. I also like explicit quantification because it is more transparent, making it easier to examine assumptions, identify main uncertainties, and therefore improve estimates in the future. With approaches associated with cluster thinking, I think it is often unclear which assumptions are driving the decisions, or whether the decisions being made actually follow from the assumptions.