Number of meaningfully different heuristics considered (the criteria in the columns of a WFM)
Number of meaningfully different solutions considered (the options to be evaluated in the rows of a WFM)
For cost-effectiveness analyses (CEAs) and a WFM covering the same variables and options, would the CEAs be preferable? I think so. For example, I like that GiveWell uses cost-effectiveness analyses of the most promising countries instead of a WFM.
I think it depends quite a bit on the quality of the CEA. I would take a sub-5-hour WFM as more useful than a sub-5-hour CEA every time. At 50 hours, I think it becomes a lot less clear. CEAs are much more error-prone and more punishing of those errors compared to WFMs, thus the risk of weaker CEAs. We have more writing on WFMs and CEAs that go into depth about their comparative strengths and weaknesses.
I also think the assessment of GW as CEA-focused is a bit misleading. They have four criteria, two of which they do not explicitly model in their CEA, and many blog posts express their skepticism about taking CEAs literally (my favorite of these, though old).
I think it depends quite a bit on the quality of the CEA. I would take a sub-5-hour WFM as more useful than a sub-5-hour CEA every time. At 50 hours, I think it becomes a lot less clear. CEAs are much more error-prone and more punishing of those errors compared to WFMs, thus the risk of weaker CEAs.
I agree the value of CEAs relative to a WFM increases with time invested.
I also think the assessment of GW as CEA-focused is a bit misleading. They have four criteria, two of which they do not explicitly model in their CEA, and many blog posts express their skepticism about taking CEAs literally (my favorite of these, though old).
Elie Hassenfeld (GiveWell’s co-founder and CEO) mentioned on the Clearer Thinking podcast that (emphasis mine):
GiveWell cost- effectiveness estimates are not the only input into our decisions to fund malaria programs and deworming programs, there are some other factors, but they’re certainly 80% plus of the case.
Isabel Arjmand (GiveWell’s special projects officer at the time) also said (Isabel’s emphasis):
The numerical cost-effectiveness estimate in the spreadsheet is nearly always the most important factor in our recommendations, but not the only factor. That is, we don’t solely rely on our spreadsheet-based analysis of cost-effectiveness when making grants.
Thanks, Joey and Ben.
For cost-effectiveness analyses (CEAs) and a WFM covering the same variables and options, would the CEAs be preferable? I think so. For example, I like that GiveWell uses cost-effectiveness analyses of the most promising countries instead of a WFM.
I think it depends quite a bit on the quality of the CEA. I would take a sub-5-hour WFM as more useful than a sub-5-hour CEA every time. At 50 hours, I think it becomes a lot less clear. CEAs are much more error-prone and more punishing of those errors compared to WFMs, thus the risk of weaker CEAs. We have more writing on WFMs and CEAs that go into depth about their comparative strengths and weaknesses.
I also think the assessment of GW as CEA-focused is a bit misleading. They have four criteria, two of which they do not explicitly model in their CEA, and many blog posts express their skepticism about taking CEAs literally (my favorite of these, though old).
Thanks for clarifying, Joey!
I agree the value of CEAs relative to a WFM increases with time invested.
Elie Hassenfeld (GiveWell’s co-founder and CEO) mentioned on the Clearer Thinking podcast that (emphasis mine):
Isabel Arjmand (GiveWell’s special projects officer at the time) also said (Isabel’s emphasis):