GiveWell has dozens of researchers putting tens of thousands of hours of work into coming up with better models and variable estimates. Their most critical inputs are largely determined by RCTs, and they are constantly working to get better data. A lot of their uncertainty comes from differences in moral weights in saving vs. improving lives.
Founders Pledge makes models using monte carlo simulations on complex theory of change models where the variables ranges are made up because they are largely unknowable. It’s mostly Johannes, with a few assistant researchers, putting in a few hundreds of hours into model choice and parameter selection—with many more hours spent on writing and coding for their monte carlo analysis (which Givewell doesn’t have to do, because they’ve got much simpler impact models in spreadsheets). FP has previously made 1/mtCO2e cost-effectiveness claims based on models like this, which was amplified in MacAskill’s WWOTF. This model is wildly optimistic. FP now disowns that particularly model, but won’t take it down or publicly list it as a mistake. They no longer publish their particular intervention CEAs publicly, though they may resume soon. My biggest criticism is that when making these complex theory-of-change models, the structure of model often matters more than than the variable inputs. While FP tries to pick “conservative” variable value assumptions (they rarely are), the model structure is wildly optimistic for their chosen interventions (generally technology innovation policy). For model feedback, FP doesn’t have a good culture or process in place that deals with criticism well, a complaint that I’ve heard from several in the EA climate space. I think FP’s uncertainty work has promise as a tool, but I think the recommendations they come up with are largely wrong given their chosen model structure and inputs.
GiveWell’s recommendations in the health space are of vastly higher quality and certainty than FP’s in the climate space.
GiveWell has dozens of researchers putting tens of thousands of hours of work into coming up with better models and variable estimates. Their most critical inputs are largely determined by RCTs, and they are constantly working to get better data. A lot of their uncertainty comes from differences in moral weights in saving vs. improving lives.
Founders Pledge makes models using monte carlo simulations on complex theory of change models where the variables ranges are made up because they are largely unknowable. It’s mostly Johannes, with a few assistant researchers, putting in a few hundreds of hours into model choice and parameter selection—with many more hours spent on writing and coding for their monte carlo analysis (which Givewell doesn’t have to do, because they’ve got much simpler impact models in spreadsheets). FP has previously made 1/mtCO2e cost-effectiveness claims based on models like this, which was amplified in MacAskill’s WWOTF. This model is wildly optimistic. FP now disowns that particularly model, but won’t take it down or publicly list it as a mistake. They no longer publish their particular intervention CEAs publicly, though they may resume soon. My biggest criticism is that when making these complex theory-of-change models, the structure of model often matters more than than the variable inputs. While FP tries to pick “conservative” variable value assumptions (they rarely are), the model structure is wildly optimistic for their chosen interventions (generally technology innovation policy). For model feedback, FP doesn’t have a good culture or process in place that deals with criticism well, a complaint that I’ve heard from several in the EA climate space. I think FP’s uncertainty work has promise as a tool, but I think the recommendations they come up with are largely wrong given their chosen model structure and inputs.
GiveWell’s recommendations in the health space are of vastly higher quality and certainty than FP’s in the climate space.