My best guess is that the All or Nothing theorist associates numbers with mathematical certainty. So, to use numbers to present one’s best estimate inherently “projects absolute confidence”
I think a version of this critique is still entirely fair. My problem here is that the numbers are often presented or spread without uncertainty qualifications.
As of July 2022, GiveWell estimates that AMF can deliver a LLIN at a cost of about $5, and that a donation to AMF has an average cost-effectiveness of $5,500 per life saved.[7][8][9]
This statement gives no information about how sure they are about the $5 or $5500 figure. Is givewell virtually certain the cost effectiveness it’s in the range of $5000 to $6000? Or do they think it could be between $2000 and $9000? Givewell explains it’s methodology in detail, but their uncertainty ranges are dropped when this claim is spread (do you know of the top of your head what their uncertainty is?). Absent these ranges, I see these claims repeated all over the place as if $5000 really is an objectively correct answer and not a rough estimate.
I actually think that’s fine. You can always look it up if you’re interested in the details, but for the casual consumer of charity-evaluation information, the bottom-line best estimate is the info that’s decision-relevant, not the uncertainty range. I think it’s completely fine for people to share core info like this without simultaneously sharing all the fine print. Just like it’s OK for public health experts to promote simple pro-vax messaging that doesn’t include all the fine print.
(See moral misdirection for my principled account of when it is or isn’t OK to leave out information.)
Absent these ranges, I see these claims repeated all over the place as if $5000 really is an objectively correct answer and not a rough estimate.
Here you just seem to be repeating the mistake of assuming that presenting a best estimate without also presenting the uncertainty range is thereby to present it as certain. I disagree with that interpretative norm. There is no “as if” being presented. That’s on you.
I’ll take an intermediate position: most readers will at least unconsciously infer an uncertainty range when presented with a point estimate only. If my mechanic tells me their best estimate for fixing my car is $1000 without saying more, I should understand from that $1200 is a reasonable possibility but would legitimately be upset if presented with a $2000 bill even if $1000 were provably the mean, median, mode, and likely outcome.
Here, I think the reader is on notice that estimating cost to save a life is likely to involve some imprecision, plus it is presented as an estimate, it is linked to a more detailed explanation, and it is rounded off.
There would be cases in which more should be said about the uncertainty range, for instance if it were between $500 and $50K! In that kind of scenario, you would need to say more to clue the reader into the degree of imprecision.
Yeah. The words “estimates” and “about” are right there in the quote. There is no pretension of certainty here, unless you think mere use of numbers amounts to pretended certainty.
But what is decision relevant is the expected value. So by best estimate do they mean expected value, or maximum likelihood estimate, or something else? To my ear, “best estimate” sounds like it means the estimate most likely to be right, and not the mean of the probability distribution. For instance, take the (B) option in “Why it can be OK to predictably lose”, where you have a 1% chance of saving 1000 people, and a 99% chance of saving no one, and the choice is non-repeatable. I would think the “best estimate” of the effectiveness of option (B) is that you will save 0 lives. But what matters for decision making is the expected value which is 10 lives.
Sorry if this is a stupid question, I’m not very familiar with GiveWell.
Fair question! I don’t know the answer. But I’d be surprised if the two came apart too sharply in this case (even though, as you rightly note, they can drastically diverge in principle). My sense is that GiveWell aims to recommend relatively “safe” bets, rather than a “hits-based” EV-maximizing approach. (I think it’s important to be transparent when recommending the latter, just because I take it many people are not in fact so comfortable with pursuing that strategy, even if I think they ought to be.)
I think a version of this critique is still entirely fair. My problem here is that the numbers are often presented or spread without uncertainty qualifications.
For example, the EA page on the against malaria foundation states:
This statement gives no information about how sure they are about the $5 or $5500 figure. Is givewell virtually certain the cost effectiveness it’s in the range of $5000 to $6000? Or do they think it could be between $2000 and $9000? Givewell explains it’s methodology in detail, but their uncertainty ranges are dropped when this claim is spread (do you know of the top of your head what their uncertainty is?). Absent these ranges, I see these claims repeated all over the place as if $5000 really is an objectively correct answer and not a rough estimate.
I actually think that’s fine. You can always look it up if you’re interested in the details, but for the casual consumer of charity-evaluation information, the bottom-line best estimate is the info that’s decision-relevant, not the uncertainty range. I think it’s completely fine for people to share core info like this without simultaneously sharing all the fine print. Just like it’s OK for public health experts to promote simple pro-vax messaging that doesn’t include all the fine print.
(See moral misdirection for my principled account of when it is or isn’t OK to leave out information.)
Here you just seem to be repeating the mistake of assuming that presenting a best estimate without also presenting the uncertainty range is thereby to present it as certain. I disagree with that interpretative norm. There is no “as if” being presented. That’s on you.
I’ll take an intermediate position: most readers will at least unconsciously infer an uncertainty range when presented with a point estimate only. If my mechanic tells me their best estimate for fixing my car is $1000 without saying more, I should understand from that $1200 is a reasonable possibility but would legitimately be upset if presented with a $2000 bill even if $1000 were provably the mean, median, mode, and likely outcome.
Here, I think the reader is on notice that estimating cost to save a life is likely to involve some imprecision, plus it is presented as an estimate, it is linked to a more detailed explanation, and it is rounded off.
There would be cases in which more should be said about the uncertainty range, for instance if it were between $500 and $50K! In that kind of scenario, you would need to say more to clue the reader into the degree of imprecision.
Agreed!
Yeah. The words “estimates” and “about” are right there in the quote. There is no pretension of certainty here, unless you think mere use of numbers amounts to pretended certainty.
But what is decision relevant is the expected value. So by best estimate do they mean expected value, or maximum likelihood estimate, or something else? To my ear, “best estimate” sounds like it means the estimate most likely to be right, and not the mean of the probability distribution. For instance, take the (B) option in “Why it can be OK to predictably lose”, where you have a 1% chance of saving 1000 people, and a 99% chance of saving no one, and the choice is non-repeatable. I would think the “best estimate” of the effectiveness of option (B) is that you will save 0 lives. But what matters for decision making is the expected value which is 10 lives.
Sorry if this is a stupid question, I’m not very familiar with GiveWell.
Fair question! I don’t know the answer. But I’d be surprised if the two came apart too sharply in this case (even though, as you rightly note, they can drastically diverge in principle). My sense is that GiveWell aims to recommend relatively “safe” bets, rather than a “hits-based” EV-maximizing approach. (I think it’s important to be transparent when recommending the latter, just because I take it many people are not in fact so comfortable with pursuing that strategy, even if I think they ought to be.)