Potential downsides of using explicit probabilities

Epistemic sta­tus: This is ba­si­cally meant as a col­lec­tion and anal­y­sis of ex­ist­ing ideas, not as any­thing brand new. I’m not an ex­pert on the top­ics cov­ered. I’d ap­pre­ci­ate feed­back or com­ments in re­la­tion to any mis­takes, un­clear phras­ings, etc. (and just in gen­eral!).

In var­i­ous com­mu­ni­ties (in­clud­ing the EA and ra­tio­nal­ist com­mu­ni­ties), it’s com­mon to make use of ex­plicit, nu­mer­i­cal prob­a­bil­ities.[1]

At the ex­treme end, this may in­volve ex­plicit at­tempts to calcu­late what would max­imise ex­pected util­ity, and then do that thing.

It could also in­volve at­tempts to cre­ate ex­plicit, prob­a­bil­is­tic mod­els (EPMs), per­haps in­volv­ing ex­pected value calcu­la­tions, and use this as an in­put into de­ci­sion-mak­ing. (So the EPM may not nec­es­sar­ily be the only in­put, or nec­es­sar­ily be in­tended to in­clude ev­ery­thing that’s im­por­tant.) Ex­am­ples of this in­clude the cost-effec­tive­ness analy­ses cre­ated by GiveWell or ALLFED.

Most sim­ply, a per­son may gen­er­ate just a sin­gle ex­plicit prob­a­bil­ity (EP; e.g., “I have a 20% chance of get­ting this job”), and then use that as an in­put into de­ci­sion-mak­ing.

(For sim­plic­ity, in this post I’ll of­ten say “us­ing EPs” as a catchall term for us­ing a sin­gle EP, us­ing EPMs, or max­imis­ing ex­pected util­ity. I’ll also of­ten say “al­ter­na­tive ap­proaches” to re­fer to more qual­i­ta­tive or in­tu­itive meth­ods, rang­ing from sim­ply “trust­ing your gut” to ex­ten­sive de­liber­a­tions where you don’t ex­plic­itly quan­tify prob­a­bil­ities.)

Many ar­gu­ments for the value of us­ing EPs have been cov­ered el­se­where (and won’t be cov­ered here). I find many of these quite com­pel­ling, and be­lieve that one of the ma­jor things the EA and ra­tio­nal­ist com­mu­ni­ties get right is rely­ing on EPs more than the gen­eral pub­lic does.

But use of EPs is also of­ten crit­i­cised. And it’s cer­tainly the case that I (and I sus­pect most EAs and ra­tio­nal­ists) don’t use EPs for most ev­ery­day de­ci­sions, at least, and I think that that’s prob­a­bly of­ten a good thing.

So the first aim of this post is to ex­plore some po­ten­tial down­sides of us­ing EPs (com­pared to al­ter­na­tive ap­proaches) that peo­ple have pro­posed. I’ll fo­cus on not the case of ideal ra­tio­nal agents, but of ac­tual hu­mans, in prac­tice, with our bi­ases and limited com­pu­ta­tional abil­ities. Speci­fi­cally, I dis­cuss the fol­low­ing (non-ex­haus­tive) list of po­ten­tial down­sides:

  1. Time and effort costs

  2. Ex­clud­ing some of one’s knowl­edge (which could’ve been lev­er­aged by al­ter­na­tive ap­proaches)

  3. Caus­ing overconfidence

  4. Un­der­es­ti­mat­ing the value of information

  5. The op­ti­mizer’s curse

  6. An­chor­ing (to the EP, or to the EPM’s out­put)

  7. Caus­ing rep­u­ta­tional issues

As I’ll dis­cuss, these down­sides will not always ap­ply when us­ing EPs, and many will also some­times ap­ply when us­ing al­ter­na­tive ap­proaches. And when these down­sides do ap­ply to uses of EPs, they may of­ten be out­weighed by the benefits of us­ing EPs. So this post is not meant to defini­tively de­ter­mine the sorts of situ­a­tions one should vs shouldn’t use EPs in. But I do think these down­sides are of­ten at least im­por­tant fac­tors to con­sider.

Some­times peo­ple go fur­ther, and link dis­cus­sion of these po­ten­tial down­sides of us­ing EPs as hu­mans, in prac­tice, to claims like that there’s an ab­solute, bi­nary dis­tinc­tion be­tween “risk” and “(Knigh­tian) un­cer­tainty”, or be­tween situ­a­tions in which we “have” vs “don’t have” prob­a­bil­ities, or some­thing like that. Here’s one state­ment of this sort of view (from Do­minic Roser, who dis­agrees with it):

Ac­cord­ing to [one] view, cer­tainty has two op­po­sites: risk and un­cer­tainty. In the case of risk, we lack cer­tainty but we have prob­a­bil­ities. In the case of un­cer­tainty, we do not even have prob­a­bil­ities. [...] Ac­cord­ing to a pop­u­lar view, then, how we ought to make policy de­ci­sions de­pends cru­cially on whether we have prob­a­bil­ities.

I’ve pre­vi­ously ar­gued that there’s no ab­solute, bi­nary risk-un­cer­tainty dis­tinc­tion, and that be­liev­ing that there is such a dis­tinc­tion can lead to us­ing bad de­ci­sion-mak­ing pro­ce­dures. I’ve also ar­gued that we can always as­sign prob­a­bil­ities (or at least use some­thing like an un­in­for­ma­tive prior). But I didn’t ad­dress the idea that it might be valuable for hu­mans to act as if there’s a bi­nary risk-un­cer­tainty dis­tinc­tion, or as if it’s im­pos­si­ble to as­sign prob­a­bil­ities in some cases.

Thus, the sec­ond aim of this post is to ex­plore whether that’s a good idea. I ar­gue that it is not (with the one po­ten­tial, par­tial ex­cep­tion of rep­u­ta­tional is­sues).

So each sec­tion will:

  • out­line a po­ten­tial down­side of us­ing EPs

  • dis­cuss whether that down­side re­ally ap­plies more to us­ing EPs than to al­ter­na­tive approaches

  • ex­plain why I be­lieve this down­side does not sug­gest one should even act as if there’s a bi­nary risk-un­cer­tainty distinction

Time and effort costs

The most ob­vi­ous down­side of us­ing EPs (or at least EPMs) is that it may of­ten take a lot of time and en­ergy to use them well enough to get bet­ter re­sults than one would get from al­ter­na­tive ap­proaches (e.g., trust­ing your gut).

For ex­am­ple, GiveWell’s re­searchers col­lec­tively spend “hun­dreds of hours [...] per year on cost-effec­tive­ness anal­y­sis”. I’d ar­gue that that’s worth­while when the stakes are as high as they are in GiveWell’s case (i.e., de­ter­min­ing which char­i­ties re­ceive tens of mil­lions of dol­lars each year).

But what if I’m just de­cid­ing what head­phones to buy? Is it worth it for me to spend a few hours con­struct­ing a de­tailed model of all the fac­tors rele­vant to the ques­tion, and then find­ing (or es­ti­mat­ing) val­ues for each of those fac­tors, for each of a broad range of differ­ent head­phones?

Here, the stakes in­volved are quite low, and it’s also fairly un­likely that I’ll use the EPM again. (In con­trast, GiveWell con­tinues to use its mod­els, with mod­ifi­ca­tions, year af­ter year, mak­ing the ini­tial in­vest­ment in con­struct­ing the mod­els more worth­while.) It seems the ex­pected value of me both­er­ing to do this EPM is lower than the ex­pected value of me just read­ing a few re­views and then “go­ing with my gut” (and thus sav­ing time for other things).[2][3]

Does this mean that we must be deal­ing with “Knigh­tian un­cer­tainty” in this case, or must be ut­terly un­able to “know” the rele­vant prob­a­bil­ities?

Not at all. In fact, I’d ar­gue that the head­phones ex­am­ple is ac­tu­ally one where, if I did spend a few hours do­ing re­search, I could come up with prob­a­bil­ities that are much more “trust­wor­thy” than many of the prob­a­bil­ities in­volved in situ­a­tions like GiveWell’s (when it is use­ful for peo­ple to con­struct EPMs). So I think the is­sue of time and effort costs may be quite sep­a­rate even from the ques­tion of how trust­wor­thy our prob­a­bil­ities are, let alone the idea that there might be a bi­nary risk-un­cer­tainty dis­tinc­tion.

Ex­clud­ing some of one’s knowledge

Let’s say that I’m an ex­pe­rienced fire­fighter in a burn­ing build­ing (un­true on both counts, but go with me on this). I want to know the odds that the floor I’m on will col­lapse. I could (quite ar­bi­trar­ily) con­struct the fol­low­ing EPM:

Prob­a­bil­ity of col­lapse = How hot the build­ing is (on a scale from 0-1) * How non-stur­dily the build­ing seems to have been built (on a scale from 0-1)

I could also (quite ar­bi­trar­ily) de­cide on val­ues of 0.6 and 0.5, re­spec­tively. My model would then tell me that the prob­a­bil­ity of the floor col­laps­ing is 0.3.

It seems like that could be done quite quickly, and while do­ing other things. So it seems that the time and effort costs in­volved in us­ing this EPM are prob­a­bly very similar to the costs in­volved in us­ing an al­ter­na­tive ap­proach (e.g., trust­ing my gut). Does this mean con­struct­ing an EPM here is a wise choice?

In­tu­itive expertise

There’s em­piri­cal ev­i­dence that the an­swer is “No” for ex­am­ples like this; i.e., ex­am­ples which meet the “con­di­tions for in­tu­itive ex­per­tise”:

  • an en­vi­ron­ment in which there’s a sta­ble re­la­tion­ship be­tween iden­ti­fi­able cues and later events or out­comes of actions

  • “ad­e­quate op­por­tu­ni­ties for learn­ing the en­vi­ron­ment (pro­longed prac­tice and feed­back that is both rapid and un­equiv­o­cal)” (Kah­ne­man & Klein)

In such situ­a­tions, our in­tu­itions may quite re­li­ably pre­dict later events. Fur­ther­more, we may not con­sciously, ex­plic­itly know the fac­tors that in­formed these in­tu­itions. As Kah­ne­man & Klein write: “Skil­led judges are of­ten un­aware of the cues that guide them”.

Klein de­scribes the true story that in­spired my ex­am­ple, in which a team of fire­fighters were deal­ing with what they thought was a typ­i­cal kitchen fire, when the lieu­tenant:

be­came tremen­dously un­easy — so un­easy that he or­dered his en­tire crew to va­cate the build­ing. Just as they were leav­ing, the liv­ing room floor col­lapsed. If they had stood there an­other minute, they would have dropped into the fire be­low. Un­be­knownst to the fire­fighters, the house had a base­ment and that’s where the fire was burn­ing, right un­der the liv­ing room.

I had a chance to in­ter­view the lieu­tenant about this in­ci­dent, and asked him why he gave the or­der to evac­u­ate. The only rea­son he could think of was that he had ex­trasen­sory per­cep­tion. He firmly be­lieved he had ESP.

Dur­ing the in­ter­view I asked him what he was aware of. He men­tioned that it was very hot in the liv­ing room, much hot­ter than he ex­pected given that he thought the fire was in the kitchen next door. I pressed him fur­ther and he re­called that, not only was it hot­ter than he ex­pected, it was also quieter than he ex­pected. Fires are usu­ally noisy but this fire wasn’t. By the end of the in­ter­view he un­der­stood why it was so quiet: be­cause the fire was in the base­ment, and the floor was muffling the sounds.

It seems that the lieu­tenant wasn’t con­sciously aware of the im­por­tance of the quiet­ness of the fire. As such, if he’d con­structed and re­lied on an EPM, he wouldn’t have in­cluded the quiet­ness as a fac­tor, and thus may not have pul­led his crew out in time. But through a great deal of ex­per­tise, with re­li­able feed­back from the en­vi­ron­ment, he was in­tu­itively aware of the im­por­tance of that fac­tor.

So when the con­di­tions for in­tu­itive ex­per­tise are met, meth­ods other than EPM may re­li­ably out­perform EPM, even ig­nor­ing costs in time and en­ergy, be­cause they al­low us to more fully lev­er­age our knowl­edge.[4]

But, again, does this mean that we must be deal­ing with “Knigh­tian un­cer­tainty” in this case, or must be ut­terly un­able to “know” the rele­vant prob­a­bil­ities? Again, not at all. In fact, the con­di­tions for in­tu­itive ex­per­tise would ac­tu­ally be met pre­cisely when we could have rel­a­tively trust­wor­thy prob­a­bil­ities—there have to be fairly sta­ble pat­terns in the en­vi­ron­ment, and op­por­tu­ni­ties to learn these pat­terns. The is­sue is sim­ply that, in prac­tice, we of­ten haven’t learned these prob­a­bil­ities on a con­scious, ex­plicit level, even though we the­o­ret­i­cally could have.

On the flip­side, us­ing EPMs may of­ten beat al­ter­na­tive meth­ods when the con­di­tions for in­tu­itive ex­per­tise aren’t met, and this may be most likely when we face es­pe­cially _un_trust­wor­thy prob­a­bil­ities. Re­lat­edly, it’s worth not­ing that just the fact that, in a par­tic­u­lar situ­a­tion, we feel more con­fi­dent in our in­tu­itive as­sess­ment than in an EPM doesn’t nec­es­sar­ily mean our in­tu­itive as­sess­ment is ac­tu­ally more re­li­able in that situ­a­tion. As Kah­ne­man & Klein note:

True ex­perts, it is said, know when they don’t know. How­ever, non­ex­perts (whether or not they think they are) cer­tainly do not know when they don’t know. Sub­jec­tive con­fi­dence is there­fore an un­re­li­able in­di­ca­tion of the val­idity of in­tu­itive judg­ments and de­ci­sions.

[...] Although true skill can­not de­velop in ir­reg­u­lar or un­pre­dictable en­vi­ron­ments, in­di­vi­d­u­als will some times make judg­ments and de­ci­sions that are suc­cess­ful by chance. Th­ese “lucky” in­di­vi­d­u­als will be sus­cep­ti­ble to an illu­sion of skill and to over­con­fi­dence (Arkes, 2001). The fi­nan­cial in­dus­try is a rich source of ex­am­ples.

Less mea­surable or leg­ible things

An ad­di­tional ar­gu­ment is that us­ing EPs may make it harder to lev­er­age knowl­edge about things that are less mea­surable and/​or leg­ible (with leg­i­bil­ity seem­ing to ap­prox­i­mately mean sus­cep­ti­bil­ity to be­ing pre­dicted, un­der­stood, and mon­i­tored).

For ex­am­ple, Alice is de­cided whether to donate to the Cen­tre for Pes­ti­cide Suicide Preven­tion (CPSP), which fo­cuses on ad­vo­cat­ing for policy changes, or to GiveDirectly, which sim­ply gives un­con­di­tional cash trans­fers to peo­ple liv­ing in ex­treme poverty. She may de­cide CPSP’s im­pacts are “too hard to mea­sure”, and “just can’t be es­ti­mated quan­ti­ta­tively”. Thus, if she uses EPs, she might ne­glect to even se­ri­ously con­sider CPSP. But if she con­sid­ered in-depth, qual­i­ta­tive ar­gu­ments, she might de­cide that CPSP seems a bet­ter bet.

I think it’s very plau­si­ble that this is a sort of situ­a­tion where, in or­der to lev­er­age as much of one’s knowl­edge as pos­si­ble, it’s wise to use qual­i­ta­tive ap­proaches. But we can still use EPs in these cases—we can just give our best guesses about the value of vari­ables we can’t mea­sure, and about what vari­ables to con­sider and how to struc­ture our model. (And in fact, GiveWell did con­struct a quan­ti­ta­tive cost-effec­tive­ness model for CPSP.) And it’s not ob­vi­ous to me which of these ap­proaches would typ­i­cally make it eas­ier for us to lev­er­age our knowl­edge in these less mea­surable and leg­ible cases.

Fi­nally, what im­pli­ca­tions might this is­sue have for the idea of a bi­nary risk-un­cer­tainty dis­tinc­tion? I dis­agree with Alice’s view that CPSP’s im­pacts “just can’t be es­ti­mated quan­ti­ta­tively”. The re­al­ity is sim­ply that CPSP’s im­pacts are very hard to es­ti­mate, and that the prob­a­bil­ities we’d ar­rive at if we es­ti­mated them would be quite un­trust­wor­thy. In con­trast, our es­ti­mates of GiveDirectly’s im­pact would be rel­a­tively more trust­wor­thy. That’s all we need to say to make sense of the idea that this is (per­haps) a situ­a­tion in which we should use ap­proaches other than EPs; I don’t think we need to even act as if there’s a bi­nary risk-un­cer­tainty dis­tinc­tion.

Caus­ing over­con­fi­dence; un­der­es­ti­mat­ing the value of information

Two com­mon cri­tiques of us­ing EPs are that:

  • Us­ing EPs tends to make one over­con­fi­dent about their es­ti­mates (and their mod­els’ out­puts); that is, it makes them un­der­es­ti­mate how un­cer­tain these es­ti­mates or out­puts are.[5]

  • There­fore, us­ing EPs tends to make one un­der­es­ti­mate the value of (ad­di­tional) in­for­ma­tion (VoI; here “in­for­ma­tion” can be seen as in­clud­ing just do­ing more think­ing, with­out ac­tu­ally gath­er­ing more em­piri­cal data)

Th­ese cri­tiques are closely re­lated, so I’ll dis­cuss both in this sec­tion.

An ex­am­ple of the first of those cri­tiques comes from Chris Smith. Smith dis­cusses one par­tic­u­lar method for deal­ing with “poorly un­der­stood un­cer­tainty”, and then writes:

Cal­ling [that method] “mak­ing a Bayesian ad­just­ment” sug­gests that we have some­thing like a gen­eral, math­e­mat­i­cal method for crit­i­cal think­ing. We don’t.

Similarly, tak­ing our hunches about the plau­si­bil­ity of sce­nar­ios we have a very limited un­der­stand­ing of and treat­ing those hunches like well-grounded prob­a­bil­ities can lead us to be­lieve we have a well-un­der­stood method for mak­ing good de­ci­sions re­lated to those sce­nar­ios. We don’t.

Many peo­ple have un­war­ranted con­fi­dence in ap­proaches that ap­pear math-heavy or sci­en­tific. In my ex­pe­rience, effec­tive al­tru­ists are not im­mune to that bias.

An ex­am­ple of (I think) both of those cri­tiques to­gether comes from Daniela Wald­horn:

The ex­ist­ing gaps in this field of re­search en­tail that we face sig­nifi­cant con­straints when as­sess­ing the prob­a­bil­ity that an in­ver­te­brate taxon is con­scious. In my opinion, the cur­rent state of knowl­edge is not ma­ture enough for any in­for­ma­tive nu­mer­i­cal es­ti­ma­tion of con­scious­ness among in­ver­te­brates. Fur­ther­more, there is a risk that such es­ti­mates lead to an over­sim­plifi­ca­tion of the prob­lem and an un­der­es­ti­ma­tion of the need for fur­ther in­ves­ti­ga­tion.

I’m some­what sym­pa­thetic to these ar­gu­ments. But I think it’s very un­clear whether ar­gu­ments about over­con­fi­dence and VoI should push us away from rather than to­wards us­ing EPs; it re­ally seems like it could go ei­ther way. This is for two rea­sons.

Firstly, we can clearly rep­re­sent low con­fi­dence in our EPs, by:

  • us­ing a prob­a­bil­ity dis­tri­bu­tion, rather than just a point estimate

  • giv­ing that dis­tri­bu­tion (ar­bi­trar­ily) wide con­fi­dence intervals

  • choos­ing the shape of that dis­tri­bu­tion to fur­ther rep­re­sent the mag­ni­tude (and na­ture) of our un­cer­tainty. (See this com­ment for di­a­grams.)

  • con­duct­ing sen­si­tivity analy­ses, which show the ex­tent to which plau­si­ble (given our un­cer­tainty) vari­a­tions in our model’s in­puts can af­fect our model’s outputs

  • vi­su­ally rep­re­sent­ing these prob­a­bil­ity dis­tri­bu­tions and sen­si­tivity analy­ses (which may make our un­cer­tainty more strik­ing and harder to ig­nore)

Se­condly, if we do use EPs (and ap­pro­pri­ately wide con­fi­dence in­ter­vals), this un­locks ways of mov­ing be­yond just the gen­eral idea that fur­ther in­for­ma­tion would be valuable; it lets us also:

  • ex­plic­itly calcu­late how valuable more info seems likely to be;

  • iden­tify which un­cer­tain­ties it’d be most valuable to gather more info on.

In fact, there’s an en­tire body of work on VoI anal­y­sis, and a nec­es­sary pre­req­ui­site for con­duct­ing such an anal­y­sis is hav­ing an EPM.

It does seem plau­si­ble to me that, even if we do all of those things, we or oth­ers will pri­mar­ily fo­cus on our (per­haps im­plicit) point es­ti­mate, and over­es­ti­mate its trust­wor­thi­ness, just due to hu­man psy­chol­ogy (or EA/​ra­tio­nal­ist psy­chol­ogy). But that doesn’t seem ob­vi­ous. Nor does it seem ob­vi­ous that the over­con­fi­dence that may re­sult from us­ing EPs will tend to be greater than the over­con­fi­dence that may re­sult from other ap­proaches (like rely­ing on all-things-con­sid­ered in­tu­itions; re­call Kah­ne­man & Klein’s com­ments from ear­lier).

And in any case, this whole dis­cus­sion was easy to have just in terms of very un­trust­wor­thy or low-con­fi­dence prob­a­bil­ities—there was no need to in­voke the idea of a bi­nary risk-un­cer­tainty dis­tinc­tion, or the idea that there are some mat­ters about which we can sim­ply can’t pos­si­bly es­ti­mate any prob­a­bil­ities.[6]

The op­ti­mizer’s curse

Smith gives a “rough sketch” of the op­ti­mizer’s curse:

Op­ti­miz­ers start by calcu­lat­ing the ex­pected value of differ­ent ac­tivi­ties.

Es­ti­mates of ex­pected value in­volve un­cer­tainty.

Some­times ex­pected value is over­es­ti­mated, some­times ex­pected value is un­der­es­ti­mated.

Op­ti­miz­ers aim to en­gage in ac­tivi­ties with the high­est ex­pected val­ues.

Re­sult: Op­ti­miz­ers tend to se­lect ac­tivi­ties with over­es­ti­mated ex­pected value.

[...] The op­ti­mizer’s curse oc­curs even in sce­nar­ios where es­ti­mates of ex­pected value are un­bi­ased (roughly, where any given es­ti­mate is as likely to be too op­ti­mistic as it is to be too pes­simistic).

[...] As un­cer­tainty in­creases, the de­gree to which the cost-effec­tive­ness of the op­ti­mal-look­ing pro­gram is over­stated grows wildly.

The im­pli­ca­tions of, and po­ten­tial solu­tions to, the op­ti­mizer’s curse seem to be com­pli­cated and de­bat­able. For more de­tail, see this post, Smith’s post, com­ments on Smith’s post, and dis­cus­sion of the re­lated prob­lem of Good­hart’s law.

As best I can tell:

  • The op­ti­mizer’s curse is likely to be a per­va­sive prob­lem and is worth tak­ing se­ri­ously.

  • In many situ­a­tions, the curse will just in­di­cate that we’re prob­a­bly over­es­ti­mat­ing how much bet­ter (com­pared to the al­ter­na­tives) the op­tion we es­ti­mate is best is—it won’t in­di­cate that we should ac­tu­ally change what op­tion we pick.

  • But the curse can in­di­cate that we should pick an op­tion other than that which we es­ti­mate is best, if we have rea­son to be­lieve that our es­ti­mate of the value of the best op­tion is es­pe­cially un­cer­tain, and we don’t model that in­for­ma­tion.

I’ve de­liber­ately kept the above points brief (again, see the sources linked to for fur­ther ex­pla­na­tions and jus­tifi­ca­tions). This is be­cause those claims are only rele­vant to the ques­tion of when to use EPs if the op­ti­mizer’s curse is a larger prob­lem when us­ing EPs than when us­ing al­ter­na­tive ap­proaches, and I don’t think it nec­es­sar­ily is. For ex­am­ple, Smith notes:

The op­ti­mizer’s curse can show up even in situ­a­tions where effec­tive al­tru­ists’ pri­ori­ti­za­tion de­ci­sions don’t in­volve for­mal mod­els or ex­plicit es­ti­mates of ex­pected value. Some­one in­for­mally as­sess­ing philan­thropic op­por­tu­ni­ties in a lin­ear man­ner might have a thought like:

“Thing X seems like an awfully big is­sue. Fund­ing Group A would prob­a­bly cost only a lit­tle bit of money and have a small chance lead­ing to a solu­tion for Thing X. Ac­cord­ingly, I feel de­cent about the ex­pected cost-effec­tive­ness of fund­ing Group A.

Let me com­pare that to how I feel about some other fund­ing op­por­tu­ni­ties…”

Although the think­ing is in­for­mal, there’s un­cer­tainty, po­ten­tial for bias, and an op­ti­miza­tion-like pro­cess. (quote marks added be­cause I couldn’t dou­ble-in­dent)

This makes a lot of sense to me. But Smith also adds:

In­for­mal think­ing isn’t always this lin­ear. If the in­for­mal think­ing con­sid­ers an op­por­tu­nity from mul­ti­ple per­spec­tives, draws on in­tu­itions, etc., the risk of [over­es­ti­mat­ing the cost-effec­tive­ness of the op­ti­mal-look­ing pro­gram] may be re­duced.

I’m less sure what he means by this. I’m guess­ing he sim­ply means that us­ing mul­ti­ple, differ­ent per­spec­tives means that the var­i­ous er­rors and un­cer­tain­ties are likely to “can­cel out” to some ex­tent, re­duc­ing the effec­tive un­cer­tainty, and thus re­duc­ing the im­pacts the amount by which one is likely to over­es­ti­mate the value of the best-seem­ing thing. But if so, it seems that this par­tial pro­tec­tion could also be achieve by us­ing mul­ti­ple, differ­ent EPMs, mak­ing differ­ent as­sump­tions in them, get­ting mul­ti­ple peo­ple to es­ti­mate val­ues for in­puts, etc.

So ul­ti­mately, I think that the prob­lem Smith raises is sig­nifi­cant, but I’m quite un­sure if it’s a down­side of us­ing EPs in par­tic­u­lar.

I also don’t think that the op­ti­mizer’s curse sug­gests it’d be valuable to act as if there’s a bi­nary risk-un­cer­tainty dis­tinc­tion. It is clear that the curse gets worse as un­cer­tainty in­creases (i.e., when one’s prob­a­bil­ities are less trust­wor­thy), but it does so in a grad­ual, con­tin­u­ous man­ner. So it seems to me that, again, we’re best off speak­ing just in terms of more and less trust­wor­thy prob­a­bil­ities, and not imag­in­ing that to­tally differ­ent be­havi­ours are war­ranted if we’re fac­ing “risk” rather than “Knigh­tian un­cer­tainty”.[7]


An­chor­ing or fo­cal­ism is a cog­ni­tive bias where an in­di­vi­d­ual de­pends too heav­ily on an ini­tial piece of in­for­ma­tion offered (con­sid­ered to be the “an­chor”) when mak­ing de­ci­sions. (Wikipe­dia)

One cri­tique of us­ing EPs, or at least mak­ing them pub­lic, seems to effec­tively be that peo­ple may be­come an­chored on the EPs given. For ex­am­ple, Ja­son Schukraft writes:

I con­tend that pub­lish­ing spe­cific es­ti­mates of in­ver­te­brate sen­tience (e.g., as­sign­ing each taxon a ‘sen­tience score’) would be, at this stage of in­ves­ti­ga­tion, at best un­helpful and prob­a­bly ac­tively coun­ter­pro­duc­tive. [...]

Of course, hav­ing stud­ied the topic for some time now, I ex­pect that my es­ti­mates would be bet­ter than the es­ti­mates of the av­er­age mem­ber of the EA com­mu­nity. If that’s true, then it’s tempt­ing to con­clude that mak­ing my es­ti­mates pub­lic would im­prove the com­mu­nity’s over­all po­si­tion on this topic. How­ever, I think there are at least three rea­sons to be skep­ti­cal of this view.

[One rea­son is that] It’s difficult to pre­sent ex­plicit es­ti­mates of in­ver­te­brate sen­tience in a way in which those es­ti­mates don’t steal the show. It’s hard to imag­ine a third party sum­ma­riz­ing our work (ei­ther to her­self or to oth­ers) with­out men­tion­ing lines like ‘Re­think Pri­ori­ties think there is an X% chance ants have the ca­pac­ity for valenced ex­pe­rience.’ There are very few se­ri­ous es­ti­mates of in­ver­te­brate sen­tience available, so mem­bers of the com­mu­nity might re­ally fas­ten onto ours.

I think that this cri­tique has sub­stan­tial merit, but that this is most clear in re­la­tion to mak­ing EPs pub­lic, rather than just in re­la­tion to us­ing EPs one­self. As Schukraft writes:

To be clear: I don’t be­lieve it’s a bad idea to think about prob­a­bil­ities of sen­tience. In fact, any­one di­rectly work­ing on in­ver­te­brate sen­tience ought to be pe­ri­od­i­cally record­ing their own es­ti­mates for var­i­ous groups of an­i­mals so that they can see how their cre­dences change over time.[8]

I ex­pect that one can some­what miti­gate this is­sue by pro­vid­ing var­i­ous strong caveats when EPs are quite un­trust­wor­thy. And (at least some­what) similar is­sues can also oc­cur when not us­ing EPs (e.g., if just say­ing some­thing is “very likely”, or giv­ing a gen­eral im­pres­sion of dis­ap­proval of what a cer­tain or­gani­sa­tion is do­ing). But I think caveats wouldn’t re­move the is­sue en­tirely.[9] And I’d guess that the an­chor­ing would be worse if us­ing EPs than if not.

Fi­nally, an­chor­ing does seem a more im­por­tant down­side when one’s prob­a­bil­ities are less trust­wor­thy (be­cause then the odds peo­ple will be an­chored to a bad es­ti­mate are higher). But again, it seems easy, and best, to think about this in terms of more and less trust­wor­thy prob­a­bil­ities, rather than in terms of a bi­nary risk-un­cer­tainty dis­tinc­tion.

Rep­u­ta­tional issues

Fi­nally, in the same post, Schukraft notes an­other is­sue with us­ing EPs:

Sen­tience scores might re­duce our cred­i­bil­ity with po­ten­tial collaborators

[....] sci­ence, es­pe­cially peer-re­viewed sci­ence, is an in­her­ently con­ser­va­tive en­ter­prise. Scien­tists sim­ply don’t pub­lish things like prob­a­bil­ities of sen­tience. For a long time, even the topic of non­hu­man sen­tience was taboo be­cause it was seen as un­ver­ifi­able. Without a clear, em­piri­cally-val­i­dated method­ol­ogy be­hind them, such es­ti­mates would prob­a­bly not make it into a rep­utable jour­nal. In­tu­itions, even in­tu­itions con­di­tioned by care­ful re­flec­tion, are rarely ad­mit­ted in the court of sci­en­tific opinion.

Re­think Pri­ori­ties is a new, non-aca­demic or­ga­ni­za­tion, and it is part of a move­ment that is—frankly—sort of weird. To col­lab­o­rate with sci­en­tists, we first need to con­vince them that we are a le­gi­t­i­mate re­search out­fit. I don’t want to make that task more challeng­ing by pub­lish­ing es­ti­mates that in­tro­duce the per­cep­tion that our re­search isn’t rigor­ous. And I don’t think that per­cep­tion would be en­tirely un­war­ranted. When­ever I read a post and en­counter an overly pre­cise pre­dic­tion for a com­plex event (e.g., ‘there is a 16% chance Latin Amer­ica will dom­i­nate the plant-based seafood mar­ket by 2025’), I come away with the im­pres­sion that the au­thor doesn’t suffi­ciently ap­pre­ci­ate the com­plex­ity of the forces at play. There may be no sin­gle sub­ject more com­pli­cated than con­scious­ness. I don’t want to re­duce that com­plex­ity to a num­ber.

Some of my thoughts on this po­ten­tial down­side mir­ror those I made with re­gards to an­chor­ing:

  • This does seem like it would of­ten be a real down­side, and worth tak­ing se­ri­ously.

  • This seems most clearly a down­side of mak­ing EPs pub­lic, rather than of us­ing EPs in one’s own think­ing (or within a spe­cific or­gani­sa­tion or com­mu­nity).

  • This down­side does seem more promi­nent the less trust­wor­thy one’s prob­a­bil­ities would be.

But un­like all the other down­sides I’ve cov­ered, this one does seem like it might war­rant act­ing (in pub­lic) as if there is a bi­nary risk-un­cer­tainty dis­tinc­tion. This is be­cause the peo­ple one wants to main­tain a good rep­u­ta­tion with may think as though there is such a dis­tinc­tion. But it should be noted that this only re­quires pub­li­cly act­ing as if there’s such a dis­tinc­tion; you don’t have to think as if there’s such a dis­tinc­tion.

One last thing to note is that it also seems pos­si­ble that similar rep­u­ta­tional is­sues could re­sult from not us­ing EPs. For ex­am­ple, if one re­lies on qual­i­ta­tive or in­tu­itive ap­proaches, one’s think­ing may be seen as “hand-wavey”, “soft”, and/​or im­pre­cise by peo­ple from a more “hard sci­ence” back­ground.


  • There are some real down­sides that can oc­cur in prac­tice when ac­tual hu­mans use EPs (or EPMs, or max­imis­ing ex­pected util­ity)

  • But some down­sides that have been sug­gested (par­tic­u­larly caus­ing over­con­fi­dence and un­der­stat­ing the VoI) might ac­tu­ally be more pro­nounced for ap­proaches other than us­ing EPs

  • Some down­sides (par­tic­u­larly re­lat­ing to the op­ti­mizer’s curse, an­chor­ing, and rep­u­ta­tional is­sues) may be more pro­nounced when the prob­a­bil­ities one has (or could have) are less trustworthy

  • Other down­sides (par­tic­u­larly ex­clud­ing one’s in­tu­itive knowl­edge) may be more pro­nounced when the prob­a­bil­ities one has (or could have) are more trustworthy

  • Only one down­side (rep­u­ta­tional is­sues) seems to provide any ar­gu­ment for even act­ing as if there’s a bi­nary risk-un­cer­tainty distinction

    • And even in that case the ar­gu­ment is quite un­clear, and wouldn’t sug­gest we should use the idea of such a dis­tinc­tion in our own thinking

  • The above point, com­bined with ar­gu­ments I made in an ear­lier post, makes me be­lieve that we should aban­don the con­cept of the risk-un­cer­tainty dis­tinc­tion in our own think­ing (and at least most com­mu­ni­ca­tion), and that we should think in­stead in terms of:

    • a con­tinuum of more to less trust­wor­thy probabilities

    • the prac­ti­cal up­sides and down­sides of us­ing EPs, for ac­tual hu­mans.

I’d be in­ter­ested in peo­ple’s thoughts on all of the above; one mo­ti­va­tion for writ­ing this post was to see if some­one could poke holes in, and thus im­prove, my think­ing.

  1. I should note that this post ba­si­cally takes as a start­ing as­sump­tion the Bayesian in­ter­pre­ta­tion of prob­a­bil­ity, “in which, in­stead of fre­quency or propen­sity of some phe­nomenon, prob­a­bil­ity is in­ter­preted as rea­son­able ex­pec­ta­tion rep­re­sent­ing a state of knowl­edge or as quan­tifi­ca­tion of a per­sonal be­lief” (Wikipe­dia). But I think at least a de­cent amount of what I say would hold for other in­ter­pre­ta­tions of prob­a­bil­ity (e.g., fre­quen­tism). ↩︎

  2. Of course, I could quickly and eas­ily make an ex­tremely sim­plis­tic EPM, or use just a sin­gle EP. But then it’s un­clear if that’d do bet­ter than similarly quick and easy al­ter­na­tive ap­proaches, for the rea­sons dis­cussed in the fol­low­ing sec­tions. ↩︎

  3. This seems analo­gous to the idea that util­i­tar­i­anism it­self may of­ten recom­mend against the ac­tion of try­ing to ex­plic­itly calcu­late what ac­tion util­i­tar­i­anism would recom­mend (given that that’s likely to slow one down mas­sively). Amanda Askell has writ­ten a post on that topic, in which she says: “As many util­i­tar­i­ans have pointed out, the act util­i­tar­ian claim that you should ‘act such that you max­i­mize the ag­gre­gate wellbe­ing’ is best thought of as a crite­rion of right­ness and not as a de­ci­sion pro­ce­dure. In fact, try­ing to use this crite­rion as a de­ci­sion pro­ce­dure will of­ten fail to max­i­mize the ag­gre­gate wellbe­ing. In such cases, util­i­tar­i­anism will ac­tu­ally say that agents are for­bid­den to use the util­i­tar­ian crite­rion when they make de­ci­sions.” ↩︎

  4. Along similar lines, Holden Karnofsky (of GiveWell, at the time) writes: “It’s my view that my brain in­stinc­tively pro­cesses huge amounts of in­for­ma­tion, com­ing from many differ­ent refer­ence classes, and ar­rives at a prior; if I at­tempt to for­mal­ize my prior, count­ing only what I can name and jus­tify, I can worsen the ac­cu­racy a lot rel­a­tive to go­ing with my gut.” ↩︎

  5. This is differ­ent to the idea that peo­ple may tend to over­es­ti­mate EPs, or over­es­ti­mate cost-effec­tive­ness, or things like that. That claim is also of­ten made, and is prob­a­bly worth dis­cussing, but I leave it out of this post. Here I’m fo­cus­ing in­stead on the sep­a­rate pos­si­bil­ity of peo­ple be­ing over­con­fi­dent about the ac­cu­racy of what­ever es­ti­mate they’ve ar­rived at, whether it’s high or low. ↩︎

  6. Here’s Nate Soares mak­ing similar points: “In other words, even if my cur­rent cre­dence is 50% I can still ex­pect that in 35 years (af­ter en­coun­ter­ing a black swan or two) my cre­dence will be very differ­ent. This has the effect of mak­ing me act un­cer­tain about my cur­rent cre­dence, al­low­ing me to say “my cre­dence for this is 50%” with­out much con­fi­dence. So long as I can’t pre­dict the di­rec­tion of the up­date, this is con­sis­tent Bayesian rea­son­ing.

    As a bounded Bayesian, I have all the be­hav­iors recom­mended by those ad­vo­cat­ing Knigh­tian un­cer­tainty. I put high value on in­creas­ing my hy­poth­e­sis space, and I of­ten ex­pect that a hy­poth­e­sis will come out of left field and throw off my pre­dic­tions. I’m happy to in­crease my er­ror bars, and I of­ten ex­pect my cre­dences to vary wildly over time. But I do all of this within a Bayesian frame­work, with no need for ex­otic “im­mea­surable” un­cer­tainty.” ↩︎

  7. Smith’s own views on this point seem a bit con­fus­ing. At one point, he writes: “we don’t need to as­sume a strict di­chotomy sep­a­rates quan­tifi­able risks from un­quan­tifi­able risks. In­stead, real-world un­cer­tainty falls on some­thing like a spec­trum.” But at var­i­ous other points, he writes things like “The idea that all un­cer­tainty must be ex­plain­able in terms of prob­a­bil­ity is a wrong-way re­duc­tion [i.e., a bad idea; see his post for de­tails]”, and “I don’t think ig­no­rance must cash out as a prob­a­bil­ity dis­tri­bu­tion”. ↩︎

  8. While I think this is a good point, I also think it may some­times be worth con­sid­er­ing the risk that one might an­chor one­self to one’s own es­ti­mate. This could there­fore be a down­side of even just gen­er­at­ing an EP one­self, not just of mak­ing EPs pub­lic. ↩︎

  9. I briefly dis­cuss em­piri­cal find­ings that are some­what rele­vant to these points here. ↩︎