• What I am say­ing is that they don’t ad­dress the op­ti­mizer’s curse just by in­clud­ing them, and I sus­pect they won’t help at all on their own in some cases.

You seem to be us­ing “peo­ple all agree” as a stand-in for “the op­ti­mizer’s curse has been ad­dressed”. I don’t get this. Ad­dress­ing the op­ti­mizer’s curse has been math­e­mat­i­cally demon­strated. Differ­ent peo­ple can dis­agree about the spe­cific in­puts, so peo­ple will dis­agree, but that doesn’t mean they haven’t ad­dressed the op­ti­mizer’s curse.

Maybe check­ing sen­si­tivity to pri­ors and fur­ther pro­mot­ing in­ter­ven­tions whose value de­pends less on them (among some set of “rea­son­able” pri­ors) would help. You could see this as a spe­cial case of Chris’s sug­ges­tion to “En­ter­tain mul­ti­ple mod­els”.
Per­haps you could even use an ex­plicit model to com­bine the es­ti­mates or pos­te­ri­ors from mul­ti­ple mod­els into a sin­gle one in a way that ei­ther pe­nal­izes sen­si­tivity to pri­ors or gives less weight to more ex­treme es­ti­mates, but a sim­pler de­ci­sion rule might be more trans­par­ent or oth­er­wise prefer­able.

I think com­bin­ing into a sin­gle model is gen­er­ally ap­pro­pri­ate. And the sub-mod­els need not be fully, ex­plic­itly laid out.

Sup­pose I’m demon­strat­ing that poverty char­ity > an­i­mal char­ity. I don’t have to build one model as­sum­ing “1 hu­man = 50 chick­ens”, an­other model as­sum­ing “1 hu­man = 100 chick­ens”, and so on.

In­stead I just set a gen­eral stan­dard for how ro­bust my claims are go­ing to be, and I feel suffi­ciently con­fi­dent say­ing “1 hu­man = at least 60 chick­ens”, so I use that rather than my mean ex­pec­ta­tion (e.g. 90).

• You seem to be us­ing “peo­ple all agree” as a stand-in for “the op­ti­mizer’s curse has been ad­dressed”. I don’t get this. Ad­dress­ing the op­ti­mizer’s curse has been math­e­mat­i­cally demon­strated. Differ­ent peo­ple can dis­agree about the spe­cific in­puts, so peo­ple will dis­agree, but that doesn’t mean they haven’t ad­dressed the op­ti­mizer’s curse.

Maybe we’re think­ing about the op­ti­mizer’s curse in differ­ent ways.

The pro­posed solu­tion of us­ing pri­ors just pushes the prob­lem to se­lect­ing good pri­ors. It’s also only a solu­tion in the sense that it re­duces the like­li­hood of mis­takes hap­pen­ing (dis­cov­ered in hind­sight, and un­der the as­sump­tion of good pri­ors), but not prov­ably to its min­i­mum, since it does not elimi­nate the im­pacts of noise. (I don’t think there’s any com­plete solu­tion to the op­ti­mizer’s curse, since, as long as es­ti­mates are at least some­what sen­si­tive to noise, “lucky” es­ti­mates will tend to be favoured, and you can’t tell in prin­ci­ple be­tween “lucky” and “bet­ter” in­ter­ven­tions.)

If you’re pre­sented with mul­ti­ple pri­ors, and they all seem similarly rea­son­able to you, but de­pend­ing on which ones you choose, differ­ent ac­tions will be favoured, how would you choose how to act? It’s not just a mat­ter of differ­ent peo­ple dis­agree­ing on pri­ors, it’s also a mat­ter of com­mit­ting to par­tic­u­lar pri­ors in the first place.

If one ac­tion is preferred with al­most all of the pri­ors (per­haps rare in prac­tice), isn’t that a rea­son (per­haps in­suffi­cient) to pre­fer it? To me, us­ing this could be an im­prove­ment over just us­ing pri­ors, be­cause I sus­pect it will fur­ther re­duce the im­pacts of noise, and if it is an im­prove­ment, then just us­ing pri­ors never fully solved the prob­lem in prac­tice in the first place.

I agree with the rest of your com­ment. I think some­thing like that would be use­ful.

• The pro­posed solu­tion of us­ing pri­ors just pushes the prob­lem to se­lect­ing good pri­ors.

The prob­lem of the op­ti­mizer’s curse is that the EV es­ti­mates of high-EV-op­tions are pre­dictably over-op­ti­mistic in pro­por­tion with how un­re­li­able the es­ti­mates are. That prob­lem doesn’t ex­ist any­more.

The fact that you don’t have guaran­teed ac­cu­rate in­for­ma­tion doesn’t mean the op­ti­mizer’s curse still ex­ists.

I don’t think there’s any com­plete solu­tion to the op­ti­mizer’s curse

Well there is, just spend too much time wor­ry­ing about model un­cer­tainty and other peo­ple’s pri­ors and too lit­tle time wor­ry­ing about ex­pected value es­ti­ma­tion. Then you’re solv­ing the op­ti­mizer’s curse too much, so that your char­ity se­lec­tions will be less ac­cu­rate and pre­dictably bi­ased in fa­vor of low EV, high re­li­a­bil­ity op­tions. So it’s a bad idea, but you’ve solved the op­ti­mizer’s curse.

If you’re pre­sented with mul­ti­ple pri­ors, and they all seem similarly rea­son­able to you, but de­pend­ing on which ones you choose, differ­ent ac­tions will be favoured, how would you choose how to act?

Max­i­mize the ex­pected out­come over the dis­tri­bu­tion of pos­si­bil­ities.

If one ac­tion is preferred with al­most all of the pri­ors (per­haps rare in prac­tice), isn’t that a rea­son (per­haps in­suffi­cient) to pre­fer it?

What do you mean by “the pri­ors”? Other peo­ple’s pri­ors? Well if they’re other peo­ple’s pri­ors and I don’t have rea­son to up­date my be­liefs based on their pri­ors, then it’s triv­ially true that this doesn’t give me a rea­son to pre­fer the ac­tion. But you seem to think that other peo­ple’s pri­ors will be “rea­son­able”, so ob­vi­ously I should up­date based on their pri­ors, in which case of course this is true—but only in a ba­nal, triv­ial sense that has noth­ing to do with the op­ti­mizer’s curse.

To me, us­ing this could be an im­prove­ment over just us­ing pri­ors

Hm? You’re just sug­gest­ing up­dat­ing one’s prior by look­ing at other peo­ple’s pri­ors. As­sum­ing that other peo­ple’s pri­ors might be ra­tio­nal, this is ba­nal—of course we should be rea­son­able, epistem­i­cally mod­est, etc. But this has noth­ing to do with the op­ti­mizer’s curse in par­tic­u­lar, it’s equally true ei­ther way.

I ask the same ques­tion I asked of OP: give me some guidance that ap­plies for es­ti­mat­ing the im­pact of max­i­miz­ing ac­tions that doesn’t ap­ply for es­ti­mat­ing the im­pact of ran­domly se­lected ac­tions. So far it still seems like there is none—aside from the ba­sic idea given by Muelhauser.

just us­ing pri­ors never fully solved the prob­lem in prac­tice in the first place

Is the prob­lem the lack of guaran­teed knowl­edge about char­ity im­pacts, or is the prob­lem the op­ti­mizer’s curse? You seem to (in­cor­rectly) think that chip­ping away at the former nec­es­sar­ily means chip­ping away at the lat­ter.

• It’s always worth en­ter­tain­ing mul­ti­ple mod­els if you can do that at no cost. How­ever, do­ing that of­ten comes at some cost (money, time, etc). In situ­a­tions with lots of un­cer­tainty (where the op­ti­mizer’s curse is li­able to cause sig­nifi­cant prob­lems), it’s worth pay­ing much higher costs to en­ter­tain mul­ti­ple mod­els (or do other things I sug­gested) than it is in cases where the op­ti­mizer’s curse is un­likely to cause se­ri­ous prob­lems.

• In situ­a­tions with lots of un­cer­tainty (where the op­ti­mizer’s curse is li­able to cause sig­nifi­cant prob­lems), it’s worth pay­ing much higher costs to en­ter­tain mul­ti­ple mod­els (or do other things I sug­gested) than it is in cases where the op­ti­mizer’s curse is un­likely to cause se­ri­ous prob­lems.

I don’t agree. Why is the un­cer­tainty that comes from model un­cer­tainty—as op­posed to any other kind of un­cer­tainty—uniquely im­por­tant for the op­ti­mizer’s curse? The op­ti­mizer’s curse does not dis­crim­i­nate be­tween es­ti­mates that are too high for mod­el­ing rea­sons, ver­sus es­ti­mates that are too high for any other rea­son.

The mere fact that there’s more un­cer­tainty is not rele­vant, be­cause we are talk­ing about how much time we should spend wor­ry­ing about one kind of un­cer­tainty ver­sus an­other. “Do more to re­duce un­cer­tainty” is just a plat­i­tude, we always want to re­duce un­cer­tainty.

• I made a long top-level com­ment that I hope will clar­ify some prob­lems with the solu­tion pro­posed in the origi­nal pa­per.

I ask the same ques­tion I asked of OP: give me some guidance that ap­plies for es­ti­mat­ing the im­pact of max­i­miz­ing ac­tions that doesn’t ap­ply for es­ti­mat­ing the im­pact of ran­domly se­lected ac­tions.

This is a good point. Some­how, I think you’d want to ad­just your pos­te­rior down­ward based on the set or the num­ber of op­tions un­der con­sid­er­a­tion and how un­likely the data that makes the in­ter­ven­tion look good. This is not re­ally use­ful, since I don’t know how much you should ad­just these. Maybe there’s a way to model this ex­plic­itly, but it seems like you’d be try­ing to model your se­lec­tion pro­cess it­self be­fore you’ve defined it, and then you look for a se­lec­tion pro­cess which satis­fies some prop­er­ties.

You might also want to spend more effort look­ing for ar­gu­ments and ev­i­dence against each op­tion the more op­tions you’re con­sid­er­ing.

When con­sid­er­ing a larger num­ber of op­tions, you could use some ran­dom­ness in your se­lec­tion pro­cess or spread fund­ing fur­ther (al­though the lat­ter will be vuln­er­a­ble to the satis­ficer’s curse if you’re us­ing cut­offs).

What do you mean by “the pri­ors”?

If I haven’t de­cided on a prior, and mul­ti­ple differ­ent pri­ors (even an in­finite set of them) seem equally rea­son­able to me.

• Some­how, I think you’d want to ad­just your pos­te­rior down­ward based on the set or the num­ber of op­tions un­der con­sid­er­a­tion and how un­likely the data that makes the in­ter­ven­tion look good.

That’s the ba­sic idea given by Muelhauser. Cor­rected pos­te­rior EV es­ti­mates.

You might also want to spend more effort look­ing for ar­gu­ments and ev­i­dence against each op­tion the more op­tions you’re con­sid­er­ing.

As op­posed to equal effort for and against? OK, I’m satis­fied. How­ever, if I’ve done the cor­rected pos­te­rior EV es­ti­ma­tion, and then my spe­cific search for ar­gu­ments-against turns up short, then I should in­crease my EV es­ti­mates back to­wards the origi­nal naive es­ti­mate.

When con­sid­er­ing a larger num­ber of op­tions, you could use some ran­dom­ness in your se­lec­tion pro­cess

As I re­call, that post found that ran­dom­ized fund­ing doesn’t make sense. Which 100% matches my pre­sump­tions, I do not see how it could im­prove fund­ing out­comes.

I don’t see how that would im­prove fund­ing out­comes.

If I haven’t de­cided on a prior, and mul­ti­ple differ­ent pri­ors (even an in­finite set of them) seem equally rea­son­able to me.

In Bayesian ra­tio­nal­ity, you always have a prior. You seem to be con­sid­er­ing or defin­ing things differ­ently.

Here we would prob­a­bly say that your ac­tual prior ex­ists and is sim­ply some kind of ag­gre­gate of these pos­si­ble pri­ors, there­fore it’s not the case that we should leap out­side our own pri­ors in some sort of vi­o­la­tion of stan­dard Bayesian ra­tio­nal­ity.

• The pro­posed solu­tion of us­ing pri­ors just pushes the prob­lem to se­lect­ing good pri­ors.

+1

In con­ver­sa­tions I’ve had about this stuff, it seems like the crux is of­ten the ques­tion of how easy it is to choose good pri­ors, and whether a “good” prior is even an in­tel­ligible con­cept.

Com­pare Chris’ piece (“se­lect­ing good pri­ors is re­ally hard!”) with this piece by Luke Muehlhauser (“the op­ti­mizer’s curse is triv­ial, just choose an ap­pro­pri­ate prior!”)

• it seems like the crux is of­ten the ques­tion of how easy it is to choose good pri­ors

Be­fore any­thing like a crux can be iden­ti­fied, com­plainants need to iden­tify what a “good prior” even means, or what strate­gies are bet­ter than oth­ers. Un­til then, they’re not even wrong—it’s not even pos­si­ble to say what dis­agree­ment ex­ists. To airily talk about “good pri­ors” or “bad pri­ors”, be­ing “easy” or “hard” to iden­tify, is just empty phras­ing and sug­gests con­fu­sion about ra­tio­nal­ity and prob­a­bil­ity.

• Hey Kyle, I’d stopped re­spond­ing since I felt like we were well be­yond the point where we were likely to con­vince one an­other or say things that those read­ing the com­ments would find in­sight­ful.

I un­der­stand why you think “good prior” needs to be defined bet­ter.

As I try to com­mu­ni­cate (but may not quite say ex­plic­itly) in my post, I think that in situ­a­tions where un­cer­tainty is poorly un­der­stood, it’s hard to come up with pri­ors that are good enough that choos­ing ac­tions based ex­plicit Bayesian calcu­la­tions will lead to bet­ter out­comes than choos­ing ac­tions based on a com­bi­na­tion of care­ful skep­ti­cism, in­for­ma­tion gath­er­ing, hunches, and crit­i­cal think­ing.

• As a real world ex­am­ple:

Ven­ture cap­i­tal­ists fre­quently fund things that they’re ex­tremely un­cer­tain about. It’s my im­pres­sion that Bayesian calcu­la­tions rarely play into these situ­a­tions. In­stead, smart VCs think hard and crit­i­cally and come to con­clu­sions based on pro­cesses that they prob­a­bly don’t full un­der­stand them­selves.

It could be that VCs have just failed to re­al­ize the amaz­ing­ness of Bayesi­anism. How­ever, given that they’re smart & there’s a ton of money on the table, I think the much more plau­si­ble ex­pla­na­tion is that hard­core Bayesi­anism wouldn’t lead to bet­ter re­sults than what­ever it is that suc­cess­ful VCs ac­tu­ally do.

• Again, none of this is to say that Bayesi­anism is fun­da­men­tally bro­ken or that high-level Bayesian-ish things like “I have a very skep­ti­cal prior so I should not take this es­ti­mate of im­pact at face value” are crazy.

• Ven­ture cap­i­tal­ists fre­quently fund things that they’re ex­tremely un­cer­tain about. It’s my im­pres­sion that Bayesian calcu­la­tions rarely play into these situ­a­tions. In­stead, smart VCs think hard and crit­i­cally and come to con­clu­sions based on pro­cesses that they prob­a­bly don’t full un­der­stand them­selves.

I in­terned for a VC, albeit a small and un­known one. Sure, they don’t do Bayesian calcu­la­tions, if you want to be re­ally pre­cise. But they make ex­ten­sive use of quan­ti­ta­tive es­ti­mates all the same. If any­thing, they are cruder than what EAs do. As far as I know, they don’t bother cor­rect­ing for the op­ti­mizer’s curse! I never heard it men­tioned. VCs don’t pri­mar­ily rely on the quan­ti­ta­tive mod­els, but other ar­eas of fi­nance do. If what they do is OK, then what EAs do is bet­ter. This is con­sis­tent with what fi­nance pro­fes­sion­als told me about the fi­nan­cial mod­el­ing that I did.

Plus, this is not about the op­ti­mizer’s curse. Imag­ine that you told those VCs that they were no longer choos­ing which star­tups are best, in­stead they now have to se­lect which ones are bet­ter-than-av­er­age and which ones are worse-than-av­er­age. The op­ti­mizer’s curse will no longer in­terfere. Yet they’re not go­ing to start rely­ing more on ex­plicit Bayesian calcu­la­tions. They’re go­ing to use the same way of think­ing as always.

And ex­plicit Bayesian calcu­la­tion is rarely used by any­one any­where. Hu­mans en­counter many prob­lems which are not about op­ti­miz­ing, and they still don’t use ex­plicit Bayesian calcu­la­tion. So clearly the op­ti­mizer’s curse is not the is­sue. In­stead, it’s a mat­ter of which kinds of cog­ni­tion and calcu­la­tion peo­ple are more or less com­fortable with.

• it’s hard to come up with pri­ors that are good enough that choos­ing ac­tions based ex­plicit Bayesian calcu­la­tions will lead to bet­ter out­comes than choos­ing ac­tions based on a com­bi­na­tion of care­ful skep­ti­cism, in­for­ma­tion gath­er­ing, hunches, and crit­i­cal think­ing.

Ex­plicit Bayesian calcu­la­tion is a way of choos­ing ac­tions based on a com­bi­na­tion of care­ful skep­ti­cism, in­for­ma­tion gath­er­ing, hunches, and crit­i­cal think­ing. (With math too.)

I’m guess­ing you mean we should use in­tu­ition for the fi­nal se­lec­tion, in­stead of quan­ti­ta­tive es­ti­mates. OK, but I don’t see how the origi­nal post is sup­posed to back it up; I don’t see what the op­ti­mizer’s curse has to do with it.