That’s use­ful info, and sounds to me like a fair point. Thanks :)

But then this strikes me as ty­ing back into the idea that “Per­haps [ALLFED] seem­ingly not hav­ing been funded by the EA Long-Term Fu­ture Fund, Open Phil, and var­i­ous other fun­ders is ev­i­dence that there’s some rea­son not to sup­port them, which I just haven’t recog­nised?”

Here that ques­tion can take a more con­crete form: If Open Phil chose to fund a group that’d work on al­ter­na­tive foods that ALLFED thinks will be less promis­ing than the al­ter­na­tive foods ALLFED fo­cuses on, but didn’t choose to fund ALLFED (at least so far), does that mean:

1. Open Phil are mak­ing a mis­take?

2. ALLFED are wrong about which foods are most promis­ing?

   • Per­haps be­cause they’re wrong about the rel­a­tive costs, or be­cause there are other con­sid­er­a­tions which out­weigh the cost con­sid­er­a­tion?

3. ALLFED are right about which foods are most promis­ing, but there’s some other over­rid­ing rea­son why the other team was a bet­ter dona­tion op­por­tu­nity?

   • E.g., per­haps at the pre­sent mar­gin, what’s most needed is more aca­demic cred­i­bil­ity and that team could get it bet­ter than ALLFED could?

4. There’s some al­ter­na­tive ex­pla­na­tion such that Open Phil’s de­ci­sions are sound but also ALLFED is a good dona­tion op­por­tu­nity?

   • E.g., per­haps there’s some rea­son why Open Phil in par­tic­u­lar shouldn’t fund ALLFED at this stage, even if it thought ALLFED was a good op­por­tu­nity for other donors?

I don’t re­ally know how likely each of those pos­si­ble im­pli­ca­tions are (and thus I don’t have strong rea­son to be­lieve 2 or 3 are the most likely im­pli­ca­tions). So this is just a con­fus­ing thing and a po­ten­tial ar­gu­ment against donat­ing to ALLFED, rather than a clearly de­ci­sive ar­gu­ment.

I’d be in­ter­ested in your (or other peo­ple’s) thoughts on this—but would also un­der­stand if this is in­ap­pro­pri­ate to dis­cuss pub­li­cly.

(Btw, I wouldn’t want read­ers to in­ter­pret this as a ma­jor cri­tique or an ex­pres­sion of strong doubt. I’d ex­pect to have at least some doubt or reser­va­tion with re­gards to ba­si­cally any place I choose to donate to, work for, etc. - pri­ori­ti­sa­tion is hard! - and I’m still plan­ning to give ~4% of my in­come this year to ALLFED.)

Thanks, MichaelA! On ne­glect­ed­ness, it is true that $3 mil­lion is very large in this space. How­ever, the Open Phil funded group de­cided to pro­pose to work on al­ter­na­tive foods that they already had ex­per­tise in. This in­cludes cel­lu­losic sugar, duck­weed, for­est prod­ucts in­clud­ing in­ner bark, mush­rooms, and sprouts. With the ex­cep­tion of cel­lu­losic sugar, these al­ter­na­tive foods are higher cost than the ones that ALLFED is pri­ori­tiz­ing. Low cost is im­por­tant for feed­ing nearly ev­ery­one and main­tain­ing sta­bil­ity of civ­i­liza­tion. There­fore, we don’t be­lieve that the high­est pri­or­ity sun-block­ing solu­tions (cel­lu­losic sugar, methane sin­gle cell pro­tein, hy­dro­gen sin­gle cell pro­tein, cold tol­er­ant crops, green­houses, sea­weed, and leaf pro­tein con­cen­trate) are sig­nifi­cantly less ne­glected now. Fur­ther­more, the Open Phil funded pro­ject is gen­er­ally not work­ing on in­ter­ven­tions for los­ing elec­tric­ity/​in­dus­try, so that re­mains highly ne­glected.

Hmm, I feel like you may be fram­ing things quite differ­ently to how I would, or some­thing. My ini­tial re­ac­tion to your com­ment is some­thing like:

It seems use­fully to con­cep­tu­ally sep­a­rate data col­lec­tion from data pro­cess­ing, where by the lat­ter I mean us­ing that data to ar­rive at prob­a­bil­ity es­ti­mates and de­ci­sions.

I think Bayesi­anism (in the sense of us­ing Bayes’ the­o­rem and a Bayesian in­ter­pre­ta­tion of prob­a­bil­ity) and “math and tech­ni­cal patches” might tend to be part of the data pro­cess­ing, not the data col­lec­tion. (Though they could also guide what data to look for. And this is just a rough con­cep­tual di­vide.)

When Ozzie wrote about go­ing with “an ap­proach that in-ex­pec­ta­tion does a de­cent job at ap­prox­i­mat­ing the math­e­mat­i­cal ap­proach”, he was speci­fi­cally refer­ring to deal­ing with the op­ti­mizer’s curse. I’d con­sider this part of data pro­cess­ing.

Mean­while, my in­tu­itions (i.e., gut re­ac­tions) and what ex­perts say are data. At­tend­ing to them is data col­lec­tion, and then we have to de­cide how to in­te­grate that with other things to ar­rive at prob­a­bil­ity es­ti­mates and de­ci­sions.

I don’t think we should see our­selves as de­cid­ing be­tween ei­ther Bayesi­anism and “math and tech­ni­cal patches” or pay­ing at­ten­tion to my in­tu­itions and do­main ex­perts. You can feed all sorts of ev­i­dence into Bayes the­o­rem. I doubt any EA would ar­gue we should form con­clu­sions from “Bayesi­anism and math alone”, with­out us­ing any data from the world (in­clud­ing even their in­tu­itive sense of what num­bers to plug in, or whether peo­ple they share their find­ings with seem skep­ti­cal). I’m not even sure what that’d look like.

And I think my in­tu­itions or what do­main ex­perts says can very eas­ily be made sense of as valid data within a Bayesian frame­work. Gen­er­ally, my in­tu­itions and ex­perts are more likely to in­di­cate X is true in wor­lds where X is true than where it’s not. This effect is stronger when the con­di­tions for in­tu­itive ex­per­tise are met, when ex­perts’ in­cen­tives seem to be well al­igned with seek­ing and shar­ing truth, etc. This effect is weaker when it seems that there are strong bi­ases or mis­al­igned in­cen­tives at play, or when it seems there might be.

(Per­haps this is talk­ing past you? I’m not sure I un­der­stood your ar­gu­ment.)

For con­text, can you ex­plain what kind of job you are look­ing for? How time-con­sum­ing does each ap­pli­ca­tion tend to be?

Thanks, fixed!

Thanks, Michael, that de­serves an en­try in https://​​docs.google.com/​​doc­u­ment/​​d/​​1OTCQlWE-GkY_V4V-OfJAr7Q-vJyIR8ZATpeMrLkm­lAo/​​edit#

I am grate­ful for all the peo­ple in the com­mu­nity, who are always happy to help with minor things. Every­time I ask some­one for train­ing ad­vice at a con­fer­ence, or for a, ex­pla­na­tion of any word on Dank EA Memes, or for ca­reer ad­vice in the Fo­rum, I always got re­ally nice and de­tailed an­swers. I feel re­ally ex­cepted through that, es­pe­cially con­sid­er­ing, how rare these things are on the in­ter­net.

Be­ing born into the 1%.

Just saw this com­ment, I’m also su­per late to the party re­spond­ing to you!

It ac­tu­ally seems to me it might have been worth em­pha­sis­ing more, as I think a ca­sual reader could think this post was a cri­tique of for­mal/​ex­plicit/​quan­ti­ta­tive mod­els in par­tic­u­lar.

To­tally agree! Hon­estly, I had sev­eral goals with this post, and I al­most com­plete failed on two of them:

• Ar­gu­ing why util­i­tar­i­anism can’t be the foun­da­tion of ethics.

• Without talk­ing much about AI, ex­plain­ing why I don’t think peo­ple in the EA com­mu­nity are be­ing rea­son­able when they sug­gest there’s a de­cent chance of an AGI be­ing de­vel­oped in the near fu­ture.

In­stead, I think this post came off as pri­mar­ily a crit­i­cism of cer­tain kinds of mod­els and a crit­i­cism of GiveWell’s ap­proach to pri­ori­ti­za­tion (which is un­for­tu­nate since I think the Op­ti­mizer’s Curse isn’t as big an is­sue for GiveWell & global health as it is for many other EA orgs/​cause ar­eas).

--
On the sec­ond piece of your com­ment, I think we mostly agree. In­for­mal/​cluster-style think­ing is prob­a­bly helpful, but it definitely doesn’t make the Op­ti­mizer’s Curse a non-is­sue.

Just found this post, com­ing in to com­ment a year late—Thanks Michael for the thought­ful post and Ozzie for the thought­ful com­ments!

I’m not say­ing that these are easy to solve, but rather, there is a math­e­mat­i­cal strat­egy to gen­er­ally fix them in ways that would make sense in­tu­itively. There’s no bet­ter ap­proach than to try to ap­prox­i­mate the math­e­mat­i­cal ap­proach, or go with an ap­proach that in-ex­pec­ta­tion does a de­cent job at ap­prox­i­mat­ing the math­e­mat­i­cal ap­proach.

I might agree with you about what’s (in some sense) math­e­mat­i­cally pos­si­ble (in prin­ci­ple). In prac­tice, I don’t think peo­ple try­ing to ap­prox­i­mate the ideal math­e­mat­i­cal ap­proach are go­ing to have a ton of suc­cess (for rea­sons dis­cussed in my post and quoted in Michael’s pre­vi­ous com­ment).

I don’t think search­ing for “an ap­proach that in-ex­pec­ta­tion does a de­cent job at ap­prox­i­mat­ing the math­e­mat­i­cal ap­proach” is prag­matic.

In most im­por­tant sce­nar­ios, we’re un­cer­tain what ap­proaches work well in-ex­pec­ta­tion. Our un­cer­tainty about what works well in-ex­pec­ta­tion is the kind of un­cer­tainty that’s hard to hash out in prob­a­bil­ities. A strict Bayesian might say, “That’s not a prob­lem—with even more math, the un­cer­tainty can be han­dled....”

While you can keep adding more math and tech­ni­cal patches to try and ground de­ci­sion mak­ing in Bayesi­anism, prag­ma­tism even­tu­ally pushes me in other di­rec­tions. I think David Chap­man ex­plains this idea a hell of a lot bet­ter than I can in Ra­tion­al­ism’s Re­sponses To Trou­ble.

Get­ting more con­crete:
Trust­ing my gut or listen­ing to do­main ex­perts might turn out to be ap­proaches that work well in some situ­a­tion. If one of these ap­proaches works, I’m sure some­one could ar­gue in hind­sight that an ap­proach works be­cause it ap­prox­i­mates an ideal­ized math­e­mat­i­cal ap­proach. But I’m skep­ti­cal of the mer­its of work done in the re­verse (i.e., try­ing to dis­cover non-math ap­proaches by look­ing for things that will ap­prox­i­mate ideal­ized math­e­mat­i­cal ap­proaches).

Its true that this is prob­a­bly most suited to a fund­ing scheme aimed at early re­searchers due to the limi­ta­tions men­tioned by you. How­ever, I might think that the grant suc­cess might go up if you use a model were you sort out all bad re­search first, be­cause your 20 % is prob­a­bly re­late to the over­all num­ber of ap­pli­ca­tions. Or maybe you could give peo­ple more tick­ets in the lot­tery if they have proven they can pro­duce good re­search. How­ever, this might in­tro­duce new bi­ases.

In ad­di­tion, it might still be a good ap­proach for in­ter­me­di­ate re­searchers be­cause the over­all time for the whole grant pro­cess gets re­duced dra­mat­i­cally if you can cut out most of the peer re­view, which might lead to more calls for re­search pro­pos­als.

Con­cern­ing the Na­ture ar­ti­cle and the mod­ified lot­tery sys­tem: I read con­flict­ing opinions on this. While the Na­ture ar­ti­cle states that very good re­search can be iden­ti­fied eas­ily, there are also oth­ers that state that re­searchers can only re­li­ably iden­tify bad re­search, but have a hard time to sort good re­search in any re­pro­ducible way.

Minor typo—The DOI for Eval­u­at­ing use cases for hu­man challenge tri­als in ac­cel­er­at­ing SARS-CoV-2 vac­cine de­vel­op­ment. Clini­cal In­fec­tious Diseases has a trailing e in the url which causes the link to fail

Meta: I sug­gest that when up­votes are used to im­ple­ment a poll, the post au­thor would also provide a com­ment to be down­voted by poll par­ti­ci­pants in or­der to bal­ance the the au­thor’s karma (so as to not cre­ate a norm that might be ex­ploited by oth­ers for gain­ing karma).

Also, I sug­gest plac­ing all the poll com­ments un­der one par­ent com­ment.

If I owe some­one a small amount of money, I some­times sug­gest toss­ing a coin: With a 50% chance we just both for­get about the debt, oth­er­wise, we dou­ble the debt. The ex­pected money stays the same, the var­i­ance is in­signifi­cant as the amount is small and with a 50% chance, we do not need to make the has­sle of ex­chang­ing coins. How­ever, it seems that a lot of peo­ple are un­will­ing to do money bet­ting. Even with amounts less than 10 Euro.

Another pos­si­ble ad­van­tage of lot­ter­ies: You sim­ply need to put in less work in de­ci­sion-mak­ing. For ex­am­ple in the donor lot­tery: https://​​app.effec­tivealtru­ism.org/​​lotteries

I’m gen­er­ally in favour of ex­per­i­ment­ing with differ­ent grant­ing mod­els and am glad to hear that fun­ders are start­ing to ex­per­i­ment with ran­dom al­lo­ca­tion. How­ever, I’d be a lit­tle bit cau­tious about mov­ing to a sys­tem based solely on ran­dom grant as­sign­ment. Depend­ing on the ac­tual grant suc­cess rate per round (cur­rently of­ten <20%), it seems likely that one would get awarded grants quite in­fre­quently, which would in­ter­rupt the con­ti­nu­ity of re­search. For in­stance, if some­body gets a ran­dom grant and makes an in­ter­est­ing dis­cov­ery, it seems silly to then ex­pect to wait sev­eral years for an­other ran­dom grant as­sign­ment to fol­low up on it. So I feel that ran­dom as­sign­ment is prob­a­bly bet­ter used for as­sign­ing fund­ing for early-ca­reer re­searchers or pi­lot pro­jects.

With re­spect to qual­ity con­trol, the Na­ture news ar­ti­cle linked above notes:

as­sess­ment pan­els spend most of their time sort­ing out the spe­cific or­der in which to place mid-rank­ing ideas. Low- and high-qual­ity ap­pli­ca­tions are easy to rank, she says. “But most ap­pli­ca­tions are in the mid­field, which is very big.”

The cur­rent mod­ified lot­tery sys­tems just re­move the low-rank­ing ap­pli­ca­tions, but if it’s easy to pick high-rank­ing ap­pli­ca­tions, surely they should be given fund­ing pri­or­ity?

I’m not op­ti­mistic. When will more rea­son­able voices with differ­ent bi­ases en­ter so­cial me­dia? Al­most the whole world is already on so­cial me­dia.

I love be­ing in a com­mu­nity which helps me live up to my val­ues:

When I first came across Peter Singer, I just kind of as­sumed I wouldn’t end up donat­ing much be­cause it seemed like even though clearly we ought to, no-one else did. It makes it so much eas­ier be­ing sud­denly in a group where donat­ing 10% is just the ac­cepted norm.

The com­mu­nity also seems great at shar­ing knowl­edge to make this eas­ier, and sup­port­ing each other. An ex­am­ple I par­tic­u­larly loved was Rob Wiblin helping me ring­ing for the elec­tion: he wrote up and easy how to guide, and then for peo­ple who still found it aver­sive (which in­cludes me be­cause I hate mak­ing phone calls), he in­vited us over to all do it to­gether and egg each other on, and so that he could help trou­bleshoot.

Sorry I think I didn’t ad­dress the mea­sure­ment is­sue very well, and as­sumed your no­tion of user in­ter­ests meant sim­ply op­ti­miz­ing for views, when maybe it isn’t. I still think through user re­search you can learn to de­velop good mea­sures. For ex­am­ple: sur­veys, co­hort tests (e.g. if you dis­count rat­ings over time within a view­ing ses­sion, to down weight lower agency views, do you see changes such as users search­ing more in­stead of just let­ting au­to­play), is there a re­la­tion­ship be­tween how much a user feels net­flix is im­prov­ing their life (in a sur­vey) and how much they are sucked in by au­to­play? Learn­ing these higher or­der be­havi­oural in­di­ca­tors can help give users a bet­ter long-term ex­pe­rience, if that’s what the com­pany op­ti­mizes for.

Many times I make mis­takes. Some­times those mis­takes are bad enough that the wronged par­ties would have cause to de­mand ex­pen­sive rec­om­pense, or to weaken their ties to me. Won­der­fully, my friends and col­leagues by and large value their re­la­tion­ship with me enough to over­look my flaws. From my Chris­tian up­bring­ing, I’d call that grace. On Thanks­giv­ing this year, when many of us have needed lit­tle more grace from our friends than be­fore, I’m grate­ful for mine.

Thanks for rais­ing this. I ap­pre­ci­ate speci­fi­ca­tion is hard, but I think there’s a broader lens on ‘user in­ter­ests’ with more ac­knowl­edge­ment for the be­havi­oural side.

What users want in one mo­ment isn’t always the same as what they might en­dorse when in a less slip­pery be­havi­oural set­ting or upon re­flec­tion. You might say this is a hu­man not a tech­ni­cal prob­lem. True, but we can de­sign sys­tems to that help us op­ti­mize for our long-term goals and that is a differ­ent task to op­ti­miz­ing for what we click on in a given mo­ment. Sure it’s much harder to spec­ify, but I think user re­search can be done. Think­ing about the user more holis­ti­cally could open up new in­no­va­tions too. Imag­ine a per­son has watched sev­eral videos in a row about weight loss and rather than keep­ing them on the couch longer, it learns to re­spond with good nudges: prompts them to get up and go for a run, re­minds them of their per­sonal goals for the day (be­cause it has such in­te­gra­tions), mes­sages your run­ning buddy, closes it­self (and has nice con­figurable set­tings with good de­faults), or ad­ver­tises join­ing a lo­cal run­ning group (right now the lo­cal run­ning group would not af­ford the ad­vert, but in a world where recom­menders weight ad qual­ity to some­how in­clude long-term prefer­ences of the user, that might be differ­ent).

I un­der­stand the mea­sure­ment frus­tra­tion is­sue, the task is harder than just op­ti­mis­ing for views and clicks though (not just tech­ni­cally, also to al­ign to the com­pany’s bot­tom line). How­ever, I do think lit­tle steps to­wards bet­ter speci­fi­ca­tion can help, and I’d love to read fu­ture user re­search on it at Net­flix.