Long-Term Future Fund: April 2020 grants and recommendations

Link post

Note: This pay­out re­port was pub­lished on the EA Funds web­site on April 27, 2020, but it hadn’t been pub­lished on the Fo­rum be­fore now.

Matt Wage

80,000 Hours ($100,000)

Gen­eral support

80,000 Hours is an or­ga­ni­za­tion that “pro­vides re­search and sup­port to help peo­ple switch into ca­reers that effec­tively tackle [the world’s most press­ing prob­lems]”. They pri­mar­ily fo­cus on longter­mist cause ar­eas. You can see their most re­cent self-re­view here.

80,000 Hours is one of the or­ga­ni­za­tions in the EA com­mu­nity that I am most op­ti­mistic about. This is partly be­cause they seem to be ad­dress­ing an im­por­tant bot­tle­neck with a high-cal­iber team, and partly be­cause of a promis­ing track record (e.g. “From 2016, the num­bers of EAs re­port­ing first hear­ing about EA from 80,000 Hours ex­plode from 9% to 13% to 25%, mak­ing them the largest sin­gle source of EA re­cruit­ment in 2018.”)

80,000 Hours had not man­aged to hit their fundrais­ing tar­get and seemed to be run­ning out of plau­si­ble fun­ders, so it seemed like a good time for us to make a grant and help fill their re­main­ing gap.

Re­think Pri­ori­ties ($68,200)

Sup­port­ing re­search on nu­clear weapons and AI arms con­trol.

This is a re­stricted grant to Re­think Pri­ori­ties to sup­port their re­search on nu­clear weapons and AI arms con­trol. You can see Re­think Pri­ori­ties’ past work and most re­cent self-re­view here.

The main case for fund­ing them (as I see it) is:

I think their ap­proach of work­ing on tractable re­search ques­tions and fre­quently, pub­li­cly writ­ing up their re­sults is a promis­ing model which I’d like to see more of in EA.

Ex­ter­nal refer­ences I trust have been pos­i­tive on the qual­ity of their past work.

Due to the pub­lic na­ture of their out­put, I think it will be pretty clear whether they’re do­ing good re­search. (This al­lows them to learn faster what re­search is use­ful and also al­lows the EA com­mu­nity to learn faster whether we should stop in­vest­ing in the pro­ject.)

Their longter­mist work seems to be less well-funded than that of most other or­ga­ni­za­tions in the space, and it’s un­likely they would be able to grow that work with­out this grant from us.

Jonas Vollmer

Note: Jonas is an ad­vi­sor to the Long-Term Fu­ture Fund. Jonas’s rea­son­ing al­igned well with our own, and he seemed best-placed to provide the ba­sic case for the be­low grant (a case which caused us to ul­ti­mately make the grant).

Pablo Staffor­ini ($17,000)

Writ­ing the pre­limi­nary con­tent for an en­cy­clo­pe­dia of effec­tive al­tru­ism.

Pablo Staffor­ini ap­plied for a grant of $17,000 over six months to write the pre­limi­nary con­tent for an en­cy­clo­pe­dia of effec­tive al­tru­ism.

I ad­vised in fa­vor of this grant be­cause I be­lieve it is im­por­tant for the EA com­mu­nity to or­ga­nize its knowl­edge and make it eas­ily ac­cessible to the com­mu­nity. If suc­cess­ful, this re­source would make it sig­nifi­cantly eas­ier for com­mu­nity mem­bers to:

fa­mil­iarize them­selves with EA knowl­edge,

in­tro­duce new­com­ers to EA ideas (see also some of the ar­gu­ments out­lined in The Value of Wikipe­dia Con­tri­bu­tions in So­cial Sciences), and

eas­ily re­trieve pub­li­ca­tions on a par­tic­u­lar topic (cur­rently spread out across many jour­nals, blogs, re­search agen­das, and web­sites).

In Staffor­ini’s own words: “It is widely ac­cepted that hav­ing ex­ist­ing re­search in a par­tic­u­lar field or­ga­nized sys­tem­at­i­cally and suc­cinctly is of high value (…).”

Sev­eral pre­vi­ous EA en­cy­clo­pe­dia pro­jects have not taken off or stopped be­ing main­tained for var­i­ous rea­sons. In the ap­pli­ca­tion, Staffor­ini did not ad­dress the im­ple­men­ta­tion and long-term main­te­nance of the pro­ject, which led to ini­tial skep­ti­cism among fund man­agers. I tried to ad­dress these po­ten­tial is­sues by send­ing fol­low-up ques­tions and sug­ges­tions to Staffor­ini. I also sug­gested he reach out to the Cen­tre for Effec­tive Altru­ism so his pro­ject can serve to up­date or re­place EA Con­cepts. The re­sponses re­duced my skep­ti­cism sub­stan­tially: I be­lieve Pablo will de­liver high-qual­ity writ­ten con­tent and in­te­grate it ap­pro­pri­ately with ex­ist­ing re­sources. I as­sign a cre­dence of 35% that this pro­ject will be main­tained well in the longer term, which seems high enough for this to be worth try­ing.

Staffor­ini’s track record sug­gests he is well-suited for this pro­ject: He was one of the main au­thors and the pro­ject man­ager of EA Con­cepts (in my view the most suc­cess­ful EA en­cy­clo­pe­dia so far), one of the main ed­i­tors of EA-re­lated Wikipe­dia ar­ti­cles, and a re­search as­sis­tant for Will MacAskill’s book Do­ing Good Bet­ter. Based on in-per­son in­ter­ac­tions, I be­lieve Staffor­ini to be par­tic­u­larly fa­mil­iar with the aca­demic and EA-ad­ja­cent liter­a­ture on many EA top­ics and un­usu­ally truth-seek­ing and un­likely to pre­sent one-sided per­spec­tives on top­ics he feels strongly about. Also, this ap­pears to be a rel­a­tively in­ex­pen­sive grant.

Oliver Habryka

[Meta]: Th­ese grant ra­tio­nales are some­what less de­tailed than in pre­vi­ous LTFF pay­out re­ports that I’ve con­tributed to. I was also hop­ing to have pub­lished a set of longer grant write­ups for last round by now, but sadly a global pan­demic hap­pened and threw off a lot of my plans, and I’ve also de­cided to re­duce my time in­vest­ment in the Long-Term Fu­ture Fund since I’ve be­come less ex­cited about the value that the fund can provide at the mar­gin (for a va­ri­ety of rea­sons, which I also hope to have time to ex­pand on at some point). I do still hope that I can find time to pro­duce longer grant write­ups, but would now as­sign only around 40% cre­dence that I cre­ate longer write­ups for ei­ther this round or next round.

As a re­sult, the fol­low­ing re­port con­sists of a rel­a­tively straight­for­ward list of the grants we made, with short ex­pla­na­tions of the rea­son­ing be­hind them.

Will Brad­shaw ($25,000)

Ex­plor­ing cru­cial con­sid­er­a­tions for de­ci­sion-mak­ing around in­for­ma­tion haz­ards.

Will Brad­shaw has been work­ing with An­ders Sand­berg from the Fu­ture of Hu­man­ity In­sti­tute (FHI) on anal­y­sis of good de­ci­sion-mak­ing pro­to­cols around in­for­ma­tion haz­ards. I trust the judge­ment of many of the peo­ple who have re­viewed his work at FHI, and they ex­pressed sig­nifi­cant ex­cite­ment about his work. I also per­son­ally think that work on bet­ter in­for­ma­tion haz­ard pro­to­cols is quite valuable, and that there has been rel­a­tively lit­tle pub­lic work on an­a­lyz­ing var­i­ous sources of in­fo­haz­ards and how to nav­i­gate them, mak­ing marginal work quite valuable.

After we made this recom­men­da­tion, Will reached out to us and asked whether it’s pos­si­ble for him to broaden the grant to also in­clude some im­me­di­ate crisis re­sponse re­lat­ing to the coro­n­avirus pan­demic, in par­tic­u­lar try­ing to make sure that this crisis trans­lates into more long-term work on biorisk. We de­cided that it would be fine for him to work on ei­ther this or his work on in­fo­haz­ards, de­pend­ing on his judge­ment and the judge­ment of his col­lab­o­ra­tors.

Sofia Ja­tiva Vega ($7,700)

Devel­op­ing a re­search pro­ject on how to in­fer hu­man’s in­ter­nal men­tal mod­els from their be­havi­our.

Sofia Ja­tiva Vega wants to work to­gether with Stu­art Arm­strong (from the Fu­ture of Hu­man­ity In­sti­tute) on de­vel­op­ing meth­ods for AI agents to in­fer hu­man men­tal mod­els and use those to pre­dict hu­man prefer­ences. I have many dis­agree­ments with Stu­art’s agenda, but over­all trust his method­ol­ogy and judge­ment, and have been fol­low­ing the re­search he has been post­ing on­line for a long time. Stu­art was very ex­cited about this op­por­tu­nity, and Sofia seems to have the rele­vant back­ground to make progress on these prob­lems (with a PhD in neu­ro­science).

An­thony Aguirre ($65,000)

Mak­ing Me­tac­u­lus use­ful and available to EA and other organizations

I’ve writ­ten up ra­tio­nales for grants to Me­tac­u­lus in the past, which you can see here.

In my last writeup on Me­tac­u­lus, I said the fol­low­ing:

My cur­rent model is that Me­tac­u­lus will strug­gle as a plat­form with­out a fully ded­i­cated team or at least in­di­vi­d­ual cham­pion, though I have not done a thor­ough in­ves­ti­ga­tion of the Me­tac­u­lus team and pro­ject, so I am not very con­fi­dent of this. One of the ma­jor mo­ti­va­tions for this grant is to en­sure that Me­tac­u­lus has enough re­sources to hire a po­ten­tial new cham­pion for the pro­ject (who ideally also has pro­gram­ming skills or UI de­sign skills to al­low them to di­rectly work on the plat­form). That said, Me­tac­u­lus should use the money as best they see fit.

I am also con­cerned about the over­lap of Me­tac­u­lus with the Good Judg­ment Pro­ject, and cur­rently have a sense that it suffers from be­ing in com­pe­ti­tion with it, while also hav­ing ac­cess to sub­stan­tially fewer re­sources and peo­ple.

The re­quested grant amount was for $150k, but I am cur­rently not con­fi­dent enough in this grant to recom­mend filling the whole amount. If Me­tac­u­lus finds an in­di­vi­d­ual new cham­pion for the pro­ject, I can imag­ine strongly recom­mend­ing that it gets fully funded, if the new cham­pion seems com­pe­tent.

I’ve since thought a lot more about Me­tac­u­lus, have used the plat­form more my­self, and have broadly been very happy with the progress that the plat­form has made since I wrote the sum­mary above. As far as I know, Me­tac­u­lus now has a full-time cham­pion for the pro­ject (Ta­may Be­siroglu), and has demon­strated to me sig­nifi­cant ad­van­tages over plat­forms like the Good Judge­ment Pro­ject, in par­tic­u­lar with its abil­ity to en­ter prob­a­bil­ity dis­tri­bu­tions over events and a ques­tion cu­ra­tion team that is much bet­ter at pro­duc­ing fore­casts that I care about and seem im­por­tant to me (and, I would guess, oth­ers work­ing on global catas­trophic risk).

This over­all makes me ex­cited about Me­tac­u­lus’s fu­ture and the effects of this grant.

Tushant Jha (TJ) ($40,000)

Work­ing on long-term macros­trat­egy and AI Align­ment, and up-skil­ling and ca­reer tran­si­tion to­wards that goal.

Tushant Jha wants to visit mul­ti­ple top x-risk or­ga­ni­za­tions while work­ing on a broad range of re­search ques­tions. They were also ac­cepted to the FHI Re­search Schol­ars Pro­gram (though were un­able to par­ti­ci­pate due to im­mi­gra­tion pro­cess re­lated de­lays), and have also re­ceived a large num­ber of highly pos­i­tive refer­ences for their work. They sadly haven’t pro­duced much pub­lic work, though I ex­pect that to change over the com­ing months.

I recom­mended this grant mostly on the ba­sis of those strong refer­ences, and a small num­ber of con­ver­sa­tions I had with TJ in which they said rea­son­able things and gen­er­ally dis­played (as far as I can tell) good judge­ment on some open re­search ques­tions.

He­len Toner

Shin-Shin Hua and Haydn Belfield ($32,000)

Iden­ti­fy­ing and re­solv­ing ten­sions be­tween com­pe­ti­tion law and long-term AI strat­egy.

Shin-Shin Hua and Haydn Belfield pro­posed a re­search pro­ject on the im­pli­ca­tions of com­pe­ti­tion law for long-term AI gov­er­nance. One im­por­tant set of ques­tions in AI gov­er­nance re­volves around the num­ber and type of ac­tors (e.g. com­pa­nies, gov­ern­ments, in­ter­na­tional or­ga­ni­za­tions) in­volved in de­vel­op­ing ad­vanced AI sys­tems. Com­pe­ti­tion law likely af­fects the types of in­ter­ven­tion that are pos­si­ble on this axis, e.g. whether mul­ti­ple re­search groups could com­bine into a con­sor­tium-like struc­ture. This broad topic has been dis­cussed in the long-term AI gov­er­nance space for a while, but so far very lit­tle se­ri­ous work has been done on it.

Shin-Shin has 7 years of ex­pe­rience as a com­pe­ti­tion lawyer, and also has ex­pe­rience in aca­demic le­gal re­search in­volv­ing AI. Haydn is an Aca­demic Pro­ject Man­ager at the Cen­tre for the Study of Ex­is­ten­tial Risk.

I think good qual­ity work on this topic would be valuable for the field of long-term AI gov­er­nance, and I be­lieve this is a promis­ing team to un­der­take such work.

MIRI ($100,000)

Gen­eral Support

The Ma­chine In­tel­li­gence Re­search In­sti­tute (MIRI) does com­puter sci­ence and math re­search on AI al­ign­ment. MIRI is a challeng­ing or­ga­ni­za­tion to eval­u­ate, both be­cause their pub­lic work is hard to eval­u­ate (be­cause much of it does not fit neatly within an ex­ist­ing dis­ci­pline, and be­cause much of the case for why the re­search mat­ters is based on hy­pothe­ses about how AI re­search will de­velop in the fu­ture) and be­cause they have de­cided to make much of their re­search non-pub­lic, for strate­gic rea­sons.

Nonethe­less, I be­lieve there is suffi­cient ev­i­dence that MIRI is do­ing good work that it makes sense for the Fund to sup­port them. This ev­i­dence in­cludes the track record of their team, the qual­ity of some re­cent hires, and some promis­ing signs about their abil­ity to pro­duce work that well-es­tab­lished ex­ter­nal re­view­ers con­sider to be very high-qual­ity—most no­tably, the ac­cep­tance of one of their de­ci­sion the­ory pa­pers to a top philos­o­phy jour­nal, The Jour­nal of Philos­o­phy.

Adam Gleave

Adam Gleave tri­alled as a mem­ber of the Fund man­age­ment team dur­ing this grant round. Adam has sub­se­quently been con­firmed as a per­ma­nent mem­ber of the Long-Term Fu­ture Fund man­age­ment team.

Dan Hendrycks ($55,000)

A hu­man value mod­el­ing benchmark

Dan Hendrycks is a sec­ond-year AI PhD stu­dent at UC Berkeley, ad­vised by Dawn Song and Ja­cob Stein­hardt. This is a re­stricted grant to sup­port cre­ation of a bench­mark for NLP model’s pre­dic­tive power for hu­man mod­els. In par­tic­u­lar, it sup­ports pay­ing con­trac­tors via plat­forms such as Me­chan­i­cal Turk to gen­er­ate and val­i­date ques­tion-an­swer pairs.

I am gen­er­ally ex­cited about bench­marks as a route for progress on AI safety, es­pe­cially for fo­cus­ing the at­ten­tion of the AI re­search com­mu­nity. Their im­pact is heavy-tailed, with many bench­marks see­ing lit­tle adop­tion while oth­ers be­ing ex­tremely in­fluen­tial, so this is definitely a “hits-based” grant. How­ever, Dan does have a strong track record of cre­at­ing sev­eral bench­marks in the field of ro­bust ma­chine learn­ing which makes me op­ti­mistic.

The topic, test­ing whether NLP mod­els im­plic­itly cap­ture no­tions of hu­man moral­ity, is very novel. Lan­guage is a nat­u­ral chan­nel by which to ex­press prefer­ences, es­pe­cially over more ab­stract con­cepts, so it seems im­por­tant that NLP mod­els can rep­re­sent prefer­ences. It is com­mon in the field to train un­su­per­vised lan­guage mod­els on large cor­pora and then fine-tune for spe­cific ap­pli­ca­tions. So test­ing whether pre­ex­ist­ing mod­els have already learned some in­for­ma­tion about prefer­ences is a nat­u­ral start­ing point.

A key con­cern re­gard­ing this pro­ject is that be­ing able to pre­dict hu­man moral judge­ments in text does not di­rectly trans­late to bet­ter al­igned AI sys­tems. We ex­pect most of the pro­ject’s im­pact to come from gain­ing a bet­ter qual­i­ta­tive un­der­stand­ing of lan­guage mod­els defi­cien­cies, and from in­creased at­ten­tion on hu­man val­ues in gen­eral in the NLP com­mu­nity. How­ever, there is some risk that the limi­ta­tions of the bench­mark are not suffi­ciently rec­og­nized, and the NLP com­mu­nity wrongly be­lieves that value learn­ing is “solved”.

Vin­cent Luczkow ($10,000)

Coun­ter­fac­tual im­pact minimization

Vin­cent Luczkow is an MSc stu­dent at Mila, soon to start a PhD in AI, in­ter­ested in con­duct­ing re­search on AI safety. This grant is for a re­search pro­ject on coun­ter­fac­tual im­pact min­i­miza­tion. We an­ti­ci­pate Vin­cent spend­ing it in part on at­tend­ing con­fer­ences, and in part on free­ing up time for this re­search pro­ject (for ex­am­ple, by not need­ing to TA sup­ple­ment his in­come).

My ex­pe­rience in­ter­act­ing with US PhD stu­dents sug­gests many stu­dents are less pro­duc­tive due to fi­nan­cial con­straints, es­pe­cially those with­out sav­ings or other in­come sources. Mila seems to have be­low-av­er­age stipends, of be­tween 25,000 and 30,000 CAD (~18,000 to 21,500 USD). By con­trast, US uni­ver­si­ties typ­i­cally offer a pack­age of at least 40,000 USD/​year. While Mon­treal has lower costs of liv­ing, over­all we find it likely that a stu­dent at MILA would benefit from sup­ple­men­tal fund­ing.

Since Vin­cent is an early-stage re­searcher, he has a limited track record to eval­u­ate, mak­ing this a slightly risky grant. How­ever, get­ting into MILA (one of the lead­ing AI labs) is a strong sig­nal, and refer­ees spoke pos­i­tively about his mo­ti­va­tion. Since we view there be­ing lit­tle down­side risk (be­yond the op­por­tu­nity cost of the dona­tion) and a sig­nifi­cant chance of a larger up­side at lit­tle cost, we de­cided to make the grant.

Michael Dick­ens ($33,000)

Con­duct­ing in­de­pen­dent re­search on cause prioritization

This is a grant to Michael Dick­ens to con­duct in­de­pen­dent re­search on cause pri­ori­ti­za­tion, fo­cus­ing on in­vest­ment strate­gies for effec­tive al­tru­ists and long-ter­mists. Speci­fi­cally, he in­tends to re­fine his work on how al­tru­is­tic in­vestors may want to in­vest differ­ently than self-in­ter­ested mar­ket par­ti­ci­pants. Ad­di­tion­ally, he in­tends to fo­cus more on giv­ing now vs later: that is, whether to donate in the short term, or save and donate later.

We were im­pressed by his prior es­says analysing in­vest­ment strate­gies. I pre­vi­ously worked as a quan­ti­ta­tive trader, and I saw Michael’s es­says as de­scribing straight­for­ward but use­ful ap­pli­ca­tions of as­set pric­ing mod­els and pro­vid­ing a good liter­a­ture re­view of in­vest­ment ad­vice. While I dis­agreed with some of the claims he made, he had ex­plic­itly ac­knowl­edged in the es­say that those claims were con­tro­ver­sial.

Whether to give now or later is an im­por­tant ques­tion for long-ter­mists that has re­ceived limited at­ten­tion. Based on his track record, we ex­pect Michael will both be able to make re­search progress and com­mu­ni­cate these re­sults clearly.