Long-Term Future Fund: September 2020 grants

Link post

In this re­port, alongside in­for­ma­tion about our lat­est grants, we have fur­ther news to share about the Long-Term Fu­ture Fund.

Changes to Fund management

We wel­come two new Fund man­agers to our team: Adam Gleave and Asya Ber­gal. Adam is a PhD can­di­date at UC Berkeley, work­ing on tech­ni­cal AI safety with the Cen­ter for Hu­man-Com­pat­i­ble AI (CHAI). Asya Ber­gal works as a re­searcher at AI Im­pacts and writes for the AI Align­ment Newslet­ter.

We de­cided to seek new Fund man­agers with strong back­grounds in AI safety and strat­egy re­search to in­crease our ca­pac­ity to care­fully eval­u­ate grants in these ar­eas, es­pe­cially given that Alex Zhu left the Long-Term Fu­ture Fund (LTFF) early this year. We ex­pect the num­ber of high-qual­ity grant ap­pli­ca­tions in these ar­eas to in­crease over time.

The new Fund man­agers were ap­pointed by Matt Wage, the Fund’s chair­per­son, af­ter a search pro­cess with con­sul­ta­tion from ex­ist­ing Fund man­agers and ad­vi­sors. They both tri­aled for one grant round (Adam in March, Asya in July) be­fore be­ing con­firmed as per­ma­nent mem­bers. We are ex­cited about their broad-rang­ing ex­per­tise in longter­mism, AI safety, and AI strat­egy.

Adam is a PhD can­di­date ad­vised by Stu­art Rus­sell and has pre­vi­ously in­terned at Deep­Mind. He has pub­lished sev­eral safety-rele­vant ma­chine learn­ing pa­pers. Over the past six years, he was deeply in­volved with the effec­tive al­tru­ism com­mu­nity (run­ning an EA group at the Univer­sity of Cam­bridge, earn­ing to give as a quan­ti­ta­tive trader) and demon­strated care­ful judg­ment on a broad range of longter­mist pri­ori­ti­za­tion ques­tions (see, e.g., his donor lot­tery re­port).

Asya brings a broad-rang­ing AI back­ground to the Fund. At AI Im­pacts, she has worked on a va­ri­ety of em­piri­cal pro­jects and de­vel­oped novel per­spec­tives on far-rang­ing strat­egy ques­tions (see, e.g., this pre­sen­ta­tion on AI timelines). She has demon­strated tech­ni­cal profi­ciency in AI, writ­ing sum­maries and opinions of pa­pers for the AI Align­ment Newslet­ter. More re­cently, she has been work­ing on hard­ware fore­cast­ing ques­tions at the Cen­tre for the Gover­nance of AI. Asya has also re­searched a broad range of longter­mist pri­ori­ti­za­tion ques­tions, in­clud­ing at the Open Philan­thropy Pro­ject (where she looked into whole brain em­u­la­tion, an­i­mal welfare, and biose­cu­rity).

Adam and Asya both had very pos­i­tive ex­ter­nal refer­ences, and both ap­pear to be es­teemed and trust­wor­thy com­mu­nity mem­bers. Dur­ing the trial, they demon­strated care­ful judg­ment and deep en­gage­ment with the grant ap­pli­ca­tions. We are ex­cited to have them on board and be­lieve their con­tri­bu­tions will fur­ther im­prove the qual­ity of our grants.

In other news, Jonas Vol­lmer re­cently joined CEA as Head of EA Funds. He pre­vi­ously served as an ad­vi­sor to the LTFF. In his new role, he will make de­ci­sions on be­half of EA Funds and ex­plore longer-term strat­egy for the en­tire EA Funds pro­ject, in­clud­ing the LTFF. EA Funds may be spun out of CEA’s core team within 6--12 months.

Other updates

Long-Term Fu­ture Fund

  • We plan to con­tinue to fo­cus on grants to small pro­jects and in­di­vi­d­u­als rather than large or­ga­ni­za­tions. We think the Fund has a com­par­a­tive ad­van­tage in this area: In­di­vi­d­ual donors can­not eas­ily grant to re­searchers and small pro­jects, and large grant­mak­ers such as the Open Philan­thropy Pro­ject are less ac­tive in the area of small grants.

  • We would like to take some con­crete ac­tions to in­crease trans­parency around our grant­mak­ing pro­cess, partly in re­sponse to feed­back from donors and grantseek­ers. Over the next few months, we plan to pub­lish a doc­u­ment out­lin­ing our pro­cess and run an “Ask Me Any­thing” ses­sion with our new Fund team on the Effec­tive Altru­ism Fo­rum.

  • We are ten­ta­tively con­sid­er­ing ex­pand­ing into more ac­tive grant­mak­ing, which would en­tail pub­li­cly ad­ver­tis­ing the types of work we would be ex­cited to fund (de­tails TBD).

EA Funds

Ear­lier this year, EA Funds ran a donor sur­vey elic­it­ing feed­back on the Long Term Fu­ture Fund.

  • Over­all, the LTFF re­ceived a rel­a­tively low Net Pro­moter score: when asked “How likely is it that you would recom­mend the Long-Term Fu­ture Fund to a friend or col­league?”, donors re­sponded with an av­er­age of 6.5 on a scale from 1 to 10. How­ever, some donors gave a low score de­spite be­ing satis­fied with the Fund be­cause their friends and col­leagues are gen­er­ally un­in­ter­ested in longter­mism. In fu­ture sur­veys, EA Funds in­tends to ask ques­tions that more di­rectly ad­dress how donors them­selves feel about the LTFF.

  • Some donors were in­ter­ested in how the Fund ad­dresses con­flicts of in­ter­est, so EA Funds has been de­vel­op­ing a con­flict of in­ter­est policy and in­tends to have stric­ter rules around grants to the per­sonal ac­quain­tances of Fund man­agers.

  • Some donors were sur­prised by the Fund’s large num­ber of AI risk-fo­cused grants. While the Fund man­agers are in fa­vor of these grants, we want to make sure that donors are aware of the work they are sup­port­ing. As a re­sult, we changed the EA Funds dona­tion in­ter­face such that donors have to opt into sup­port­ing their cho­sen Funds. (Pre­vi­ously, the web­site sug­gested a de­fault al­lo­ca­tion for each Fund.) EA Funds also plans to offer a dona­tion op­tion fo­cused on cli­mate change for in­ter­ested donors.

  • Some donors ex­pressed a prefer­ence for more leg­ible grants (e.g., to es­tab­lished, rep­utable in­sti­tu­tions). EA Funds will con­sider offer­ing a sep­a­rate dona­tion op­tion for those donors; while we are still de­vel­op­ing our plans, this might take the form of a sep­a­rate Fund that pri­mar­ily sup­ports Open Philan­thropy’s longter­mist grant re­cip­i­ents.

Grant recipients

Each grant re­cip­i­ent is fol­lowed by the size of the grant and a one-sen­tence de­scrip­tion of their pro­ject. All of these grants have been paid out.

Grants made dur­ing our stan­dard cy­cle:

  • Robert Miles ($60,000): Creat­ing qual­ity videos on AI safety, and offer­ing com­mu­ni­ca­tion and me­dia sup­port to AI safety orgs.

  • Cen­ter for Hu­man-Com­pat­i­ble AI ($75,000): Hiring a re­search en­g­ineer to sup­port CHAI’s tech­ni­cal re­search pro­jects.

  • Joe Col­l­man ($25,000): Devel­op­ing al­gorithms, en­vi­ron­ments and tests for AI safety via de­bate.

  • AI Im­pacts ($75,000): An­swer­ing de­ci­sion-rele­vant ques­tions about the fu­ture of ar­tifi­cial in­tel­li­gence.

  • Alexis Car­lier ($5,000): Sur­vey­ing ex­perts on AI risk sce­nar­ios and work­ing on other pro­jects re­lated to AI safety.

  • Gavin Tay­lor ($30,000): Con­duct­ing a com­pu­ta­tional study on us­ing a light-to-vibra­tions mechanism as a tar­geted an­tiviral.

  • Cen­ter for Elec­tion Science ($50,000): Sup­port­ing the use of bet­ter vot­ing meth­ods in U.S. elections

  • Char­lie Rogers-Smith ($7,900): Sup­port­ing re­search and job ap­pli­ca­tions re­lated to AI al­ign­ment.

Off-cy­cle grants:

  • Clau­dia Shi ($5,000): Or­ga­niz­ing a “Hu­man-Aligned AI” event at NeurIPS.

  • Gopal Sarma ($5,000): Or­ga­niz­ing a work­shop aimed at high­light­ing re­cent suc­cesses in the de­vel­op­ment of ver­ified soft­ware.

  • Alex Turner ($30,000): Un­der­stand­ing when and why pro­posed AI de­signs seek power over their en­vi­ron­ment.

  • Cam­bridge Sum­mer Pro­gramme in Ap­plied Rea­son­ing (CaSPAR) ($26,300): Or­ga­niz­ing im­mer­sive work­shops on meta skills and x-risk for STEM stu­dents at top uni­ver­si­ties.

Grant reports

Oliver Habryka

Robert Miles ($60,000)

Creat­ing qual­ity videos on AI safety, and offer­ing com­mu­ni­ca­tion and me­dia sup­port to AI safety orgs.

We’ve funded Rob Miles in the past, and since Rob’s work has con­tinued to find trac­tion and main­tain a high qual­ity bar, I am view­ing this mostly as a grant re­newal. Back then, I gave the fol­low­ing ra­tio­nale for the grant:

The videos on [Rob’s] YouTube chan­nel pick up an av­er­age of ~20k views. His videos on the offi­cial Com­put­er­phile chan­nel of­ten pick up more than 100k views, in­clud­ing for top­ics like log­i­cal un­cer­tainty and cor­rigi­bil­ity (in­ci­den­tally, a term Rob came up with).

More things that make me op­ti­mistic about Rob’s broad ap­proach:

  • He ex­plains that AI al­ign­ment is a tech­ni­cal prob­lem. AI safety is not pri­mar­ily a moral or poli­ti­cal po­si­tion; the biggest chunk of the prob­lem is a mat­ter of com­puter sci­ence. Reach­ing out to a tech­ni­cal au­di­ence to ex­plain that AI safety is a tech­ni­cal prob­lem, and thus di­rectly re­lated to their pro­fes­sion, is a type of ‘out­reach’ that I’m very happy to en­dorse.

  • He does not make AI safety a poli­ti­cized mat­ter. I am very happy that Rob is not need­lessly trib­al­is­ing his con­tent, e.g. by talk­ing about some­thing like “good vs bad ML re­searchers”. He seems to sim­ply por­tray it as a set of in­ter­est­ing and im­por­tant tech­ni­cal prob­lems in the de­vel­op­ment of AGI.

  • His goal is to cre­ate in­ter­est in these prob­lems from fu­ture re­searchers, and not to sim­ply get as large of an au­di­ence as pos­si­ble. As such, Rob’s ex­pla­na­tions don’t op­ti­mize for views at the ex­pense of qual­ity ex­pla­na­tion. His videos are clearly de­signed to be en­gag­ing, but his ex­pla­na­tions are sim­ple and ac­cu­rate. Rob of­ten in­ter­acts with re­searchers in the com­mu­nity (at places like Deep­Mind and MIRI) to dis­cuss which con­cepts are in need of bet­ter ex­pla­na­tions. I don’t ex­pect Rob to take unilat­eral ac­tion in this do­main.

Rob is the first skil­led per­son in the X-risk com­mu­nity work­ing full-time on pro­duc­ing video con­tent. Be­ing the very best we have in this skill area, he is able to help the com­mu­nity in a num­ber of novel ways (for ex­am­ple, he’s already helping ex­ist­ing or­ga­ni­za­tions pro­duce videos about their ideas).

Since then, the av­er­age views on his videos ap­pear to have quin­tu­pled, usu­ally eclips­ing 100k views on YouTube. While I have a lot of un­cer­tainty about what level of en­gage­ment those views rep­re­sent, it would not sur­prise me if more than 15% of peo­ple in­tro­duced to the topic of AI al­ign­ment in the last year dis­cov­ered it through Rob’s YouTube chan­nel. This would be a sub­stan­tial figure, and I also con­sider Rob’s ma­te­rial one of the best ways to be in­tro­duced to the topic (in terms of ac­cu­rately con­vey­ing what the field is about).

In most wor­lds where I think this grant turns out to be bad, it is be­cause it is cur­rently harm­ful for the field of AI al­ign­ment to grow rapidly, be­cause it might cause the field to be­come harder to co­or­di­nate, cause more bad ideas to be­come pop­u­lar, or lead too many peo­ple to join who don’t have suffi­cient back­ground or tal­ent to make strong con­tri­bu­tions. I think it is rel­a­tively un­likely that we are in that world, and I con­tinue to think that the type of out­reach Rob is do­ing is quite valuable, but I still think there’s at least a 5% prob­a­bil­ity to it be­ing bad for the AI Align­ment field to grow right now.

I trust Rob to think about these con­sid­er­a­tions and to be care­ful about how he in­tro­duces peo­ple to the field; thus, I ex­pect that if we were to end up in a world where this kind of out­reach is more harm­ful than use­ful, Rob would take ap­pro­pri­ate ac­tion.

Cen­ter for Hu­man-Com­pat­i­ble AI ($75,000)

Hiring a re­search en­g­ineer to sup­port CHAI’s tech­ni­cal re­search pro­jects.

Over the last few years, CHAI has hosted a num­ber of peo­ple who I think have con­tributed at a very high qual­ity level to the AI al­ign­ment prob­lem, most promi­nently Ro­hin Shah, who has been writ­ing and up­dat­ing the AI Align­ment Newslet­ter and has also pro­duced a sub­stan­tial num­ber of other high-qual­ity ar­ti­cles, like this sum­mary of AI al­ign­ment progress in 2018-2019.

Ro­hin is leav­ing CHAI soon, and I’m un­sure about CHAI’s fu­ture im­pact, since Ro­hin made up a large frac­tion of the im­pact of CHAI in my mind.

I have read a num­ber of pa­pers and ar­ti­cles from other CHAI grad stu­dents, and I think that the over­all ap­proach I see most of them tak­ing has sub­stan­tial value, but I also main­tain a rel­a­tively high level of skep­ti­cism about re­search that tries to em­bed it­self too closely within the ex­ist­ing ML re­search paradigm. That paradigm, at least in the past, hasn’t re­ally pro­vided any space for what I con­sider the most valuable safety work (though I think most other mem­bers of the Fund don’t share my skep­ti­cism). I don’t think I have the space in this re­port to fully ex­plain where that skep­ti­cism is com­ing from, so the be­low should only be seen as a very cur­sory ex­plo­ra­tion of my thoughts here.

A con­crete ex­am­ple of the prob­lems I have seen (cho­sen for its sim­plic­ity more than its im­por­tance) is that, on sev­eral oc­ca­sions, I’ve spo­ken to au­thors who, dur­ing the pub­li­ca­tion and peer-re­view pro­cess, wound up hav­ing to re­move some of their pa­pers’ most im­por­tant con­tri­bu­tions to AI al­ign­ment. Often, they also had to add ma­te­rial that seemed likely to con­fuse read­ers about the pa­per’s pur­pose. One con­crete class of ex­am­ples: adding em­piri­cal simu­la­tions of sce­nar­ios whose out­come is triv­ially pre­dictable, where the speci­fi­ca­tion of the sce­nario adds a sub­stan­tial vol­ume of un­nec­es­sary com­plex­ity to the pa­per, while dis­tract­ing from the gen­er­al­ity of the over­all ar­gu­ments.

Another con­cern: Most of the im­pact that Ro­hin con­tributed seemed to be driven more by dis­til­la­tion and field-build­ing work than by novel re­search. As I have ex­pressed in the past (and el­se­where in this re­port), I be­lieve dis­til­la­tion and field-build­ing to be par­tic­u­larly ne­glected and valuable at the mar­gin. I don’t cur­rently see the rest of CHAI en­gag­ing in that work in the same way.

On the other hand, since it ap­pears that CHAI has prob­a­bly been quite im­pact­ful on Ro­hin’s abil­ity to pro­duce work, I am some­what op­ti­mistic that there are more peo­ple whose work is am­plified by the ex­is­tence of CHAI, even if I am less fa­mil­iar with their work, and I am also rea­son­ably op­ti­mistic that CHAI will be able to find other con­trib­u­tors as good as Ro­hin. I’ve also found en­gag­ing with An­drew Critch’s think­ing on AI al­ign­ment quite valuable, and I am hope­ful about more work from Stu­art Rus­sell, who ob­vi­ously has a very strong track record in terms of gen­eral re­search out­put, though my sense is that marginal fund­ing to CHAI is un­likely to in­crease Stu­art’s out­put in par­tic­u­lar (and might in fact de­crease it, since man­ag­ing an or­ga­ni­za­tion takes time away from re­search).

While I eval­u­ated this fund­ing re­quest pri­mar­ily as un­re­stricted fund­ing to CHAI, the spe­cific pro­ject that CHAI is re­quest­ing money for seems also quite rea­son­able to me. Given the pro­saic na­ture of a lot of CHAI’s AI al­ign­ment works, it seems quite im­por­tant for them to be able to run en­g­ineer­ing-heavy ma­chine learn­ing pro­jects, for which it makes sense to hire re­search en­g­ineers to as­sist with the as­so­ci­ated pro­gram­ming tasks. The re­ports we’ve re­ceived from stu­dents at CHAI also sug­gest that past en­g­ineer hiring has been valuable and has en­abled stu­dents at CHAI to do sub­stan­tially bet­ter work.

Hav­ing thought more re­cently about CHAI as an or­ga­ni­za­tion and its place in the ecosys­tem of AI al­ign­ment,I am cur­rently un­cer­tain about its long-term im­pact and where it is go­ing, and I even­tu­ally plan to spend more time think­ing about the fu­ture of CHAI. So I think it’s not that un­likely (~20%) that I might change my mind on the level of pos­i­tive im­pact I’d ex­pect from fu­ture grants like this. How­ever, I think this holds less for the other Fund mem­bers who were also in fa­vor of this grant, so I don’t think my un­cer­tainty is much ev­i­dence about how LTFF will think about fu­ture grants to CHAI.

(Re­cusal note: Due to be­ing a grad stu­dent at CHAI, Adam Gleave re­cused him­self from the dis­cus­sion and vot­ing sur­round­ing this grant.)

Adam Gleave

Joe Col­l­man ($25,000)

Devel­op­ing al­gorithms, en­vi­ron­ments and tests for AI safety via de­bate.

Joe was pre­vi­ously awarded $10,000 for in­de­pen­dent re­search into ex­ten­sions to AI safety via de­bate. We have re­ceived pos­i­tive feed­back re­gard­ing his work and are pleased to see he has formed a col­lab­o­ra­tion with Beth Barnes at OpenAI. In this round, we have awarded $25,000 to sup­port Joe’s con­tinued work and col­lab­o­ra­tion in this area.

Joe in­tends to con­tinue col­lab­o­rat­ing with Beth to fa­cil­i­tate her work in test­ing de­bate in hu­man sub­ject stud­ies. He also in­tends to de­velop sim­plified en­vi­ron­ments for de­bate, and to de­velop and eval­u­ate ML al­gorithms in this en­vi­ron­ment.

In gen­eral, I ap­ply a fairly high bar to fund­ing in­de­pen­dent re­search, as I be­lieve most peo­ple are more pro­duc­tive work­ing for a re­search or­ga­ni­za­tion. In this case, how­ever, Joe has demon­strated an abil­ity to make progress in­de­pen­dently and forge col­lab­o­ra­tions with es­tab­lished re­searchers. I hope this grant will en­able Joe to fur­ther de­velop his skills in the area, and to pro­duce re­search out­put that can demon­strate his abil­ities to po­ten­tial em­ploy­ers and/​or fun­ders.

AI Im­pacts ($75,000)

An­swer­ing de­ci­sion-rele­vant ques­tions about the fu­ture of ar­tifi­cial in­tel­li­gence.

AI Im­pacts is a non­profit or­ga­ni­za­tion (fis­cally spon­sored by MIRI) in­ves­ti­gat­ing de­ci­sion-rele­vant ques­tions about the fu­ture of ar­tifi­cial in­tel­li­gence. Their work has and con­tinues to in­fluence my out­look on how and when ad­vanced AI will de­velop, and I of­ten see re­searchers I col­lab­o­rate with cite their work in con­ver­sa­tions. Notable re­cent out­put in­cludes an in­ter­view se­ries around rea­sons why benefi­cial AI may be de­vel­oped “by de­fault” and con­tinued work on ex­am­ples of dis­con­tin­u­ous progress.

I would char­ac­ter­ize much of AI Im­pacts’ re­search as things that are fairly ob­vi­ous to look into but which, sur­pris­ingly, no one else has. In part this is be­cause their re­search is of­ten sec­ondary, sum­ma­riz­ing rele­vant ex­ist­ing sources, and in­ter­dis­ci­plinary—both of which are un­der-in­cen­tivized in academia. Choos­ing the right ques­tions to in­ves­ti­gate also re­quires con­sid­er­able skill and fa­mil­iar­ity with AI re­search.

Over­all, I would be ex­cited to see more re­search into bet­ter un­der­stand­ing how AI will de­velop in the fu­ture. This re­search can help fun­ders to de­cide which pro­jects to sup­port (and when), and re­searchers to se­lect an im­pact­ful re­search agenda. We are pleased to sup­port AI Im­pacts’ work in this space, and hope this re­search field will con­tinue to grow.

We awarded a grant of $75,000, ap­prox­i­mately one fifth of the AI Im­pacts bud­get. We do not ex­pect sharply diminish­ing re­turns, so it is likely that at the mar­gin, ad­di­tional fund­ing to AI Im­pacts would con­tinue to be valuable. When fund­ing es­tab­lished or­ga­ni­za­tions, we of­ten try to con­tribute a “fair share” of or­ga­ni­za­tions’ bud­gets based on the Fund’s over­all share of the fund­ing land­scape. This aids co­or­di­na­tion with other donors and en­courages or­ga­ni­za­tions to ob­tain fund­ing from di­verse sources (which re­duces the risk of fi­nan­cial is­sues if one source be­comes un­available).

(Re­cusal note: Due to work­ing as a con­trac­tor for AI Im­pacts, Asya Ber­gal re­cused her­self from the dis­cus­sion and vot­ing sur­round­ing this grant.)

Asya Bergal

Alexis Car­lier ($5,000)

Sur­vey­ing ex­perts on AI risk sce­nar­ios and work­ing on other pro­jects re­lated to AI safety.

We awarded Alexis $5,000, pri­mar­ily to sup­port his work on a sur­vey aimed at iden­ti­fy­ing the ar­gu­ments and re­lated be­liefs mo­ti­vat­ing top AI safety and gov­er­nance re­searchers to work on re­duc­ing ex­is­ten­tial risk from AI.

I think the views of top re­searchers in the AI risk space have a strong effect on the views and re­search di­rec­tions of other effec­tive al­tru­ists. But as of now, only a small and po­ten­tially un­rep­re­sen­ta­tive set of views ex­ist in writ­ten form, and many are stated in im­pre­cise ways. I am hope­ful that a widely-taken sur­vey will fill this gap and have a strong pos­i­tive effect on fu­ture re­search di­rec­tions.

I thought Alexis’s pre­vi­ous work on the prin­ci­pal-agent liter­a­ture and AI risk was use­ful and thought­fully done, and showed that he was able to col­lab­o­rate with promi­nent re­searchers in the space. This col­lab­o­ra­tion, as well as de­tails of the ap­pli­ca­tion, sug­gested to me that the sur­vey ques­tions would be writ­ten with lots of in­put from ex­ist­ing re­searchers, and that Alexis was likely to be able to get wide­spread sur­vey en­gage­ment.

Since recom­mend­ing this grant, I have seen the sur­vey cir­cu­lated and taken it my­self. I thought it was a good sur­vey and am ex­cited to see the re­sults.

Gavin Tay­lor ($30,000)

Con­duct­ing a com­pu­ta­tional study on us­ing a light to vibra­tions mechanism as a tar­geted an­tiviral.

We awarded Gavin $30,000 to work on a com­pu­ta­tional study as­sess­ing the fea­si­bil­ity of us­ing a light to vibra­tions (L2V) mechanism as a tar­geted an­tiviral. Light to vibra­tions is an emerg­ing tech­nique that could de­stroy viruses by vibrat­ing them at their res­o­nant fre­quency us­ing tuned pulses of light. In an op­ti­mistic sce­nario, this study would iden­tify a set of viruses that are the­o­ret­i­cally sus­cep­ti­ble to L2V in­ac­ti­va­tion. Re­sults would be pub­lished in aca­demic jour­nals and would pave the way for fur­ther ex­per­i­men­tal work, pro­to­types, and even­tual com­mer­cial pro­duc­tion of L2V an­tiviral equip­ment. L2V tech­niques could be gen­er­al­iz­able and rapidly adapt­able to new pathogens, which would provide an ad­van­tage over other tech­niques used for large-scale con­trol of fu­ture viral pan­demics.

On this grant, I largely deferred to the ex­per­tise of col­leagues work­ing in physics and biorisk. My ul­ti­mate take af­ter talk­ing to them was that the de­scribed ap­proach was plau­si­ble and could mean­ingfully af­fect the course of fu­ture pan­demics, al­though oth­ers have also re­cently started work­ing on L2V ap­proaches.

My im­pres­sion is that Gavin’s aca­demic back­ground is well-suited to do­ing this work, and I re­ceived pos­i­tive per­sonal feed­back on his com­pe­tence from other EAs work­ing in biorisk.

My main un­cer­tainty recom­mend­ing this grant was in how the LTFF should com­pare rel­a­tively nar­row biorisk in­ter­ven­tions with other things we might fund. I ul­ti­mately de­cided that this pro­ject was worth fund­ing, but still don’t have a good way of think­ing about this ques­tion.

Matt Wage

Cen­ter for Elec­tion Science ($50,000)

Sup­port­ing the use of bet­ter vot­ing meth­ods in U.S. elec­tions.

This is an un­re­stricted grant to the Cen­ter for Elec­tion Science (CES). CES works to im­prove US elec­tions by pro­mot­ing ap­proval vot­ing, a vot­ing method where vot­ers can se­lect as many can­di­dates as they like (as op­posed to the tra­di­tional vot­ing method where you can only se­lect one can­di­date).

Aca­demic ex­perts on vot­ing the­ory widely con­sider ap­proval vot­ing to be a sig­nifi­cant im­prove­ment over our cur­rent vot­ing method (plu­ral­ity vot­ing), and our un­der­stand­ing is that ap­proval vot­ing on av­er­age pro­duces out­comes that bet­ter re­flect what vot­ers ac­tu­ally want by pre­vent­ing is­sues like vote split­ting. I think that pro­mot­ing ap­proval vot­ing is a po­ten­tially promis­ing form of im­prov­ing in­sti­tu­tional de­ci­sion mak­ing within gov­ern­ment.

CES is a rel­a­tively young or­ga­ni­za­tion, but so far they have a rea­son­able track record. Pre­vi­ously, they passed a bal­lot ini­ti­a­tive to adopt ap­proval vot­ing in the 120,000-per­son city of Fargo, ND, and are now re­peat­ing this effort in St. Louis. Their next goal is to get ap­proval vot­ing adopted in big­ger cities and then even­tu­ally states.

Char­lie Rogers-Smith ($7,900)

Sup­port­ing re­search and job ap­pli­ca­tions re­lated to AI al­ign­ment.

Char­lie ap­plied for fund­ing to spend a year do­ing re­search with Jan Brauner, Sören Min­der­mann, and their su­per­vi­sor Yarin Gal (all at Oxford Univer­sity), while ap­ply­ing to PhD pro­grams to even­tu­ally work on AI al­ign­ment. Char­lie is cur­rently finish­ing a mas­ter’s in statis­tics at Oxford and is also par­ti­ci­pat­ing in the Fu­ture of Hu­man­ity In­sti­tute’s Sum­mer Re­search Fel­low­ship.

We think Pro­fes­sor Gal is in a bet­ter po­si­tion to eval­u­ate this pro­posal (and our un­der­stand­ing is that his group is ca­pa­ble of pro­vid­ing fund­ing for this them­selves), but it will take some time for this to hap­pen. There­fore, we de­cided to award Char­lie a small “bridge fund­ing” grant to give him time to try to fi­nal­ize the pro­posal with Pro­fes­sor Gal or find an al­ter­na­tive po­si­tion.

Off-cy­cle grants

The fol­low­ing grants were made out­side of our reg­u­lar sched­ule, and weren’t in­cluded in pre­vi­ous pay­out re­ports, so we’re in­clud­ing them here.

He­len Toner

Clau­dia Shi ($5,000)

Or­ga­niz­ing a “Hu­man-Aligned AI” event at NeurIPS.

Grant date: Novem­ber 2019

Clau­dia Shi and Vic­tor Veitch ap­plied for fund­ing to run a so­cial event themed around “Hu­man-al­igned AI” at the ma­chine learn­ing con­fer­ence NeurIPS in De­cem­ber 2019. The aim of the event was to provide a space for NeurIPS at­ten­dees who care about do­ing high-im­pact pro­jects and/​or about long-term AI safety to gather and dis­cuss these top­ics.

I be­lieve that hold­ing events like this is an easy way to do a very ba­sic form of “field-build­ing,” by mak­ing it eas­ier for ma­chine learn­ing re­searchers who are in­ter­ested in longter­mism and re­lated top­ics to find each other, dis­cuss their work, and per­haps work to­gether in the fu­ture or change their re­search plans. Our fund­ing was mainly used to cover cater­ing for the 100-per­son event, which we hoped would make the event more en­joy­able for par­ti­ci­pants and there­fore more effec­tive in fa­cil­i­tat­ing dis­cus­sions and con­nec­tions. After the event, the or­ga­niz­ers had $1863 left over, which they re­turned to the Fund.

Matt Wage

Gopal Sarma ($5,000)

Or­ga­niz­ing a work­shop aimed at high­light­ing re­cent suc­cesses in the de­vel­op­ment of ver­ified soft­ware.

Grant date: Jan­uary 2020.

Gopal ap­plied for a grant to run a work­shop called “For­mal Meth­ods for the In­for­mal Eng­ineer” (FMIE) at the Broad In­sti­tute of MIT and Har­vard, on the topic of for­mal meth­ods in soft­ware en­g­ineer­ing. More in­for­ma­tion on the work­shop is here.

We made this grant be­cause we know a small set of AI safety re­searchers are op­ti­mistic about for­mal ver­ifi­ca­tion tech­niques be­ing use­ful for AI safety, and we thought this grant was a rel­a­tively in­ex­pen­sive way to sup­port progress in that area.

Un­for­tu­nately, the work­shop has now been post­poned be­cause of COVID-19.

Oliver Habryka

Alex Turner ($30,000)

Un­der­stand­ing when and why pro­posed AI de­signs seek power over their en­vi­ron­ment.

Grant date: Jan­uary 2020

We pre­vi­ously made a grant to Alex Turner at the be­gin­ning of 2019. Here is what I wrote at the time:

My thoughts and reasoning

I’m ex­cited about this be­cause:

Alex’s ap­proach to find­ing per­sonal trac­tion in the do­main of AI Align­ment is one that I would want many other peo­ple to fol­low. On LessWrong, he read and re­viewed a large num­ber of math text­books that are use­ful for think­ing about the al­ign­ment prob­lem, and sought pub­lic in­put and feed­back on what things to study and read early on in the pro­cess.

He wasn’t in­timi­dated by the com­plex­ity of the prob­lem, but started think­ing in­de­pen­dently about po­ten­tial solu­tions to im­por­tant sub-prob­lems long be­fore he had “com­pre­hen­sively” stud­ied the math­e­mat­i­cal back­ground that is com­monly cited as be­ing the foun­da­tion of AI Align­ment.

He wrote up his thoughts and hy­pothe­ses in a clear way, sought feed­back on them early, and ended up mak­ing a set of novel con­tri­bu­tions to an in­ter­est­ing sub-field of AI Align­ment quite quickly (in the form of his work on im­pact mea­sures, on which he re­cently col­lab­o­rated with the Deep­Mind AI Safety team)

Po­ten­tial concerns

Th­ese in­tu­itions, how­ever, are a bit in con­flict with some of the con­crete re­search that Alex has ac­tu­ally pro­duced. My in­side views on AI al­ign­ment make me think that work on im­pact mea­sures is very un­likely to re­sult in much con­crete progress on what I per­ceive to be core AI al­ign­ment prob­lems, and I have talked to a va­ri­ety of other re­searchers in the field who share that as­sess­ment. I think it’s im­por­tant that this grant not be viewed as an en­dorse­ment of the con­crete re­search di­rec­tion that Alex is pur­su­ing, but only as an en­dorse­ment of the higher-level pro­cess that he has been us­ing while do­ing that re­search.

As such, I think it was a nec­es­sary com­po­nent of this grant that I have talked to other peo­ple in AI al­ign­ment whose judg­ment I trust, who do seem ex­cited about Alex’s work on im­pact mea­sures. I think I would not have recom­mended this grant, or at least this large of a grant amount, with­out their en­dorse­ment. I think in that case I would have been wor­ried about a risk of di­vert­ing at­ten­tion from what I think are more promis­ing ap­proaches to AI Align­ment, and a po­ten­tial dilu­tion of the field by in­tro­duc­ing a set of (to me) some­what du­bi­ous philo­soph­i­cal as­sump­tions.

Over­all, while I try my best to form con­crete and de­tailed mod­els of the AI al­ign­ment re­search space, I don’t cur­rently de­vote enough time to it to build de­tailed mod­els that I trust enough to put very large weight on my own per­spec­tive in this par­tic­u­lar case. In­stead, I am mostly defer­ring to other re­searchers in this space that I do trust, a num­ber of whom have given pos­i­tive re­views of Alex’s work.

In ag­gre­gate, I have a sense that the way Alex went about work­ing on AI al­ign­ment is a great ex­am­ple for oth­ers to fol­low, I’d like to see him con­tinue, and I am ex­cited about the LTF Fund giv­ing out more grants to oth­ers who try to fol­low a similar path.

I’ve been fol­low­ing Alex’s work closely since then, and over­all have been quite happy with its qual­ity. I still have high-level con­cerns about his ap­proach, but have over time be­come more con­vinced that Alex is aware of some of the philo­soph­i­cal prob­lems that work on im­pact mea­sures seems to run into, and so am more con­fi­dent that he will nav­i­gate the difficul­ties of this space cor­rectly. His work also up­dated me on the tractabil­ity of im­pact-mea­sure ap­proaches, and though I am still skep­ti­cal, I am sub­stan­tially more open to in­ter­est­ing in­sights com­ing out of an anal­y­sis of that space than I was be­fore. (I think it is gen­er­ally more valuable to pur­sue a promis­ing ap­proach that many peo­ple are skep­ti­cal about, rather than one already known to be good, be­cause the former is much less likely to be re­place­able).

I’ve also con­tinued to get pos­i­tive feed­back from oth­ers in the field of AI al­ign­ment about Alex’s work, and have had mul­ti­ple con­ver­sa­tions with peo­ple who thought it made a differ­ence to their think­ing on AI al­ign­ment.

One other thing that has ex­cited me about Alex’s work is his ped­a­gog­i­cal ap­proach to his in­sights. Re­searchers fre­quently pro­duce ideas with­out pay­ing at­ten­tion to how un­der­stand­able those ideas are to other peo­ple, and en­shrine for­mu­la­tions that end up be­ing clunky, un­in­tu­itive or un­wieldy, as well as ex­pla­na­tions that aren’t ac­tu­ally very good at ex­plain­ing. Over time, this poor com­mu­ni­ca­tion of­ten re­sults in sub­stan­tial re­search debt. Alex, on the other hand, has put large amounts of effort into ex­plain­ing his ideas clearly and in an ap­proach­able way, with his “Refram­ing Im­pact” se­quence on the AI Align­ment Fo­rum.

This grant would fund liv­ing ex­penses and tu­ition, helping Alex to con­tinue his cur­rent line of re­search dur­ing his grad­u­ate pro­gram at Ore­gon State.

Cam­bridge Sum­mer Pro­gramme in Ap­plied Rea­son­ing (CaSPAR) ($26,300)

Or­ga­niz­ing im­mer­sive work­shops for STEM stu­dents at top uni­ver­si­ties.

Grant date: Jan­uary 2020

From the ap­pli­ca­tion:

We want to build on our mo­men­tum from CaSPAR 2019 by run­ning an­other in­ten­sive week-long sum­mer camp and alumni re­treat for math­e­mat­i­cally tal­ented Cam­bridge stu­dents in 2020, and in­crease the co­hort size by 13 from 12 to 16.

At CaSPAR, we at­tract young peo­ple who are tal­ented, al­tru­is­ti­cally mo­ti­vated and think transver­sally to show us what we might be miss­ing. We find them at Cam­bridge Univer­sity, in math­e­mat­ics and ad­ja­cent sub­jects, and fun­nel them via our se­lec­tion pro­cess to our week-long in­ten­sive sum­mer camp. After the camp, we wel­come them to the CaSPAR Alumni. In the alumni we fur­ther sup­port their plan changes/​ideas with them as peers, and send them op­por­tu­ni­ties at a de­ci­sion-rele­vant time of their lives.

CaSPAR is a sum­mer camp for Cam­bridge stu­dents that tries to cover a va­ri­ety of ma­te­rial re­lated to ra­tio­nal­ity and effec­tive al­tru­ism. This grant was origi­nally in­tended for CaSPAR 2020, but since COVID has made most in-per­son events like this in­fea­si­ble, this grant is in­stead in­tended for CaSPAR 2021.

I con­sider CaSPAR to be in a similar refer­ence class as SPARC or ESPR, two pro­grams with some­what similar goals that have been sup­ported by other fun­ders in the long-term fu­ture space. I cur­rently think in­ter­ven­tions in this space are quite valuable, and have been im­pressed with the im­pact of SPARC; mul­ti­ple very promis­ing peo­ple in the long-term fu­ture space cite it as the key rea­son they be­came in­volved.

The pri­mary two vari­ables I looked at while eval­u­at­ing CaSPAR were its staff com­po­si­tion and the refer­ences we re­ceived from a num­ber of peo­ple who worked with the CaSPAR team or at­tended their 2019 event. Both of those seemed quite solid to me. The team con­sists of peo­ple I think are pretty com­pe­tent and have the right skills for a pro­ject like this, and the refer­ences we re­ceived were pos­i­tive.

The biggest hes­i­ta­tion I have about this grant is mostly the size of the pro­gram and the num­ber of par­ti­ci­pants. Com­pared to SPARC or ESPR, the pro­gram is shorter and has sub­stan­tially fewer at­ten­dees. From my ex­pe­rience with those pro­grams, the size of the pro­gram and the length both seemed in­te­gral to their im­pact (I think there’s a sweet spot around 30 par­ti­ci­pants—enough peo­ple to take ad­van­tage of net­work effects and form lots of con­nec­tions, while still main­tain­ing a high-trust at­mo­sphere).