I agree this is a very helpful com­ment. I would add: these roles in my view are not *lesser* in any sense, for a range of rea­sons and I would en­courage peo­ple not to think of them in those terms.

• You might have a big­ger im­pact on the mar­gins be­ing the only—or one of the first few—peo­ple think­ing in EA terms in a philan­thropic foun­da­tion than by adding to the pool of ex­cel­lence at OpenPhil. This goes for any role that in­volves in­fluenc­ing how re­sources are al­lo­cated—which is a LOT, in char­ity, gov­ern­ment, in­dus­try, aca­demic foun­da­tions etc.

• You may not be in the pres­i­den­tial cab­i­net, or a spad to the UK prime minister, but those peo­ple are sup­ported and en­abled by peo­ple build­ing up the re­sources, ca­pac­ity, over­ton win­dow ex­pan­sion el­se­where in gov­ern­ment and civil ser­vice. The ‘se­nior per­son’ on their own may not be able to achieve pur­chase with key policy ideas and in­fluence.

• A lot of xrisk re­search, from biose­cu­rity to cli­mate change, draws on and de­pends on a huge body of work on biol­ogy, pub­lic policy, cli­mate sci­ence, re­new­able en­ergy, in­su­la­tion in homes, and much more. Often there are gaps in re­search on ex­treme sce­nar­ios due to lack of in­cen­tives for this kind of work, and other rea­sons—and this may make it par­tic­u­larly im­pact­ful at times. But that spe­cific work can’t be done well with­out draw­ing on all the un­der­ly­ing work. E.g., biorisk miti­ga­tion needs not just the peo­ple figur­ing out how to defend against the ex­treme sce­nar­ios, but also ev­ery­thing from peo­ple test­ing birds in viet­nam for H5N1 and seals in the north sea for H7, to peo­ple plan­ning for overflow ca­pac­ity in re­gional hos­pi­tals, to peo­ple push­ing for the value of pre­pared­ness funds in the rein­surance in­dus­try to much more. Same for cli­mate+en­vi­ron­ment, same will be true for AI policy etc.

• I think there’s prob­a­bly a good case to be made that in many or per­haps most in­stances the most use­ful place for the next gen­er­ally ca­pa­ble EA to be is *not* an EA org. And for all 80k’s great work, they can’t sur­vey and re­view ev­ery­thing, nor tai­lor to per­sonal fit for the thou­sands, or hun­dreds of thou­sands of differ­ent-skil­lset peo­ple who can play a role in mak­ing the fu­ture bet­ter.

For EA to re­ally make the fu­ture bet­ter to the ex­tent that it has the po­ten­tial, it’s go­ing to need a *much* big­ger global team. And that team’s go­ing to need to be in­ter­spersed ev­ery­where, some­times do­ing glamorous stuff, some­times do­ing more stan­dard stuff that is just as im­por­tant in that it makes the glamorous stuff pos­si­ble. To an­noy ev­ery­one with a sports anal­ogy, the defense and mid­field po­si­tions are ev­ery bit as im­por­tant as the glamorous striker po­si­tions, and if you’ve got a team made up pri­mar­ily of star strik­ers and wannabe star strik­ers, that team’s go­ing to un­der­perform.

The Global Challenges Foun­da­tion ( https://​​globalchallenges.org/​​ ) has a lot of in­for­ma­tion re­ally rele­vant for EAs. They care about im­prov­ing co­op­er­a­tion among coun­tries to re­duce global catas­trophic risks. I strongly recom­mend adding them in those up­dates. They have a newslet­ter.

Also, what do you think of karma as a mea­sure to the con­tri­bu­tion of a post to the com­mu­nity? I re­al­ize that I am con­flat­ing this with a mea­sure of trust, but these are not the same.

When I up­vote, I usu­ally think of how use­ful I think of the post for the com­mu­nity. Say, down­vote a post be­cause it was a waste of my time

Yea, makes sense. I guess that I’m just pig­gy­back­ing on the karma to make trade eas­ier.

Maybe you can have To­tal gained Karma and Unused Karma, where the site’s trust is based on theTo­tal, and you can only pay us­ing the Unused but any gain is to them both. This still leaves an op­tion for two mem­bers to juts trans­fer ea­chother karma and ar­tifi­cially in­crease thie trust level. This is not that bad as it only amounts to 2 times as large, and I do not re­aly think that peo­ple on the fo­rum would do that.

Very helpful sum­mary, thanks

So, af­ter I read this com­ment I left think­ing that Re­serve performed ex­cep­tion­ally poorly, but it seems that al­most all cryp­tocur­ren­cies have gone down about the same amount since June 19th (the time of Re­serve’s launch, from what I can tell). Here are some ran­dom cur­ren­cies that I clicked on, on the coin­mar­ket­cap web­site that you linked. This is a com­pre­hen­sive list, so I re­port the price change since June 19th for ev­ery cur­rency that I looked at:

• Bit­coin Cash:

  • June 19th price: $416

  • Price now: $244

  • Change: −41.3%

• XRP:

  • June 19th price: $0.448

  • Price now: $0.25

  • Change: −44.1%

• Lite­coin:

  • June 19th price: $135

  • Price now: $55

  • Change: −59.2%

• Monero

  • June 19th price: $100

  • Price now: $58

  • Change: −42%

You are also in­cor­rect that Bit­coin has re­turned 1% over the same time pe­riod. On June 19th, the price of Bit­coin was $9273, and it now is $8027. So while you are cor­rect that Bit­coin went down sig­nifi­cantly less than Re­serve, it performed dras­ti­cally bet­ter than al­most all other cryp­tocur­ren­cies, and still went down by about 13%.

I don’t think Re­serve is over­all a su­per great idea, but I think the statis­tics you cited seem mis­lead­ing to me, and it seems that Re­serve over­all is perform­ing similarly to the rest of the non-Bit­coin crypto-mar­ket.

Se­conded.

I think of my­self as mak­ing a lot of gam­bles with my ca­reer choices. And I sus­pect that re­gard­less of which way the propo­si­tions turn out, I’ll have an in­cli­na­tion to think that I was an idiot for not re­al­iz­ing them sooner. For ex­am­ple, I of­ten have both the fol­low­ing thoughts:

• “I have a bunch of com­par­a­tive ad­van­tage at helping MIRI with their stuff, and I’m not go­ing to be able to quickly re­duce my con­fi­dence in their re­search di­rec­tions. So I should stop wor­ry­ing about it and just do as much as I can.”

• “I am not sure whether the MIRI re­search di­rec­tions are good. Maybe I should spend more time eval­u­at­ing whether I should do a differ­ent thing in­stead.”

But even if it feels ob­vi­ous in hind­sight, it sure doesn’t feel ob­vi­ous now.

So I have big gam­bles that I’m mak­ing, which might turn out to be wrong, but which feel now like they will have been rea­son­able-in-hind­sight gam­bles ei­ther way. The main two such gam­bles are think­ing AI al­ign­ment might be re­ally im­por­tant in the next cou­ple decades and work­ing on MIRI’s ap­proaches to AI al­ign­ment in­stead of some other ap­proach.

When I ask my­self “what things have I not re­ally con­sid­ered as much as I should have”, I get an­swers that change over time (be­cause I ask my­self that ques­tion pretty of­ten and then try to con­sider the things that are im­por­tant). At the mo­ment, my an­swers are:

• Maybe I should think about/​work on s-risks much more

• Maybe I spend too much time in­vent­ing my own ways of solv­ing de­sign prob­lems in Haskell and I should study other peo­ple’s more.

• Maybe I am much more pro­duc­tive work­ing on out­reach stuff and I should do that full time.

• (This one is only on my mind this week and will prob­a­bly go away pretty soon) Maybe I’m not se­ri­ously enough en­gag­ing with ques­tions about whether the world will look re­ally differ­ent in a hun­dred years from how it looks to­day; per­haps I’m sub­ject to some bias to­wards sen­sa­tion­al­ism and ac­tu­ally the world will look similar in 100 years.

Ok, but note that this de­pends cru­cially on whether you de­cide that your util­ity looks more like log(GDP), or more like (GDP)^0.1, say. I don’t know how we can be con­fi­dent that it is one and not the other.

It’ll have [Link] au­to­mat­i­cally added to the ti­tle, and the link will ap­pear in ital­i­cized text at the top. Not sure that’s ex­haus­tive.

My val­ues be­ing differ­ently ex­pressed seems very im­por­tant, though. If I feel as if I value the welfare of dis­tant peo­ple, but I stop tak­ing ac­tions in line with that (e.g. mak­ing dona­tions to global poverty char­i­ties), do I still value it to the same ex­tent?

That said, my ex­am­ple wasn’t about ex­ter­nal be­havi­our changes, so you prob­a­bly weren’t re­spond­ing with that in mind.

I’ve inar­guably ex­pe­rienced drift in the leg­i­bil­ity of my val­ues to my­self, since I no longer have the same emo­tional sig­nal for them. I find the the term “Value Drift” a use­ful short­hand for that, but it sounds like you find it makes things un­clear?

I hadn’t ac­tu­ally no­ticed that.

One fac­tor here is that a lot of AI safety re­search seems to need ML ex­per­tise, which is one of my least fa­vorite types of CS/​en­g­ineer­ing.

Another is that com­pared to many EAs I think I have a com­par­a­tive ad­van­tage at roles which re­quire tech­ni­cal knowl­edge but not do­ing tech­ni­cal re­search day-to-day.

I’m em­pha­siz­ing strat­egy 1 be­cause I think that AI safety re­search needs re­ally great en­g­ineers at AI safety re­search orgs.

I worry very lit­tle about los­ing the op­por­tu­nity to get ex­ter­nal crit­i­cism from peo­ple who wouldn’t en­gage very deeply with our work if they did have ac­cess to it. I worry more about us do­ing worse re­search be­cause it’s harder for ex­tremely en­gaged out­siders to con­tribute to our work.

A few years ago, Holden had a great post where he wrote:

For nearly a decade now, we’ve been putting a huge amount of work into putting the de­tails of our rea­son­ing out in pub­lic, and yet I am hard-pressed to think of cases (es­pe­cially in more re­cent years) where a pub­lic com­ment from an un­ex­pected source raised novel im­por­tant con­sid­er­a­tions, lead­ing to a change in views. This isn’t be­cause no­body has raised novel im­por­tant con­sid­er­a­tions, and it cer­tainly isn’t be­cause we haven’t changed our views. Rather, it seems to be the case that we get a large amount of valuable and im­por­tant crit­i­cism from a rel­a­tively small num­ber of highly en­gaged, highly in­formed peo­ple. Such peo­ple tend to spend a lot of time read­ing, think­ing and writ­ing about rele­vant top­ics, to fol­low our work closely, and to have a great deal of con­text. They also tend to be peo­ple who form re­la­tion­ships of some sort with us be­yond pub­lic dis­course.

The feed­back and ques­tions we get from out­side of this set of peo­ple are of­ten rea­son­able but fa­mil­iar, seem­ingly un­rea­son­able, or difficult for us to make sense of. In many cases, it may be that we’re wrong and our ex­ter­nal crit­ics are right; our lack of learn­ing from these ex­ter­nal crit­ics may re­flect our own flaws, or difficul­ties in­her­ent to a situ­a­tion where peo­ple who have thought about a topic at length, form­ing their own in­tel­lec­tual frame­works and pre­sup­po­si­tions, try to learn from peo­ple who bring very differ­ent com­mu­ni­ca­tion styles and pre­sup­po­si­tions.

The dy­namic seems quite similar to that of academia: aca­demics tend to get very deep into their top­ics and in­tel­lec­tual frame­works, and it is quite un­usual for them to be moved by the ar­gu­ments of those un­fa­mil­iar with their field. I think it is some­times jus­tified and some­times un­jus­tified to be so un­moved by ar­gu­ments from out­siders.

Re­gard­less of the un­der­ly­ing rea­sons, we have put a lot of effort over a long pe­riod of time into pub­lic dis­course, and have reaped very lit­tle of this par­tic­u­lar kind of benefit (though we have reaped other benefits—more be­low). I’m aware that this claim may strike some as un­likely and/​or dis­ap­point­ing, but it is my lived ex­pe­rience, and I think at this point it would be hard to ar­gue that it is sim­ply ex­plained by a lack of effort or in­ter­est in pub­lic dis­course.

My sense is pretty similar to Holden’s, though we’ve put much less effort into ex­plain­ing our­selves pub­li­cly. When we’re think­ing about top­ics like de­ci­sion the­ory which have a whole aca­demic field, we seem to get very lit­tle out of in­ter­act­ing with the field. This might be be­cause we’re ac­tu­ally in­ter­ested in differ­ent ques­tions and aca­demic de­ci­sion the­ory doesn’t have much to offer us (eg see this Paul Chris­ti­ano quote and this com­ment).

I think that MIRI also em­piri­cally doesn’t change its strat­egy much as a re­sult of talk­ing to highly en­gaged peo­ple who have very differ­ent world views (eg Paul Chris­ti­ano), though in­di­vi­d­ual re­searchers (eg me) of­ten change their minds from talk­ing to these peo­ple. (Per­son­ally, I also change my mind from talk­ing to non-very-en­gaged peo­ple.)

Maybe talk­ing to out­siders doesn’t shift MIRI strat­egy be­cause we’re to­tally con­fused about how to think about all of this. But I’d be sur­prised if we figured this out soon given that we haven’t figured it so far. So I’m pretty will­ing to say “look, ei­ther MIRI’s onto some­thing or not; if we’re onto some­thing, we should go for it whole­heart­edly, and I don’t se­ri­ously think that we’re go­ing to up­date our be­liefs much from more pub­lic dis­course, so it doesn’t that seem costly to have our pub­lic dis­course be­come costlier”.

I guess I gen­er­ally don’t feel that con­vinced that ex­ter­nal crit­i­cism is very helpful for situ­a­tions like ours where there isn’t an es­tab­lished re­search com­mu­nity with taste that is rele­vant to our work. Physi­cists have had a lot of time to de­velop a rea­son­ably healthy re­search cul­ture where they no­tice what kinds of ar­gu­ments are wrong; I don’t think AI al­ign­ment has that re­source to draw one. And in cases where you don’t have an es­tab­lished base of knowl­edge about what kinds of ar­gu­ments are helpful (some­times peo­ple call this “be­ing in a preparadig­matic field”; I don’t know if that’s cor­rect us­age), I think it’s plau­si­ble that peo­ple with differ­ent in­tu­itions should do di­ver­gent work for a while and hope that even­tu­ally some of them make progress that’s per­sua­sive to the oth­ers.

By not en­gag­ing with crit­ics as much as we could, I think MIRI is prob­a­bly in­creas­ing the prob­a­bil­ity that we’re bark­ing com­pletely up the wrong tree. I just think that this gam­ble is worth tak­ing.

I’m more con­cerned about costs in­curred be­cause we’re more care­ful about shar­ing re­search with highly en­gaged out­siders who could help us with it. Eg Paul has made some sig­nifi­cant con­tri­bu­tions to MIRI’s re­search, and it’s a shame to have less ac­cess to his ideas about our prob­lems.

Off-the-cuff ob­jec­tion: this works inas­much as karma is a game to get a fancy high num­ber. But if you think of karma as the mea­sure of the site’s trust in a user, I don’t want the site to trust the user less be­cause they have elected to re­ward other users.

Sorry I missed this! I should have used the sub­scrip­tion fea­ture on the sub­scrip­tion an­nounce­ment post. Yes, (to my sur­prise, tbh) it is limited at 5. I’ll for­ward the ques­tion to the au­thor of the fea­ture.

in­creíble tra­bajo, muchísi­mas gra­cias por tanta ded­i­cación en es­tos artícu­los.

Amaz­ing work, thank you very much for your ded­i­ca­tion in these ar­ti­cles

Oh, sure. Are there any other differ­ences be­tween how link posts and reg­u­lar posts are treated?

Not sure which ex­pla­na­tion will work bet­ter so I’ve in­cluded both a prose de­scrip­tion of what I think is go­ing on and a model. Look at whichever seems most use­ful.

Words

This figure is then in­come ad­justed so that it is com­pa­rable with pre­sent day Amer­i­cans (it might ac­tu­ally down­ward ad­just the dam­ages of rich coun­tries which will be richer than pre­sent day US by the end of the cen­tury). Be­cause then it can be in­ter­preted as “this is what the In­dian gov­ern­ment should spend right now in USD to avert per tonne”, which seems to make most sense.

I think this is the crux of the dis­crep­ancy. I in­ter­pret the coun­try-level so­cial cost of car­bon as ex­press­ing the cost in terms of that coun­try’s cit­i­zens’ con­sump­tion. More con­cretely, if the so­cial cost of car­bon is $85/​tonne of CO2 in In­dia and ev­ery­one in In­dia emits 5 tonnes of CO2, ev­ery­one’s welfare in In­dia is as though their in­come is ac­tu­ally $425 (5 * 85) lower. I un­der­stand you to be say­ing that if ev­ery­one in In­dia emits 5 tonnes at $85/​tonne, it’s ac­tu­ally as though their in­come is re­duced by $12.5 (5 * 85 /​ 34 where 34 is the poverty mul­ti­plier us­ing an eta of 1.5 and an In­dian in­come of $2000). The lower num­ber comes be­cause you be­lieve the $85/​tonne to be quoted in terms of dol­lars for the Amer­i­can me­dian in­come and it needs to be ad­justed to ac­count for the fact that each dol­lar “means more” in lower-in­come In­dia. Is that de­scrip­tion in­tel­ligible? Is it an ac­cu­rate ac­count of your views?

Sup­pos­ing that’s right, I can now ex­plain why I be­lieve my ver­sion. I think the rele­vant parts of the pa­per and model are the dam­ages mod­ule and the dis­count­ing mod­ule.

I don’t think the sort of ad­just­ment-to-Amer­ica (i.e. “in­come ad­justed so that it is com­pa­rable with pre­sent day Amer­i­cans”) would hap­pen in the dam­ages mod­ule. In­deed, a quick skim of the dam­age func­tion refer­ence shows it to be talk­ing purely about macroe­co­nomic in­di­ca­tors. So the out­put of the dam­ages mod­ule very likely is in lo­cal terms (e.g. a 1% de­cline in In­dia’s GDP) rather than Amer­i­can terms.

Based just on the name, if this sort of ad­just­ment-to-Amer­ica were pre­sent, I’d ex­pect it to take place in the dis­count­ing mod­ule. But the Ricke pa­per doesn’t de­scribe any sort of ad­just­ment for dis­parate in­comes at time T=0. The in­come-rele­vant ad­just­ments they perform are an: (1) an op­tional rich-poor dam­age speci­fi­ca­tion which af­fects how dam­ages grow over time for coun­tries in each bin; (2) an op­tional elas­tic­ity of marginal util­ity ad­just­ment to ac­count for how the cost diminishes as economies grow ^[1]. So it doesn’t look like the dis­count­ing mod­ule does the re­quired ad­just­ment-to-Amer­ica ei­ther.

Model

I’ve also made a quick spread­sheet model try­ing to ex­plain what I think is go­ing on.

The first two tabs are just data—the coun­try level so­cial costs of car­bon and the GDP per cap­ita (me­dian in­come prob­a­bly would have been bet­ter but alas).

The next tab to look at is prob­a­bly “Hille­brandt Analogue”. It repli­cates your calcu­la­tion us­ing coun­try-level data. It ap­plies your uniform “Real­is­tic” poverty mul­ti­plier to each coun­try and then sums rather than sum­ming be­fore ap­ply­ing the poverty mul­ti­plier. Re­as­sur­ingly, we get ba­si­cally the same re­sult: a global so­cial cost of car­bon of $0.32.

The other two tabs show roughly how I think the mod­el­ing should be done.

“CSSC in Amer­i­can Dol­lars” uses Amer­i­can GDP per cap­ita as a refer­ence in­come to com­pute sep­a­rate poverty mul­ti­pli­ers for each coun­try based on its GDP per cap­ita. The mul­ti­plier is 1 for Amer­ica, less than 1 for coun­tries richer than Amer­ica and much greater than 1 for coun­tries much poorer than Amer­ica. Ap­ply­ing each coun­try’s poverty mul­ti­plier to its coun­try-level so­cial cost of car­bon and sum­ming gives us a global so­cial cost of car­bon $36,834. If 100% of the so­cial cost of a tonne of CO2 fell on an Amer­i­can mak­ing $62,641 (US GDP per cap­ita), this would be the welfare-equiv­a­lent of a $36,834 re­duc­tion in in­come.

“CSSC in GiveDirectly Dol­lars” fol­lows the same pro­ce­dure but uses GiveDirectly an­nual con­sump­tion ($180) as the refer­ence in­come. Thus the poverty mul­ti­plier is near 0 for most de­vel­oped coun­tries and is only over 0.5 for ex­tremely poor coun­tries like Bu­rundi. Ap­ply­ing each coun­try’s poverty mul­ti­plier to its coun­try-level so­cial cost of car­bon and sum­ming gives us a global so­cial cost of car­bon of $5.67. If 100% of the cost of a tonne of CO2 fell on a typ­i­cal GiveDirectly re­cip­i­ent, this would be the welfare-equiv­a­lent of a $5.67 re­duc­tion in in­come.

In­ter­est­ingly, this $5.67 figure makes cli­mate change in­ter­ven­tions within a fac­tor of 2 of GiveDirectly in terms of cost-effec­tive­ness (as­sum­ing $10 tonne/​CO2 averted).

I think it’s plau­si­ble that “solv­ing the al­ign­ment prob­lem” isn’t a very clear way of phras­ing the goal of tech­ni­cal AI safety re­search. Con­sider the ques­tion “will we solve the rocket al­ign­ment prob­lem be­fore we launch the first rocket to the moon”—to me the in­ter­est­ing ques­tion is whether the first rocket to the moon will in­deed get there. The prob­lem isn’t re­ally “solved” or “not solved”, the rocket just gets to the moon or not. And it’s not even ob­vi­ous whether the goal is to al­ign the first AGI; maybe the ques­tion is “what pro­por­tion of re­sources con­trol­led by AI sys­tems end up be­ing used for hu­man pur­poses”, where we care about a weighted pro­por­tion of AI sys­tems which are al­igned.

I am not sure whether I’d bet for or against the propo­si­tion that hu­mans will go ex­tinct for AGI-mis­al­ign­ment-re­lated-rea­sons within the next 100 years.