TagLast edit: 14 Jul 2022 17:51 UTC by

The academia tag is for posts relevant to topics such as how (in)efficient academia is, what influence academia has had and could have, the best ways to leverage or direct academia, and whether people should go into academic or academia-related careers.

Whittlestone, Jess (2018) Academic research, 80,000 Hours

# The aca­demic con­tri­bu­tion to AI safety seems large

30 Jul 2020 10:30 UTC
115 points

# More un­der­grad­u­ate or just-grad­u­ated stu­dents should con­sider get­ting jobs as re­search techs in aca­demic labs

1 Sep 2021 9:02 UTC
30 points

# Writ­ing about my job: Eco­nomics Professor

28 Jul 2021 19:41 UTC
86 points

# Writ­ing about my job: Re­search Fel­low, FHI

29 Jul 2021 13:53 UTC
96 points

# My first PhD year

31 Aug 2021 11:31 UTC
64 points

# EA megapro­jects continued

3 Dec 2021 10:33 UTC
175 points

# New EA Cause Area: Run Black­well’s Bookstore

5 Feb 2022 5:57 UTC
77 points

# Work­shop for un­der­rep­re­sented groups in GPR—In­tel­lec­tual and cul­tural di­ver­sity in GPR

29 Mar 2022 15:55 UTC
47 points

# Re­sults of a sur­vey of in­ter­na­tional de­vel­op­ment pro­fes­sors on EA

22 Jun 2022 15:49 UTC
175 points

# Why EA needs Oper­a­tions Re­search: the sci­ence of de­ci­sion making

21 Jul 2022 0:47 UTC
66 points

# How to PhD

28 Mar 2021 19:56 UTC
100 points

# Im­prov­ing the EA-al­igned re­search pipeline: Se­quence introduction

11 May 2021 17:57 UTC
60 points

# In­gre­di­ents for cre­at­ing dis­rup­tive re­search teams

16 May 2019 16:23 UTC
153 points

# In­ter­ven­tion op­tions for im­prov­ing the EA-al­igned re­search pipeline

28 May 2021 14:26 UTC
42 points

# Long-Term Fu­ture Fund: May 2021 grant recommendations

27 May 2021 6:44 UTC
110 points

# My cur­rent im­pres­sions on ca­reer choice for longtermists

4 Jun 2021 17:07 UTC
364 points

# Why sci­en­tific re­search is less effec­tive in pro­duc­ing value than it could be: a mapping

14 Jun 2021 8:24 UTC
107 points

# Eco­nomics PhD ap­pli­ca­tion sup­port—be­come a mentee!

8 Jul 2021 15:40 UTC
60 points

# Mak­ing im­pact researchful

19 Jul 2021 4:30 UTC
9 points

# Im­prov­ing EAs’ use of non-EA op­tions for re­search train­ing, cre­den­tials, test­ing fit, etc.

11 Sep 2021 13:52 UTC
22 points

# How to Ac­cel­er­ate Tech­nolog­i­cal Progress—New Things Un­der the Sun

3 Oct 2021 17:49 UTC
13 points
(www.newthingsunderthesun.com)

# [Question] How do EAs deal with hav­ing a “weird” ap­pear­ance?

7 Nov 2021 0:49 UTC
21 points

# New Effec­tive Th­e­sis ser­vices and op­por­tu­ni­ties to get involved

15 Nov 2021 18:15 UTC
48 points

# Sci-Hub sued in India

30 Dec 2021 2:48 UTC
107 points

# Hits-based de­vel­op­ment: fund­ing de­vel­op­ing-coun­try economists

1 Jan 2022 0:28 UTC
75 points

# New study on whether an­i­mal welfare re­forms af­fect wider at­ti­tudes to­wards animals

2 May 2022 10:00 UTC
52 points

# [Question] What are the coolest top­ics in AI safety, to a hope­lessly pure math­e­mat­i­cian?

7 May 2022 7:18 UTC
82 points

# How to ap­ply for a PhD

11 May 2022 19:18 UTC
70 points

# [Question] What’s the causal effect of a PhD?

5 Jun 2022 8:37 UTC
26 points

# Read­ing the ethi­cists 2: Hunt­ing for AI al­ign­ment papers

6 Jun 2022 15:53 UTC
9 points
(www.lesswrong.com)

# Break­ing Up Elite Colleges

7 Jun 2022 5:21 UTC
4 points
(sahajsharda.substack.com)

# Un­jour­nal: Call for par­ti­ci­pants and re­search

13 Jun 2022 11:11 UTC
40 points
(effective-giving-marketing.gitbook.io)

# NeurIPS ML Safety Work­shop 2022

26 Jul 2022 15:33 UTC
72 points
(neurips2022.mlsafety.org)

# The His­tory of AI Rights Research

27 Aug 2022 8:14 UTC
41 points
(www.sentienceinstitute.org)

# [Cause Ex­plo­ra­tion Prizes] Re­search Communication

30 Aug 2022 3:13 UTC
5 points

# Smart Move­ments Start Aca­demic Dis­ci­plines

25 Sep 2022 15:58 UTC
115 points

# Es­ti­ma­tion of prob­a­bil­ities to get tenure track in academia: baseline and pub­li­ca­tions dur­ing the PhD.

20 Sep 2020 18:32 UTC
39 points

# Nar­ra­tion: Im­prov­ing the EA-al­igned re­search pipeline: Se­quence introduction

27 Jul 2021 0:56 UTC
3 points
(anchor.fm)

# New Global Pri­ori­ties In­sti­tute work­ing pa­pers—and an up­dated ver­sion of “The case for strong longter­mism”

9 Aug 2021 16:57 UTC
46 points

# Self-Sus­tain­ing Fields Liter­a­ture Re­view: Tech­nol­ogy Fore­cast­ing, How Aca­demic Fields Emerge, and the Science of Science

6 Sep 2021 15:04 UTC
27 points

# In­ves­ti­gat­ing how tech­nol­ogy-fo­cused aca­demic fields be­come self-sustaining

6 Sep 2021 15:04 UTC
42 points

20 Sep 2021 14:54 UTC
18 points
(universalprior.substack.com)

# New Work­ing Paper Series of the Le­gal Pri­ori­ties Project

18 Oct 2021 10:30 UTC
60 points

# [Job – Academia] As­sis­tant Pro­fes­sor (Philos­o­phy/​Ethics) at Ling­nan Univer­sity (Hong Kong)

26 Oct 2021 9:25 UTC
5 points

# Ap­ply to be a Stan­ford HAI Ju­nior Fel­low (As­sis­tant Pro­fes­sor- Re­search) by Nov. 15, 2021

31 Oct 2021 2:21 UTC
15 points

# Pro­posal: al­ter­na­tive to tra­di­tional aca­demic jour­nals for EA-rele­vant re­search (multi-link post)

3 Nov 2021 20:16 UTC
28 points

# New Publi­ca­tion: Effec­tive Altru­ism and Reli­gion

25 Jan 2022 9:22 UTC
71 points

# [Question] What are some ex­am­ples of EA <-> Academia col­lab­o­ra­tions?

18 Feb 2022 10:18 UTC
21 points

# [Question] Do we have any *lists* of ‘aca­demics/​re­search groups rele­vant/​ad­ja­cent to EA’ … and open sci­ence?

30 Mar 2022 15:36 UTC
15 points

# [Question] What “pivotal” and use­ful re­search … would you like to see as­sessed? (Bounty for sug­ges­tions)

28 Apr 2022 15:49 UTC
34 points

# Should you do an eco­nomics PhD (or mas­ter’s)?

19 Apr 2022 20:04 UTC
56 points

# Paper sum­mary: The case for strong longter­mism (Hilary Greaves and William MacAskill)

29 Apr 2022 15:52 UTC
43 points
(globalprioritiesinstitute.org)

# Paper sum­mary: Mo­ral de­mands and the far fu­ture (An­dreas Mo­gensen)

29 Apr 2022 15:54 UTC
21 points
(globalprioritiesinstitute.org)

# BERI is seek­ing new col­lab­o­ra­tors (2022)

17 May 2022 17:31 UTC
21 points

# Apollo Aca­demic Surveys

20 May 2022 13:38 UTC
65 points

# Vael Gates: Risks from Ad­vanced AI (June 2022)

14 Jun 2022 0:49 UTC
45 points

# As­sis­tant pro­fes­sor­ship place­ments at top eco­nomics departments

8 Jul 2022 12:38 UTC
28 points

# Talk­ing EA to my philos­o­phy friends

14 Jul 2022 8:48 UTC
10 points

24 Jul 2022 20:20 UTC
47 points
(www.jefftk.com)

# A re­view of the Sch­warz­man Schol­ars pro­gram for EAs

15 Aug 2022 12:07 UTC
40 points

# EJOR spe­cial is­sue on role of op­er­a­tional re­search in fu­ture pandemics

17 Aug 2022 13:34 UTC
20 points
(www.sciencedirect.com)

# Hyper­bolic Science: Or, Play­ing Games with Growth

26 Aug 2022 2:41 UTC
2 points
(markov.bio)

# We’re search­ing for mean­ing, not hap­piness (et al.)

1 Sep 2022 7:34 UTC
17 points

# [Question] How can we se­cure more re­search po­si­tions at our uni­ver­si­ties for x-risk re­searchers?

6 Sep 2022 14:41 UTC
3 points

# What are the biggest EA Ques­tions in Amer­i­can Poli­tics?

11 Sep 2022 3:59 UTC
2 points

# In­put Sought For Roundtable with Peter Singer, Toby Ord, and Allan Dafoe

11 Sep 2022 22:13 UTC
23 points

# An­nounc­ing Ap­pli­ca­tions Now Open for Johns Hop­kins Cen­ter for Health Se­cu­rity’s PhD Opportunities

21 Sep 2022 14:27 UTC
30 points
• Vaidehi and Amber—very helpful and insightful post, with good suggestions.

Another obstacle is that busy academics in EA-adjacent fields face several career incentives against forum posts—especially if they’re tenure-track, teaching big courses, or running big lab groups.

Every hour we spend writing EA Forum posts or comments is an hour that we’re not writing a grant application, a journal paper, or a book. Those count for our tenure, promotion, and annual reviews. Forum posts don’t really count for anything in our academic jobs.

How to overcome this? I’m not sure, but it might be good to brainstorm about how to lay down a smoother path from EA Forum posts to academic journal articles , e.g. for academics writing posts to flag them with something explicit like ‘This is a rough draft of some ideas I might turn into a journal article for journal X or Y; I’d especially welcome feedback that helps with that goal’.

Another option is to develop a couple of online academic journals called ‘Effective Altruism’ or ‘Longtermism’ or ‘Existential Risk Review’ or whatever, which would basically publish polished, referenced, peer-reviewed versions of EA posts. The article selection criteria and review process could be quite streamlined, but to most academics, if it looks like a journal, and has a journal-style website and submission procedure, and it’s genuinely peer-reviewed to some reasonable degree, then it’s considered a legit journal. Also, the editors of such journals could keep their eyes on which EA Forum posts look interesting, upvoted, and much commented-upon, and could invite the writers of those posts to revise their post into a contribution to the journal.

Basically, if I write a 9,000 word post for EA Forum, I can’t list it on my academic CV, and it counts for absolutely nothing in an academic career. But if I publish exactly the same post as a peer-reviewed article in an EA journal, it counts for a lot.

The downside is that formal EA academic journals would be a departure from the usual EA ethos of very fast, effective, interactive, community-wide discussion, because traditional journals involve a huge amount of wasted time and effort (non-public reviews, slow review times, slow publication times, journals behind paywalls, little opportunity for visible feedback appended to the articles, etc). So we’d need to develop some new models for ‘peer-reviewed academic articles’ that combine the best of EA Forum communication style with the career-building credibility of traditional journal articles.

There are probably some other downsides to this suggestion, e.g. it would require some pretty smart and dedicated EAs to devote a fair amount of time to being journal editors and reviewers. However, we do get academic credit for doing those jobs! And it would not be very expensive to top up an aspiring academic’s pay with an editorship supplement. (I know lots of junior academics who would happily spend 30 hours of month editing a new journal if they could make an extra 30k a year doing so.) • 4 Oct 2022 0:27 UTC 2 points 0 ∶ 0 Globally, climate change “could force 216 million people… to move within their countries by 2050.” This seems like a remarkably small number to me. In 2019 around 7.4 million people moved state within the USA alone (source); over the next 28 years, with 0.4% annual population growth, that is 218 million people-moves. Spread out over the entire world this seems like a quite small amount of migration. • This is a fantastic forecast, thank you. I understand why this would be beyond the scope of this forecast, and maybe it’s simply too chaotic to forecast reasonably, but I think from the perspective of making a personal decision what I wish there were here is a forecast on the likely regional outcome of a nuclear weapon detonating on London (or any other major city). Something like a forecast, conditional of a nuclear weapon detonating on London, that an escaped Londoner stays alive >1 year somewhere not directly hit in the UK vs >1 year in New Zealand, given baseline but minimal disaster prep. Mostly I just bring this up because the cost of leaving the direct blast/​fallout zone of a major city by travelling an hour or two is significantly lower than the cost of say, moving a family back and forth from New Zealand each time there is an escalatory step taken. If the only reasonable escape option is a significant trip to the Southern Hemisphere then it changes the cost side of the calculus pretty drastically for a personal decision (vs. just getting out of the immediate danger radius). • I think the EA forum wiki should allow longer and more informative articles. I think that it would get 5x traffic. So I’ve created a market to bet. • I fail to be convinced. Many of your arguments seem like fully general arguments about why to worry about anything as a longtermists and thus wash out. For example, you argue Climate change would interfere with our ability to address extinction risks. States burdened by climate change could not expend efforts on preventing other catastrophic risks But there’s a great many things that fall under the category of impacting the ability of modern civilization to address arbitrary risks. For example, someone could just as easily argue that our failure to produce a communist revolution results in an inability to effectively coordinate on what really matters because states are burdened with excess profit motive. This is a bit of a caricature, sure, but I think it illustrates that you’ve made too general an argument in that I could sub in arbitrary things and someone will argue it as reasonable. I’d be much more interested in, for example, pointed evidence that climate change poses an extinction risk in-and-of itself and isn’t just another generic source of background risk that is very much not neglected. • I wish there was a way to do granular disagreement on this. While this is a clear case for forecasting and estimation, all issues boil down to how big quantities are. There is no way for me to comment on specific lines or specific numbers. And technically speaking I don’t think this can be too difficult since LessWrong has a docs-style editor. I suggest that inline comment threads are hidden by default and users can turn them on if they want to. Perhaps they get brighter the more upvoted they are. Secondly, I wish estimation was a first-class citizen of the forum. Imagine if for each of these models, users could add their own numbers and we could see a community median across all supplied values. For many articles, I think that would be really valuable/​game-changing. I reckon that it’s gonna happen by 2030 and I’m happy to bet to that effect • I am lower on use in Ukraine the next month (~4%) and much lower in the next year (6%). I guess I sense it’s just still a big norm we need reasons to push us away from this. What’s more, even if Putin does want to launch a nuke, which I guess I think is unlikely (10?%) it still needs to happen soon and the chain of command need to agree with him. And his lieutenants need to think that this is a situation that gains them something rather than thinking that NATO will support Ukraine with conventional weapons. What I find more compelling is that they might launch/​detonate a low-megaton nuke on the battlefield where they are losing. But again, that has to go off without a hitch, via some chain of command, it has to happen in the next month. And again, what does Russia gain from this, other than showing that their bluff wasn’t a bluff. I am happy to bet here. • Where can one find a PDF download of this handbook? That would be helpful for those who cannot always be online. • Thanks for doing this! In this squiggle you use “ableToEscapeBefore = 0.5”. Does that assume that you’re following the policy “escape if you see any tactical nuclear weapons being used in Ukraine”? (Which someone who’s currently on the fence about escaping London would presumably do.) If yes, I would have expected it to be higher than 50%. Do you think very rapid escalation is likely, or am I missing something else? • In this squiggle you use “ableToEscapeBefore = 0.5” I was just using 0.5 as a default value. In our March estimate, we were at 0.75, a critic was at 0.3; Zvi Moskovitz was at solomonic 0.5. This time this wasn’t really the focus of our estimate, because I was already giving forecasters many questions to estimate, and the situation for that sub-estimate doesn’t seem to have been changed as much. • Thanks for the links! (Fyi the first two points to the same page.) The critic’s 0.3 assumes that you’ll stay until there’s nuclear exchanges between Russia and NATO. Zvi was at 75% if you leave as soon as a conventional war between NATO and Russia starts. I’m not sure how to compare that situation with the current situation, where it seems more likely that the next escalatory step will be a nuke on a non-NATO target than conventional NATO-Russia warfare. But if you’re happy to leave as soon as either a nuke is dropped anywhere or conventional NATO/​Russia warfare breaks out, I’m inclined to aggregate those numbers to something closer to 75% than 50%. • On the other hand, the critic updated me towards higher numbers on p(nuke london|any nuke). Though I assume Samotsvety have already read it, so not sure how to take that into account. But given that uncertainty, given that that number only comes into play in confusing worlds where everyone’s models are broken, and given Samotsvety’s 5x higher unconditional number, I will update at least a bit in that direction. • 3 Oct 2022 21:53 UTC 3 points 0 ∶ 0 In the row for “Nuclear conflict scales beyond Ukraine in the next month after the initial nuclear weapon use” it says that the probability is 0.36%. I think that is a typo and should say 0.386%. • One thing I intuitively think could help a ton is an opt-out-only “First Post” (and also maybe “Second Post” and “Third Post”) tag placed on such posts. I think it’s both good and true that Forum users would/​do shift their qualitative feedback and voting in a more charitable direction if and when they knew that a particular post was a user’s first. But it currently takes a couple clicks to discern this information which I assume most people don’t actually go through upon reading every post by an author they don’t recognize. Of course charitable voting/​commenting could go too far, but that seems like a fixable problem To share a bit of my motivation for this idea, here’s my comment on “Your first posts will be cringe. It’s fine.” : Literally true for me, down to the physical motion of cringing (maybe more like a grimace lol), a minute ago when I stared into the abyss. Thanks to the 13 people who gave it a like to spare me some trauma at the time, though; in all seriousness, if it had gotten 3 karma instead of 20 I don’t think I would have tried again • Thank you! It’s such a pleasure to read high quality work in this forum. • Thanks, though your complement seems oddly double-sided against the rest of the forum’s content • That was not my intention. I just wanted to communicate the ideas that 1. this is high quality work 2. I appreciate it more when I read high quality work here, instead of somewhere else. Makes me happier about the community. • Oh man that is not a pretty graph… Same data, better resolution (and feel free to move this to the main post Vaidehi or Amber) • ## Thoughts and Notes: October 3rd 0002022 (1) I have been working on a post which introduces a framework for existential risks that I have not seen covered on the either LW or EAF, but I think I’ve impeded my progress by setting out to do more than I originally intended. Rather than simply introduce the framework and compare it to the Bostrom’s 2013 framework and the Wikipedia page on GCRs, I’ve tried to aggregate all global and existential catastrophes I could find under the “new” framework. Creating an anthology of global and existential catastrophes is something I would like to complete at some point, but doing so in the post I’ve written would be overkill and would not in line with the goal of”making the introduction of this little known framework brief and simple”. To make my life easier, I am going to remove the aggregated catastrophes section of my post. I will work incrementally (and somewhat informally) on accumulating links and notes for and thinking about each global and/​or existential catastrophe through shortform posts. Each shortform post in this vein will pertain to a single type of catastrophe. Of course, I may post other shortforms in between, but my goal generally is to cover the different global and existential risks one by one via shortform. As was the case in my original post, I include DALLE-2 art with each catastrophe, and the loose structure for each catastrophe is Risk, Links, Forecasts. Here is the first catastrophe in the list. Again note that I am not aiming for comprehensiveness here, but rather am trying to get the ball rolling for a more extensive review of the catastrophic or existential risks that I plan to complete at a later date. The forecasts were observed on October 3 0002022 and represent the community’s uniform median forecast. ### Use of Nuclear Weapons(Anthropogenic, Current, Preventable) • Time-poor long-time deep EA with imposter syndrome here with a forum post draft now probably over a year old. One point you didn’t hit on that I think strongly applies to me (and probably others like me) is just when I think I’ve struck upon some sort of insight or found a topic I want to dive into to write about… I find a lengthy well-written EA Forum post has already been written on the topic and not only that, that I agree with it and that there is nothing new I would add (or if there is it is better suited as a quick comment rather than a followup post). In other online communities I have at times found myself writing profusely because I find areas of disagreement , areas where I can add value, influence discourse etc. In EA… I’m just “another EA” that has roughly the same views and values as so many other EAs. I’m weird and interesting and insightful outside of EA. Within EA… Not so much. And what I find myself wanting to say so often has already been said. It actually seems to be a paradox of sorts. Due to being an EA “insider” I’m less likely to generate any unique valuable insight compared to someone who is more EA-adjacent or an EA “outsider.” • That’s a really interesting point—thanks for mentioning it! (Edited) i think there can be value to the community or individuals in people saying similar things than what has been said before, if some of the following are true (not all of these apply to your situation specifically): • You posting the idea later means that different people come across the idea and engage with it (see The forum is a newsfeed) • You personally learn from the process of writing the post. You can flag that you aren’t saying anything original, but I think the act of actually publishing something forces you to crystallize ideas (even if they’re wrong) and can help with improving reasoning transparency. I think it’s plausible if you actually wrote a post you could find something to add to the conversation (I like Holden’s post Learning by Writing which goes into this more) • Assumption: you don’t have to aim to have impact by writing on the Forum—you can use it for upskilling yourself. • You say the same thing, but in a different way—e.g. use different /​ better /​ more recent examples, or emphasize different points, say it more simply (and “explain like i’m 5” version of a concept). • You situate what they’re writing within the existing literature (e.g. “I broadly agree with post X, but am confused about concept Y”) or red team a critique that you disagree with I also think that these kind of insights might be interesting to write in a personal blog where people outside the community could read it—your insights could be valuable to people who haven’t heard them before. (Edited the first paragraph since to be more accurate to my belief about the claim) • It looks like Nir Eyal just joined the forum! Is there a way to invite him to the subforum? • 3 Oct 2022 19:30 UTC 22 points 1 ∶ 0 Thanks for this! It was really useful and will save 80,000 Hours a lot of time. • I recently made my first forum post and ran into some formatting barriers. I was able to overcome them with experimentation that cost 30-60 minutes of my time. For example, making a line completely bold automatically puts the text in the outline and putting asterixes (I put the * around this word in mobile and it just ended up italicizing it) centers it and puts it in the outline. I had some sections I wanted to bold, but leave out of the outline so I found a workaround by unbolding just the colon at the end. I was also a bit confused on how to use the link post option and had to search up examples to find out if I should be using it. That added another 10 minutes of time. It would be great if there was a video tutorial for using the interface. It could significantly reduce the barrier for first time posters. • What would have been really interesting is if someone wrote a piece critiquing the EA movement for showing little to no interest in scrutinizing the ethics and morality of Sam Bankman-Fried’s wealth. To put a fine point on it, has any of his wealth come from taking fees from the many scams, Ponzi schemes, securities fraud, money laundering, drug trafficking, etc. in the crypto markets? FTX has been affiliated with some shady actors (such as Binance), and seems to be buying up more of them (such as BlockFi, known for securities fraud). Why isn’t there more curiosity on the part of EA, and more transparency on the part of FTX? Maybe there’s a perfectly good explanation (and if so, I’ll certainly retract and apologize), but it seems like that explanation ought to be more widely known. • Someone has written something related: https://​​medium.com/​​@sven_rone/​​the-effective-altruism-movement-is-not-above-conflicts-of-interest-25f7125220a5 # Summary Sam Bankman-Fried, founder of the cryptocurrency exchange FTX, is a major donator to the Effective Altruism ecosystem and has pledged to eventually donate his entire fortune to causes aligned with Effective Altruism. By relying heavily on ultra-wealthy individuals like Sam Bankman-Fried for funding, the Effective Altruim community is incentivized to accept political stances and moral judgments based on their alignment with the interests of its wealthy donators, instead of relying on a careful and rational examination of the quality and merits of these ideas. Yet, the Effective Altruism community does not appear to recognize that this creates potential conflicts with its stated mission of doing the most good by adhering to high standards of rationality and critical thought. In practice, Sam Bankman-Fried has enjoyed highly-favourable coverage from 80,000 Hours, an important actor in the Effective Altruism ecosystem. Given his donations to Effective Altruism, 80,000 Hours is, almost by definition, in a conflict of interest when it comes to communicating about Sam Bankman-Fried and his professional activities. This raises obvious questions regarding the trustworthiness of 80,000 Hours’ coverage of Sam Bankman-Fried and of topics his interests are linked with (quantitative trading, cryptocurrency, the FTX firm…). In this post, I argue that the Effective Altruism movement has failed to identify and publicize its own potential conflicts of interests. This failure reflects poorly on the quality of the standards the Effective Altruism movement holds itself to. Therefore, I invite outsiders and Effective Altruists alike to keep a healthy level of skepticism in mind when examining areas of the discourse and action of the Effective Altruism community that are susceptible to be affected by incentives conflicting with its stated mission. These incentives are not just financial in nature, they can also be linked to influence, prestige, or even emerge from personal friendships or other social dynamics. The Effective Altruism movement is not above being influenced by such incentives, and it seems urgent that it acts to minimize conflicts of interest. • That’s a great start, and conflict of interest (or fear of annoying a major funder) probably explains the relative lack of interest in what could be a more concerning issue—to what extent is FTX itself on the up-and-up? For example, consider this bit from the origin story of FTX: In 2019, he took some of the profits from Alameda and8 million raised from a few smaller VC firms and launched FTX. He quickly sold a slice to Binance, the world’s biggest crypto exchange by volume, for about $70 million. So what was Binance doing in 2019? Generating headlines like this: “How crypto giant Binance became a hub for hackers, fraudsters and drug traffickers”: During this period, Binance processed transactions totalling at least$2.35 billion stemming from hacks, investment frauds and illegal drug sales, Reuters calculated from an examination of court records, statements by law enforcement and blockchain data, compiled for the news agency by two blockchain analysis firms. Two industry experts reviewed the calculation and agreed with the estimate.

Separately, crypto researcher Chainalysis, hired by U.S. government agencies to track illegal flows, concluded in a 2020 report that Binance received criminal funds totalling $770 million in 2019 alone, more than any other crypto exchange. Or consider FTX’s hiring of Daniel Friedberg as a chief compliance officer. This article claims that he had been involved in previous cheating/​fraud at other businesses: Crypto’s ongoing addiction to the Tether stablecoin is nearly as alarming as the sector’s questionable embrace of lawyers linked to online gambling fraud. . . . If one ever doubted the insincerity of SBF’s compliance commitment, . . . the company’s former GC, Daniel S. Friedberg, is now FTX’s new chief compliance officer, a role for which Friedberg is almost comically inappropriate. . . . Friedberg’s presence on FTX’s payroll means Sam Bankman-Fried (SBF) either didn’t do his due diligence before hiring, or he knew of Friedberg’s past sins and didn’t care. Neither of these options paints Sam Bankman-Fried in an overly flattering light.” Then there’s all the recent flurry of activity buying (or offering to bailout) other companies that are definitely shady: For example, in July, FTX signed a deal to buy BlockFi for up to$240 million, and to give it $400 million in revolving credit. BlockFi is most famous for having agreed to pay$100 million in penalties for its securities fraud. It’s not clear why FTX would want to spend this amount of money on buying a fraudulent firm.

Just last week, there was a story that FTX is thinking about buying Celsius, another fraudulent firm.

Another story from July had the remarkable claim that SBF is even thinking of putting his own cash into bailing out other crypto firms:

On one or two occasions, Bankman-Fried, who made billions arbitraging cryptocurrency prices in Asia beginning in 2017, said he has used his own cash to backstop failing crypto companies when it didn’t make sense for FTX to do so.

“FTX has shareholders and we have a duty to do reasonable things by them and I certainly feel more comfortable incinerating my own money,” he said.

Why is FTX and perhaps SBF himself putting so much money into buying up other people’s scams? I would hope it’s because they intend to reform the crypto industry and put it on more of a moral and legitimate footing, although that would reduce the crypto market size considerably.

It is just fascinating that for all the ethical scrutiny given to everything else about society, one’s personal life, and even the feelings of shrimp, hardly anyone betrays the least curiosity about any of this. It’s probably foolish and entirely against my self-interest to mention it as well.

But as it stands now, the crypto industry seems to me somewhat analogous to, say, Pornhub, Backpage, Silk Road, or any number of industries that have a very large component of criminal activity. If someone showed up from Pornhub bearing billions in grants to EA, I hope someone would at least express some moral qualms about how much of the money was due to this sort of thing. Maybe the money is legit, and maybe it’s much better for the world for it to be devoted to EA causes rather than being tied up in Pornhub (or crypto), but it seems like a conversation worth having, no?

That’s an unfair comparison, one might say. Fair point. But look at all the lives ruined by crypto scams—look at the people driven to suicidal thoughts and despair. If someone made money on those deals, it wasn’t entirely innocent. https://​​ez.substack.com/​​p/​​the-consequences-of-silence?sd=pf

If FTX is indeed different—which I very much hope! -- then they should be more transparent about exactly where all the money comes from, and what, if anything, they are doing to compensate victims of the many companies that are scams, particularly the companies that FTX is currently trying to take over.

• [ ]
[deleted]
• Thanks for sharing this! I’d never heard of this Tetsu Nakamura before. I’d love to hear more stories like this.

• We’re thinking of naming an office “Focal Point”—let me know what you think !

• Thanks for referencing Community Posts, which I wrote about here.

This would entail:

• Posts anyone could edit

• They could be on any EA post topic

Would you use this feature if it existed? Agree using ✅, disagree using ❌. Don’t worry about brutally x-voting me, I’ll be fine.

• Here’s a list of tail events which have significant political and economic implications, but you may not easily find models of the political or economic implications. (This is not to say the models don’t exist) All of these are greater than 1% probability imo.

• viable fusion energy

• viable quantum computing

• geopolitical conflicts or significant accidents in space (low earth orbit mostly)

• a mass pandemic atleast as infectious and lethal as covid

• notable advances in ability to engineer pandemics

• Russia /​ insert country X here detonating a nuke

• viable human gene editing, viable embryo selection + adoption in some countries at large scale

• cryptocurrency becoming bigger or more influential than gold

• notable advances in atomically precise manufacturing or nanotech

• <insert country X> unilaterally attempting solar geoengineering

• engineering of biomes at a scale significantly larger than gene editing of individual organisms (mosquito gene drives would be an example but many others exist)

What is different about EA is that it claims AGI is more likely, more soon and more impactful (morally) than most of the other moonshots. To someone who hasn’t studied them all, it’s easy to put all such things into the same reference class

• 3 Oct 2022 15:07 UTC
13 points
0 ∶ 0

If someone considers this line of work, I’d be keen to chat and outline some considerations. lennart[ät]heim.xyz

Just a heads up my latest estimate is here in footnote 15: https://​​www.effectivealtruism.org/​​articles/​​introduction-to-effective-altruism#fn-15

I went for 300 technical researchers though say the estimate seems more likely to be too high than too low, so seems like we’re pretty close.

(My old Twitter thread was off the top of head, and missing the last year of growth.)

Glad to see more thorough work on this question :)

• [Nuance]

While I buy the housing theory of everything and support more housing, I don’t think it is nearly one of the most pressing problems.

But I think it’s very cheap for us to occasionally say “while it’s not the most important thing, it would be good if we fixed it”.

I feel similarly about homelessness, sexism in politics, human rights abuses in small countries.

• What does he think about rowing versus steering in AI safety? Ie does he think we are basically going in the right direction and we just need to do more of it, or do we need to do more thinking about the direction in which we are heading?

• Note that talking to Tom, he would have included a PDF if we had a way to upload those.

• (h/​t Eva Vivalt)

• Thanks for this post! Future Fund has removed this project from our projects page in response.

• Here’s a relevant set of estimates from a couple of years ago, which has a guesstimate model you might enjoy. Your numbers seem to be roughly consistent with theirs. They were trying to make a broader argument that “1. EA safety is small, even relative to a single academic subfield. 2. There is overlap between capabilities and short-term safety work. 3. There is overlap between short-term safety work and long-term safety work. 4. So AI safety is less neglected than the opening quotes imply. 5. Also, on present trends, there’s a good chance that academia will do more safety over time, eventually dwarfing the contribution of EA.”

• Thanks for writing this post! I would love to see how elements of what is reported here could potentially improve CEA’s virtual introductory program and be more discussed among/​taught to community builders on a consistent basis. I look forward to digesting these topics more properly to see how it can benefit the work I do as university group supporter.

• For the sake of using the handbook for an intro fellowship—where is the “earlier estimate of your future income” located, if anywhere, in the handbook? Does it refer to some exercise or to a graph in one of the articles on economic inequality? And if there is no such thing, perhaps an alternative phrasing or including the estimate in the exercise would be great!

• I really liked this!

• “I also think that engagement on Twitter is still pretty underdeveloped and neglected (especially relative to the more nascent Progress Studies movement) as it seems like a lot of intellectuals frequent there and can be pretty moved by the content they see there regularly.”

Curious about this! You’re saying progress studies folks are more widely read? I think that’s true, though I think in part it’s because they slot more neatly into already-going-on political things, and I’m not sure we want to do that.

• Thanks, this was fun to read and highlighted several interesting posts I wouldn’t have otherwise found!

• Amplify is a great name.

• As a broader point, which Tom is brave to have reached out in this way, this is a terrible user flow. I wonder how many people want to work with the EA community but have no idea how to interact with our ecosystem.

• I’ve messaged two people about it. Feel free to nudge me on @nathanpmyoung if noone gets in touch. Thanks for reaching out, big fan of the TBI’s pragmatic approach to policy change.

• I might be biased because I had an idea for something very similar, but I think this is amazing and I think hit on something very, very interesting. I found the calibration training game very addictive (in a good way) and actually played it for for a few hours.

I think it might be because I play it in particular way though:

• I always set it to 90%.

• Then, I only put in orders of magnitudes, even when the prompt and mask doesn’t force the user to do this. So for instance, ‘What percent of the world’s population was killed by the 1918 flu pandemic?’ I put in: 90% Confidence Interval, Lower Bound: 1%, Upper Bound: 10%. This has two advantages:

1. I can play the game very quickly—I can do a rough BOTEC in my head.

2. I’m almost always accurate but not very precise but when I’m not, I’m literally orders of magnitude off and I get this huge prediction error signal—and that is very memorable (and I feel a bit dumb! :D). This might also guide people towards those parts of my model of the world, where I have biggest gaps in my knowledge (certain scientific subjects). ‘It’s better to be roughly right than precisely wrong’. I think you could implement a spaced repetition feature based on how many orders of magnitude you’re off, where the more OOMs you’re off, the earlier it prompts you with the same question again (so if you’re say >3 orders of magnitude off it prompts you within the same session, if you’re 2 orders of magnitude of within 24 hours, 1 within in 3 days (from Remnote)). You could preferentially prioritize displaying questions that people often get wrong, perhaps even personalize it using ML.

With that in mind, here are some feature suggestions:

1. You’re already pretty good at getting people to make rough orders of magnitude estimations, by often using scientific notation, but you could zero in on this aspect of the game.

• Add even higher confidence setting like 95% and 99%, and perhaps make that the default. This will get users to answer questions faster.

• Restrict the input to orders of magnitude or make that the default. It might also be good to select million, 10 million, 100M from a drop down menu, so that people gets faster and is more reinforcing.

• While I appreciate that I got more of an intuitive grasp of scientific notation playing the game (how many 0s does a trillion have again?), have the word ‘billion’ displayed when putting in the 10^12.

• When possible, try to contextualize where possible (I do this in this post on trillion dollar figures: ‘So how can you conceptualize $1 trillion? 1 trillion is 1,000 billion. 1 billion is 1,000 million. Houses often costs ~1 million. So 1 trillion ≈ 1 million houses—a whole city.’) • I like the timer feature, but perhaps consider either reducing the time per question even further or give more point if one answers faster. If you gamify this properly, I think this could be the next Sporcle (but much more useful better). • [ ] [deleted] • Ha, this is interesting. There might be something around “growing up in EA” where you get to a more grounded place after going through a pretty normal “explore with tons of energy” phase. Though relative to you I seem much more invested in EA and leaning on it to provide a lot for me. So far, so good! • I really liked the tone of this relative to posts on similar topics, felt more grounded and less panicked. (Maybe the others were right to be panicked, but this one resonated with me better). • Broadly, I agree. I think that social connections bringing benefit is a deeply human trait, but I think we can do better. My suggestions: • Acknowledge the situation. Yes, we are a community that works a lot on interpersonal networks and we lose information and talent from people who don’t have access to those. We gain the ability to move quickly often without grifters. What are the costs and benefits in lives saved? • Relentlessly build systems that scale. Personal connections are much higher fidelity than submission forms, people you vibe with are usually better employees than random people. But at scale these heuristics break down—there are ideas you’ll miss without submission forms and better candidates who you won’t vibe with. I am attempting to build systems in forecasting question generation and prize generation that don’t require people to know someone to get something done. Superlinear is the org that gets this best in my opinion • 3 Oct 2022 10:05 UTC 3 points 2 ∶ 0 Weak downvoted because I think the title is somewhat misleading given the content of the post (and I think you know this, since you replied below: ‘Perhaps I oversold the provocative title.’) I think we should discourage clickbait of this nature on the Forum. • I think my highschool might like this kind of thing. Any thoughts on whether this would work well in a non-English-speaking country? Highschoolers generally have good enough English here to engage with general texts, but obviously it’s harder for them and takes them longer to read, write etc. So basically the question is, how many hours of work do you think this requires from each student (if they were a native speaker)? If it’s not too high an amount, I’ll reach out to my school and pitch it as an opportunity for English class. • 3 Oct 2022 9:31 UTC 12 points 0 ∶ 0 For whatever it’s worth, it looks like Carrick himself has chosen to donate$2900 to the Salinas campaign, and to publicly announce it via his official Twitter account:

Today I donated the maximum amount, $2900, to #OR06′s @AndreaRSalinas. I earned less than$45k last year, so my money is where my mouth is when I say that I believe she will do an excellent job representing Oregonians in DC. [1/​2]

This is a tight race and we must win it not only to get Andrea into office but also to keep Congress blue. Please consider digging deep and donating to her campaign here: https://​​tinyurl.com/​​2p8m9nwh. And for those planning to help GOTV, I’m right here with you. [2/​​2]

• I suspect this is because there isn’t a globally credible/​legible consensus body generating or validating the forecasts, akin to the IPCC for climate forecasts that are made with even longer time horizons.

• Cool, I might be spending a few weeks in Belgrade sometime next year! I’ll reach out if that ends up happening. (Writing from Dubrovnik now, and I met up with some rationalists/​EAs in Zagreb ~1mo ago.)

• [ ]
[deleted]
• I would not recommend dyeing your hair at home. And if you want to dye your hair, this is generally a good idea, regardless of why this idea came to your head.

• 3 Oct 2022 6:54 UTC
4 points
0 ∶ 0

Strong upvote! Thank you for making these important distinctions.

Here is an add-on from effectiveness research.
Some studies show that approaches integrating different types of techniques might be more effective than approaches using just one type, such as just CBT or just inner parts work. This makes sense if it is true that different individuals at different times in different situations benefit most from different methods.
These are exemplary meta-analyses:

* Integrative approaches even can help with very difficult problems and cases: such as Dialectical Behavioral Therapy and schema therapy can help people with Borderline Personality tendencies (Cristiea et. al, 2017); Mindfulness-based approaches can help with treatment-resistant depression (Chen, et. al, 2021).
* Integrating different evidence-based methods might work a little better in coaching for improving wellbeing (Wang et. al, 2021).
* Transdiagnostic (mostly also integrative) treatment approaches such as ACT or mindfulness-based treatments might work a little better for reducing depression than disorder-specific CBT (Newby, 2015)

Personal note: What might limit the effectiveness of IFS is the lack of evidence-based structure offered for separating the parts in a meaningful and helpful way. Both the approaches explained in this article do this well, drawing from what we already know from science about how the human mind works.

• It will take me some time to digest the information in this post, and I’ve only skimmed through part of it so far, but I want to express my appreciation for providing such a resource-dense post.

• 3 Oct 2022 5:06 UTC
2 points
1 ∶ 0

Maybe hiring nonEAs for certain roles (like “communications assistant” and not like “board member”) could improve communications/​appearances/​maybe outreach?

• 3 Oct 2022 4:52 UTC
−11 points
1 ∶ 0

This post has been zapped off the forum (it’s downvoted so negative that few will see it):

I don’t like the post, but I think people should know it exists (for a few days or something), so I’m posting it here.

(Commenting in this post because, why not?)

https://​​forum.effectivealtruism.org/​​posts/​​ALMX8BbR5HfhwbPwp/​​

• (Commenting in this post because, why not?)

I’m not super familiar with forum norms personally, but a brief (~15 second) search suggests low relevancy might be a good candidate here. If you commented because you felt the post was relevant to the criticism/​red-teaming contest, one suggestion might be to include why you think it is relevant, since it’s hard to tell on skimming the post, and might be misinterpreted.

I’m not personally very excited about an ongoing discussion on this, but if you have more questions or thoughts feel free to DM me.

• On the vegan thing:

I’m not actively involved in EA but sometimes I read this forum and try to live a utilitarian lifestyle (or like to tell myself that at least). I hope to become mostly vegan at some point, but given the particular moment I am at in my career/​life, it strikes me as a terrible idea for me to try to be vegan right now. I’m working 100+ hours per week with virtually no social life. Eating and porn are basically the only fun things I do.

If I were to try to go vegan, it would take me a lot longer to eat meals because I’d have to force it down, and I would probably not get full and would end up being hungry and thus less productive throughout the day. I think I would also lose up mental energy and willpower by removing fun from my day and would be less productive. If I am productive now, I can eventually potentially make a big impact on various things including something like animals stuff or other things.

Is this just selfish rationalization? I don’t think so, though there is some of that.

I try to look for good veggie/​vegan dishes/​restaurants and have ~2/​3 of my meals vegan but the remainder just doesn’t seem even close to worth it right now. Since I have very little social contact and am not “important” yet, the signaling value is low.

I think it’s great that people have made being vegan work for them, but I don’t think it’s right for everyone at every time in their lives.

• I struggled a lot with it until I learned how to cook in that particular style (roughly: way more oil, MSG, nutritional yeast, two proteins in every recipe). Good luck!

• 3 Oct 2022 4:49 UTC
1 point
0 ∶ 0

Although these seem like positive signals, 50K total is a fairly marginal increase at the scale of big tech salaries, and the increase in new group members did not lead to a noticeable uptick in engagement or attendance for future events. This also all came at the cost of a fairly large time and effort investment from our organizers and external speakers. I agree with the assessment that given tech salaries/​average hourly rates vs total time, $50K (really$25K from new donors, who were hoping to influence the most, since us organizers could’ve just maxed the donation matching if it came to it) is a lower ROI given how many organizers put time into it.

But there’s the potential greater impact that’s harder to measure:

• All of the people donation were not part of the core organizers and were likely people recently introduced to EA, and influenced enough by the ideas to donate. It’s possible this leads to future repeated donations (one person has mentioned this intention so far), and engagement with EA

• The increased group membership could pay off in the future, though no noticeable improvements now. Having double the eyes on all the future promotions of EA content we post might pay off in terms of increased engagement or donations. Possibly it might already have—ie the rate of newly engaged faces doesn’t seem to have changed, but maybe the return to in-person events has made our exclusively virtual events less appealing, and without give month the new-person rate would’ve gone down

• Wow, opinion on this post went really negative since I last checked. I figured its karma would hang out around 0–15…

• 3 Oct 2022 4:29 UTC
2 points
0 ∶ 0

Many of these individuals took donation pledges, several helped us organize events and raise money, attended EAGs, and some considered or are taking career pivots.

Just wanna add more detail to this one.

• I’d say those that took the donation pledges, attended EAGs, and considered career pivots we’re already part of the core organizers. I’d classify them (us) as already predisposed to EA, which slightly reduces the counterfactual impact vs just laypeople being persuaded this strongly. Still mostly counterfactual, by the establishing of community and shared purpose at Meta

• The donations, however, were fully counterfactual imo. None of the core organizers donated to the drive since they were already going to donate and didn’t want to distort the results in measuring impact

• [ ]
[deleted]
• To support your point, Holden signal-boosted this in his aptitudes over paths post:

Basic profile: advancing into some high-leverage role in government (or some other institution such as the World Bank), from which you can help the larger institution make decisions that are good for the long-run future of the world.

Essentially any career that ends up in an influential position in some government (including executive, judicial, and legislative positions) could qualify here (though of course some are more likely to be relevant than others).

Examples:

Richard Danzig (former Secretary of the Navy, author of Technology Roulette); multiple people who are pursuing degrees in security studies at Georgetown and aiming for (or already heading into) government roles.

...

On track?

As a first pass, the answer to “How on track are you?” seems reasonably approximated by “How quickly and impressively is your career advancing, by the standards of the institution?” People with more experience (and advancement) at the institution will often be able to help you get a clear idea of how this is going (and I generally think it’s important to have good enough relationships with some such people to get honest input from them—this is an additional indicator for whether you’re “on track”).

• 3 Oct 2022 3:46 UTC
3 points
1 ∶ 0

Re optimism bias

Towards the top of the post I think you made a claim that EAs are often very optimistic (particularly agentic one’s doing ambitious things or in ‘elitist’ positions).

I just wanted to flag that this isn’t my impression of many EAs who I think are doing ambitious projects, I think a disproportionate number of agentic people I know in EA are pretty pessimistic in general.

I think the optimism thing and something like desire to try hard/​motivation/​ enthusiasm for projects are getting a bit confused here, but low confidence.

• 3 Oct 2022 3:13 UTC
38 points
7 ∶ 0

First, I want to say I did not donate to Carrick Flynn. Second, as someone who has been fairly involved in politics, primaries happen in every race every year and generally you shouldn’t expect much more than an endorsement from your opponents should you win, and even that isn’t guaranteed.

Third, telling another candidate’s supporters it’s their fault if your campaign is unsuccessful is pretty much the opposite of building a winning coalition. I think a much better strategy is to demonstrate commitment to the issue(s) those constituents care about so that they want to support you, if you truly believe their support is critical to your success.

• 3 Oct 2022 3:03 UTC
3 points
1 ∶ 0

Interesting post! Something I’ve thought about in a different direction is how EA seems to share the dedication to doing good of many religions but not the emphasis on a particular practices aimed at reducing greed/​negative traits and building discipline/​compassion/​positive traits/​emotional resources that help sustain such altruistic action. For example, various forms of meditation like loving-kindness and insight, reflection, etc.

• In a way I suppose EA does rather heavily emphasise against greed in a few directions (giving what we can, earning to give) but it certainly doesn’t emphasise many personal behaviours beyond (adjacently perhaps) rationalism.

I do wonder if, as you say, some of the strength of effective religious movements was an emphasis on personal behaviours the community should strive to include and praise. This was somewhat Tyler’s point about being Mormon—they have a bunch of behaviours that push the group towards long-term success (for instance, banning alcohol consumption from members and encouraging lots of children).

• Yeah, that’s a good point about giving. Giving itself can be a transformative action. I guess if I were to put it another way, EA shares many religions’ emphasis on works/​charity but not necessarily experiences and practices of ‘faith’/​transcendence.

In terms of the Mormon stuff, I think importing cultural habits is maybe different but adjacent to what I mean. Although maybe some aren’t that separate. It seems like abstaining from alcohol exists in a lot of different religions so maybe that particular behavior was found to be helpful in some way to personal transformation (I am assuming certain kinds of personal growth as a kind of distinct goal to be paired with charity/​good deeds in the world). And in terms of having kids, it seems like many religions distinguish between a “lay” path that may or may not emphasize having kids, and a more hardcore “monastic” (to use Buddhist terminology) path that emphasizes celibacy as a virtue to clear the mind and have more time to focus on transformation and service.

• 3 Oct 2022 2:49 UTC
3 points
0 ∶ 0

Something I think is really valuable is being upfront about mistakes and uncertainties. I really admire CEA’s mistakes page, for example. Cults often try to portray themselves and their leaders as infallible. Whereas admitting mistakes helps dispel that illusion and encourage critical thinking and a diversity of ideas.

• Late to the party here, but I was wondering why these organizations need aligned engineering talent. Anthropic seems like the kind of org that talented, non-aligned people would be interested in...

• The choice of readers of this forum to fund Flynn in the primary plus the massive influx of money from PACs guided by EA principles made it more likely that she will lose this general election...

Unfortunately, the contributions you made plus the many millions from the PAC funded by Bankman-Fried required Salinas supporters to dig deeply in their pockets to respond to a tsunami of ads, including untrue attacks.

I think giving persuasive detail about this claim would contribute to the goals of this ask, especially given how central this idea is to this post.

• What special actions did Salinas perform or have performed for her that resulted in depletion, in response to the Flynn candidate? For example, did she spend or deploy favors that were earmarked for the general?

• If Salinas has been depleted, can you explain or illustrate this with some anecdotal information (a few sentences/​paragraphs) that show why Salinas has been put in a worse place?

• Are you suggesting that election funding resembles a fixed pot of spending each cycle? Doesn’t the perceived closeness of elections and other specific factors strongly drive election spending?

In general, can you explain how people from the EA did harm and explain what EA’s duty of care is here as a result for actions?

• Let’s say that EA did not donate money and instead, Flynn did well because EAs got out the vote and worked hard for him. Salinas had to spend a lot of money as a result. Would EAs need to compensate the winning candidate Salinas then?

• If in the scenario above, you don’t think EAs need to compensate Salinas, then why would the use of money be a special case, compared to the other substantial resources used, and how fundamental money is to US political campaigns, including Salinas (exemplified by your post)?

• In all close primaries, are the losing candidate’s supporters expected to compensate the winner?

• Are you asking individual EAs who made personal donations, or are you asking SBF for money?

• If you’re asking EAs personally for money, can you explain/​show why these individual donations was abnormal, and if it was abnormal, why it was “bad”?

• As an aside, basically, no individual normal EA “wanted or approved” the 8 figure donation strategy.

• In the aftermath, isn’t it common to believe/​speculate that much of the SBF money was neutral or negative to the Flynn campaign?

• The other candidates orchestrated a major press event, uniting against the Flynn campaign

• The regional paper began a sustained series of hostile, critical takes on Flynn.

• Doesn’t it seem plausible that the absolute, net result of the SBF money, was a strong reaction that unified and concentrated support behind Salinas, who has a powerful narrative that embodies many traits (background, political career) that Flynn lacked in his narrative?

• There was at least one other well-funded candidate (millions of dollars of personal wealth) who literally complained that this crowded out his strategy.

• Didn’t the House PAC give $1M, comparable to the total of the EA donation amounts? Do they need to compensate for their behavior too? • I’m not sure I understand why this is the best donation target, even for people who want to donate specifically to a political race. For one, it seems all prediction markets and forecasters like FiveThirtyEight give R’s a ~75%+ chance of taking back the house, so this single race seems unlikely to be particularly impactful. What’s more, the Salinas Erickson race seems relatively safe for D’s and I’ve seen no mention of it anywhere being a tossup. This feels like something I’d get in a campaign email down to the closing line pulling on heartstrings and then immediately asking for a donation. • Interesting post, and some valid points. I would also add: cults tend to micro-manage the sexual relationships and reproductive strategies of their members. Sometimes this involves minimizing sexual activity, so cult members direct all of their energy and time into cult-propagation rather than mating effort. Sometimes it involves maximizing sexual connections or endogamous marriages within the cult, so people don’t feel any tension between their relationship commitments and their cult commitments. Sometimes cults are anti-natalist and strongly discourage reproduction in order to maximize energy and time directed into cultural cult-propagation (i.e. ‘horizontal cultural transmission’). Sometimes they’re pro-natalist and strongly encourage reproduction in order to create new recruits for the next generation (i.e. ‘vertical cultural transmission’). An implication is that the more ‘normal’ EA seems in terms of relationship formation (e.g. a nice mix of ‘cultural inbreeding’ within the group and outbreeding outside the group), and family formation (e.g. people having kids, but not crazy numbers of kids), the less cult-like we’ll seem. • Downvoted because • There’s little attempt to justify why Salinas is a great donation-target, much less one of the best available. • I believe supporting Salinas is quite clearly and robustly less valuable than alternatives. • There are better donation-targets for causing Democrats to keep Congress. • Salinas’s district is substantially more Democratic than the median, so it’s quite unlikely to be the tipping point. • There are more important, tractable, and neglected causes than causing Democrats to keep Congress. • Miscellaneous phrases (including “As one of the few new congressional districts in the nation” and “say that they value” and “Having tipped the balance the wrong way”) are unnecessarily misleading, rude, and anti-truth-seeking. • E.g. it’s almost totally irrelevant that this is a new congressional district, and the author presumably knows that. • Relatedly this is a one-sided pitch; I prefer posts like this to seek to inform rather than convince. • I believe we should think in terms of marginal effectiveness rather than offsetting particular harms we (individually or as a community) cause (see the author’s “you will have contributed in a small way to this failure” argument). If you want to offset harm that you have done or if you feel guilty, there’s little reason to do good in that particular domain (in this case, by donating to Salinas) rather than doing good in a more effective manner. (Good luck to Salinas.) • I think it’s worth engaging with Carol, the Salinas campaign, and more generally people who have been adversely affected by EA efforts. If EA wants win elections in party politics, it will require working together with people who run those parties. Narrowly speaking, you might think that they’re not focused on the most important issues or that you have better policy ideas, and you might be right. But the ability to build coalitions, working together despite disagreements to accomplish common goals, is a central challenge of party politics. I’m not convinced that EAs should donate to the Salinas campaign. FiveThirtyEight gives her a 78% chance of winning her race, meaning that closer races would offer a better chance for donations to tip the scales. Salinas also doesn’t list pandemic preparedness on her Issues page, which was the key issue of the Flynn campaign and I believe an important and neglected cause. But if the argument for the cost-effectiveness of donations to the Salinas campaign were to change, or if EAs found a more cost-effective way to offset possible harms of the Flynn campaign by continuing to engage with Oregonian or Democratic politics, I would consider supporting such an effort. More simply, EAs should be kind and understanding in our discussions with Carol and others affected by our work. Maybe they’re interested in the EA mindset, but they’re unsure how to interpret our actions. We should show them good examples of how we think. I believe we should think in terms of marginal effectiveness rather than offsetting particular harms we (individually or as a community) cause (see the author’s “you will have contributed in a small way to this failure” argument). If you want to offset harm that you have done or if you feel guilty, there’s little reason to do good in that particular domain (in this case, by donating to Salinas) rather than doing good in a more effective manner. I think many people would disagree, and I expect that they’ll interpret your unwillingness to offset direct harms as a moral failure and an inability to cooperate with others. There are some domains that call for ruthless cost-effectiveness, and others that call for building relationships and trust with people with whom you might not always agree. I think politics is the latter. • I believe we should think in terms of marginal effectiveness rather than offsetting particular harms we (individually or as a community) cause (see the author’s “you will have contributed in a small way to this failure” argument). If you want to offset harm that you have done, there’s little reason to do so by donating to Salinas rather than doing good in a more effective manner. I have no involvement in the Oregon race, but I disagree with this particular line of reasoning. Even setting aside traditional non-consequentialist arguments for compensating for harm (which I happen to believe in, and which I think are perfectly fine for EAs to act upon while still being EAs), this line of reasoning only works if one adopts causal decision theory. If we instead adopt functional decision theory, then there are much stronger reasons to consistently act as a harm-compensating agent. In particular, it can disincentivize harmful strategic behavior by others who try to influence you by simulating what might do in the future. If you cannot be simulated to harm some party without compensating them later, then you cannot be influenced to do so by others. It also enables co-operation with others who can now trust you will compensate them for harm (necessary even for everyday economic interactions). I think one could disagree as to whether FDT applies in this case (and also disagree with FDT in general), but I want to push back against the general argument that we should always be marginal thinkers, without consideration for the history of past events. (S/​O to particlemania for having first explained this argument to me. There’s also an argument to be made that conventional morality evolved FDT-like characteristics precisely to solve these strategic problems, but I won’t get into that here.) • I don’t buy that CDT vs FDT matters here? It’s seems like you’ll do better to always try to do what’s best (and appropriately take into account how actors may try to influence you) than focus on compensating for harm. And perfect altruists (at least) are able to cooperate without compensating one another’s harms. And it’s not like there’s potential cooperation with Salinas here—donating to her won’t affect her actions. And there are some cases where you should act differently if you thought you were being simulated, but those seem to be the exception for general harm-offsetting decisions. (I probably can’t continue a discussion on this now, sorry, but if there’s something explaining this argument in more detail I’d try to read it.) P.S. thinking in terms of contractualism, I think rational agents would prefer good-maximizing over harm-compensation policies, e.g. from behind a veil of ignorance. • I initially downvoted for many of the same reasons. And tbh, I still don’t like this post, as it does give off big “give us money please” vibes without really justifying some of its key claims. But ultimately I rescinded the downvote because it (sort of) raises a good point: IF people actually believe Flynn/​EA undermined Salinas enough to cost the Dems the election, it might really look bad for EA. This leads me to wonder if it isn’t worth just paying some to offset part of the (supposed) damages. Personally, I’m reluctant to caving to mud throwing and misrepresenting rhetoric used to extort money out of a good cause. Probably if Salinas came out genuinely in favor of pandemic prevention policies I’d probably be quite supportive of providing funds—but otherwise I’d be iffy on it. But it’s not my money… Update: Actually, I’m becoming much more pessimistic about funding Salinas unless she clearly supported pandemic preparedness/​prevention, because otherwise it would come across as more partisan (“we’re funding this Democrat because… we were told we hurt this Democrat…”). And appeals to political slogans like “short term business success instead of long term sustainability of our world” is honestly a bit intellectually insulting to me. • Cf. your update, I’d guess the second order case should rely on things being bad rather than looking bad. The second-order case in the OP looks pretty slim, and little better than the direct EV case: it is facially risible supporters of a losing candidate owe the winning candidate’s campaign reparations for having the temerity to compete against them in the primary. The tone of this attempt to garner donations by talking down to these potential donors as if they were naughty children who should be ashamed of themselves for their political activity also doesn’t help. I’d guess strenuous primary contests within a party does harm the winning candidate’s chances for the general (sort of like a much watered down version of third party candidates splitting the vote for D or R), but competitive primaries seem on balance neutral-to-good for political culture, thus competing in them when one has a fair chance of winning seems fair game. It seems the key potential ‘norm violation you owe us for’ is the significant out-of-state fundraising. If this was in some sense a ‘bug’ in the political system, taking advantage of it would give Salinas and her supporters a legitimate beef (and would defray the potential hypocrisy of supporters of Salinas attacking Flynn in the primary for this yet subsequently hoping to solicit the same to benefit Salinas for the general—the latter is sought to ‘balance off’ the former). This looks colorable but dubious by my lights: not least, nationwide efforts for both parties typically funnel masses of out-of-state support to candidates in particular election races, and a principled distinction between the two isn’t apparent to me. • a principled distinction between the two isn’t apparent to me. Indeed, there was an explicit tit-for-tat donation of out-of-state money to Salinas intended to offset donations to Flynn: Bold PAC, the Congressional Hispanic Caucus (CHC) campaign arm, is preparing the seven-figure independent expenditure in favor of its endorsed candidate, state Rep. Andrea Salinas (D), in the race for Oregon’s newly created 6th District. … The new investment is a tit-for-tat blow against the Democratic leadership PAC, House Majority PAC (HMP), which earlier this month pledged a similar investment to prop up political newcomer Carrick Flynn in that race. • Maybe it’s just a matter of degree but the Protect our Future PAC spent unprecedented levels on Carrick’s campaign, and, maybe this more of a principled distinguishing feature, they seem to have spent$1.75M on attack ads against Salinas, which maybe biggest ‘within party’ attack ad budget in a primary. Seems understandable this can be seen as a norm violation (attack ads are more sticky) and perhaps it’s poor ‘cooperation with other value systems’.

• Yeah, the language in your comment really resonates with me/​my emotions and also gives me a more negative view of the OP, yet I am worried about being overly influenced by 1) the quality of the OP (relative to the legitimacy of the underlying points), and 2) my emotions on this.

Ultimately, I think the second order effects still dominate and warrant someone somewhere giving this request a good think-through separate from emotional reactions:

1. Does not providing some funds to Salinas hurt the chances of future EA candidates or advocacy (especially those who might run for or target the Democratic Party) due to Dem opposition/​bitterness (regardless of how legitimate such feelings may be)?

2. Does providing funds to Salinas hurt the chances of future EA candidates or advocacy (especially those who who might run for or target the Republican Party) due to Republicans portraying EA as a “fund blue no matter who” movement (regardless of how legitimate such a label may be)?

• we tech/​ea/​ai people are overly biased in the actual relevance of our own field (I’m CS student)?

You can just as easily say that global institutions are biased about the relevance of their own fields, and I think that is a good enough explanation: Traditional elite fields (press, actors, lawyers, autocrats) don’t teach AI, and so can’t influence the development of AGI. To perform the feats of confidence that gains or defends career capital in those fields, or to win the agreement and flattery of their peers, they have to avoid acknowledging that AGI is important, because if it’s important, then none of them are important.

But, I think this dam will start to break, generally. Economists know better than the other specializations, they have the background in decision theory to know what superintelligence will mean, and they see what’s happening in industry. Military is also capable of sometimes recognizing and responding to emerging risks. They’re going to start to speak up, and then maybe the rest of the elite will have to face it.

• This post reads a little like someone pushing at an open door to me. So you write that FTX should ask themselves whether humanity should create AGI. The feeling I get from that is that you think FTX assume that AGI will be good. But the reason they’ve announced the contest is that they think the development of AGI carries a serious risk of global catastrophe.

Two of the propositions focus on when AGI will arrive. This makes it seem like AGI is a natural event, like an asteroid strike or earthquake. But AGI is something we will create, if we create it.

There are immense (economical, other) incentives to build AGI, so while humanity can simply choose not to build AGI, FTX (or any other single actor) is not in a position to choose not to build AGI. I expect FTX is open to considering interventions aimed at making that happen (not least as there’s been some discussion on whether to try to slow down AI progress recently). But whether those work at all is not obvious.

How would know we had successful AGI if/​when we created it? It would nothing like human intelligence, which is shaped not only by information processing, but by embodiment and the emotions central to human existence. … So AGI cannot be like human intelligence.

As far as I’m aware, writers on AGI risk have been clear from the beginning that there’s no reason to expect an AGI to take the same form as a human mind (unless it’s the result of whole-brain emulation). E.g. Bostrom roughly defines superintelligence as “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest” (Superintelligence ch. 2). There’s no reason to think a highly capable but alien-to-us intelligence poses less of a threat than one that’s similar to us.

AGI might be helpful for thinking about complex human problems, but it is doubtful that it would be better than task specific AI. Task specific AI has already proven successful at useful, difficult jobs (such as cancer screening for tissue samples and hypothesizing protein folding structures). Part of what has enabled such successful applications is the task specificity. That allows for clear success/​fail training and ongoing evaluation measures.

There are advantages to generality too, like reducing the need for task-specific data. There’s at least one example of a general intelligence being extremely successful, and that is our own, as evidenced by the last few billions’ years of evolutionary history. An example of fairly successful general-ish AI is GPT-3, which was just trained on next-word prediction but ended up being capable of everything from translation and spell-checking to creative writing and chess-playing.

• Let me then be more specific. Take Bostrom’s definition. What are all the cognitive tasks in all the domains of human interest? I think this is super vague and ill-defined. We can train up AI on specific tasks we want to accomplish (and have ways of training AI to do so, because success or failure can be made clear). But there is no “all cognitive tasks in the domains of human interest” training because we have no such list, and for some crucial tasks (e.g. ethics) we cannot even define success clearly.

GPT-3 is impressive for writing, other AI is impressive for image production, but such systems also produce amusingly wrong outputs at times. What is more impressive is useful AI for tasks like the protein folding success.

Our success has been evolutionarily framed (survival, spread of species), and tested against a punishing world. But AGI will not be embodied. So what counts as success or failure for non-task specific AI? Back to Bostrom’s definition, we have no such set of cognitive tasks defined or delineated.

So what are the incentives for such a creation? You say they are immense. I want to press on the idea that they are substantial.

• Thanks for responding! I think I now understand better what you’re getting at, though I’m still a bit unsure about how much work each of these beliefs are doing:

1. We shouldn’t build AGI.

2. We can’t build AGI (because there’s no coherent reward function we can give it, since many of the tasks it’d have to do have fuzzy success criteria).

3. We won’t build AGI (because the incentives mean narrow AI will be far more useful).

Could you clarify whether you agree with these and how important you think each point is? Or is it something else entirely that’s key?

• I think we could try to build AGI, but I am skeptical it could be anything useful or helpful (a broad alignment problem) because of vague or inapt success criteria, and because of the lack of embodiment of AGI (so it won’t get beat up on by the world generally or have emotional/​affective learning). Because of these problems, I think we shouldn’t try (1).

Further, I am trying this line of argument out to see if it will encourage (3) (not building AGI), because these concerns cast doubt on the value of AGI to us (and thus the incentives to build it).

This takes on additional potency if we embrace the shift to thinking about “should” and not just “can” in scientific and technological development generally. So that brings us to the questions I think we should be asking, which is how to encourage a properly responsible approach to AI, rather than shifting credences on the Future Funds’ propositions about.

Does that make sense?

• and because of the lack of embodiment of AGI (so it won’t get beat up on by the world generally or have emotional/​affective learning).

There are two ways to plausibly embody AGI.

1. as supervisors of dumb robot bodies, the AGI remotely controls a robot body, processing a portion or all of the robot’s sensor data.

2. as host of an AGI, the AGI’s hardware is resident in the robot body.

• I think it’s 1. Plenty of ideas are reasonable but not mainstream. E.g. the idea to not attack Ukraine in Russia. The experts probably don’t get into the technical arguments and dismiss AGI as hype.

• My apologies but I had to strong downvote because this is the sort of content that I want to stay far away from the forum. I would have maybe given a weak downvote or maybe even none if:

• it was nonpartisan, nonpolarized, and neutral

• there was a transcript of the video (I watched only a couple minutes, too long)

• there was a specific theory of change with an expected value calculation for a given amount of resources to improve a specific problem

• it compared this to other possible uses of those resources within the same or different cause areas

(Here is an example of a post from today that seems somewhat more neutral and specific, though still not as mechanistic as I’d like, but I only skimmed it: https://​​forum.effectivealtruism.org/​​posts/​​FtHhC7CfN4r5xD3Lm/​​easy-fixing-voting)

• Hey Dony, looking back at it, I can see why this appears partisan even though that was not my intention. My post was not meant to demonize my compatriots who happen to vote Republican or identify as conservative. It was meant to educate and bring awareness to an EA cause area.

I definitely could’ve used a better term than “Donald Trump supporters” in my first paragraph as not all of his supporters want to commit election fraud.

However, facts are facts. Regardless of someone’s politics, acknowledging the undermining of liberal democracy is necessary, especially when there are such detrimental consequences.

As I said, protecting liberal democracy isn’t one of my main interests or expertise so I don’t have the background to do a specific theory of change with an expected value calculation.

I appreciate the feedback and will learn from this post.

• Jeffrey—I think getting involved in this sort of partisan political issue is extremely dangerous for the EA movement, in terms of our movement-building, public relations, outreach, and image.

I thought our remit in EA was to focus on large-scale, tractable, neglected problems.

Partisan political squabbles between the US Democratic and Republican parties are moderate in scale, but they’re very far from tractable or neglected.

• Hey Geoffrey, looking back at it, I can see why this appears partisan even though that was not my intention. My post was not meant to demonize my compatriots who happen to vote Republican or identify as conservative. It was meant to educate and bring awareness to an EA cause area, protecting liberal democracy.

According to the 80k hyperlink I included in my post, it is a pressing issue. This is a large-scale issue as the United States is arguably the most influential geopolitical power in the world and has cultural and political influence around the globe on ensuring or protecting liberal democracies. (I’m not taking a side on whether US intervention in bringing liberal democracies is a net good or net bad, but history and the present (Ukraine) have shown how impactful U.S. involvement can be.)

I definitely could’ve used a better term than “Donald Trump supporters” in my first paragraph as not all of his supporters want to commit election fraud.

However, facts are facts. Regardless of someone’s politics, acknowledging the undermining of liberal democracy is necessary, especially when there are such detrimental consequences.

I appreciate your constructive criticism and will take it into account in future posts and edit this post to make it seem less partisan.

1. This is not necessarily between Democrats and Republicans—there’s a split within the Republican party too.

2. “Dangerous for EA” is a consideration, but a bad enough threat for democracy, assuming this is one, can still be a stronger consideration.

I’m not educated enough on American politics to know if the assumption is correct. But on a global scale, it certainly looks like democracy is backsliding (and has been for decades). It’s true locally here in Israel, it’s true in several European countries, and it looks like it’s true on average in the world in general, going by Freedom House democracy scores.

The current US political situation is very complex, volatile, emotional, and polarized. There are very low levels of trust in public schools, newspapers, congress, TV news, the presidency, big tech companies, etc.

From the Left’s point of view, Trump supporters are a major threat to democracy.

From the Right’s point of view, the liberal media/​academia/​big tech hegemony is a major threat to democracy.

For EA to wade into the most polarized and volatile American political landscape I’ve ever seen, seems foolish—and there’s no way we could get involved without upsetting tens of millions of highly passionate voters.

• I felt so strongly that this deserved another upvote that I was moved to create myself a forum account. To the extent that this work is really original I would actually encourage you to consider working it up into a journal article. As long as that isn’t going to take all the fun out of it.

• A lot of good potential answers are discussed in this earlier post’s comments. My favored explanation is because AGI is a minority concern even within CS academia, so we shouldn’t expect it to have much impact outside.

• 2 Oct 2022 20:46 UTC
4 points
0 ∶ 0

I like this post. It also seems to me that your point is really about community, not religion. The good things about the Quakers you mention don’t seem to be fundamentally about God or mysticism or something. What do you think?

• Yes, I’m not advocating for a belief in God, but perhaps for practices that arose due to a particular belief in God. For instance, Quaker style meetings might have a bunch of unknown positive effects that we just can’t reason or realise a priori. It’s worth looking at their practices and seeing which ones we consider might be useful for the community (or parts of the community) to adopt

• For the same reason that e.g. net electricity generation from fusion power is not the “number one single factor debated in every single argument on any economic/​political topic with medium-length scope”: Until it exists, it is fictional – why should everyone focus so much on fictional technology? It remains a narrow, academic field. The difference is that there is actual progress towards fusion.

• 2 Oct 2022 20:07 UTC
5 points
0 ∶ 0

Nice to see new people in the Balkans! I’d be down to chat sometime about how EA Croatia started off :)

• I’d find this post much more valuable if it argued that some parts of the EA community were bad, rather than arguing that they’re cultish. Cultish is an imperfect proxy for badness. Sure, cults are bad and something being a thing which cults do is weak evidence of its badness (see Reversed Stupidity Is Not Intelligence). Is, say, advertising EA too aggressively bad? Probably! But it is bad for specific reasons, not because it is also a thing cults do.

A particular way cultishness could be bad, which would make it directly bad for EA to be cultish, is if cults are an attractor in the space of organizations. This would mean that organizations with some properties of cults would feel pressure to gain more and more properties of cults. Still, I currently don’t think is the case, and so I think direct criticisms are much more valuable than insinuations of cultishness.

--------

# TL;DR

• Keep and use the word belief in your vocabulary distinct from your use of hypothesis or conclusion.

• Unweighted beliefs are useful distinctions to employ in your life.

• Constrain beliefs rather than decrease your epistemic confidence in them.

• Challenge beliefs to engage with underlying (true) knowledge.

• Take additional actions to ensure preconditions for the consequences of your actions.

• Match ontologies with real-world information to create knowledge used to make predictions.

• Use an algebraic scoring of altruism and selfishness of actions to develop some simple thought experiments.

• My red team critique is that EA folks can effectively rely on unweighted beliefs where they now prefer to use Bayesian probabilities.

# Introduction: Effective Altruists want to be altruistic

EA community members want to improve the welfare of others through allocation of earnings or work toward charitable causes. Effective Altruists just want to do good better.

There’s a couple ways to know that your actions are actually altruistic:

• Believe that your actions are altruistic.

• Confirm your actions are altruistic before you believe it.

I believe that EA folks confirm the consequences of their altruistic actions in the near term. For example, they might rely on:

• expert opinion

• careful research

• financial audits

• cost-effectiveness models

• big-picture analyses

and plausibly other processes to ensure that their charities do what they claim.

# The essentials of my red team critique

## Bayesian subjective probabilities don’t substitute for unweighted beliefs

The community relies on Bayesian subjective probabilities when I would simply rely on unweighted beliefs. Unweighted beliefs let you represent assertions that you consider true.

Why can’t Bayesian probabilities substitute for all statements of belief? Because:

• Humans do not consciously access all their beliefs or reliably recall formative evidence for every belief that they can articulate. I consider this self-evident.

• Beliefs are important to human evaluation of morality, especially felt beliefs. I will explore this in a couple of thought experiments about EA guilt.

• Beliefs can represent acceptance of a conclusion or validation of a hypothesis, but sometimes they appear as seemingly irrational feelings or intuitions.

• Any strong assertion that you make quickly is a belief. For example, as a medic, you could say “Look, this epipen will help!” to someone guarding their anaphylactic friend from you.

## Distinguish beliefs from conclusions from hypotheses

Use the concept of an unweighted belief in discussion of a person’s knowledge. Distinguish beliefs from conclusions from hypotheses, like:

• Belief Qualifiers: “I believe that X.”, “X, or so I believe.”, “X.”

• Conclusion Qualifiers: “Therefore X.”, “I conclude that X.”, “X is my conclusion.”

• Hypothesis Qualifiers: “I hypothesize that X.”, “X, with 99% confidence.”

I don’t mean to limit your options. These are just examples.

## Challenge the ontological knowledge implicit in a belief

If I say, “I believe that buying bednets is not effective in preventing malaria.” you can respond:

• “Why do you believe that?”

• “What led you to that conclusion?”

• “Based on what evidence?”

I believe that you should elicit ontology and knowledge from whom you challenge, rather than letting the discussion devolve into exchanges of probability estimates.

### Partial list of relationship types in ontologies

You will want information about entities that participate in relationships like:

• part-whole relationships: an entity is part of another entity

• meaning relationships: a defining, entailment, or referent relationship

• causal relationships: a causal relationship, necessary, sufficient, or both

• set-subset relationships: a relationship between sets

I will focus on causal relationships in a few examples below. I gloss over the other types of relationships though they deserve a full treatment.

### A quick introduction to ontologies and knowledge (graphs)

An ontology is a list of labels for things that could exist and some relationships between them. A knowledge graph instantiates the labels of the relationship. The term knowledge graph is confusing, so I will use the word knowledge as a shorthand. For example:

• Ontology: Tipping_Points cause Bad_Happenings, where Tipping_Points and Bad_Happenings are the labels, and the relationship is causes.

• Knowledge: “Melting Arctic Ice causes Ocean Heating At the North Pole.” where Melting Arctic Ice instantiates Tipping_Points and Ocean Heating At The North Pole instantiates Bad_Happenings.

I am discussing beliefs about the possible existence of things or possible causes of events in this red team. So do EA folks. Longtermists in Effective Altruism believe in making people happy and making happy people.”iii Separate from possibilities or costs of longtermist futures is the concept of moral circles, meant to contain beings with moral status. As you widen your moral circle, you include more beings of more types in your moral calculations. These beings occupy a place in an ontology of beings said to exist at some point in present or future time[1]. Longtermists believe that future people have moral status. An open question is whether they actually believe that those future people will necessarily exist. In order for me to instantiate knowledge of that type of person, I have to know that they will exist.

### A quick introduction to cause-effect pathways

A necessary cause is required for an effect to occur, while a sufficient cause is sufficient, but not necessary, for an effect to occur. There are also necessary and sufficient causes, a special category.

When discussing altruism, I will consider actions taken, consequences achieved, and the altruistic value[2] of the consequences. Lets treat actions as causes, and consequences as effects.

In addition, we can add to our model preconditions, a self-explanatory concept. For example, if your action has additional preconditions for its performance that are necessary in order for the action to cause a specific consequence, then those preconditions are necessary causes contributing to the consequence.

# Using ontologies to make predictions

“Excuse me. Zero! Zero! There is a zero percent chance that your subprime losses will stop at five percent.” -The character Mark Baum in The Big Short

I will offer my belief that real-world prediction is not done with probabilities or likelihoods. Prediction outside of games of chance is done by matching an ontology to real-world knowledge of events. The events, whether actual or hypothetical, contain a past and a future, and the prediction is that portion of the cause-effect pathways and knowledge about entities that exist in the future.

## Some doubts about subjective probability estimates

I doubt the legitimacy and meaningfulness of most subjective probability estimates. My doubts include:

• betting is not a model of neutral interest in outcomes. Supposedly betting money on predictions motivates interest in accuracy, but I think it motivates interest in betting.

• you can convey relative importance through probabilities just as easily as you can convey relative likelihoods. I think that’s the consequence or even the intent behind subjective probability estimates in many cases.

• subjective probabilities might be used with an expected value calculation. I believe that humans have no reliable sense of judgement for relative subjective probabilities, and that expected value calculations that rely on any level of precision will yield expected values without any real significance.

To reframe the general idea behind matching ontologies to events, I will use Julia Galef’s well-known model of scouts and soldiers[3]:

• scouts: collect knowledge and refine their ontology as they explore their world.

• soldiers: adopt an (arbitrary) ontology and assert its match to the real-world.

## An example of a climate prediction using belief filters and a small ontology

This example is pieces of internal dialog interspersed with pseudo-code. The dialog describes beliefs and questions. The example shows the build-up of knowledge by application of an ontology to research information. I gloss over all relationships except causal ones, relying on the meanings of ontology labels rather than making relationships like part-whole or meaning explicit.

The ontology content is mostly in the names of variables and what they can contain as values while the causal pathways are created by causal links. A variable looks like x? and → means causes. So x?y? means some entity x causes some entity y.

Internal dialog while researching:Climate change research keeps bringing tipping point timing closer to the present as the required GAST increases for tipping of natural systems drop in value. Climate tipping points are big and cause big bad consequences. So are there climate system tipping points happening now? Yes, the ice of the arctic melts a lot in the summer.

Knowledge instantiation:

• small_GAST_increase? → Tipping_Points?

• 1.2C GAST Increase → Ice-free Arctic

Internal dialog while researching:Discussions about tipping points could be filtered by media or government. So is there near term-catastrophic threat from tipping points? No, but the arctic melt is a positive feedback for other tipping points.

Knowledge instantiation:

• Tipping_points? → catastrophe?

• Ice-free Arctic → catastrophe?

• Ice-free Arctic → permafrost melt

• Permafrost melt → catastrophe?

• Permafrost melt → uninhabitable land and 1+ trillion tons carbon emitted

• Ice-free Arctic → methane hydrate/​thermogenic methane emissions

• methane hydrate/​thermogenic methane emissions → catastrophe?

• methane hydrate/​thermogenic methane emissions → 1 degree Celsius GAST rise

• Ice-free Arctic → accelerating Greenland melt

• accelerating Greenland melt → catastrophe?

• accelerating Greenland melt → <7m sea level rise possible

• sea level rise → catastrophe?

• Sea level rise → estimate:1+ meters sea level rise destabilizes coastal populations, 2+ meters wipes out some countries

Internal dialog while researching:Is the threat from tipping points immediate? Well, researchers are surprised by acceleration of changes at tipping points. Right now fast and surprising threats depend on the movements of the meandering jet stream. Is the jet stream a tipping point? I’ll call it a climate phenomenon.

Knowledge instantiation:

• Tipping_points? → immediate_threat?

• Climate_phenomenon? → immediate threat?

• Jet stream → severe heat waves and cold snaps

Internal dialog while researching:Is stated policy matching the situation severity? No. Climate scientists discussing the issues are frustrated or bitter. It’s clear global commitments are inadequate and either way ignored.

Knowledge instantiation:

• Situation? → Good_Policy?

• Severe situation → mediocre policy response

• Paris Agreement → Paris Agreement ignored

Internal dialog while researching: Policy change around the environment, population, and resource use has not progressed much. Are there other pressures on systems than direct climate change pressure? Yes. There is plastic pollution killing marine life; a 6th great extinction cascading through species; new pesticides killing insects; lack of clean water causing migration, disease, or death; shills and skeptics shaping climate policy.

Knowledge instantiation:

• Other pressures? → Synergism with Climate change pressures?

• lots of plastic pollution in the ocean → death of marine life

• a 6th great extinction → loss of habitats and loss of species

• .new pesticide stresses on insect species → continuing insect species/​biomass losses

• lack of clean water availability → migration, disease, death

• climate skeptics and industry shills in government → reactive, bogus, or null climate change mitigation or adaptation policies

• [OK, this list could go on for a while, so I will stop now]

Internal dialog while researching:My goal was to understand the big picture of climate change in future. What is the big picture of consequences? We are on a path toward extinguishing life on the planet.

### Keeping a small ontology or prefiltering for relevance

A prediction is a best-fit match of an ontology to a set of (hypothetical) events. Your beliefs let you filter out what you consider relevant in events. Correspondingly, you match relatively small ontologies to real-world events. The more you filter for relevance, the smaller the matching requirements for your ontology. You need to filter incoming information about events for credibility or plausibility as well, but that is a separate issue involving research skills and critical thinking skills.

In my example above:

• the internal dialog sections presumed my beliefs. For example, “Discussions about tipping points could be filtered by media or government. ” presupposes that organizations might filter discussions of tipping points.

• questions presupposed beliefs. For example, ” Is the threat from tipping points immediate?” presupposes that tipping points pose a threat.

These beliefs, the content of the dialog sections, defines the ontology that I instantiate with the information that I gather during research. For example, I believe in climate tipping points that have bad and big consequences, and that is one of the first relationships I list in the example. The ontological relationships that I instantiate with information about tipping points are few and general so my example ontology was small but let me reach a useful conclusion: “We are on a path toward extinguishing life on the planet.”[4].

### Superforecasters have smaller ontologies with better relevance and good research and critical thinking skills

“A hundred percent he’s there. Ok, fine, ninety-five percent because I know certainty freaks you guys out, but it’s a hundred.”
-Intelligence Analyst Maya in the movie Zero Dark Thirty

Superforecasters might be good at:

• revising their ontologies quickly by adding new entities or relationships and subtracting others.

• making predictions based on matches to a relatively small but well-curated ontology[5].

I suspect that forecasters continue to use subjective probabilities because forecasters are not asked to justify their predictions by explaining their ontology.

A plausible problem for domain experts charged with forecasting is that they develop larger ontologies than they should. Then, when they try to match the ontology against real-world information, they fail to collect only relevant data. They then know a lot, but don’t believe much about what qualifies as relevant.

I am by no means an expert in prediction or its skills. I am offering my beliefs here.

How you live your life can be consequential for other people, for the environment, for other species, and for future generations. The big question is “What consequences do you cause?”

I suggest that you discuss assertions about your consequences without the deniability that conclusions or hypotheses allow. Assert what you believe that you cause, at least to yourself. Don’t use the term consequence lightly. An outcome occurred in some correspondence to your action. A consequence occurred because your action caused it[6].

That I would advocate for this epistemic position might seem confusing. You’re probably wondering what benefit it has for deciding the altruism of your consequences or the validity of causal models that you employ to decide the altruism of your consequences.

## Ordinary pressures to hide beliefs

I believe that there are a number of pressures on discussions of beliefs about moral actions. Those pressures include:

• social expectations and norms

• your attention and actual commitment

To be a good scout involved in intersubjective validation of your beliefs, don’t lie to others or yourself. If you start off by asserting what you believe, then you might be able to align your beliefs with the best evidence available. What you do by expressing your beliefs is:

1. ignore expectations of rationality or goodness-of-fit to your beliefs.

3. commit to the discussion of your beliefs

provided that describing your unweighted beliefs is tolerable to those involved.

On way to make discussing your actual beliefs tolerable to yourself is to provide a means to constrain them. You can limit their applicability or revise some of their implications. Here are two short examples:

• I start out believing that God creating the universe 6000 years ago. I learn enough about evolution and planetary history and theories of the start of the universe that I revise my belief. Yes, God created everything, and God created us, but did it 4+ billion years ago in some kind of primordial chemical soup.

• I start out believing that climate change is a liberal hoax. I learn about climate science and recognize that climate researchers are mostly sincere. Now I believe that only the most liberal climate researchers make up how bad climate change will be, and I can tell those liberals apart from the truthful majority at the IPCC.

You might think these are examples of poor epistemics or partially updated beliefs, not something to encourage, but I disagree.

To simply make belief statements probabilistic implies that we are in doubt a lot, but with reference to unconstrained forms of our original beliefs. Constraining a belief, or chipping away at the details of the belief, is a better approach than chipping away at our confidence in the belief. We curate our ontologies and knowledge more carefully if we constrain beliefs rather than manipulate our confidence in beliefs.

Lets consider a less controversial example. Suppose some charity takes action to build a sewage plant in a small town somewhere. Cement availability becomes intermittent and prices rise, so the charity’s effectiveness drops. Subjective confidence levels assigned for the altruistic value of the charity’s actions correspondingly drop. However, constraining those actions to conditions in which cement is cheap and supply is assured can renew interest in the actions. Better to constrain belief in the action’s effectiveness than to reduce your confidence level without examining why. We’ll return to this shortly.

## A summary of advantages of unweighted beliefs

At the risk of being redundant, let me summarize the advantages of unweighted beliefs. Preferring unweighted beliefs to weighted beliefs with epistemic confidence levels does offer advantages, including:

• an unambiguous statement of what you hold true. An unweighted belief is not hedged. “I believe in god” gives me information that “I ninety-five percent believe in god.” does not[7].

• an intuitive use of the concept of beliefs. Beliefs, as I think most people learned about them, are internal, sometimes rational, sometimes taken on faith, knowledge of what is true.

• a distinction of believed truths from contingent conclusions or validated hypotheses, as noted earlier.

• an opportunity to add constraints to your belief when evidence warrants.

• a way to identify that you think true something that contradicts your epistemic best practices[8].

• a way to distinguish hunches, intuitions, or other internal representations of beliefs from conclusions or hypotheses.

• A way to name an assertion that you can revise or revoke rather than assign to a probability.

• an opportunity to focus on the ontology or knowledge implicit in your belief.

# The nature of the problem

Tuld:“So what you’re saying is that this has already happened.”

Peter:“Sort of.”

Tuld: “Sort of. And, Mr. Sullivan, what does your model say that that means for us here?”

- CEO John Tuld and Analyst Peter Sullivan in the movie Margin Call

A community of thinkers, intellectuals, and model citizens pursuing altruism is one that maintains the connection between:

• intersubjective validation of the consequences of one’s actions.

• taking responsibility for changing the consequences of one’s actions for others.

The altruistic mission of the EA community is excellent and up to the task. I will propose a general principle for you, and that’s next.

## Altruism valuations lack truth outside their limiting context

If you decide that your actions produce a consequence, then that is true in a context, and limited to that context. By implication, then, you have to assure that the context is present in order to satisfy your own belief that the action is producing the consequence that you believe it does.

### An example of buying cement for a sewage-processing facility

For example, if you’re buying cement from a manufacturer to build a sewage-processing facility, and the payments are made in advance, and the supplier claims they shipped, but the building site didn’t receive the cement, then the charitable contributions toward the purchase of cement for the sewage treatment plant do not have the consequence you believe. Your belief in what you cause through your charity is made false. You don’t want to lose your effectiveness, so what do you do? You:

1. add the precondition that the cement reaches the building site to your list of what causes the sewage plant to be built. You bother because that precondition is no longer assured.

2. add actions to your causal model that are sufficient to ensure that cement deliveries complete. For example, possible actions might include:

• to send a representative to the supplier’s warehouses.

• to send employees to ride the freight train and verify delivery.

• to create a new contract with the cement manufacturer that payment is only made after delivery.

Lets say that cement deliveries are only reliable if you take all those actions. In that case your beliefs about paying for cement change:

• original belief: If we pay for cement, we can build our sewage plant.

• revised belief: If we arrange a cement purchase, and send a representative to the supplier, and that representative rides the freight train carrying the cement to the construction location before we send payment for the cement, then we can build our sewage plant.“

You could do something more EAish involving subjective probabilities and cost effectiveness analyses, such as:

1. track the decline in subjective probabilities that your build site receives cement and take action below a specific probability, just sufficient to raise subjective estimates of delivery efficiency above a certain percentage.

2. develop a cost-effectiveness model around cement deliveries. Ask several employees to supply subjective probabilities for future delivery frequencies based on proposed alternatives to correct delivery failures. Weight their responses to choose a cost-effective means to solve delivery problems.

3. rather than send a representative to the supplier or have that person ride the freight train, just get a new contract with a supplier that lets you specify that payment is made after cement is freighted to the build site. If successful, it is the cheapest action to start.

If that process also results in a revised true belief, then good enough. There might be other advantages to your seemingly more expensive options that actually reduce costs later[9].

You can take your causal modeling of your actions further[10]. You can:

• look further upstream in causes. For example, explore what caused the need for a charity to pay for a sewage treatment plant

• assess causes of preconditions that you have already established, or their preconditions. Maybe your employees wont ride the freight train without hazard pay.

• Look at your other actions to see if they cause the undesirable preconditions that make your charitable effort necessary. Are you contributing to the reasons that the town cannot afford a sewage treatment plant?[11]

### Your altruism is limited to the contexts where you have positive consequences

Yes, your actions only work to bring about consequences in specific contexts in which certain preconditions hold. You cannot say much about your altruism outside those contexts unless you consider your actions outside those contexts. Effective altruists leverage money to achieve altruism, but that’s not possible in all areas of life or meaningful in all opportunities for altruistic action. This has implications that we can consider through some thought experiments.

# Scoring altruistic and selfish consequences of actions

“Is that figure right?”
-Manager Sam Rogers in the movie Margin Call

Actions can be both good and evil in their consequences for yourself or others. To explore this, lets consider a few algebra tools that let you score, rank, scale, and compare the altruistic value of your actions in a few different ways. I think the following four-factor model will do.

Here’s a simple system of scoring actions with respect to their morality, made up of a:

• benefit score

• harm score

• self-benefit score

• self-harm score

Lets use a tuple of (benefit score, harm score, self-benefit score, self-harm score) to quantify an action in terms of its consequences. For my purposes, I will consider the action of saving a life to have the maximum benefit and self-benefit and give it a score of 10. Analogously, the maximum harm and self-harm score is also 10. All other scores relative to that maximum of 10 are made up.

## A Thought Experiment: Two rescues and a donation

### The first rescue: A woman saves two children from a building fire

Consider a stranger unknown to anyone in a neighborhood. She walks into a burning building and saves two children inside but dies herself from smoke inhalation. She intended well but caused the two children she rescued to suffer smoke inhalation because of her poor rescue technique.

Lets suppose saving a life gets a score of 10. The tuple of scores for this person’s life-saving action is (20,2,0,10). To explain, here a breakdown of the scores:

• benefit score: 20. There were two rescues, so 10 points for each life saved.

• harm score: 2. The rescue caused a bit of smoke inhalation to each person saved, worth 1 point of harm each, because the rescuer held the people over her shoulders after pulling them off the floor where they were lay screaming below the smoke.

• Self-benefit score: 0. The rescue did nothing for the rescuer, who died immediately afterward.

• Self-harm score: 10. As a result of inhalation caused by the rescuer searching the building for the children before finding them, the rescuer died on the steps of the building.

### The second rescue: a woman saves a child drowning in mud on a street

Now how about a person walking along that sees and rescues a child drowning in mud, like Peter Singer’s original thought experiment to motivate students to give what they can? We might assign that action scores like (10,0,2,1). The act is scored like:

• benefit score: 10. The rescuer saved a child’s life.

• harm score: 0. The rescuer didn’t hurt anyone.

• self-benefit score: 2. The rescuer raised the her reputation among others in the town and with the child’s family

• self-harm score: 1. The rescuer ruined her mother’s faux suede boots with mud.

### Comparing the two rescues

A quick subtraction of the fire rescue values from the drowning rescue values gives:

(20,1,0,10) - (10,0,2,1) = (10,1,-2,9)

lets you compare the two actions. The fire rescue action has:

• benefit score difference: +10. It was much more altruistic than the drowning rescue.

• harm score difference: +1. It did a bit more harm to others than the drowning rescue..

• self-benefit score difference: −2. It was less helpful to the rescuer than the drowning rescue.

• self-harm score difference: +9. It was far more harmful to the rescuer than the drowning rescue.

There’s another way to compare the fire rescue and the drowning rescue actions, and its through distance from a hypothetical null action (0,0,0,0) and from each other.

• Distance of fire rescue from null:

• Distance of drowning rescue from null:

• Distance of fire rescue from drowning rescue:

These distance scores could show that the actions had considerably different outcomes overall. The distances of each action from null show that the fire rescue action had more impacts overall. The distance of each action from each other shows that the impact of each rescue action is different.

### The donation with wildly altruistic consequences

EA is cool because you can accomplish goals like reducing suffering from malaria by paying for relatively cheap bednets.

Paying for a couple hundred bednets might deserve a score like (1000,0,0,0.2). This score, with all the validity of a thought experiment, shows:

• benefit score: 1000. the donation removed malaria suffering for 200 people, so a score of: 5 * 200 = 1000.

• harm score: 0. the donation did not cause anyone any harm.

• self-benefit score: 0. the donation did not help the giver in any way.

• self-harm score: 0.2. the donation cost the giver a trivial fraction of his yearly discretionary spending budget.

### Comparing the donation to the fire rescue

Lets compare the scores of a donation to an EA foundation (1000,0,0,0.2) with the scores from the previous fire rescue action of (20,2,0,10).

(1000,0,0,0.2) – (20,2,0,10) = (980,-2,0,-9.8)

Compared to the fire rescue, the donation is different by:

• benefit score difference: +980. The donation had far more altruistic consequences overall.

• harm score difference: −2. The donation caused less harm than the fire rescue.

• self-benefit score difference: 0. The donation didn’t benefit the giver more or less than the fire rescuer’s action..

• self-harm score difference: −9.8. The donation caused the giver far less harm than the fire rescue woman’s action caused the fire rescue woman.

Distance from null of donation action:

The donation was clearly impactful. Compare that distance (1000) to the earlier distances of the fire rescue (22.4) and the drowning rescue (10.2).

Distance of donation from fire rescue:

Clearly, the donation action is far different from the fire rescue.

This should be enough to see how this simple scoring system can develop your intuitions about actions.

Disagreements over these thought experiments might include:

• the individual scores are subjective and relative. Is preventing a death (a benefit score of 10) really worth two cases of malaria (a benefit score of 5 * 2)?

• the individual scores conflate incompatible types of altruism. Is death comparable to ruining a pair of boots?

• The causal models are debatable. Was the large donation actually sufficient to generate new bednet deliveries to Uganda? In the fire rescue thought experiment, are we sure the children suffered from smoke inhalation because of how the rescuer carried them out?

• Some scores are not possible in practice. For example, is it possible to have null benefit or harm ever?

If you have these concerns, then you can bring them up to improve your use of such a thought experiment. For example, you could disallow absolute 0 values of any individual score.

## The influence of beliefs about causation on your sense of responsibility

EA folks complain that they suffer depression or frustration when they compare their plans for personal spending against the good they could do by donating their spending money. Let’s explore this issue.

Continuing with our earlier donation example, lets say it takes $x to cause (1000,0,0,0.2) through the action of donating toward bednet purchases to prevent malaria infection. That same$x affords you a week-long rejuvenating vacation at an expensive spa resort, an action with a score of lets say (0,0,3,0). The vacation action gets a self-benefit score of 3 with no other believed consequences. You could go on to calculate the difference of the two scores (about 1000) and compare their distances from null (1000 vs 3). What do you believe happens to your morality or the world though, if you take that day trip to the spa?

To answer this question, lets consider how actions cause consequences, one more time. If you believe that your action to withhold money from your charity to pay for your spa vacation causes 200 people to suffer malaria, then you need to score your vacation action again to reflect that. Instead of the vacation deserving a score of (0,0,3,0), it should get a score of (0,1000,3,0), because along with rejuvenating at the spa for a week, you caused 200 hundred in Uganda to suffer malaria by your withholding bednets from them. The question comes down to what you believe about what you cause. Does that vacation deserve a scoring of (0,0,3,0), or of (0,1000,3,0)? Did you cause 200 people to contract malaria or not?

Let’s return to the example of the woman running into the burning building. This simpler example will ground your intuition about taking responsibility for outcomes.

A woman is walking home from a day at the local animal shelter on a roundabout route, getting in some exercise and fresh air while she ponders increasing her monthly Givewell payment. About halfway through her walk, she sees smoke and flame rising from a building at the end of a cul-de-sac and hears screams from inside the building. She doesn’t see anyone else around. There’s no cars in driveways, no lights on in homes, no other sound, light, or motion. She assumes that there is no one else there aware of the fire, and that no one is coming to the rescue. She races toward the building, and you know the rest. She rescues two children but dies of smoke inhalation herself.

Suppose that woman had instead called 911, waited for a fire truck to arrive, and then continued on her walk. A few days later, she reads a news report that two children were found dead in a fire at the location she passed. In her belief, she could have rescued those people trapped inside. She recalls that one else was around to rescue the two children. She concludes and then believes that she was necessary and sufficient to save those children, Furthermore, she believes that her calling 911 and waiting for a fire truck actually killed the two children inside. Therefore, the action of her walking home has a four-factor score like (0,20,1,0).

• benefit score: 0 She helped no one else.

• harm score: 20. She believes that she caused two deaths.

• self-benefit score: 1. She got some mild exercise on her roundabout walk.

• self-harm score: 0. She didn’t cause herself any harm by walking home.

All this example illustrates is our limitations in assigning ourselves a causal role in events. With a few small changes in in the narrative, we can change the fire rescuer’s beliefs and our own assessment as well. For example, suppose the woman remembers that her lungs are scarred from an accident many years ago and reconsiders whether she could have navigated the building in the thick smoke to find the children.

I have emphasized beliefs throughout this red team exercise because:

• beliefs assert what you consider to be consequences of your actions.

• beliefs decide what goes onto your own ledger of your good, evil, or selfish actions.

• beliefs guide your decisions and feelings in many situations of moral significance.

### The flow of time carrying your actions defines a single path of what exists

Again, I offer my beliefs. Your actions precede their consequences. What you believe that you cause is found in the consequences of your actions. You can look forward in time and predict your future consequences. You can look at diverging tracks of hypothetical futures preceded by actions you could have taken but did not take. However, those hypothetical futures are nothing more than beliefs. On those hypothetical tracks, you can perceive the consequences of what you could have done, like the woman in the fire rescue who elected to call 911 and wait for the fire truck to arrive. Later on, she’s haunted by what would have happened if she had rushed into the building and carried out two living healthy children instead of letting them die in a fire. What would have happened is not an alternate reality though. It is just a belief. The poor woman, haunted by guilt, will never know if her belief is true. Anyone suffering regret or enjoying relief about what they didn’t do or did do is responding to their beliefs, not some kind of escaped reality or alternate set of facts.

# Conclusion: This was a red-team criticism, not just an exploration of ideas

If I did my job here, then the content and my intent were clear. I am critical of the overuse of subjective probabilities inside EA. A renewal of unqualified use of assertions that indicate unweighted belief will make it easier for you to apply useful critical thinking and prediction skills.

## A few recommendations

I offered a few recommendations, including that you:

• remember that beliefs are more or different than conclusions and hypotheses.

• qualify your conclusions and hypotheses as such.

• state your unweighted beliefs with confidence.

• challenge beliefs to learn the particulars of their implicit ontology or knowledge.

• match causal pathways and ontological lists to make predictions.

• qualify a prediction with a discussion of the causal pathway it matches or the ontology it presumes, instead of with a prediction probability.

• model preconditions, actions, and consequences to achieve your goals.

• explore the context of any actions of yours that you want to perform.

I attribute your assessment of your action’s consequences to your beliefs, not your rationality, regardless of how you formed that assessment and any confidence that you claim in it. Or so I believe.

I believe that you can succeed to the benefit of others inside your moral circle and outside it as well. That said, these are difficult times, and we could all use some good luck. Take care!

I am a geophysics and mathematics graduate from the 1990′s out of UCSC. I’m an older man now, who spent time working with software and has done various types of work in between. My career interests are open but I expect to seek and find work in the software industry again at some point. I don’t have credentials to bolster your belief in what I share here. My formal education in related academic fields consists of several years of software training, some industry work, and a few philosophy and linguistics courses. Plus a lot of learning on my own time because I have hobbies.

This critique represents my beliefs about the topics involved, including the Effective Altruist community. It is not a scholarly work. My conclusions are not drawn from a history of the same that I know about, although there are many related areas of study. Ethics, philosophy, pragmatics, knowledge representation, all have something to say about the issues that I address here.

I should credit the Goldratt methods of analysis and discovery for inspiring my discussion of precondition identification and support here. If any reader wants to model causality to discover root causes, resolve conflicts, or solve problems, look into Goldratt Consulting’s work.

# Bibliography

Alciem LLC. Flying Logic User Guide v.3.0.9 . Arciem LLC. 2020.

Cawsey, Allison, The Essence of Artificial Intelligence. Pearson Education Limited, 1998.

Daub, Adrian, What Tech Calls Thinking. FSG Originals, 2020.

Fisher, Roger, and William Ury. Getting to Yes. Boston: Houghton Mifflin, 1981.

Galef, Julia, The Scout Mindset. Penguin, 2021.

Goldratt, Eliyahu, M., et al, The Goal. North River Press, Inc. 2012.

Johnstone, Albert A., Rationalized Epistemology. SUNY Press, 1991.

Pearl, Judea and Dana Mackenzie, The Book of Why. Hachette UK, 2018.

1. ^

In fact any time that you can affect causally. At least, so far, I have not heard any direct arguments for the immorality of not causing beings to exist.

2. ^

Altruistic value is the value to others of some effect, outcome, or consequence. I call the value altruistic because I consider it with respect to what is good for others that experience an effect, outcome, or consequence. In different contexts, the specifics of altruistic value will be very different. For example, EA folks talk about measuring altruistic value in QALY’s when evaluating some kinds of interventions. Another example is that I think of my altruistic value as present in how I cause other’s work to be more efficient in time or energy required. There is the possibility of anti-altruistic value, or harm to others, or evil consequences. Some actions have no altruistic value. They neither contribute to nor subtract from the well-being others. Those actions have null altruistic value.

3. ^

Sorry, Julia, if I butchered your concepts.

4. ^

I don’t believe that the mind uses an inference system comparable to production rules or forward-chaining or other AI algorithms. However, I think that matching is part of what we do when we gather information or make predictions or understand something. I don’t want to defend that position in this critique.

5. ^

There could be intuitive matching algorithms put to use that let a person quantify the match they determine between the various causal pathways in their ontology and real-world events. I am just speculating, but those algorithms could serve the same purpose as subjective probabilities in forecasting.

6. ^

Of course, no matter what confidence value (2%, 95%, very, kinda, almost certainly) that you give a assertion, if you don’t believe it, then it’s not your belief. Conversely, if you do believe an assertion, but assert a low confidence in it, then it is still your belief.

7. ^

And no, I am not interested in betting in general.

8. ^

Of course, no matter what confidence value (2%, 95%, very, kinda, almost certainly) that you give an assertion, if you don’t believe it, then it’s not your belief. Conversely, if you do believe an assertion, but assert a low confidence in it, then it is still your belief.

9. ^

The narrative details determine what’s cost effective, but a plausible alternative is that the area is subject to theft from freight trains, and an employee riding the freight train could identify the problem early, or plausibly prevent the thefts, for example with bribes given to the thieves.

10. ^

If you’re interested, there are various methods of causal analysis without probabilities. You can find them discussed in a manual from the company behind Flying Logic software, or in books about the Goldratt methods of problem-solving, and perhaps in other places.

11. ^

It would be difficult to learn that your actions in your career increase poverty in or force migration from a country receiving financial support for an EA charity, particularly when you also contribute a good chunk of your income to that charity. If you work for some banks or finance institutions, then it is plausible that you are causing some of the problem that you intend to correct.

• The results of a solarpunk art contest from last year.

That song! Easily one of my favourites now and I also absolutely love the visual artwork, this aesthetic really speaks to me, thank you 😊

• Nicole Noemi gathers some forecasts about AI risk (a) from Metaculus, Deepmind co-founders, Eliezer Yudkowsky, Paul Christiano, and Aleja Cotra’s report on AI timelines.

h/​t Nuño

• Terri Griffith [thinks](https://​​econjwatch.org/​​File+download/​​1236/​​UnderappreciatedWorksSept2022.pdf?mimetype=pdf Research Team Design and Management for Centralized R&D is their most neglected paper. They summarize it as follows:

It is a field study of 39 research teams within a global Fortune 100 science/​technology company. As we write in the abstract, we demonstrate that “teams containing breadth of both research and business unit experience are more effective in their innovation efforts under two conditions: 1) there must be a knowledge-sharing climate in the team (arguably allowing the team to have access to the knowledge developed through the members’ breadth of experience) and 2) the team leader also has a breadth of research and business experience allowing for the member breadth to be knowledgably managed.” With 13 years perspective, I still find these results valuable and often share them in my innovation management courses.

• I created a Zapier to post Pablo’s ea.news feed of EA blogs and website to this subreddit:

https://​​reddit.com/​​r/​​eackernews

I wonder how much demand there’d be for a ‘Hackernews’ style high-frequency link only subreddit. I feel there’s too much of a barrier to post links on the EA forum. Thoughts?

• “Price’s Law says that half of the contributions in a field come from the square root of the number of contributors. In other words, productivity increases linearly as the number of contributors increases exponentially.” This is incorrect; the square root of an exponential is still an exponential… it’s just exp(0.5 * x) rather than exp(x).

• Thanks, I think you’re right. I’ll have to edit that section.

• Hi Dushan—this is wonderful news! I look forward to watching the growth of EA Serbia. You will be excellent in guiding this from its beginnings.

• More edits:
- DeepMind: 5 → 10.
- OpenAI: 5 → 10.
- Moved GoodAI from the non-technical to technical table.
- Added technical research organization: Algorithmic Alignment Group (MIT): 4-7.
- Merged ‘other’ and ‘independent researchers’ into one group named ‘other’ with new manually created (accurate) estimate.

• To clarify, all pain-free time is assumed to be morally neutral (due to technical difficulties, I am not able to edit the post).

• I do think it’d be interesting to have an AGI-pilled economist talk to one of the economists that do GWP forecasting to see if they can find cruxes.

• Bostrom selects his most neglected paper here.

• Also might be worth paging radiobostrom.com

• crossposted from my blog

In 2018, Nick Bostrom published an anthology of his papers in German under “The Future of Humanity”:

Some other good papers by him:

• 2 Oct 2022 15:45 UTC
29 points
9 ∶ 0

There are two totally valid conclusions to draw from the structure you’ve drawn up: that CS people or EA people are deluded, or that the world at large, including extremely smart people, is extremely bad at handling weird or new things.

• @1 seems unreasonable, because as soon as the first AI-economics people would come up with these arguments, if they were reasonable, they would become mainstream

1 seems the most plausible to me. Reasonable arguments might eventually become mainstream, but that doesn’t mean they would do so immediately.

In particular (a) there may not be many AI-economics people, so the signal could get lost in the noise and (b) economics journals may tend to favour research that focuses on established topics or that uses clever methodology, rather than topics that are important/​valuable.

• Matt—I also strongly agree with this. I worked in an economics department for 4 years, as the token psychologist. Academic economics is largely obsessed with getting clever theorems and quantitative results published in one of the few highest-impact econ journals. Anything that challenges core assumptions in econ (e.g. the Rational Man hypothesis, rapid convergence on Nash equilibrium play in complex games, importance of positional goods in advanced market economies, continuity in rates of economic growth) is usually rejected, because it would undermine one’s chances of getting that American Economic Review paper accepted, and it would distract from the tenure track....

• There’s a lot going on here, but this person’s take on economics seems bad, and also is a reductive take that is very common.

To calibrate, this take is actually incredibly similar how someone critiquing EA would say “all EA is obsessed with esoteric philosophical arguments and captured by AI and the billionaire donors”[1].

Reasons:

• A decent chunk of economics is concerned with meta-economics and disliking economics. These ideas are held by many of the key figures.

• In addition to these negative meta views, which are common, entire subdisciplines or alt schools are concerned with worldview changes in economics

• See behavioral economics (which is well, sort of not super promising because it seems to be a repacking of anecdotes/​psychology)

• See heterodox economics (which also does poorly basically for similar reasons as above, as well as challenges with QA/​QC/​talent supply, because diluting disciplines wholesale doesn’t really work)

• As a plus, economics has resisted the extremes of the culture wars and kept its eye on the work, while internally, to most students and some faculty, IMO proportionately giving disadvantaged people some equity or leg up (obviously not complete).

• The effort/​practice given to diversity is pretty similar to the level EA orgs have chosen and I suspect that’s not a coincidence.

• Economics has avoided the replication crisis, the event which drives a lot of negative opinion about mainstream science in EA

• To me, it’s obvious economics would...it’s very hard to communicate why, e.g. to show EAs the environment in an empirical seminar (argument in labor economics between senior faculty)

• The amount of respect senior/​mainstream economics give to reality and talking to people on the ground in empirical matters is large, and many ideas about unobserved/​quality/​social models has come out of this (although these ideas themselves can be attacked as repackaging the obvious)

• Some work in economics like environmental economics (not the same as “Ecological economics” , which is one of the unpromising heterodox schools) and practical work like kidney donation (Al Roth) are highly cherished by almost all economists

• Health economics and developmental economics is basically the entire cornerstone of the most concrete/​publicized sector of EA, that is, global health, e.g. GiveWell.

• GiveDirectly was literally founded and driven by economists, the entire methodology/​worldview is an economics one.

There’s (a lot) more but I have to do some work and I got tired of writing

1. ^

The issues with economics are similar/​literally isomorphic/​and in one case identical with EA (the math, dissenting subcultures and decision theory). I always wanted to write up but it seemed ancillary, hard to do well (for the social reality reasons) and embarrassing in multiple ways.

• I don’t usually add this, but writing this, because this person seems to be setting themselves up for a bit of a public figure role in EA, and mentions credentials a bit:

Some of the content in previous comments and this comment isn’t a great update in regards to those goals above and I would tap the brakes here. In this comment, it’s not the general take in the content itself (negative takes on economics are fine and good ones are informative) but the intellectual depth/​probably quality of the context of these specific ideas (“Rational Man hypothesis, rapid convergence on Nash equilibrium play in complex games, importance of positional goods in advanced market economies, continuity in rates of economic growth”) patterns matches not great.

• Hi Charles, you seem to be putting a lot of weight on a short, quick note that I made as a comment on a comment on an EA Forum post, based on my personal experiences in an Econ department (I wasn’t ‘mentioning credentials’, I was offering observations based on experience).

(You also included some fairly vague criticisms of my previous posts and comments that could be construed as rather ad hominem.)

You are correct that there are many subfields within Econ, some of which challenge standard models, and that Econ has some virtues that other social sciences often don’t have. Fair enough.

The question remains: why is Econ largely ignoring the likely future impact of AI on the economy (apart from some specific issues such as automation and technological unemployment), and treating most of the economy as likely to carry on more or less as it is today?

Matt and I offered some suggestions based on what we see as a few intellectual biases and blind spots in Econ. Do you have any other suggestions?

• I agree strongly! It would be interesting to research how economists have looked upon the creation of the internet. I guess that there is in fact little research on how the internet would change the world pre-1990.

• Animal Advocacy Careers’ “skilled volunteering” board has a few things that might be relevant in the “other technical” section.

• This seems cool!

When I saw the word “app” I assumed ‘oh cool I can download this on my phone and maybe I’ll be tempted to fiddle with it in spare moments similarly to how I get tempted to scroll social media.’ Seems it’s just on a website for now? I’m less optimistic that I’ll remember /​ get tempted to use it in this format.

(Not a criticism, just a reflection.)

• I enjoyed this post a lot!

I’m really curious about your mention of the “schism” pattern because I both haven’t seen it and I sort of believe a version of it. What were the schism posts? And why are they bad?

I don’t know if what you call “schismatics” want to burn the commons of EA cooperation (which would be bad), or if they just want to stop the tendency in EA (and really, everywhere) of people pushing for everyone to adopt convergent views (the focus of “if you believe X you should also believe Y” which I see and dislike in EA, versus “I don’t think X is the most important thing, but if you believe X here are some ways you could can do it more effectively” which I would like to see more).

Though I can see myself changing my mind on this, I currently like the idea of a more loose EA community with more moving parts that has a larger spectrum of vaguely positive-EV views. I’ve actually considered writing something about it inspired by this post by Eric Neyman https://​​ericneyman.wordpress.com/​​2021/​​06/​​05/​​social-behavior-curves-equilibria-and-radicalism/​​ which quantifies, among other things, the intuition that people are more likely to change their mind/​​behavior in a significant way if there is a larger spectrum of points of view rather than a more bimodal distribution.

• It seems bad in a few ways, including the ones you mentioned. I expect it to make longtermist groupthink worse, if (say) Kirsten stops asking awkward questions under (say) weak AI posts. I expect it to make neartermism more like average NGO work. We need both conceptual bravery and empirical rigour for both near and far work, and schism would hugely sap the pool of complements. And so on.

Yeah the information cascades and naive optimisation are bad. I have a post coming on a solution (or more properly, some vocabulary to understand how people are already solving it).

DMed examples.

• I’m the author of a (reasonably highly upvoted) post that called out some problems I see with all of EA’s different cause areas being under the single umbrella of effective altruism. I’m guessing this is one of the schism posts being referred to here, so I’d be interested in reading more fleshed out rebuttals.

The comments section contained some good discussion with a variety of perspectives—some supporting my arguments, some opposing, some mixed—so it seems to have struck a chord with some at least. I do plan to continue making my case for why I think these problems should be taken seriously, though I’m still unsure what the right solution is.

• Good post!

I doubt I have anything original to say. There is already cause-specific non-EA outreach. (Not least a little thing called Lesswrong!) It’s great, and there should be more. Xrisk work is at least half altruistic for a lot of people, at least on the conscious level. We have managed the high-pay tension alright so far (not without cost). I don’t see an issue with some EA work happening sans the EA name; there are plenty of high-impact roles where it’d be unwise to broadcast any such social movement allegiance. The name is indeed not ideal, but I’ve never seen a less bad one and the switching costs seem way higher than the mild arrogance and very mild philosophical misconnotations of the current one.

Overall I see schism as solving (at really high expected cost) some social problems we can solve with talking and trade.

• If I had to make a criticism, it’s that EA’s ideas of improving morality only exist if moral realism is true.

Now to define moral realism, I’m going to define it as moral rules that are crucially mind independent, ala how physical laws are mind- independent.

If it isn’t true (which I suspect will happen with 50% probability) than EA has no special claim to morality, although everyone else doesn’t either. But moral realism is a big crux here, at least for universal EA.

• I see this criticism a lot, but I don’t understand where it cashes out. In the 50% case where moral realism is false, then the expected value of all actions is zero. So the expected value of our actions is determined only by what happens in the 50% case where moral realism is true, and shrinking the EV of all actions by 50% doesn’t change our ordering of which actions have the highest EV. More generally than EV-based moralities, any morality that proposes an ordering of actions will have that ordering unchanged by a <100% probability that moral realism is false. So why does it matter if moral realism is false with probability 1% or 50% or 99%?

• If you define moral antirealism as mind-dependence of moral facts, then people can still attach value judgments to actions or consequences. One doesnt have to assign zero.

What is different is that if two people attach different value judgments, there is no longer a process by which those two could eventually come to agree on what the “right” value judgments are.

• Admittedly that is a good argument against the idea that moral realism actually matters too much, albeit I would say that the EV of your actions can be very different depending on your perspective (if moral realism is false).

Also, this is a case where non-consequentialist moralities fail badly at probability, because it’s asking for an infinite amount of evidence in order to update one’s view away from the ordering, which is equivalent to asking for mathematical proof that you’re wrong.

• Even though you disagreed with my post, I was touched to see that it was one of the “top” posts that you disagreed with :). However, I’m really struggling to see the connection between my argument and Deutsch’s views on AI and universal explainers. There’s nothing in the piece that you link to about complexity classes or efficiency limits on algorithms.

• The basic answer is, computational complexity matters less than you think, primarily because it makes very strong assumptions, and even one of those assumptions failing weakens it’s power.

The assumptions are:

1. Worst case scenarios. In this setting, everything matters, so anything that scales badly will impact the overall problem.

2. Exactly optimal, deterministic solutions are required.

3. You have only one shot to solve the problem.

5. Linear returns are the best you can do.

This is a conjunctive argument, where if one of the premises are wrong, than the entire argument gets weaker.

And given the conjunction fallacy, we should be wary of accepting such a story.

https://​​www.gwern.net/​​Complexity-vs-AI#complexity-caveats

• You are totally right, Deutsch’s argument is computability, not complexity. Pardon!

Serves me right for trying to recap 1 of 170 posts from memory.

• [ ]
[deleted]
• I think if you replace ‘aesthetic sense’ with ‘System 1’/​‘gut feelings’ the post makes more sense. Taking ethics specifically, I think a lot of cases of “naive consequentialism” could be seen as following one’s rational sense despite the protests of one’s aesthetic sense.

Maybe we shouldn’t completely conflate ‘aesthetic sense’ with ‘System 1’ but I personally really like this framing.

• One of my favourite EA Forum posts (and I didn’t expect to like it). Many thanks to Gavin for highlighting it.

• Am I right that Carlsmith (2021) is the only end-to-end model of AI Risk with numerical predictions at each stage (by end-to-end I mean there are steps in between ‘AI invented’ and ‘AI catastrophe’ which are individually predicted)? Any other examples would be really helpful so I can scope out the community consensus on the microdynamics of AI risk.

This spreadsheet (found here) has estimates on the propositions in Carlsmith by (some of?) the reviewers of that paper.

• Really interesting!

I get the impression that you do organizational consulting. I have been in various business environments where I watched organizational consultants work from my perspective as an employee.

I am curious how your approach and ethics let you handle:

• emperor wears no clothes organizational problems: everyone seems to think some X is really great, but X is a fiction and only you see that.

• elephant in the room communication situations: there’s something everyone knows about, fears, and won’t talk about and it’s the problem that needs handling.

• covert consulting needs: you’re consulting, but the problems are so obviously related to leadership or the organization, that you either leave or create organizational change covertly despite whatever management identified as problems to fix.

These situations were a test of consultant integrity, from what I saw, but they also show up in everyday life, where fictions, secrets, or politics conflict with desire for integrity.

• Update on the nutritional tests: 5 tests have been ordered, at least 3 completed, 2 have results back, 1 of which speaks to the thesis (the other person wasn’t vegan but was very motivated). I won’t have real results until people have gone through the full test-supplement-retest cycle, but so far it’s 1 of 1 vegans having one of the deficiencies you’d expect. This person had put thought into their diet and supplements and it seems to have worked because they weren’t deficient in any of the things they were supplementing, but had missed one.

I have no more budget for covering tests for people but if anyone would like to pay their own way ($613 for initial test) and share data I’m happy to share the testing instructions and the what-I’d-do supplementation doc (not medical advice, purely skilled-amateur level “things to consider trying”). • I have a few years of data from when I was vegan; any use? • I probably can’t combine it with the trial data since it’s not comparable enough, but seems very useful for estimating potential losses from veganism. • Hi, you may be able to find funding here but I recommend posting a detailed grant application style post, explaining your project in detail. • As a general point, do you believe that if women did realize the full benefits of receiving education in terms of learning outcomes, that then they would be served by their literacy, knowledge of mathematics, physics, politics, government, and software even if they could not find employment in a career field such as engineering? Your answer might help me understand your question a little better. I am also curious about what expert or organization discusses the relationship between poverty and population that you allude to when you write: To be clear, I’m aware that most experts no longer believe global overpopulation will be an issue, and I think more population growth could be good for the progress of technology. • Note to anyone still following this: I have now written up a long list of longtermist policy projects that should be funded, this gives some idea how big the space of opportunities is here: List of donation opportunities (focus: non-US longtermist policy work) • Which are your favorites? 1. Honestly, I find just the sheer diversity of dreams here really beautiful and inspiring. Each one brings to life a different element of a broadly shared vision, and while most of these point to an immense cosmic wrongness, an intolerable gash in the world, distressing and enraging, it’s very motivating to feel part of a serious collective effort to do something about it. 2. Prioritize. • (The irony here is not lost on me. At least on a certain reading—after all, it would be suspicious if each of our opinions on what to prioritise were all exactly the same. I mainly like “Prioritize” for being the most succinct way I’ve come across to explain EA.) • Hey Aron, thanks for your post! we should weight suffering today much more highly than suffering in the future, because only we can do something about it. This can be framed in terms of both the importance, tractability and neglectedness (ITN) framework and the significance, persistence and contingency (SPC) framework. Using the ITN framework, you might argue that suffering that occurs in the present is more neglected than suffering in the distant future because fewer people will ever be in a position to address it (only the present generation). By contrast, everyone from now until a given point in the future will be in a position to address suffering that occurs at that point in time. It is also more tractable because we can address it directly, whereas we can only address suffering in the future indirectly, e.g. by empowering future generations to address it when it occurs. (These considerations weigh against each other, though.) Using the SPC framework, you might argue that suffering in the distant future is not very contingent on our actions in the present because people in the future will be able to address it regardless of what we do now. These points are not fatal to longtermism, though. The idea that future people will be better positioned to address future problems is the basis of patient longtermism, “the view that individuals can have a greater positive impact by investing current altruistic resources and spending them later than by spending them now.” • Overall agreed, except that I’m not sure the idea of patient longtermism does anything to defend longtermism against Aron’s criticism? By my reading of Aron’s post, the assumptions there are that people in the future will have a lot of wealth to deal with problems of their time, compared to what we have now—which would make investing resources for the future (patient longtermism) less effective than spending them right away. I think your point is broadly valid, Aron: if we knew that the future would get richer and more altruistically-minded as you describe, then we would want to focus most of our resources on helping people in the present. But if we’re even a little unsure—say, there’s just a 1% chance that the future is not rich and altruistic—then we might still have very strong reason to put our resources toward making the future better: because the future is (in expectation) so big, if there’s anything at all we can do to influence it, that could be very important. And to me it seems pretty clear that the chance of a bad future is quite a bit more than 1%, which further strengthens the case. • Got opinions on this? (how 80k vet jobs and their transparency about it) It wasn’t officially submitted to the contest • Nice work, glad to see it’s improving things. I sympathise with them though—as an outreach org you really don’t want to make public judgments like “infiltrate these guys please; they don’t do anything good directly!!”. And I’m hesitant to screw with the job board too much, cos they’re doing something right: the candidates I got through them are a completely different population from Forumites. Adding top recommendations is a good compromise. I guess a “report job ” [as dodgy] button would work for your remaining pain point, but this still looks pretty bad to outsiders. Overall: previous state strikes me as a sad compromise rather than culpable deception. But you still made them move to a slightly less sad compromise, so hooray. • Ah and regarding “infiltrate these guys please”—I am not voicing an opinion on this making sense or not (it might) - but I am saying that if you want person X to infiltrate an org and do something there—at least TELL person X about this. wdyt? • Thanks, How about the solution/​tradeoff of having a link saying “discuss this job here”? • “As the area of our knowledge grows, so does the perimeter of our ignorance.” Neil DeGrasse Tyson • Wow, I’m glad I noticed Vegan Nutrition in among the winners. Many thanks to Elizabeth for writing, and I hope it will eventually appear as a post. A few months ago I spent some time looking around the forum for exactly this and gave up—in hindsight, I should’ve been asking why it didn’t exist! • There is a full post planned, but I wanted actual data, which means running nutrition tests on the population I think is hurting, treating any deficiencies, and retesting. I have a grant for this (thanks SFF!) but even getting the initial tests done is taking months so the real post is a very long ways out. PS. I have no more budget to pay for tests but if anyone wants to cover their own test ($613, slightly less if you already have or want to skip a genetic test) and contribute data I’d be delighted to have you. Please PM me for details.

• Maybe a more descriptive title next time e.g. “These ethics puzzles aren’t just challenges for utilitarianism” or something?

Despite this post being curated and me reading nearly all your posts, I personally ignored this one because it made me think it was something like “I’ve turned some sophisticated philosophical concepts/​arguments into fun, simplistic games that you all can understand!”, whereas it actually makes an extremely good point that I wish people recognised much more often. (I finally gave in and read it when I saw it on the EA UK newsletter just now.)

Mostly too late now as it’s no longer curated, plus I’m only one data point and not really your target audience, but maybe still worth changing and/​or a general thing to bear in mind for the future.

• In my experience, most people (~75%) become their worst selves on Twitter. Many times I have followed someone because I enjoyed their writing off Twitter, only to find that a significant portion of their Twitter content was angry culture-warring. So I think someone being interesting and thoughtful is not enough to make them a good Twitter poster, and using Twitter seems to be net negative for a lot of people.

• Hey Y’all, I’m Mar! I’m an undergraduate student studying Arts and Humanities with a focus in community engagement and a minor in Sustainable Natural Resource Recreation Management.

Learning about EA was entirely a coincidence. I started seeing more and more recommended tweets that interested and/​or inspired me. I like the emphasis on reason and rationality, the acceptance of people of diverse world views, and that everyone seems genuinely kind. I followed more accounts, and saw “EA” in many of the bios. I had literally no idea what that meant, and didn’t think much about it until I noticed the majority of people I followed were identifying as Effective Altruists. I started to read more about it, and I like many of the ideas and values.

I feel lucky to have found EA. I’ve been frustrated with my college and degree path. I’m in a small liberal arts program that produces and encourages unrealistic, unproductive, echo-chamber ideas and culture. Some examples are: a non-acceptance of political or economic ideas other than communism or socialism, valuing identity politics over factual information, none of my community engagement classes involving doing community engagement, and hypocritical ideas on the value of human lives. I initially loved this program and believed much of that myself, until living in a kinda-anarchist housing cooperative caused me challenge and re-evaluate a lot of the social and economic beliefs I held. EA addresses a lot of the concerns I have about the world, and does so in a way that feels a lot less harsh and judgemental than my my college community.

I’m drawn to EA because of the positive critical thinking, the acceptance of diverse beliefs, the kindness, and the action. I want to learn more about global sustainability, philosophy, and AI and how it can be beneficial in community engagement.

Some Concerns I Have: There seems to be a really strong emphasis on EA’s being smart and good at science things. I’m a liberal arts girl, and I’m worried that there isn’t much value in that here. Also, I could totally be wrong, but it seems to be a more male dominated community. Which isn’t necessarily bad, just confusing to me.

Random Fun Facts About Me: As if this intro isn’t long enough, here’s more.

• I have a cat! Her name is French Toast (I will gladly provide unlimited cat pics)

• I’m a vegetarian! And I have a pretty bad dairy allergy so my diet is like 99% vegan

• Some things I like to talk about: Decriminalizing drugs, gender and feminism in country music, consent/​sexual violence/​and sex work, making art

• Sociology major from a small liberal arts college here, and resonate with a lot of what you’re saying. I wrote about my journey into EA here, in case that’s helpful:

https://​​forum.effectivealtruism.org/​​posts/​​kE3FRC5gq9QxMrn3w/​​what-drew-me-to-ea-reflections-on-ea-as-relief-growth-and

Fwiw my view is thay you can add a lot, but you’ll have to work a little bit harder to figure out where the best place is for you.

In my case, I have a good fit with community building and operations-type roles, and now product management also.

I wrote my undergrad thesis on social movements, which was partially influenced by EA and I think helped me think more systematically about movement building. I think figuring out ways to explore areas you are particularly interested in through classes could be really interesting (one of the benefits of liberal arts is flexibility on coursework and essays!)

• Wow, that is a strong claim!

Could these conscious AI also have affective experience?

• Perhaps I oversold the provocative title. But I do think that affective experiences are much harder, so even if there is a conscious AI it is unlikely to have the sorts of morally significant states we care about. While I think that it is plausible that current theories of consciousness might be relatively close to complete, I’m less sympathetic that current theories of valence are plausible as relatively complete accounts. There has been much less work in this direction.

• Which makes me wonder how anyone expects to identify whether software entities have affective experience.

Is there any work in this direction that you like and can recommend?

• There is a growing amount of work in philosophy investigating the basic nature of pain that seems relevant to identifying important valenced experiences in software entities. What the body commands by Colin Klein is a representative and reasonably accessible book-length introduction that pitches one of the current major theories of pain. Applying it to conscious software entities wouldn’t be too hard. Otherwise, my impression is that most of the work is too recent and too niche to have accessible surveys yet.

Overall, I should say that not particularly sympathetic to the theories that people have come up with here, but you might disagree and I don’t think you have much reason to take my word for it. In any case they are trying to answer the right questions.

• Thirdly, the question of whether going veg*n strengthens your altruistic motivations is an empirical one which I feel pretty uncertain about. There may well be a moral licensing effect where veg*ns feel (disproportionately) like they’ve done their fair share of altruistic action; or maybe parts of you will become resentful about these constraints. This probably varies a lot for different people.

Related to this is this study finding:

Across six experiments, including one conducted with individuals involved in policymaking, we show that introducing a green energy default nudge diminishes support for a carbon tax.

• EDIT: I gave this a mild rewrite about an hour after writing it to make a few points clearer. I notice I already got one strong disagreement. If anyone would like to offer a comment as well, I’m interested in disagreements, particularly around the value of statements of epistemic confidence. Perhaps they serve a purpose that I do not see? I’d like to know, if so.

Hmm. While I agree that it is helpful to include references for factual claims when it is in the author’s interest[1], I do not agree that inclusion of those references is necessarily useful to the reader.

For example, any topic about which the reader is unfamiliar or has a strongly held point of view is also one that likely has opposing points of view. While the reader might be interested in exploring the references for an author’s point of view, I think it would make more sense to put the responsibility on the reader to ask for those references than to force the author to presume a reader without knowledge or agreement with the author’s point of view.

Should the author be responsible for offering complete coverage of all arguments and detail their own decisions about what sources to trust or lines of argument to pursue? I think not. It’s not practical or needed.

However, what seems to be the narrative here is that, if an author does not supply references, the reader assumes nothing and moves on. After all, the author didn’t use good practices and the reader is busy. On the other hand, so the narrative goes, by offering a self-written statement of my epistemic confidence, I am letting you know how much to trust my statements whether I offer extensive citations or not.

The EA approach of putting the need for references on authors up-front (rather than by request) is not a good practice and neither is deferring to experts or discounting arguments simply because they are not from a recognized or claimed expert.

In the case of a scientific paper, if I am critiquing its claims, then yes, I will go through its citations. But I do so in order to gather information about the larger arguments or evidence for the paper’s claims, regardless of the credentials or expertise that the author claims. Furthermore, my rebuttal might include counterarguments with the same or other citations.

I can see this as valuable where I know that I (could) disagree with the author. Obviously, the burden is on me to collect the best evidence for the claims an author makes as well as the best evidence against them. If I want to “steelman” the author, as you folks put it, or refute the author’s claims definitively, then I will need to ask for citations from the author and collect additional information as well.

The whole point of providing citations up-front is to allow investigation of the author’s claims, but not to provide comfort that the claims are true or that I should trust the author. Furthermore, I don’t see any point to offering an epistemic confidence statement because such a statement contributes nothing to my trust in the author. However, EA folks seem to think that proxies for rigor and the reader’s epistemic confidence in the author are valid.[2]

• scientific literature

• think-tank distillations of research or informed opinion

• public statements offered by scientists and experts on topics both inside and outside their areas of expertise

• books (with good bibliographies) written about most topics

• freely accessible journalism

• thousands of forum posts on EA topics

• and other sources

I don’t find it necessary to cite sources explicitly in forum posts, unless I particularly want to point readers to those sources. I know they can do the research themselves. If you want to steelman my arguments or make definitive arguments against them, then you should, is my view. It’s not my job to help you nor should I pretend that I am by offering you whatever supports my claims.

In some cases, you can’t provide much of the reasoning for your view, and it’s most transparent to simply say so.

Well, whether a person chooses not to offer their reasoning, or can’t offer their reasoning, you can conclude, if they don’t offer their reasoning, and you want to know what it is, that you should ask for it. After all, if I assert something you find disputable, why not dispute it? And the first step in disputing it is to ask for my reasoning. This is a much better approach than deciding whether you trust my presentation based on my write-up of my epistemic status.

For example, here is a plausible epistemic status for this comment:

• a confident epistemic status: I spent several hours thinking over this comment’s content in particular. I thought it through after browsing more than 30 statements of epistemic status presented by EA folks in the last six months. I have more than a decade of experience judging internet posts and articles on their argumentation quality (not in any professional capacity) and have personally authored about 1500-2000 posts and comments (not tweets) on academic topics in the last 20 years. My background includes a year of training in formal and informal logic, more than a year of linguistics study, and about a year of personal study of AI and semantic networks topics and an additional (approximate) year of self-study of research methods, pragmatics and argumentation, all devoted to assertion deconstruction and development, and most applied in the context of responding to or writing forum/​blog posts.

Actually, that epistemic status is accurate, but I don’t see it as relevant to my claims here. Plus how do you know its true? But suppose you thought it was true and felt reassured that my claims might have merit based on that epistemic status. I think you would be mistaken but I won’t argue it now. Interestingly, it does not appear representative of the sorts of epistemic status that I have actually read on this forum.

Here is an alternative epistemic status statement for my comment that looks like others on this forum:

• a less-confident epistemic status: I only spent a few minutes thinking over my position before I decided to write it down[implying that I’m brash]. Plus I wrote it when I was really tired. I’m not an acknowledged expert on this topic, so my unqualified statements are not reliable. Also, I don’t know of anybody else saying this. Finally, I’m not sure if I can articulate why I think what I do, at least to meet your standards, which seem high on this forum [implying that I’m intimidated]. So I guess I’m not that confident about my position and don’t want you to agree with it easily, particularly since you’ve been using your approach for so long. I’m just trying to stimulate conversation with this comment.

It seems intended to acknowledge and validate potential arguments against it that are:

• ad hominem: “[I’m brash and] I wrote this when I was really tired”

• appeals to authority: “I’m not an acknowledged expert on this topic”

• ad populum :”I don’t know of anybody else saying this”

• sunk cost: “you’ve been using your approach for so long”

and the discussion of confidence is confusing. After acknowledging that I:

• am brash and tired

• didn’t spend long formulating my position

• am not an expert

• feel intimidated by your high standards

• wouldn’t want you to reverse your position too quickly

I let you know that I’m not that confident in my assertions here. And then I go make them anyway with the stated goal that “I’m just trying to stimulate conversation.” Turned into a best practice as it is here, though, I see a different consequence for these statements of epistemic confidence.

What I have learned over my years on various forums (and blogs) includes:

• an author will offer information that is appealing to readers or that helps the readers see the author in a better light.

Either of the epistemic status offerings I gave might convince different people to see me in a better light. The first might convince readers that I am knowledgeable and have strong and well-formed opinions. The second might convince readers that I am honest and humble and would like more information. The second also compliments the reader’s high standards.

• readers that are good critical thinkers are aware of fallacies like ad hominem, ad populum, appeals to authority, and sunk cost. They also distrust flattery.

They know that to get at the truth, you need more information than is typically available in a post or comment. If they have a strong disagreement with an author, they have a forum to contest the author’s claims. If the author is really brash, tired, ill-informed, inarticulate, and has no reason for epistemic confidence in his claims , then the way to find out is not to take the author’s word for it, but to do your own research and then challenge the author’s claims directly. You wouldn’t want to encourage the author to validate irrelevant arguments against his claims by asking him for his epistemic confidence and his reasons for it. Even if the author chose on his own to hedge his claims with these irrelevant statements about his confidence, when you decide to steelman or refute the author’s argument, you will need different arguments than the four offered by the author (ad hominem, ad populum, appeals to authority, and sunk cost) .

1. ^

I agree that an author should collect their data and offer reliable access to their sources, including quotes and page numbers for citations and archived copies, when that is in the author’s interest. In my experience, few people have the patience or time to go through your sources just to confirm your claims. However, if the information is important enough to warrant assessment for rigor, and you as the author care about that assessment, then yeah. Supply as much source material as possible as effectively as possible. And do a good job in the first place, that is, reach the right conclusions with your best researcher cap on. Help yourself out.

2. ^

It should seem obvious (this must have come up at least a few times on the forum), but there’s a dark side to using citations and sources, and you’ll see it from think tanks, governments, and NGO’s, and here. The author will present a rigorous-looking but bogus argument. You have to actually examine the sources and go through them to see if they’re characterized correctly in the argument or if they are unbiased or representative of expert research in the topic area. Use of citations and availability of sources is not a proxy for correct conclusions or argument rigor, but authors commonly use just the access to sources and citations to help them persuade readers of their correctness and rigor, expecting that no one will follow up.

• 2 Oct 2022 1:55 UTC
1 point
0 ∶ 0

This is a fantastic post

• This seems like a really good post. It’s compact, honest, and clueful to a lot of social realities. It is aware of many offsetting considerations and presents these appropriate way. The post must also have been easier to write than a longer, more ornate post.

• This post is really well written and explains a lot of ideas in an approachable way.

• 2 Oct 2022 0:02 UTC
4 points
1 ∶ 0

I’m curious whether the reason why EA may be perceived as a cult while, e.g., environmentalist and social justice activism are not, is primarily that the concerns of EA are much less mainstream.

I appreciate the suggestions on how to make EA less cultish, and I think they are valuable to implement, but I don’t think they would have a significant effect on public perception of whether EA is a cult.

• Thank you for this post!

You identify two parts to developing empathy that I want to note:

1. develop knowledge about the situation that some people endure

2. develop emotional empathy for those people in that situation

I develop emotional empathy from videos, photos, and first-hand experiences, but it is not always necessary. For example, I was moved by Gail Eisnitz’ book Slaughterhouse to become a vegan for a few years[1]. Learning in depth about the troubles of factory farmed animals helped me appreciate their plight. It did contain photos, but those were not the convincing part of its content.

On a related note, I particularly enjoy stories of animals that help humans and help each other, that is, animals that show empathy. Weird as it is, I also enjoy stories of animals helping each other across species, and animals that show empathy or feeling for humans (for example, dolphins rescuing humans or tigers happy to see their former trainers). Video evidence of such interactions are very convincing for me (for example, watching a tiger hug and play with its trainer).

For me, it seems emotionally moving to learn about the commonalities with humans of animal emotions and affective experience, or to interpret animal behavior as showing those commonalities.

1. ^

I found the diet difficult, and in particular, dealing with protein requirements difficult. However, a kernel of interest in vegan protein remains in me, enough to motivate changes in my food consumption in a future where that is easier.

• Sounds good! Do you plan to publish the results each month on the forum, or if not what is a good way to get a quick summary of the results each month?

• Thanks for asking. We’re thinking of having a public dashboard showing the results for each month. At present, we’re not thinking of posting each month’s results on the Forum, but rather posting key results and intermittent updates. We think separate Forum posts every month might be unnecessary, since many of the results of the monthly tracker element of EA Pulse will make most sense in the context of multiple months having been run.

• Hey everybody!

One of my friends is interested in learning more about EA and I am trying to find good resources to recommend to her. The thing is, her English is only so-so; her preferred language is Spanish. I found a couple websites that give brief overviews of some EA ideas, but I am having a hard time finding comprehensive EA texts in Spanish.

Does anyone know of any EA resources in Spanish that could be helpful?

Thanks!

• Thanks for launching the contest!

It will be interesting to dig through this list and read through these posts, many seem to raise good points.

As the writer of the post (well, 4 of them) on energy depletion, I will try to integrate the feedback you gave here. I shall make a follow-up post addressing the common counters given against claims of energy depletion, then—and explain why I’m still worried about this topic in spite of that. While I think that societies will try to adapt, my biggest concern is that there will be little time to transition if peak oil is already past us (a likely possibility). I will try to express more clearly why I think this way.

• Congratulations, Corentin! Having just read part 1 in detail, I’m looking forward to more of your material.

Time and scale, as you said are the biggest concerns around adaptation. A virtue of not adapting with new infrastructure is that we save on carbon and energy put toward creating that infrastructure. Conservation could help more than anyone else believes, but it’s the least sexy approach. I want to comment on these issues, but now I don’t know where. Should I comment on the part 2 post, or in the google doc, or on the short contest version post?

• Thanks for the feedback!

Yeah, I’d really like conservation of energy to take place, using only what we really need. Unfortunately, we are in an economic system that values using something as precious as oil as fast as possible in order to grow—meaning it will be harder for future generations to produce stuff like mosquito nets, antiseptics or aspirin, as they are all derived from oil. I talk about it in part 3.

For comments, it depends. For general comments that everyone could find more interesting, the summary post is probably more adapted (probably where conservation could go). For comments related more particularly to data related to one topic (transition or the economy or actions), then go part 1/​2/​3. If you think there is a useful comment to make in the Google Docs, go ahead because it allows to target specifically one sentence in particular.

• OK, I will put my conservation thoughts in the comments of the summary post.

The speed of posting and conversation changes on this forum is way faster than I can match, by the time I have something decent written up, conversation will have moved on.

Keep an eye out for my reply though, I’ll come around when I can. Your work on this parallels a model I have of a pathway for our civilization coping with global warming. I call it “muddling through.” I know, catchy, right? How things go this century is very sensitive to short time scales, I’m thinking 10-20 year differences make all kinds of difference, and in my view, most needed changes are in human behavior and politics, not technology developments. So, good and bad.

• As a newcomer to EA, and a person with a fair amount of experience of cults and cult-like groups (and I’m 78 years old), I would like to report my experience.

I am very attracted to the ideas expressed in Doing Good Better. Having a science background, the idea of analyzing how effective my philanthropy may be is something I have pursued for many years, leading to many of the same conclusions.

On the other hand, many of the ideas of longtermism, trying to save human beings thousands of years in the future, being concerned about spreading to other planets, seeing malevolent AGI as among the most critical issues to address, strike me as similar to cults like Scientology and other groups whose vision and ideas seem contrary to common sense (if not downright wacky) but which seems to be common currency if not required by “members.”

In What We Owe the Future, MacAskill often expresses reservations about his ideas, points out alternatives or potential flaws, and in general shows somewhat more humility that I encounter on this Forum, for example. I certainly disagree with some of his conclusions and approaches, which I have begun to attempt to express in my few posts here to date, but I do respect his and others’ efforts to think long-term when accompanied by express recognition of our limitations in trying to impact the future (except in set-in-stone certainties) more than a couple of decades out. Without those ongoing acknowledgments of our limitations, our uncertainty, and the weirdness of our perspectives (from a “normal” viewpoint), we are bound to come across as potentially cult-like.

• I have often been fascinated watching young children expressing great confidence, even though, from my adult point of view, they have no basis for confidence other than their internal sense (i.e. they don’t understand the domain about which they are speaking in any adult way).

It is also my experience and belief that adults carry this same unwarranted sense of confidence in their opinions. Just to pick one example, 73% of Americans (including 80% of American men) believe they are better than average drivers.

Our culture selects for confidence, especially in men. This leads to overconfidence, especially among successful men. Successful people have often made at least one successful prediction (which may have led to their success), which may have simply been luck, but which reinforces their self-confidence.

I therefore strongly agree that longtermist predictions carry huge uncertainty despite expressions of confidence by those promoting them. I argue that in evaluating effective action, we should lower our expected value of any intervention based on how far in the future we are predicting, with a discount rate of 3.87% annually [/​joke].

• 1 Oct 2022 21:50 UTC
8 points
3 ∶ 0

Some quick impressions and thoughts:

1. I like historical anecdotes and forgotten/​underappreciated pieces of history. I enjoyed learning about the Quakers and some of their achievements.

2. I agree that a lot of the discussion on whether religion is good or bad is incredibly superficial. Nowadays it’s popular (among secular elites) to slam religion, but I’m quite certain that religions have played important roles in many positive developments (and, on the other hand, in many atrocities). Of course different religious groups are very different from one another and I think it’s very likely that some have been net positive while others net negative (it likely also depends on what you’d consider the counterfactual alternative to religion, given that it’s been so prevalent throughout most of human history).

3. It’s not entirely clear to me what you suggest in this post. Do you think that EAs should embrace a more religious attitude in general (and what would it mean practically, given that religions are so different)? Or do you specifically advocate for Quakerism? (Again, what would it mean in practical terms?) Or should we just be more open to learn useful lessons from historical groups wherever they happen to present themselves? If you just intended for this post to provide some inspiration and didn’t have clear action items in mind that’s also perfectly fine (I just left the reading with some uncertainty about what you really tried to say).

• Thanks for this—really appreciate your thoughts!

On 3 - I think seriously examining what made Quaker membership so impactful, as well as ahead of the moral curve, is something we should consider as a community. I think we should consider that various culture parts of Quakerism may have contributed meaningfully to their productivity—for example, I do wonder if EA meetups that emphasise silence, with occassional spoken words or passages read aloud by members who felt compelled, would actually have a bunch of unknown positive effects to the quality of debate and ideas.

I am definitely uncertain about what this would mean in a multitude of ways but I do think emulation means that you can improve a community through grabbing a series of positives that you might not, through a priori reasoning, realise are positives. Things that seem unnecessary might be very important—and we should be open to historical precedents to see if we can try any of these (at least particularly low cost examples) and see if we find a bunch of unintended positive results.

• So many of these topics seem really interesting to me personally that your saying “It was designed primarily for economics graduate students considering careers in global priorities research.” made me wonder: Is there something similar for people with a less robust background in economics? Maybe for economics (or political science) undergraduate students? :)

• Glad to hear you find the topics interesting!

First, I should emphasize that it’s not designed exclusively for econ grad students. The opening few days try to introduce enough of the relevant background material that mathematically-minded people of any background can follow the rest of it. As you’ll have seen, many of the attendees were pre-grad-school, and 18% were undergrads. My impression from the feedback forms and from the in-person experience is that some of the undergrads did struggle, unfortunately, but others got a lot out of it. Check out the materials, and if you think you’d be a good fit, you’re more than welcome to apply for next year.

That said, I agree that a more undergrad-focused version of the program would be valuable. I don’t have plans to make one myself in the near future, but I would like to at some point. In the meantime, if anyone reading this wants to do it, please feel free to reach out!

• 1 Oct 2022 20:35 UTC
10 points
2 ∶ 0

I think a great way to prevent us from turning into a cult is listening to criticism and have a diversity and opinions.

I would say EAs for the most part are open to criticism but there are EAs who will unintentionally partake in fallacious reasoning when their EA cause or EA opinion is attacked.

MacAskill once said EAs for the most part are social liberals. This is understandable considering most EAs come from fortunate backgrounds and have a college education.

Matt Yglesias and Tyler Cowen noted in their fireside chats at EA Global that their is a sense of homogeneity and a common group. Tyler Cowen said most EAs are “coastal elites.” I wouldn’t use the term coastal, but most EAs definitely come from elite families, elite colleges, elite company, or some other form of elite group.

There’s nothing wrong with being a social liberal (I am one), college educated, wealthy, or elite, but it could create an echo chamber and result in a cult or something cult-like.

I would like to see more libertarians, conservatives, and non-elites in EA so we can get different viewpoints, critiques, and a diversity of thought.

• This might be the best feedback I’ve ever gotten on a piece of writing (On the Philosophical Foundations of EA). Thanks for reading so many entries and helping make the contest happen!

• 1 Oct 2022 20:12 UTC
4 points
3 ∶ 0

I love this post. I’m a secular atheist and am strongly influenced by the New Atheism movement, but even I must admit there are habits and customs we can learn from religions and religious communities that can be beneficial if we apply it to ourselves and our communities.

Some of the acts Tyler Cowen recommended was abstaining from alcohol, being co-dependent on others (if I recall that correctly), having many children, and building private social safety nets.

(I’m just listing what Cowen said and am not saying I agree or disagree with them all).

I’ve been trying to find this book which talks about the importance of ritual for secular people (I think the author was a secular atheist too). Lmk if anyone knows the name of the book I’m talking about.

• Really great to hear—yes I don’t advocate for adopting Quaker views, but I do think Quaker practices may be of some use to us. Which ones, and in what ways, I don’t know—but I do consider its worth taking a very broad look at the Quakers and attempting some versions of key parts of their practices.

• Yeah. I like your analogy of EA and Quakers being a minority, but hopefully, EA can make an ethical revolution as the Quakers did with the abolition of slavery. We can certainly learn some tactics and habits from them while simultaneously not believing in their faith.

If someone is interested in looking more into this, I remember Willam MacAskill wrote about Quakers and their impact on the abolition movement in “What We Owe the Future.”

• I am not concerned too much with EA turning into a cult, for one reason:

Cults/​New Religious Movements are vastly less bad than most people think, and the literature on cults repudiates a lot of claims that the general population believes on cults, especially anything to do with harm.

https://​​www.lesswrong.com/​​posts/​​TiG8cLkBRW4QgsfrR/​​notes-on-brainwashing-and-cults

• I skimmed the link and it seems to be mostly about brainwashing not being effective. But cults do a lot of damage besides brainwashing. The insight that cults do provide some value to their members (otherwise why would anyone join?) is true, but does not mean that they don’t do a lot of net harm.

• Replying to this because I don’t think this is a rare view, and I’m concerned about it. Met someone this week who seemed to openly view cults as a template (flawed, but useful) and was in the process of building a large compound overseas where he could disseminate his beliefs to followers who lived onsite. By his own admission, he was using EA as a platform to launch multiple(?) new genuine religions.

In light of the Leverage Research incident, we should expect and keep an eye out for folks using the EA umbrella to actually start cults.

• My point is that contra the narrative in this post, cults are vastly less bad than the general public believes, so much so that the post is responding to a straw problem. I don’t necessarily agree with the beliefs of the New Religious Movements/​cults but the cult literature shows that they are vastly less bad then the general public thinks.

I know it’s a counterintuitive truth, but I want people to understand that the general public believing something is bad does not equal badness.

• [ ]
[deleted]
• 1 Oct 2022 19:09 UTC
1 point
0 ∶ 0

I really don’t understand the results of point 1. I guess I miss something.

Easterlin’s estimates include many countries, including a significant number of economically strong countries, right? And cash transfers are directed to the very poor, aren’t they? If so, I really don’t understand how the increase in happiness due to cash transfers is not much higher than that of Easterlin’s estimates (or at least significantly higher if only a small fraction of the countries included are rich). Isn’t it well established that the effect of wealth in happiness is much stronger for poorer people? What am I missing?

• There are two different things you could mean by “effect of wealth in happiness is much stronger for poorer people”:

1. Giving $1000 to a family in Kenya brings them much more life satisfaction than giving$1000 to a family in the United States: This is uncontroversial. Since Easterlin and O’Connor’s regressions look at the impact of percent change in GDP, they are assuming a logarithmic utility function. Therefore, a $1000 increase would represent something like a 100% GDP increase, while it would only represent something like 1.5% in the US. So the impact on happiness in Kenya would be much higher to be consistent with the regression. 2. Utility functions aren’t actually logarithmic. A 1% increase in income should bring a lot of happiness in Kenya, but almost none in the United States, because the United States has maxed out: I think Easterlin, O’Connor and some other economists believe this. Others don’t. I ran a quick regression of my own on Easterlin’s data, and it suggests that a 1% increase in income for developed countries actually increases happiness more than a 1% increase in LMICs. So that would lead us to expect that the impact from cash transfers might actually be a bit smaller than the results of Easterlin-style regression. • Thanks for the answer. I was referring to 2. I thought it was something we’ll established. But I think I was so convinced of it because I did not think much about and I probably conflated it with 1 as well. • Rachel Glennester is excellent. In so far as I know them, I’d trust her views on the effectiveness of development policy. She did great things as the UK Chief Economist for the Development department. She is a GWWC member and has talked at an ea conference in the. I’m not sure but I think she might now be in the US. • Thanks!! She looks perfect, and it wasn’t obvious from her professional page that she was EA, so this was a great find—I’ve suggested her name. • Or Ester Duflo obviously great and also sensible views on development • Thanks! She talked there last year or the year before… But just out of curiosity, are you saying that she is an effective altruist? This didn’t seem obvious to me in previous exchanges with her... • I don’t know that she’d call herself an effective altruist, but if you just want someone to talk about doing effective development spending then I’m not sure that it matters... • 1 Oct 2022 17:11 UTC 5 points 3 ∶ 0 I would find this more persuasive if it had thorough references to the existing science of consciousness. I was under the impression that we still don’t know what the necessary conditions for consciousness are, and there are many competing theories on the topic. Stating that one theory is correct doesn’t answer that key question for me. • I was under the impression that we still don’t know what the necessary conditions for consciousness are We definitely don’t, and I hope I haven’t committed myself to any one theory. The point is that the most developed views provide few obstacles. Those views tend to highlight different facets of human cognitive architecture. For instance, it may be some form of self-representation that matters, or the accessibility of representations to various cognitive modules. I didn’t stress this enough: of the many views, we may not know which is right, but it wouldn’t be technically hard to satisfy them all. After all, human cognitive architecture satisfies every plausible criterion of consciousness. On the other hand, it is controversial whether any of the developed views are remotely right. There are some people who think we’ve gone in the wrong direction. However, these people generally don’t have specific alternative proposals that clearly come down one way or another on AI systems. • 1 Oct 2022 16:06 UTC 25 points 14 ∶ 2 Needed to be said. I’m someone who gravitates to a lot of EA ideas, but I’ve avoided identifying as “an EA” for just this reason. Recently went to an EAG, which quelled some of my discomfort with EA’s cultishness, but I think there’s major room for improvement. My lightly held hypothesis is that the biggest reason for this is EA’s insularity. I think that using broader means of communication (publishing in journals and magazines, rather that just the EA forum) would go a really long way to enabling people to be merely inspired by EA, rather than “EAs” themselves. I like EA as a set of ideas and a question, not so much as a lifestyle and an all-consuming community. People should be able to publicly engage with (and cite!) EA rhetoric without having to hang out on a particular forum or have read the EA canon. • Current theories are missing important criteria that might be relevant to artificial consciousness because they’re defined primarily with the goal of distinguishing conscious from unconscious states of human brains (or possibly conscious human brains from unconscious animals, or household objects). They aren’t built to distinguish humans from crude representations of human architectures. It is open to most theorists to expand their theories to exclude digital minds. I think this is a key point that counts strongly against the possibility of building conscious AI so soon, as in the title. Some theories of consciousness are basically theories of human (and sometimes animal) consciousness, really just explaining which neural correlates predict subjective report, but do not claim those minimal neural correlates generate consciousness in any system, so building a computer or writing software that just meets their minimally stated requirements should not be taken as generating consciousness. I think Global Workspace Theory and Attention Schema Theory are theories like this, although Graziano, the inventor of AST, also describes how consciousness is generated in the general case, i.e. illusionism, and how AST relates to this, i.e. the attention schema is where the “illusion” is generated in practice in animals. (I’m not as familiar with other theories, so won’t comment on them.) If you overextend such theories of consciousness, you can get panpsychism, and we already have artificial or otherwise non-biological consciousness: • Also see section 6.2 in https://​​www.openphilanthropy.org/​​research/​​2017-report-on-consciousness-and-moral-patienthood for discussion of some specific theories and that they don’t answer the hard problem. • Some theories of consciousness are basically theories of human (and sometimes animal) consciousness, really just explaining which neural correlates predict subjective report, but do not claim those minimal neural correlates generate consciousness in any system, so building a computer or writing software that just meets their minimally stated requirements should not be taken as generating consciousness. From the 50s to the 90s, there was a lot of debate about the basic nature of consciousness and its relation to the brain. The theory that emerged from that debate as the most plausible physicalist contender suggested that it was something about the functional structure of the brain that matters for consciousness. Materials aren’t relevant, just abstract patterns. These debates were very high level and the actual functional structures responsible for consciousness weren’t much discussed, but they suggested that we could fill in the details and use those details to tell whether AI systems were conscious. From the 90s to the present, there have been a number of theories developed that look like they aim to fill in the details of the relevant functional structures that are responsible for consciousness. I think these theories are most plausibly read as specific versions of functionalism. But you’re right, the people who have developed them often haven’t committed fully either to functionalism or to the completeness of the functional roles they describe. They would probably resist applying them in crude ways to AIs. The theorists who might resist the application of modern functionalist theories to digital minds are generally pretty silent on what might be missing in the functional story they tell (or what else might matter apart from functional organization). I think this would undercut any authority they might have in denying consciousness to such systems, or even raising doubts about it. Suppose that Replika produced a system that had perceptual faculties and a global workspace, that tracked its attention and utilized higher-order representations of its own internal states in deciding what to do. Suppose they announced to the media that they had created a digital person, and charged users$5 an hour to talk to it. Suppose that Replika told journalists that they had worked hard to implement Graziano’s theory in their system, and yes, it was built out of circuits, but everything special about human consciousness was coded in there. What would people’s reactions be? What would Graziano say about it? I doubt he could come up with many compelling reasons to think it wasn’t conscious, even if he could say that his theory wasn’t technically intended to apply to such systems. This leaves curious public in the who can really say camp reminiscent of solipsism or doubts about the consciousness of dogs. I think they’d fall back on problematic heuristics.

• The theorists who might resist the application of modern functionalist theories to digital minds are generally pretty silent on what might be missing in the functional story they tell (or what else might matter apart from functional organization).

If they weren’t silent, they would be proposing further elaborated theories.

I think this would undercut any authority they might have in denying consciousness to such systems, or even raising doubts about it.

Maybe, but I don’t think we would be justified in giving much weight to the extension of their theory of consciousness to artificial entities. Taking GWT, for example, many things could qualify as a global workspace.

Suppose that Replika produced a system that had perceptual faculties and a global workspace, that tracked its attention and utilized higher-order representations of its own internal states in deciding what to do.

So far, I think this would be possible with <1000 neurons, and very probably 100 or fewer, if you interpret all of those things minimally. For example, a single neuron to represent an internal state and another another neuron for a higher-order representation of that internal state. Of course, maybe that’s still conscious, but the bar seems very low.

Suppose that Replika told journalists that they had worked hard to implement Graziano’s theory in their system, and yes, it was built out of circuits, but everything special about human consciousness was coded in there. What would people’s reactions be? What would Graziano say about it? I doubt he could come up with many compelling reasons to think it wasn’t conscious, even if he could say that his theory wasn’t technically intended to apply to such systems. This leaves curious public in the who can really say camp reminiscent of solipsism or doubts about the consciousness of dogs. I think they’d fall back on problematic heuristics.

“everything special about human consciousness was coded in there” sounds like whole brain emulation, and I think Graziano and most functionalists should and would endorse its consciousness. Generally, though, I think Graziano and other illusionists would want to test whether it treats its own internal states or information processing as having or seeming to have properties consciousness seems to have, like being mysterious/​unphysical/​ineffable. Maybe some more specific tests, too. Wilterson and Graziano have already come up with artificial agents with attention schemas, which should be conscious if you overextended the neural correlate claims of AST too literally and forgot the rest, but would not be conscious according to Graziano (or AST), since it wouldn’t pass the illusionist tests. Graziano wrote:

Suppose the machine has a much richer model of attention. Somehow, attention is depicted by the model as a Moray eel darting around the world. Maybe the machine already had need for a depiction of Moray eels, and it coapted that model for monitoring its own attention. Now we plug in the speech engine. Does the machine claim to have consciousness? No. It claims to have an external Moray eel.

Suppose the machine has no attention, and no attention schema either. But it does have a self-model, and the self-model richly depicts a subtle, powerful, nonphysical essence, with all the properties we humans attribute to consciousness. Now we plug in the speech engine. Does the machine claim to have consciousness? Yes. The machine knows only what it knows. It is constrained by its own internal information.

AST does not posit that having an attention schema makes one conscious. Instead, first, having an automatic self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. Second, the reason why such a self-model evolved in the brains of complex animals, is that it serves the useful role of modeling attention.

I suppose my claim about AST in my first comment is inaccurate. AST is in fact a general theory of consciousness, but

1. the general theory is basically just illusionism (AFAIK), and

2. having an attention schema is neither necessary nor sufficient for consciousness in general, but it is both in animals in practice.

• For example, a single neuron to represent an internal state and another another neuron for a higher-order representation of that internal state.

This requires an extremely simplistic theory of representation, but yeah, if you allow any degree of crudeness you might get consciousness in very simple systems.

I suppose you could put my overall point this way: current theories present very few technical obstacles, so there it would take little effort to build a system which would be difficult to rule out. Even if you think we need more criteria to avoid getting stuck with panpsychism, we don’t have those criteria and so can’t wield them to do any work in the near future.

“everything special about human consciousness was coded in there” sounds like whole brain emulation

I mean everything that is plausibly relevant according to current theories, which is a relatively short list. There is a big gulf between everything people have suggested is necessary for consciousness and a whole brain emulation.

Generally, though, I think Graziano and other illusionists would want to test whether it treats its own internal states or information processing as having or seeming to have properties consciousness seems to have, like being mysterious/​unphysical/​ineffable.

It has been awhile since I’ve read Graziano—but if I recall correctly (and as your quote illustrates) he likes both illusionism and an attention schema theory. Since illusionism denies consciousness, he can’t take AST as a theory of what consciousness is; he treats it instead as a theory of the phenomena that leads us to puzzle mistakenly about consciousness. If that is right, he should think that any artificial mind might be led by an AST architecture, even a pretty crude one, to make mistakes about mind-brain relations and that isn’t indicative of any further interesting phenomenon. The question of the consciousness of artificial systems is settled decisively in the negative by illusionism.

• I suppose you could put my overall point this way: current theories present very few technical obstacles, so there it would take little effort to build a system which would be difficult to rule out. Even if you think we need more criteria to avoid getting stuck with panpsychism, we don’t have those criteria and so can’t wield them to do any work in the near future.

This is also my impression of the theories with which I’m familiar, except illusionist ones. I think only illusionist theories actually give plausible accounts of consciousness in general, as far as I’m aware, and I think they probably rule out panpsychism, but I’m not sure (if small enough animal brains are conscious, and counterfactual robustness is not necessary, then you might get panspychism again).

I mean everything that is plausibly relevant according to current theories, which is a relatively short list. There is a big gulf between everything people have suggested is necessary for consciousness and a whole brain emulation.

Fair. That’s my impression, too.

It has been awhile since I’ve read Graziano—but if I recall correctly (and as your quote illustrates) he likes both illusionism and an attention schema theory. Since illusionism denies consciousness, he can’t take AST as a theory of what consciousness is; he treats it instead as a theory of the phenomena that leads us to puzzle mistakenly about consciousness. If that is right, he should think that any artificial mind might be led by an AST architecture, even a pretty crude one, to make mistakes about mind-brain relations and that isn’t indicative of any further interesting phenomenon. The question of the consciousness of artificial systems is settled decisively in the negative by illusionism.

I guess this is a matter of definitions. I wouldn’t personally take illusionism as denying consciousness outright, and instead illusionism says that consciousness does not actually have the apparently inaccessible, ineffable, unphysical or mysterious properties people often attribute to it, and it’s just the appearance/​depiction/​illusion of such properties that makes a system conscious. At any (typo) rate, whether consciousness is a real phenomenon or not, however we define it, I would count systems that have illusions of consciousness, or specifically illusions of conscious evaluations (pleasure, suffering, “conscious” preferences) as moral patients and consider their interests in the usual ways. (Maybe with some exceptions that don’t count, like giant lookup tables and some other systems that don’t have causal structures at all resembling our own.) This is also Luke Muehlhauser’s approach in 2017 Report on Consciousness and Moral Patienthood.

• I guess this is a matter of definitions.

I agree that this sounds semantic. I think of illusionism as a type of error theory, but people in this camp have always been somewhat cagey what they’re denying and there is a range of interesting theories.

At an rate, whether consciousness is a real phenomenon or not, however we define it, I would count systems that have illusions of consciousness, or specifically illusions of conscious evaluations (pleasure, suffering, “conscious” preferences) as moral patients and consider their interests in the usual ways.

Interesting. Do you go the other way too? E.g. if a creature doesn’t have illusions of consciousness, then it isn’t a moral patient?

• [ ]
[deleted]
• Also Future Academy (but maybe that’s not an org and instead a project of EA Sweden?).

• Also anything in Alignment Org Cheat Sheet that’s not in here. And maybe adding that post’s 1-sentence descriptions to the info this database has on each org listed in that post.

• Also fp21 and maybe Humanity Forward.

(Reminder: This is a database of orgs relevant to longtermist/​x-risk work, and includes some orgs that are not part of the longtermist/​x-risk-reduction community, don’t associate with those labels, and/​or don’t focus specifically on those issues.)

• 1 Oct 2022 15:17 UTC
35 points
24 ∶ 2

It’s also important to note that with many (I think 86% at last survey) EAs being nonreligious, it can be relatively easy for EA to play that role in people’s lives. The cultural template is there, and it’s powerful.

• Interestingly we both posted this on the same day but I have almost the entire opposite approach! Would appreciate your thoughts!

https://​​forum.effectivealtruism.org/​​posts/​​3Jm6tK3cfMyaan5Dn/​​ea-is-not-religious-enough-ea-should-emulate-peak-quakerism

• Yes the timing was funny (it looks like virtually everyone on the forum has something to say about EA culture nowadays :P)

• Extremely excited for this!

Think the effective altruism community should in general embrace some of the explorations and ideas in the broader crypto public goods sphere, and get more involved improving them and introduce insights and learning from effective altruism like cost effectiveness, meta cause evaluation and prioritization.

Some great efforts to highlight: https://​​www.radicalxchange.org/​​ , protocol labs with https://​​fundingthecommons.io/​​ (youtube videos) , gitcoin with quadratic donations, impact daos and decentralized science etc.

Loved this conversation with Rob and Vitalik for example https://​​80000hours.org/​​podcast/​​episodes/​​vitalik-buterin-new-ways-to-fund-public-goods/​​

• 1 Oct 2022 14:28 UTC
15 points
0 ∶ 0

It’s hard to know who to suggest without knowing anything about the audience or topic

• sorry for the lack of context! In this case, it’s a series of talks from external researchers, and the topic has to be something relevant to developmental economics, but no other specific topic otherwise.

• 1 Oct 2022 13:41 UTC
27 points
0 ∶ 0

Many of my picks narrowly missed prizes and weren’t upvoted much at the time, so check it out.

• Being effective requires monitoring performance; how well are things going before/​after a change. An understanding of how things work is needed to guide new ideas; randomly blundering around in is very ineffective.

What is the most effective way of doing software engineering?

There is lots of folklore and existing practice, but no evidence that any of it is the best way of doing things.

There is an analysis of all the available public software engineering data (as of 2020), but the 600+ datasets are not enough to build reliable models.

What is needed is more data. If anybody knows of any public software engineering data, please let me know. I’m always up for analyzing software data. I know there is lots of code on Github, and there are lots of animals in the Jungle; this is not data. The hard to find data is human related, e.g., task estimates and schedules.

• Elon Musk’s perspective on AGI safety (from the Tesla AI day, source)

• If Tesla contributes significantly to AGI, they will invest a lot in AI safety research.

• There should be a governmental AI regulatory authority like the FDA that tries to ensure public safety for AGI.

• Tesla will probably make a significant contribution to AGI because of their unique real-world data advantage once their robots roll out.

• I’m concerned my entry was not read. I submitted via the form. I pointed out two typos on a particular EA website (not as part of my criticism but clearly and prominently attached as an aside) and one was fixed and one was not. Since one was fixed I assume someone else pointed that one out. The typo is not ambiguously wrong so there would be no reason not to fix it.

• 1 Oct 2022 10:36 UTC
2 points
0 ∶ 0

DM’d with two suggestions :)

• Regarding self-determination theory. How would you relate that to agency (similarities and differences)? Agency (or the related self-efficacy) appears relatively popular among EA and has been around in psychology for a while.

• Thanks so much for writing this up.
I carefully read through the “how to teach effectively in five steps” and got a lot of value from it. I’ve yet to read through other aspects of the blog post.

The thing I love the most about this section is that it builds on robust findings from meta-analyses. Why do I love this? Because i) nothing in community-building is based on robust evidence and ii) it counteracts a tendency for EA to rely to much on what’s EA—reinventing a new paradigm rather than recalling that the majority of what’s useful in the world in the world isn’t “EA” (we’re substantially standing on the shoulders of giants).

Additionally, here’s a couple of reactions and questions to five step section specifically:
1. I appreciate the focus on defining a learning objective as a skill and then working backward from that. I think that “creating personal theory of changes and impactful career plans that one can wholeheartedly (or agentically) pursue” is a good default to have in mind. One downside to this approach is that it might lead to Goodharting and leading the teacher to go in “exploitation” mode. E.g., I worry that I might become too attached to a specific outcome on behalf of the students and tacitly start to persuade (similar to some concerns expressed by Theo Hawkins) and/​or neglect other important opportunities that might emerge during the program. How do you think of that risk?

2. Can you say anything about what forms summative assessments are particularly useful? For Future Academy, we’re contemplating pitching project ideas or presentation and discussion of career plans (although we likely wouldn’t label it as an assessment but rather as an exciting culmination of their work).

3. I think there’s a typo under 3a. (“Formative assessments” —> formative activities)?

4. While I appreciate the rigor of the evidence upon which this post is built, I worry about this being true for on average for average university students and might not generalize to the subpopulation that some portion of community-building efforts is targetted towards (e.g., people who are in the 90th percentile on various domains, including openness to experience, conscientiousness, need-for-cognition, etc.). How worried are you about this?

5. Strongly agree with the importance of role models. Humans are deeply social and our social incentives (including role models) might be the most important things to change. In fact, being generally good people (or virtuous) in addition to the unique virtues you mentioned appears important as we have some research showing that this might be off-putting. Finally, same-race role-models appear to be particularly important.

• [ ]
[deleted]
• A quick update: as far as I was able to tell, Telegram doesn’t support secret group chats (protected by end-to-end encryption), so I now use Signal for sensitive group messaging. I still prefer Telegram overall, which is my main messaging app, but I also think that Signal is far superior to Messenger and WhatsApp. I wanted to share this update because my earlier comments may suggest that I have a much more negative opinion of Signal than I in fact do.

• 1 Oct 2022 9:13 UTC
2 points
0 ∶ 0

For readers, here are links to The Malala Fund and the Global Health and Development Fund (part of EA Funds).

The consideration of whether to help people in poverty without solving the problem of poverty itself is a good one! In a slightly different formulation, it had been addressed in this amazing work—Growth and the case against randomista development.

However, the question of how exactly we can alleviate poverty is still very tough. For example, GiveWell’s analysis of educational programs in developing countries concludes that there’s very limited evidence on the impact of education on improved future earnings, and their cost-effectiveness model suggests that they are about (0.5x-3x) as cost-effective as direct cash transfers. This contrasts with the large potential effects of direct health interventions, like deworming, which could be much higher (but there’s still a lot of uncertainty and debate).

• I have lots of questions about the paper “Biological Anchors external review”.

I think this paper contains a really good explanation of the biological anchors and gives good perspective in plain English.

But later in the paper, the author seems to present ideas that seem briefly handled, and to me they appear like intuitions. Even if these are intuitions, these can be powerful, but I can’t tell how insightful they really are with the brief explanation given, and I think they need more depth to be persuasive.

One example is when the author critiques using compute and evolutionary anchors as an upper bond:

(Just to be clear I don’t actually read any of the relevant papers and I just guess the content based on the title) but the only way I can imagine “biological anchors” can be used is as a fairly abstract model, a metaphor, really.

People won’t really simulate a physical environment and physical brains literally. In a normal environment, actual evolution rarely selects for “intelligence” (no mutation/​phenotype, environment has too many/​no challenges). So you would skip a lot of features, and force mutation and construct challenges. I think a few steps along these lines means the simulation will use abstract digital entities, and this would be wildly faster.

It seems important to know more about why the author thinks that more literal, biological brains need to be simulated. This seems pretty core to the idea of her paper, where she says a brain-like models needs to be specified:

But I think it’s implausible to expect a brain-like model to be the main candidate to emerge as dangerous AI (in the AI safety worldview) or useful as AGI for commercial reasons. The actual developed model will be different.

Another issue is that (in the AI safety worldview) specifying this “AGI model” seems dangerous by itself, and wildly backwards for the purpose of this exercise. Because of normal market forces, by the time you write up this dangerous model, someone would be running it. Requiring to see something close to the final model is like requiring to see an epidemic before preparing for one.

• I don’t know anything about machine learning, AI or math, but I’m really uncertain about the technical section in the paper, on “Can 2020 algorithms scale to TAI?.”

One major issue is that in places in her paper, the author expresses doubt that “2020 algorithms” can be the basis for computation for this exercise. However, she only deals with feed forward neural nets for the technical section.

This is really off to leave out other architectures.

If you try using feed forward neural nets, and compare them to RNN/​LSTM for things like sequence like text generation, it’s really clear they have a universe of difference. I think there’s many situations where you can’t get similar functionality in a DNN (or get things to converge at all) even with much more “compute”/​parameter size. On the other hand, plain RNN/​LTSM will work fine, and these are pretty basic models today.

• Hi Charles, thanks for all the comments! I’ll reply to this one first since it seems like the biggest crux. I completely agree with you that feedforward NNs != RNN/​LSTM… and that I haven’t given a crisp argument that the latter can’t scale to TAI. But I don’t think I claim to in the piece! All I wanted to do here was to (1) push back against the claim that the UAT for feedforward networks provides positive evidence that DL->TAI, and (2) give an example of a strategy that could be used to argue in a more principled way that other architectures won’t scale up to certain capabilities, if one is able to derive effective theories for them as was done for MLPs by Roberts et al. (I think it would be really interesting to show this for other architectures and I’d like to think more about it in the future.)

• Nested inside of the above issue, another problem is that the author seems to use “proof-like” rhetoric in arguments, when she needs to provide broader illustrations that could generalize for intuition, because the proof actually isn’t there.

Sometimes some statements don’t seem to resemble how people use mathematical argumentation in disciplines like machine learning or economics.

To explain, the author begins with an excellent point that it’s bizarre and basically statistically impossible that a feed forward network can learn to do certain things through limited training, even though the actual execution in the model would be simple.

One example is that it can’t learn the mechanics of addition for numbers larger than it has seen computed in training.

Basically, the most “well trained”/​largest feed forward DNN that uses backprop training, will never add 99+1 correctly, if it was only trained on adding smaller numbers like 12+17 if these calculations never total 100. This is because in backprop, the network literally needs to see and create processes for the 100 digits. This is despite the fact that it’s simple (for a vast DNN) to “mechanically have” the capability to perform true logical addition.

Immediately starting from the above point, I think author wants to suggests that, in the same way it’s impossible to get this functionality above, this constrains what feed forward networks would do (and these ideas should apply to deep learning or 2020 technology for biological anchors).

However, everything sort of changes here. The author says:

I’s not clear what is being claimed or what is being built on above.

• What computations are foreclosed or what can’t be achieved in feed forward nets?

• While the author shows that addition with n+1 digits can’t be achieved by training with addition with numbers with n digits”, and certainly many other training to outcomes are prevented, why would this generally rule out capability, and why would this stop other (maybe very sophisticated) training strategies/​simulations from producing models that could be dangerous?

• The author says the “upshot is that the class of solutions searched over by feedforward networks in practice seems to be (approximately) the space of linear models with all possible features” and “this is a big step up from earlier ML algorithms where one has to hand-engineer the features”.

• But that seems to allow general transformations on the features. If so, that is incredibly powerful. It doesn’t seem to constrain functionality (of these feed forward networks)?

• Why would the logic which relies on a technical proof (which I am guessing relies on a “topological-like” argument that requires the smooth structure of feed forward neural nets), apply to even to RNN or LTSM, or transformers?

• Regarding the questions about feedforward networks, a really short answer is that regression is a very limited form of inference-time computation that e.g. rules out using memory. (Of course, as you point out, this doesn’t apply to other 2020 algorithms beyond MLPs.) Sorry about the lack of clarity—I didn’t want to take up too much space in this piece going into the details of the linked papers, but hopefully I’ll be able to do a better job explaining it in a review of those papers that I’ll post on LW/​AF next week.

(I also want to reply to your top-level comments about the evolutionary anchor, but am a bit short on time to do it right now (since for those questions I don’t have cached technical answers and will have to remind myself about the context). But I’ll definitely get to it next week.)

• Thanks for the responses, they give a lot more useful context.

(I also want to reply to your top-level comments about the evolutionary anchor, but am a bit short on time to do it right now (since for those questions I don’t have cached technical answers and will have to remind myself about the context). But I’ll definitely get to it next week.)

If it frees up your time, I don’t think you need to write the above, unless you specifically want to. It seems reasonable to interpret that point on “evolutionary anchors” as a larger difference on the premise, and that is not fully in scope of the post. This difference and its phrasing is more disagreeable/​overbearing to answer, so it’s also less worthy of a response.

• You may be interested in this convo I had about research on pedagogical models. The tl;dw if you just want the interventions that have replicated with large effects sizes:

1. Deliberate practice

2. Lots of low stakes quizzing

3. Elaboration of context (deliberately structuring things to give students the chance to connect knowledge areas themselves)

4. Teaching the material to others (forcing organization of the material in a way helpful to the one doing the teaching, and helping them identify holes in their own understanding)

• Including for this contest; we’d love to hear general feedback, and are also interested in hearing about any cases where a submission (or our reviews) changed your mind or actions. You might also want to tell the author(s) of the submission if this happens.

Hi, Lizka.

I’m curious about your mention of reviews. Were reviews written for each contest submission?

• 1 Oct 2022 4:12 UTC
81 points
22 ∶ 0

Just also want to emphasise Lizka’s role in organising and spearheading this, as well as her conscientiousness and clear communication at every step of the process—I’ve enjoyed being part of this, and am personally super grateful for all the work she has put into this contest.

• The prospect of Russia’s grotesque invasion of Ukraine leading the use of beyond conventional weapons has produced anxiety in the past. I don’t think this anxiety is justified.

Recently, there’s been media reports suggesting the possibility of this escalation again, after sham claims of annexing Ukraine territory.

Despite these media reports, the risk of this escalation is low. The report here from the “ISW” points this out, with detailed considerations.

https://​​www.understandingwar.org/​​backgrounder/​​special-report-assessing-putin%E2%80%99s-implicit-nuclear-threats-after-annexation

• 1 Oct 2022 3:02 UTC
34 points
2 ∶ 0

As requested, here are some submissions that I think are worth highlighting, or considered awarding but ultimately did not make the final cut. (This list is non-exhaustive, and should be taken more lightly than the Honorable mentions, because by definition these posts are less strongly endorsed by those who judged it. Also commenting in personal capacity, not on behalf of other panelists, etc):

Bad Omens in Current Community Building
I think this was a good-faith description of some potential /​ existing issues that are important for community builders and the EA community, written by someone who “did not become an EA” but chose to go to the effort of providing feedback with the intention of benefitting the EA community. While these problems are difficult to quantify, they seem important if true, and pretty plausible based on my personal priors/​limited experience. At the very least, this starts important conversations about how to approach community building that I hope will lead to positive changes, and a community that continues to strongly value truth-seeking and epistemic humility, which is personally one of the benefits I’ve valued most from engaging in the EA community.

Seven Questions for Existential Risk Studies
It’s possible that the length and academic tone of this piece detracts from the reach it could have, and it (perhaps aptly) leaves me with more questions than answers, but I think the questions are important to reckon with, and this piece covers a lot of (important) ground. To quote a fellow (more eloquent) panelist, whose views I endorse: “Clearly written in good faith, and consistently even-handed and fair—almost to a fault. Very good analysis of epistemic dynamics in EA.” On the other hand, this is likely less useful to those who are already very familiar with the ERS space.

Most problems fall within a 100x tractability range (under certain assumptions)
I was skeptical when I read this headline, and while I’m not yet convinced that 100x tractability range should be used as a general heuristic when thinking about tractability, I certainly updated in this direction, and I think this is a valuable post that may help guide cause prioritisation efforts.

The Effective Altruism movement is not above conflicts of interest
I was unsure about including this post, but I think this post highlights an important risk of the EA community receiving a significant share of its funding from a few sources, both for internal community epistemics/​culture considerations as well as for external-facing and movement-building considerations. I don’t agree with all of the object-level claims, but I think these issues are important to highlight and plausibly relevant outside of the specific case of SBF /​ crypto. That it wasn’t already on the forum (afaict) also contributed to its inclusion here.

I’ll also highlight one post that was awarded a prize, but I thought was particularly valuable:

Red Teaming CEA’s Community Building Work
I think this is particularly valuable because of the unique and difficult-to-replace position that CEA holds in the EA community, and as Max acknowledges, it benefits the EA community for important public organisations to be held accountable (and to a standard that is appropriate for their role and potential influence). Thus, even if listed problems aren’t all fully on the mark, or are less relevant today than when the mistakes happened, a thorough analysis of these mistakes and an attempt at providing reasonable suggestions at least provides a baseline to which CEA can be held accountable for similar future mistakes, or help with assessing trends and patterns over time. I would personally be happy to see something like this on at least a semi-regular basis (though am unsure about exactly what time-frame would be most appropriate). On the other hand, it’s important to acknowledge that this analysis is possible in large part because of CEA’s commitment to transparency.

• 1 Oct 2022 2:52 UTC
9 points
0 ∶ 0

I’m curating this post — thank you so much for writing it.

I agree with other commenters that replication is extremely precious, and I think this post chooses an excellent work to replicate — something that is quite influential for discussions about whether we should prioritize economic growth or more direct types of global health and wellbeing interventions. (Here’s a pretty recent related piece by Lant Pritchett.) I also really appreciate that the conclusion about economic growth seems to rely on three very different but independently strong arguments (straightforward estimation of impact given Easterlin’s values, noting that the conclusions are very sensitive to small tweaks in the methodology, and suggesting that GDP interventions might be a better approach to improving wellbeing even if Easterlin’s interpretations are accurate).

Re: the discussion on tractability, I want to note that most problems [seem to] fall within a 100x tractability range (assuming that effort on the problems has ~logarithmic returns, which seems very roughly reasonable for, say, research on economic growth or better global health interventions). (“For a problem to be 10x less tractable than the baseline, it would have to take 10 more doublings (1000x the resources) to solve an expected 10% of the problem. Most problems that can be solved in theory are at least as tractable as this; I think with 1000x the resources, humanity could have way better than 10% chance of starting a Mars colony, solving the Riemann hypothesis, and doing other really difficult things.”) If I’m interpreting things correctly, I think this means a more plausible reason other interventions might be more impactful is if they’re much more neglected (rather than much more tractable). Alternatively, we should simply not expect them to be more impactful. (Disclaimer: I read the tractability post quite a while back, didn’t follow the links in this post, and didn’t try very hard to understand the parts that I didn’t understand after a first read. I also don’t have any proper expertise in economics, so I might be getting things significantly wrong. I’m also writing quickly while tired.)

Finally, for those who like Our World in Data charts (and for those who’d appreciate a reference on what we should expect in terms of the relationship between GDP and measures of happiness) — here’s a chart showing self-reported life satisfaction vs GDP per capita in different countries (note that this is different from Easterlin’s approach for the paradox, which looks at differences in GDP and happiness within countries over time):

• Thanks so much for the kind words.

I think the question of the the plausible range for tractability is an interesting one. I suspect that most global health interventions seriously considered by EA fall within a 100x range. But I would guess that the reason this is true is that the only interventions with enough evidence are already in the process of solving more than 0.5% of the problem. At the other end of the spectrum, I suspect intervention trying to influence the very long term trajectory of human culture might fall into a range that spans at least 6 orders of magnitude. There are probably plenty of interventions we could consider that we should expect to have much less than a one in a million chance of solving 10% of the problem. Because there is little evidence and feedback for what would work in this context, we should not expect most things we consider to have a non-tiny chance of working.

I am also a little skeptical of how much information we get out of neglectedness when working with these sorts of problems. I think something being neglected might often be a sign that experts in the space don’t consider the approach plausible, or that some experts have tried it and given up on it. If that is the case, then that effect may swamp the diminishing marginal returns we might expect. Additionally, diminishing marginal returns might not be as common in fields where it’s not obvious what the next good thing to do is (because there are poor feedback mechanisms).

• 1 Oct 2022 2:44 UTC
19 points
5 ∶ 0

The BioAnchors review by Jennifer Lin is incredible. Has it ever been shared as an independent post to LessWrong or EAF? I think many people might learn from it, and further discussion could be productive.

• 1 Oct 2022 2:09 UTC
8 points
2 ∶ 0

This is very useful both for the content and for examples of how to effectively structure constructive criticism. Will there be a write up on lessons learned from the contest?

I think that could be quite useful as well since there is interest in running more contests, like the FTX FF AI Worldview Prize.

• [ ]
[deleted]
• Yes, I think you are right. Sorry, I made too broad of a statement when I only had things like strength and speed in mind.

• When will you next be open for applications?

• Maybe a feature to let Google Doc headers be switched automatically to EAF headers? This will be mildly useful to me, and considering the most common type of broken links I see from others on the forum, probably to others as well!

• In a scenario where this tech works as well as we are dreaming of and has generalized in hundreds of millions of buildings: isn’t there a risk of a general weakening of our immunity systems, making us?

Basic logic behind this question: certain/​many classes of virus eradicated from many modern buildings ⇒ our bodies are generally less prepared to encounter it in other settings ⇒ the day the setting changes, we are especially vulnerable.

(epistemic status: I’m very ignorant in these fields)

• The lethal autonomous weapons (LAW) case seems fundamentally different, unfortunately, for two reasons:

In the case of CBW we had the work of a dedicated group of scientists persuading Congress, the people, and the government that biological weapons would not be useful. It will be a lot harder to argue that LAWs cant be useful. As devices that are wholly man-made I’d think there will always be a significant group of scientists and engineers arguing they just need to be better built to provide huge value. Unlike CBWs you don’t have complicated issues of containment to worry about. Unlike CBWs, with LAWs the potential for creative fine-grained designs seems boundless since you’re not restricted by what a biological agent is capable of. LAWs are only restricted by physics and how well your AI is programmed.

Secondly, the public and Congressional pressure on Nixon I see as largely stemming from the “creep” factor of CBWs. These are weapons that bring to the public’s mind horrendous diseases gone rampant—something we as a species have deep evolutionary and culturally ingrained reasons to be creeped out about, both aesthetically and morally.

LAWs don’t give that same creep factor. If anything, they give a “cool” factor, probably as a result of being wholly man-made. So I have a hard time seeing concerned scientists having anything but a very long upward battle trying to convince the public and Congress that LAWs should be banned analogously to the biological and chemical weapons conventions. This is just making me wonder if it might be intrumentally better to argue for LAWs being banned without arguing they are analogous to CBWs. I don’t have expertise in this area, but this is what is coming to my mind.

sidenote: I don’t believe you explicitly defined “CBW.” I am inferring BW = Biological Weapons. What does the C stand for? “Conventional”?

• First just want to flag that I don’t have extremely high confidence in the last section in general, it wasn’t nearly as researched as the rest.

I agree there are a number of disanalogies, most specifically that it does seem like biological weapons are straightforwardly less useful than lethal autonomous weapons. In this sense maybe LAWS are more like chemical weapons, which were at least claimed to be useful (though probably still not as useful), but were also eventually banned.

I’m not sure I agree about the creep factor. I think it’s possible to make LAWS “creepy;” at least, watching the Slaughterbots documentary felt creepy to me. I think it’s true they could be “cooler” though; I can’t imagine a biological weapon being cool.

I don’t believe you explicitly defined “CBW.”

Thanks, fixed. It stands for “chemical and biological weapons.”

• The history here is great, thanks for posting

• The google doc link doesn’t seem to be working for me.

• Love this, great work. I especially appreciate your honest opinions on what mistakes you think you made and how the survey could have been improved. If JERIS continues next year, those thoughts will enable a lot of improvement!

• Hi Inga,
I have a degree in clinical psychology, am a therapist and have conducted research on human wellbeing. Very interested in getting involved! I keep meeting people who say that we should meet! Looking forward to getting involved and seeing where this goes!

• Is there a specific cutoff day to determine if a post has enough karma to be featured in the newsletter?

• I think this is an excellent area to focus on—though I am maybe biased in that I favor quality of life interventions over quantity of life interventions (one might say that I find the Repugnant Conclusion especially repugnant).

My main curiosity as regards iodine supplementation specifically is whether it is currently neglected enough to be a good cause area. That it can be dramatically efficient when successful is pretty clear I think, but it’s also an area where many governments do make ongoing efforts (for example, India has a National Iodine Deficiency Disorders Control Programme). Are there private organizations that do good work in filling in the gaps or compensating for the failures in these government programs?

• 30 Sep 2022 19:08 UTC
2 points
0 ∶ 0

This is great.

A few musings/​nitpicks:

I wouldn’t use the words “quick trick”. For some people it won’t be quick or easy, and in my view it’s importantly not a “trick” either. When it ‘works’, it works exactly because it’s not a trick! (Of course other factors have to line up as well.) I suspect the underlying theory of change and values that guide people will be very different if they view this as a trick rather than if they have a deeper understanding of why this way of engagement can have very high returns.

Non-self-hatingness: Thinking that you are doing something valuable, that matters to you, that you don’t have to apologize for caring about, along with its implications

I wouldn’t call this non-self-hatingness, although self-hatred may heavily hamper what you’re describing. What you’re describing sounds closer to having genuine belief in the value of what you’re doing, or perhaps some amount of willingness to trust yourself in the face of criticism. What do you think?

I basically agree with the skills you say are required to execute this well, but I would phrase it in a softer way than using the word “requirement”—it makes it sound too binary and might unwittingly discourage some people by making it sound too fancy or requiring skills they don’t have and don’t know how to develop. Whereas the core idea is pretty simple and human (albeit with a wide range of difficulty of execution). I also think people can develop these skills, even if they start at a low level.

• [ ]
[deleted]
• Thanks for this. It has rocked some of my priors—I don’t believe in Easterlin’s paradox anymore. On the other hand:

The health regression estimates how much a decrease in the number of years people in a country lose to ill health corresponds to increases in happiness. This regression produces coefficients that are either an order of magnitude smaller than the GDP regression, or negative, depending on whether we exclude countries that have less than 12 years of data.

I agree that, by modus tollens, this shows one should be suspicious of Easterlin’s overall argument. But I think that this metric is not the best proxy to measure increase in welfare due to better health. I’d be surprised if lives saved increased average life satisfaction. Even from an individual point of view, and even though most people have a strong preference for surviving and usually see their lives as net positive, I don’t think that avoiding death makes the person much happier quite often people surviving accidents and disease become traumatized or more careful about life).

Also, I’d be astonished if it made this person happier than the population mean – which is required for us to observe an increase in average life satisfaction. I guess the people most likely to die are precisely those who tend to be less “happy” in most metrics (e.g., depressive individuals, or the poor, or people with health conditions…); so one can intuitively conceive of a reduction in average happiness by decreasing some deaths. Thus, unless one adopts some weird position in population ethics (or bites the bullet and becomes a hedonic utilitarian villain willing to let sad people die), I don’t think it’s bad news that your “r-squared of the regression is essentially 0”.

• That’s an interesting point. It sounds like you are probably right that we shouldn’t expect better health to improve life satisfaction. I could try to argue that everyone being a little healthier for the first 70 years of their life would mean less unemployment, loneliness and chronic pain, and that those effects would be larger than the ones you bring up. But I don’t think that’s likely.

I also agree with you that the main value of better health is just in averting years of life lost. In fact, I think global health interventions are likely to be better than growth interventions for this reason.

The reason I ran the health regression (as it seems you surmised) was because that was something that people have brought up as something that might be better than gdp growth at boosting happiness, and we have good longitudinal data on it.

• I also agree with you that the main value of better health is just in averting years of life lost. In fact, I think global health interventions are likely to be better than growth interventions for this reason.

The reason I ran the health regression (as it seems you surmised) was because that was something that people have brought up as something that might be better than gdp growth at boosting happiness, and we have good longitudinal data on it.

Agreed, and I commend you for this.

On the other hand, I’m afraid we might be talking past each other here (though I’m not sure how important this is):

you are probably right that we shouldn’t expect better health to improve life satisfaction

I didn’t say that—my fault, I think I wasn’t clear enough. I’m pretty sure I would be less satisfied with life if my health was worse, and I think one can extrapolate that for most people—just ask them, or observe how their satisfaction decreases after they get sick, or compare sick and healthy people, etc.

What I said is that my average satisfaction wouldn’t increase if my life was saved. And I meant that your regression using “number of years people in a country lose to ill health” is probably a poor proxy for what you’re trying to measure (i.e., better health vis-a-vis life satisfaction). This is because your metric health years is probably dominated by averted deaths (especially if one is tracking marginal improvements), and average life satisfaction only measures the welfare of those who are alive—because the dead don’t speak.

Take world1: I suddenly die of disease on my 6th birthday, and world2: my life is saved on my 6th birthday. There’s no reason to believe that:

• a) I’d happier than usual (i.e., the average) on the day following my birthday in world2, or that

• b) my average happiness in world1 is smaller than in world2 (either measured on 6th or on my 30th birthday).

(I now realize this might explain why some scholars use increases in mean height to measure the effects of health improvements—because this is likely more robust)

• I think we are on the same page here. I was just using “improve life satisfaction” as shorthand for “improve average life satisfaction across the whole population.”

• Epistemic status: a few projects in ML, technically “a professional ML engineer” right now but I don’t think I’m good enough to get hired by one of the big EA-ML orgs right now.

Two points re the start of your math sequence

1. A lot of calculus is geared toward engineers and the way engineering is taught, leaving a gap in the basic language of mathematics that IME can be filled by discrete math. This pays dividends when you’re reading wikipedia and research papers later. It doesn’t take long to get ahold of sets and logic with trevtutor.

2. Single variable calculus may require exercises. I find it really odd that you think 3b1b gets the job done- it probably doesn’t. Luckily, single variable calculus is well before you’ve maxed out khanacademy’s wonderful exercises widget. Some ways of doing the rote calculations is a waste of time (like professors who don’t seem to realize that if you’ve derivative’d one polynomial you’ve taken every polynomial), but the rote computations really pay dividends in your overall sophistication level (from the symbolic find-and-replace game to mystery solving even to thinking on your feet about applications.

• I ran the UChicago x-risk fellowship this summer (we’d already started by the time I learned there was a joint ERI survey so decided to stick with our original survey form).

I just wanted to note that, for the fellows who weren’t previously aware of x-risk, we observed a dramatic increase in how important fellows thought x-risk work was and their reported familiarity with x-risk. As well, many indicated in the written responses an intention to work on x-risk related topics in the future where they previously hadn’t when responding to the same question. We exclusively advertised to UChicago students for this iteration and about 23 of our fellows were new to EA/​x-risk.

• Thanks for this comment—if you do run the UChicago fellowship again, we should definitely coordinate on joint impact assessment surveys.

Your finding about less x-risk-aware fellows becoming dramatically more so is very promising. It is also somewhat different from what the surveys show; I imagine multiple factors would affect reported familiarity with x-risk (e.g., types of events you ran, nature of the projects, etc.), and I would be keen to discuss this further with you over a call at some point.

To put this in context for CERI, we received 650+ applications for ~24 places, so we might have filtered for prior x-risk engagement to a greater extent than you did. We probably also had different theories of change, and it will be interesting to compare our approaches.

• “If the basic idea of long-termism—giving future generations the same moral weight as our own—seems superficially uncontroversial, it needs to be seen in a longer-term philosophical context. Long-termism is a form of utilitarianism or consequentialism, the school of thought originally developed by Jeremy Bentham and John Stuart Mill.

The utilitarian premise that we should do whatever does the most good for the most people also sounds like common sense on the surface, but it has many well-understood problems. These have been pointed out over hundreds of years by philosophers from the opposing schools of deontological ethics, who believe that moral rules and duties can take precedence over consequentialist considerations, and virtue theorists, who assert that ethics is primarily about developing character. In other words, long-termism can be viewed as a particular position in the time-honored debate about inter-generational ethics.

The push to popularize long-termism is not an attempt to solve these long-standing intellectual debates, but to make an end run around it. Through attractive sloganeering, it attempts to establish consequentialist moral decision-making that prioritizes the welfare of future generations as the dominant ethical theory for our times.”

This strikes me as a very common class of confusion. I have seen many EAs say that what they hope for out of “What We Owe the Future” is that it will act as a sort of “Animal Liberation for future people”. You don’t see a ton of people saying something like “caring about animals seems nice and all, but you have to view this book in context. Secretly being pro-animal liberation is about being a utilitarian sentientist with an equal consideration of equal interests welfarist approach, that awards secondary rights like life based on personhood”. This would seem either like a blatant failure of reading comprehension, or a sort of ethical paranoia that can’t picture any reason someone would argue for an ethical position that didn’t come with their entire fundamental moral theory tacked on.

On the one hand I think pieces like this are making a more forgivable mistake, because the basic version of the premise just doesn’t look controversial enough to be what MacAskill actually is hoping for. Indeed I personally think the comparison isn’t fantastic, in that MacAskill probably hopes the book will have more influence on inspiring further action and discussion than on changing minds about the fundamental issue (which again is less controversial, and which he spends less time in the book on).

On the other hand, he has been at special pains to emphasize in his book, interviews, and secondary writings, that he is highly uncertain about first order moral views, and is specifically, only arguing for longtermism as a coalition around these broad issues and ways of making moral decisions on the margins. Someone like MacAskill who is specifically arguing for a period where we hold off from irreversible changes as long as possible in order to get these moral discussions right really doesn’t fit the bill or someone trying to “make an end run around” these issues.

• Hi there, as a fellow EA, developer and avid creator of Userscripts, here are my thoughts on first seeing the site.

The design is very different from other online communities. This makes for an awkward first impression, users like familiarity in their UI.

I believe the gold standard for forums are Reddit, Facebook, StackOverflow, Discourse. By gold standard I mean some of the best minds in software UX works on these site. I particularly love Discourse.

This is a forum, yet there are no topics /​ subtopics. It tries to do too much in one place. I don’t think questions, articles and events belong in in the same listing. I am aware of the filters, my criticism still stands :-)

Everywhere I move the mouse I’m assaulted by a popup. Why do you hate me? :-D

Infinite scroll /​ load more adds uncertainty to the UX. It’s hard to track context, I can’t tell if I click somewhere all my “progress” will be lost.

Gray on gray! No gray background please!

Titles are long, yet the columns are narrow.

The comments font looks bold, it should be lighter.

Some pages have too much info. “How to use the Forum” shouldn’t have a pages long comments section, specially with unrelated discussions.

Still, thank you for taking the time in trying to innovate and contribute to the OS community!

• It does sadly look very broken for me:

It does look better on the all-posts page:

Some thoughts

• I like the idea of making the text smaller and increasing the density of the post list. Seems good to experiment with

• I think getting rid of the grey background really breaks a lot of the recent discussion section as well as the overall navigability of the UI (and also we’ve gotten tons of user feedback that people found the perfect white as the whole background to feel quite straining on their eyes).

• I do overall think the font is just too small for me to read. I expect most users would zoom in a decent amount in order to actually make it comfortable to skim.

• I think having line-breaks in the post-titles is quite bad for skimming, and also gives undue attention to posts that have longer titles, which seems quite bad.

• While I do find it easier to skim to move the post-icons to the left of the items, I think it gets the information hierarchy wrong. I think the type of post (link post, curated, personal blog) is at best a secondary piece of information, and the design you proposed gives it too much prominence.

• Yeah, it messes up a few other pages as well. To be fixed.

I think the site needs a dark mode. More and more people are favoring it. I use my monitor in a nearly yellow tone, redshift -O 2800k so I like the white background just fine. I can’t get behind the gray background though. I mean, how many sites does that? I find it harder to read.

The font I used could be one size larger, I did made an alternate screenshot to compare. Yet research suggests the current font size, not the one from my script, is ideal. I still favor higher density, as I can analyze the content faster.

Regarding skimming, I read titles by rows, not lines. I think we’ve been conditioned for this. Just look at Reddit or Medium. I find it easy to read a few words and skip to the next row. The title is too important to be trimmed away, I would sooner hide the author, date an comments count. I think it’s very hard to find a site with this few characters in a title.

I haven’t used the site enough to give a proper opinion on the icons. I think they either should be used more or hidden altogether. But I mix my feelings regarding topics, something I didn’t touch yet. They will either be on the left of the title, on the end of the line, or below the titles, in a smaller font. I can’t tell you how much I want to see 50 titles at a time and instantly know where they fit. Blue tagged AI, green tagged Animal Wellfare, etc.

I plan on enhancing my script as I spend more time here. It might take a while. I mostly wanted to take a feel if my experiences are in line with others. I’m happy to keep my preferences as a userscript and give the users another choice.

• I think the site needs a dark mode. More and more people are favoring it.

The site already has one! Or more precisely LessWrong has one, and it probably wouldn’t be too hard to adapt it to the EA Forum (which shares a codebase).

The font I used could be one size larger, I did made an alternate screenshot to compare. Yet research suggests the current font size, not the one from my script, is ideal. I still favor higher density, as I can analyze the content faster.

I am generally skeptical of research in this space, but yeah, the current font size is what seems to work pretty well in user tests I’ve done. I do also think sometimes it makes sense to have more density and smaller font sizes (and like, comment text is already almost that small)

I can’t get behind the gray background though. I mean, how many sites does that? I find it harder to read.

The pattern of “grey background with white boxes in front, occasional header or nav element on the grey background” is as far as I can tell the standard pattern to reduce eye fatigue while also ensuring high text contrast. I actually can’t think of a content heavy site that doesn’t do this.

• I’m confused why the all white background is better, grey is easier on the eyes and the non-white color gives a natural framing to the other content. Both points seem pretty normal in design.

I disagree that those other sites are superior. Also a major issue is that they use visual/​video content (reddit and FB) and have different modes of use/​seeking attention. They are designed around a scrolling feed, producing a constant stream of content, showing 1-3 items at a time.

Setting the above aside, I’m uncertain why your changes reflect ideas from them. For example, your changes to text, make posts much more compact than Reddit or SO.

• “Grey is easier” I don’t think it is. Would you disagree that most publications use a white background? Could you provide at least some examples of ones that doesn’t?

“I disagree that those other sites are superior.” We would have to define superior. For me, the best (most well paid) minds in UX + the most number of users are objective measures. That doesn’t mean we have to copy them, but it beckons to the familiarity factor.

I agree that they have a constant stream of content and this matters on design. What use is to have 50 compacted posts that I can scan in 1 second, if we have 30 posts a week? It is unfortunate that we don’t have a higher traffic. I believe in reducing barriers of entry to help on this, and making a familiar site is but a very small of those.

To your third point, open a screenshot of my version, the current design here and any of them. See you can spot the ideas I try to incorporate. I don’t know your background, but I can give you a a technical response. Fonts, spacing, that kind of thing. I basically copied the typography from them, while keeping the site identity and adding a few of my preferences.

Please note I did that in about 4 hours of work. The gross of it was very fast, some details took very long. 1 hour I spent fighting the pop ups before deciding to disable them

• “Grey is easier” I don’t think it is. Would you disagree that most publications use a white background? Could you provide at least some examples of ones that doesn’t?

I checked two sites that you listed, FB and StackExchange, and they literally use a grey/​off white background. Started with these two and I stopped after checking these two, I suspect I’ll find more.

The stackexchange site literally answered this very question and one answer pointed out that the very site is off-white (although less than grey or the EA forum). The top answer here supports grey backgrounds: https://​​ux.stackexchange.com/​​questions/​​23965/​​is-there-a-problem-with-using-black-text-on-white-backgrounds?rq=1.

Before I thought this opinion about the use of grey (and avoidance of high contrast) was normal/​standard before. Now I’m even more sure, and not knowing/​ opposing about it seems sort of strange to me.

“I disagree that those other sites are superior.” We would have to define superior. For me, the best (most well paid) minds in UX + the most number of users are objective measures. That doesn’t mean we have to copy them, but it beckons to the familiarity factor.

There’s a lot going on here, but IMO neither of those things make this view very promising. This is because they are designed for MAU/​growth hacking and the audience is different (and I don’t think this is some elite or niche thing). Also, since the business is multiple billions are year, you naturally get top talent.

As an analogy, tabloids are popular and well designed for their audience, but that doesn’t make them dominant design choices. I do agree that the design on average is good and things work for those sites.

Also, I suspect some design choices from those sites have dependencies—I think having an infinite scroll or video or picture focus would affect other design choices, such as size/​position/​font of text, so copying those design choices to a forum might not be appropriate without more sophistication.

I don’t want to be disagreeable or press too much here on you here. Honestly I want to learn about design and different perspectives, but I don’t think I am?

Some of the other things you said suggested you have strong views that seemed more personal and also that you use some unusual color filter? This makes me speculate that you are applying a personal perspective disproportionately and ignoring “the customer”, but maybe this is unfair.

• ??? Yeah, Reddit’s design literally uses a grey background. It’s darker than the EA forum.

• You’re talking about the framing. Sorry, I didn’t realize. It’s not among my concerns to the site. Yes, It’s a preference. There are a few main trends regarding framing, I’m on the one against it. Gray on gray refers to the comments section, and any other place where there is a gray background and a “gray” font. It is not an unusual choice, I just don’t find it the best. As an argument, you read articles in a white background, why comments should have gray, aside from structural purposes?

Regarding audience, I kind of disagree. Yes, the audience here is not the same of that of Reddit. And I think this should change. Still I’d like to see a site like this. It literally created its own engine! Which is awesome by the way. I love VulcanJs. Here is an example of what I would like to see on hitting the main page: https://​​forum.vuejs.org/​​.

Just for reference, I have 20 years as a developer, and I have been part in maybe hundreds of design discussions, even though I’m a front/​back end developer. So, no expert but I’m somewhat on the loop. The changes I propose are a mix of personal choices and experience/​research based opinions.

Also, any discussion of familiarity starts with mobile, which I don’t use. My focus is mainly on the 1080p 24inch desktop experience.

• Consider adding the Berkeley Existential Risk Initiative (BERI) to the list, either under Professional Services or under Financial and other material support. Suggested description: “Supports university research groups working to reduce x-risk, by providing them with free services and support.”

This strategy is particularly evident in discussions of artificial intelligences risks and benefits. Developers and investors hope that by persuading the public that the really “big” threat is being addressed, they will be sanguine about more immediate problem and shortcomings. They hope to create the impression that harms being done today are worth enduring because they will be far outweighed by the benefits promised for tomorrow when the technology matures. Such a strategy masks the possibility that the longer term risks will far outweigh the short term benefits of specific applications.

It is no coincidence that institutes working, for example, to anticipate the existential risks of artificial general intelligence get much of their funding from the very same billionaires who are enthusiastically pursuing the development of cutting-edge AI systems and applications. Meanwhile, it is much harder—if not impossible—to get funding for research on those cutting-edge applications that are being applied today in ways that boost profits but harm society.

The well-intentioned philosophy of long-termism, then, risks becoming a Trojan horse for the vested interests of a select few. Therefore we were surprised to see this philosophical position run like a red thread through “Our Common Agenda,” the new and far-reaching manifesto of United Nations Secretary-General António Guterres.

The authors are suggesting that AI safety risk research and sponsorship is similar to greenwashing, just a facade to hide the short-term goals of AI technology developers.

• 30 Sep 2022 17:04 UTC
2 points
0 ∶ 0

Huge GWWC fan here—excited to do exactly this!

• 30 Sep 2022 17:04 UTC
9 points
1 ∶ 0

Thank you so much for asking this question, @smountjoy!

Wanted to put in a pitch for Legal Impact for Chickens.

Legal Impact for Chickens would be extremely honored and excited if any donor were to consider us for a gift of any size! And I believe the money would go a long way in terms of expected value to reduce suffering.

We’re an EA-aligned litigation start-up dedicated to making factory-farm cruelty a liability.

LIC filed our first lawsuit a few months ago: A widely publicized shareholder-derivative case against Costco’s executives for neglecting chickens. Costco breeds and kills 100 million chickens each year. And of course, in the US alone, companies kill 9 billion chickens a year. If we can improve these animals’ lives, that could reduce a huge amount of suffering.

We have room for more funding. We want to grow to become a 6-person organization by 2024. Right now, we have two full-time staff: (1) me and (2) our amazing new litigator hire, Denise Morris, who starts on Monday.

My name is Alene. I’m the founder and I’m very friendly! If anyone is at all interested in potentially donating, please reach out to me, because I’d love to speak with you, get to know you, and tell you all about our organization! I think most of our donors are EAs. Thank you!!

And thank you to all the EA Forum readers who already support Legal Impact for Chickens! You rock!!

And thank you to the rest of the readers of this forum for everything you do to make the world better as effectively as possible!

Love,

Alene

PS I very much agree with @smountjoy about the importance of smaller donors.

First, that’s where a lot of money comes from!

Second, that’s where a lot of money SHOULD come from, according to the US Internal Revenue Service (IRS). If you’re a US charity, then you’ll probably want to get “public charity” status from the IRS so that you can receive tax-exempt donations and spend them on your own programs. For this status, you need to prove that a certain percentage of your support comes from SMALL donors, not just big foundations! So there’s a special legal reason to court individual donors. But even besides the legal reason, I think the IRS has a point: It’s more democratic for our charities to receive support from a large number of donors than just a few. It helps you know that your charity is listening to many opinions and being open to feedback.

So individual donors matter a lot, both for the money they provide and for the way they “vote” with their money.

LIC is proud to have a total of 213 unique donors. And the number of donors matters to us as well as the quantity of donations—because each donor represents a kind person trying to help our mission and believing that we can be entrusted with their money.

• Live reading and synthesis of (the top/​EA Forum part of) this post is up on the EA Forum podcast, read by Coleman Snell. anchor link

He/​we will be doing this for future weeks, and probably creating further content. Other ‘live reading and discussion’ of EA Forum content is in the works, stay tuned.

• What do you feel are the downsides of someone spending, say, 2 years more in skilling up than they actually should? Besides grantmaker money paying for salary (which I agree is valuable and worth preserving), I’m not sure I see a big downside. 2 years for instance is a short span of time relative to the lifetime you could devote to study, it’s also small relative to AI timelines (unless your timelines are really 2035 median or closer).

• Note that I put up a version of this summary, read and synthesized by Coleman Snell, on the EA Forum podcast (Anchor link: here Spotify link here … available on all podcast platforms).

He/​we will be doing this for future weeks, and creating further content. Other ‘live reading and discussion’ of EA Forum content is in the works, stay tuned.

Epistemic effort: Four hours of armchair thinking, and two hours of discussion. No literature review, and the equations are intended as pointers rather than anything near conclusive.

Currently, the status quo in information sharing is that a suboptimally large amount of information hazards are likely being shared. In order to decrease infohazard sharing, we have modeled out a potential system for achieving such a goal. As with all issues related to information hazards, we strongly discourage unilateral action. Below you find a rough outline of such a possible system and descriptions of its downsides. We furthermore currently believe that the for the described system, in the domain of biosecurity the disadvantages likely outweigh the advantages (that’s why we called it Bad IDEA), and in the domain of AI capabilities research the advantages outweigh the disadvantages (due to suboptimal sharing norms such as “publishing your infohazard on arXiv”). It’s worth noting that there are potentially many more downside risks that neither author thought of.

Note: We considered using the term sociohazard/​outfohazard/​exfohazard, but decided against it for reasons of understandability.

# Current Situation

• Few incentives not to publish dangerous information

• Based in previous known examples

• We’d like a system to incentivize people not to publish infohazards

# Model

• Researcher discovers infohazard

• Researcher writes up description of infohazard (longer is better)

• Researcher computes cryptographic hash of infohazard

• Researcher sends hash of description to IDEA

• Two possibilities:

• Infohazard gets published

• Researcher sends in description of infohazard

• Bad IDEA computes the cryptographic hash and compares the two

• Researcher gets rewarded according to the reward function

• Bad IDEA deletes the hash function and description of infohazard from their database

• Researcher wants intermediate payout

• All of the above steps, except IDEA doesn’t delete the hash function from the database, but does delete the the description of the infohazard

## Reward Function Desiderata

We know it’ll be Goodharted, but we can at least try.

• Reward function is dependent on

• Danger of the infohazard (d)

• Reward higher if danger is higher

• Time between discovery & cashin (t)

• Reward higher if time between discovery & cashin is longer

• The number of people who found it (n)

• Lower payout if more people found it, to discourage sharing of the idea

• Reward being total_payout/​discoveries? Or something that increases total_payout as a function of independent discoveries?

• Latter case would make sense, since that indicates the idea is “easier” to find (or at least the counterfactual probability of discovery is higher)

• Counterfactual probability of it being discovered (p)

• This is really hard to estimate

• If counterfactual discovery probability is high, we want to reward higher than if it’s low.

• How difficult the idea is to discover

• Ideas that are “very” difficult to discover would be given less than “easy” ideas. This could potentially discourage strong efforts to research new information hazards

Individual payout for a researcher then is

Alternative version punishes actively for looking for infohazards

• Base rate of information hazard discovery → publication rate

• Incentive for people to research & create infohazards

• Might be counteracted by the right reward function which incorporates counterfactual discovery probability

• Gently sharing existence of IDEA with trusted actors

• Bad IDEA observers might remember infohazard

• Repository for malign actors to go & recruit

• This could be solved by (unrealistically)

• Real-world amnestics

• AI systems trained to estimate badness

• Estimating danger of information hazard is quite difficult

• Could be overcome through rough estimates on how many people would be capable of engaging with idea to do harm + how damaging it would be

• Estimating difference between ideas are difficult

• Attracting attention to the concept of infohazard

• Amazing resource, thanks so much! I’ll add that the Effective Institutions Project is in the process of setting up an innovation fund to support initiatives like these, and we are planning to make our first recommendations and disbursements later this year. So if anyone’s interested in supporting this work generally but doesn’t have the time/​interest to do their own vetting, let us know and we can get you set up as a participant in our pooled fund (you can reach me via PM on the Forum or write info@effectiveinstitutionsproject.org).

• 30 Sep 2022 16:26 UTC
9 points
4 ∶ 0

Unsolicited feedback: I love both of the posts you have done recently. IMO they identify problems and solutions without under or over-stating them, are emphatic and measured, and conclude with actionable and proportionate advice. Thank you so much :)

• Hey, I think you’ll get more relevant candidates if you

1. Put the position name in the title, it will let candidates know if clicking on this post is relevant for them

2. Elaborate on the roles (not just behind a link), to let candidates filter easily on things like “local/​remote” and decide if they want to read more. (you absolutely did this in your actual job post, I’d consider copying it here)

I hope this is helpful and hope you manage to hire great people!

• I have always thought that there is a lot of unpicked low-hanging fruit for animation among the most popular blogposts. Examples might be “I only believe in the paranormal” and “Great minds might not think alike”from lesswrong.

• I think you raise an interesting point around:

Slow takeoff + alignment solved + competitive dynamics + xrisk due to future technologies I vented with help AGI (in competitive world)

I tend to assume world government with high probability because of the number of ways in which future tech (possibly invented with help of AGI) can be used to snowball towards one, with or without the consent of all political actors at the time.

There is a taboo around discussing ways to attempt totalitarian power grabs in general so I won’t be doing that here.

• I’d also like to see a list of orgs that could use funding. I imagine there are many but I don’t think they are collated anywhere. Maybe big funders could release lists of organizations they didn’t fund for whatever reason and other smaller funders can fund them if they think it correct.

Separately, small note: you stated that Ben Todd mentioned this in 2019, but the post you link to is from 2021.

• [ ]
[deleted]
• I like the low-HDI country idea, I’ve been really taken with something I can’t find which gives you a random person and facts about that person [kids, religion, etc], weighted by actual probabilities.

• Excited to see GWWC, OFTW and HIP (where I work) cover the space this year and how much we can collectively raise for high-impact charities!

• I’d guess that a lot of non-longtermist, non-EA-meta charities are more more likely to be funding constrained and less likely to be topped up by FTX. I also suspect FTX isn’t taking up all the opportunities for organizations to spend money, even for the ones it supports.

I suspect organizations with a research focus, such as Sentience Institute, ALLFED, and other answers on this post, are often happy to hire more researcher time with marginal donations.

Organizations that do marketing probably have room to spend more there, such as 80,000 Hours and Giving What We Can. GWWC wrote earlier this year that they were looking for funding (I’m not sure what the status of that is).

I believe the Center for Election Science is looking for more funding since approval voting has a lot of room to grow in the US—it sounds like their goal is to scale campaigns with more funding.

I’m not sure how much room they have, but probably Effective Institutions Project.

• What’s the minimum sized audience that you’d be happy to present to?

• Hi Kevin, we’d probably be able to arrange for someone to speak if there was an audience of 10 or more! Obviously the bigger the better :) Or we could try and combine several smaller groups for a virtual event!

• Will is promoting longtermism as a key moral priority—merely one of our priorities, not the sole priority. He’ll say things like (heavily paraphrased from my memory) “we spend so little on existential risk reduction—I don’t know how much we should spend, but maybe once we’re spending 1% of GDP we can come back and revisit the question”.

It’s therefore disappointing to me when people write responses like this, responding to the not-widely-promoted idea that longtermism should be the only priority.

• [ ]
[deleted]
• 30 Sep 2022 10:08 UTC
22 points
2 ∶ 0

We try to keep a page with information (including room for funding numbers) for the organisations that get founded through Charity Entrepreneurship. Many of them are in a situation where marginal, small donors could make an impact.

• For Level 3: Machine Learning, this document might be useful. It provides a quick summary/​recap of a lot of the math required for ML.

• Thanks for making this. I expect that after you make edits based on comments and such this will be the most up to date and accurate public look at this question (the current size of the field). I look forward to linking people to it!

• Hello, at your suggestion the first thing I did after making an account was to post a bio. Thanks for the guidance!

• To me it seems they understood longtermism just fine and just so happen to disagree with strong longtermism’s conclusions. We have limited resources and if you are a longtermist you think some to all of those resources should be spent ensuring the far future goes well. That means not spending those resources on pressing neartermist issues.

If EAs, or in this case the UN, push for more government spending on the future the question everyone should ask is where that spending should come from. If it’s from our development aid budgets, that potentially means removing funding for humanitarian projects that benefit the worlds poorest.

This might be the correct call, but I think it’s a reasonable thing to disagree with.

• They understand the case for longtermism but don’t understand the proposals or solutions for longtermism aspirations.

One of the UN’s main goals is sustainable development. You can still address today’s issues while having these solutions have the future in consideration.

Therefore, you don’t have to spend most funds only addressing the long term future. You can tackle both problems simultaneously.

• You can only spend your resources once. Unless you argue that there is a free lunch somewhere, any money and time spent by UN inevitably has to come from somewhere else. Arguing that longtermist concerns should be prioritized necessarily requires arguing that other concerns should be de-prioritized.

If EA’s or the UN argue that longtermism should be a priority, it’s reasonable for the authors to question from where those resources are going to come.

For what it’s worth I think it’s a no-brainer that the UN should spend more energy on ensuring the future goes well, but we shouldn’t pretend that it’s not at the expense of those who currently exist.

In the early 2000′s when climate change started seriously getting onto the multilateral agenda, there were economists like Bjørn Lomborg arguing that we instead should spend our resources on cost-effective poverty alleviation.

He was widely criticized for this, for example by Michael Grubb, an economist and lead author for several IPCC reports, who argued:

To try and define climate policy as a trade-off against foreign aid is thus a forced choice that bears no relationship to reality. No government is proposing that the marginal costs associated with, for example, an emissions trading system, should be deducted from its foreign aid budget. This way of posing the question is both morally inappropriate and irrelevant to the determination of real climate mitigation policy.

Yet today, much (if not most) multilateral climate mitigation, is funded by countries’ foreign aid budgets. The authors of this article, like Lomborg was almost two decades ago, are reasonable to worry that multilateral organizations adopting new priorities comes at the expense of the existing.

I believe we should spend much more time and money ensuring the future goes well, but we shouldn’t pretend that this isn’t at the expense of other priorities.

• Thanks for this. I read or skimmed about 60-70% I think, and found about 20% pretty valuable. I like your Community Action and Community Consumption models and some of the graphics describing them.

A couple of questions:

• What’s your intended audience for this post?

• Have you thought about condensing some of the information in this post into something that’s closer to a 10-20 minute read?

• Hello Sphor, thanks for your comment! I intentionally made this a long-form post. I had already had around 10-15 more pages that I had cut out. If I had made it just a 10-20 minute read, it would have been just as generic, non-technical, and non-actionable as the rest of the “community improvement” discourse that surrounds EA. It pops up everyday on this forum, and these generic points can all be referenced as a point I’ve tackled in this series of essays.

As for intended audience, I think that is addressed in Section 5.9 . I think each community person within EA should read this and take a step back to see if they’ve realized what they’ve built and how they’re willing to contribute moving forward. The most impactful read would be someone within CEA that wants to take actions in what I’ve written. If not, I do believe many of my predictions and impending issues they will face will come true.

The only sections that are not EA specific are Sections 1-3. They will be expanded as community education that I’m developing and is what I’m taking with me from this contest. If EA wants to fix these things, they are more than welcome to, but it is not my job or duty. :)

• Hi Diana, thanks for the reply. To clarify I was sincerely curious to hear your responses and wasn’t trying to push you to put in even more work than you’ve already done. I agree both that there are serious issues with the EA community and that it’s up to the EA community to fix its issues.

I had already had around 10-15 more pages that I had cut out. If I had made it just a 10-20 minute read, it would have been just as generic, non-technical, and non-actionable as the rest of the “community improvement” discourse that surrounds EA.

I’m curious why you think this; it doesn’t seem true to me. You have plenty of concrete models and points and go into far more depth than most discourse. I could be wrong but I feel you could create a powerful shortish essay or presentation that focuses on your key points and models. I think this could make your material much more accessible and might help with your book as well. To be clear, I mean this as general feedback, not specific to EA, and I’m also not suggesting you do actually this, just raising it as a possibility.

• Got it! Yes—it’s on my roadmap for what I’m building to create more accessible content surrounding my ideas and at different reading comprehension levels (similar to what I suggested in Section 5.8.1) :).

I literally wrote this entire thing in 20 days so I was cramming it all out. The revisions will come soon and I will share the polished models /​ website /​ workshops /​ etc that’s shareable across communities—not just the EA one. My north star is making community building education accessible and implementable, this submission was just an exercise for me to test it out.

The reason why I think the discourse is generic is that every single EA community problem addressed boils down to how faulty the community was originally built. As listed in Section 5.1, the organization is based off of the EA models provided to community builders and I describe at length why their models should be debunked and further give instruction in how to mediate and change this. I provided the tools, action items, and education necessary to level up and the community discourse is cyclical and circular. They will keep running into the same problems until they change what I’ve listed in the essays.

I had considered making a list and referencing the top 50 community building/​discourse posts and allocating them in each appropriate part of my submission to show how they need and can be resolved by my suggestions, but I ran out of time. I could of course comment on every single post and reference which part of my submission where this is covered but alas—until it is wanted—the point is moot.

• I don’t think the arguments are fallacious if you look at how strong longtermism is defined:

Positively influencing the future is not just a moral priority but the moral priority of our time.

See general discussion hereand in depth discussion here

Perhaps they should have made that distinction since not all EAs take the strong longtermist view—including MacAskill himself who doesn’t seem certain.

• The article was on MacAskill’s book which doesn’t argue for strong longtermism but longtermism.

However, I think the acknowledgement and critique of strong longtermism is necessary.

• This is great. Was looking for stuff similar to Scott Alexander’s ‘Every Bay area house party’

• I think Alignment Forum double-counts researchers as most of them are not independent, especially if you count MATS separately (which I think had about 6 mentors and 31 fellows this summer). Looking at the top posts this year:

Paul Christiano works at ARC. Yudkowsky works at MIRI. janus is from Conjecture, Kraknova is at DeepMind, I don’t know about nostalgebraist, Nanda is independent, Larsen works at MATS, Ajeya is at OpenPhil. So 4-5 of the top authors are double-counted.

• I went through all the authors from the Alignment Forum from the past ~6 months, manually researched each person and came up with a new estimate named ‘Other’ of about 80 people which includes independent researchers, other people in academia and people in programs such as SERI MATS.

• This project would be valuable if the costs outweighed the benefits.

It could be relatively expensive (in person-hours) to run (there might be a tonne of publications to vet!) and relies on us being good (low false positive, high recall) at identifying biohazards (my prior is that this is actually pretty hard and those biohazardous publications would happen anyway). We’d also need to worry about incentivising people to make it harder to tell that their work is dangerous.

Biohazards are bad but preventing biohazards might have low marginal returns when some already exist. It’s not that any new biohazard is fine; it’s that marginal biohazard might be pretty rare (like something that advances the possibilities) relative to “acceptable” sort of non-marginal biohazards (i.e., another bad genome for something as bad as what’s already public knowledge). Other work might advance what’s possible without being a biohazard persay (i.e., AlphaFold).

I think a way to verify if this is a good project might be to talk to the Spiez lab. They run a biosecurity conference every year and invite anyone doing work that could be dangerous to attend.

I’m happy to chat more about it.

• Regarding the problem being expensive (because there are many publications to vet or something like that):

I think I could help optimize that part, it sounds like an engineering problem, which is a thing I’m ok with

• I’d like to see a more comprehensive model for what biosecurity risk looks like that can motivate a comparison of project ideas. In the absence of that, it’s really hard to say where we get the most benefit.

• We don’t have to drop all our plans in favor of the one top plan

Or as Dumbledore said:

I do not say that carrying your father’s rock is the one best possible course of action, only that it is wiser to do than not.

• Some AI applications may involve AI systems that need to get aligned with the interests, values, and preferences of non-human animals (e.g. pets, livestock, zoo animals, lab animals, endangered wild animals, etc.) -- in addition to being aligned with the humans involved in their care-taking.

Are AI alignment researchers considering how this kind of alignment could happen?

Which existing alignment strategies might work best for aligning with non-human animals?

• Inga—this sounds exciting and useful, and I’m happy to help with it however I can.

I’m a psychology professor at U. New Mexico (USA) who’s taught classes on ‘The psychology of Effective Altruism’, human emotions, human sexuality, intelligence, evolutionary psychology, etc., and I’ve written 5 books and lots of papers on diverse psych topics, the most relevant of which might be the ones on mental disorders (schizophrenia, depression, autism), psych research methods, individual differences (intelligence, personality traits, behavior genetics), consumer psychology, and moral psychology (eg evolutionary origins of altruism and virtue-signaling). I also did a bunch of machine learning research (neural networks, genetic algorithms, autonomous agents) back in the 90s, and I’ve been catching up on AI alignment research.

And my web site: https://​​www.primalpoly.com/​​

• Interesting idea for a competition, but I don’t think that the contest rules as designed and, more specifically, the information hazard policy, are well thought out for any submissions that follow the below line of argumentation when attempting to make the case for longer timelines:

• Scaling current deep learning approaches in both compute and data will not be sufficient to achieve AGI, at least within the timeline specified by the competition

• This is due to some critical component missing in the design of current deep neural networks

• Supposing that this critical component is being ignored by current lines of research and/​or has otherwise been deemed intractable, AGI development is likely to proceed slower than the current assumed status quo

• The Future Fund should therefore shift some portion of their probability mass for the development of AGI further into the future

Personally, I find the above arguments one of the more compelling cases for longer timelines. However, a crux of these arguments holding true is that these critical components are in fact largely ignored or deemed intractable by current researchers. Making that claim necessarily involves explaining the technology, component, method, etc. in question, which could justifiably be deemed an information hazard, even if we are only describing why this element may be critical rather than how it could be built.

Seems like this type of submission would likely be disqualified despite being exactly the kind of information needed to make informed funding decisions, no?

• After going through my first EAG this past weekend, I am left pretty confused about the rationale for the competitive application process.

I know of two main categories of people who seem to have applied and gotten rejected — people very personally invested in EA as a cause but deemed unworthy of networking opportunities, and semi-important DC types who are maybe EA-curious but were deemed insufficiently committed to EA. But I don’t know that I really understand why the event would have been worse if it had featured more people from both of those categories. To CEA’s enormous credit they have a really good system in place of one-on-one meetings, formal office hours, and informal hangouts and there was plenty of stuff happening both formally and informally on the fringes of the event. It all seemed really thoughtful and well-designed. So thoughtful and well-designed that it wouldn’t have in any way been undermined by having more merely curious or somehow low powered people around.

I sort of get the idea of doing a light basic screening before you let people in — make sure they’re not scammers or wreckers or what have you. But rejecting people who are totally sincere in their desire to attend an EA conference seems potentially very costly. Is it just a question of money? The food was maybe nicer and more plentiful than it needed to be. In the limit, people needing to buy their own lunch seems like a much less damaging possibility than making people feel rejected.

• Re the 1st category of people, I don’t really think of being told “try an (in-person) EAGx first” as rejection, which is my understanding of the OP’s situation.

• Do you think it would have been bad if the “semi-important DC types who are maybe EA-curious” had had a bad time at the conference? Perhaps because it was weirder than they expected, or had interactions with different norms than they expected?

• Perhaps if the norms were posted somewhere upfront, then the EA curious types could self select for attendance. I’m still not sure what the norms are after doing all the research for this post. A video of the conference with narration of the setup/​interviews with attendees or just a list of norms on the EAG information page would be helpful to understand what the conference is like.

• From my understanding, it is almost certainly not a question of money. They have the option of making admissions less competitive but increasing the ticket price and then having a competitive process for financial aid, but they do not think that that is a good option.

CEA’s concern with allowing a wider range of people into the conference seems to be that they’d take up the time of the people CEA most wants to be at an EAG.

• I too find money being an issue very unlikely. They rented out a government venue for the weekend with staff serving vegan food and cleaning up after you as if you were royalty. Maybe I’m just not used to this kind of lifestyle due to me a couple of years ago having to be a busser for similar venues, but I still found it odd how an organization that is meant to be about using our ’limited resources to do the best must’ve spent big bucks on this venue.

• Besides (or after) doing his AGI Safety Fundamentals Program (and the potential future part 2 /​ advanced version of the curriculum), what does he recommend university students interested in AI safety do?

• Seems promising and I would support these efforts. One danger to watch for and try to address would be attention info-hazards e.g.:

1. if object-level rejection criteria were specific and public/​well-known, they could point less sophisticated bad actors to a curated list of topics/​papers/​ideas thought by experts to be dangerous

2. if rejected papers find an alternative publication location, that location could become suffused with a higher density of potentially dangerous material relative to other locations, maybe facilitating malicious application

But these risks can be managed, and are outweighed by the project’s benefits in my view.

• Thanks for sharing this—interesting! Would your opinion on the mandatory use of CCTV change significantly if you considered on-farm CCTV instead?

• Thanks for the positive feedback :)

If you consider on-farm (rather than slaughterhouse) CCTV, the welfare benefits increase significantly, as you’re monitoring a much longer period of each animal’s life. However, the tractability of an on-farm CCTV campaign would probably* be much lower. Farmers often have closer, more personal relationships with their farms than slaughterhouse owners do with their slaughterhouses. Proposing to install CCTV on farms would likely trigger a lot of backlash from farmers (particularly given the common public image of the ‘family farm’).

*Admittedly, this is mostly speculation (although speculation that we checked with experts). It could be the case that an on-farm CCTV campaign is more tractable than we think. If somebody tests a campaign and finds it to be tractable, then yes, my opinion of such a campaign would increase significantly. That would start to look like a highly impactful campaign. But you would first need to show that the campaign is tractable, which I don’t have high hopes for. I could imagine that you might be able to increase the tractability by only focusing on industrial/​factory farms, which would capture almost all of the impact for animals anyway.

• Here are slides from my “Writing on the Forum” workshop at EAGxBerlin.

• Arguments for

• The pace of relevant research would slow (allowing for regulation to catch up)

Arguments against

• Malevolent actor could publish or retrieve said information elsewhere than arxiv/​bioRxiv/​medRxiv/​participating publisher. Members of the International Gene Synthesis Consortium (IGSC) voluntarily apply screening standards to assess gene sequence orders and customers, and these companies today only represent approximately 80% of global commercial gene synthesis capacity so compulsory filtering, or better coordination of the industry (which stricter self-regulation at this stage may deter) is required.

• Benevolent actors would be face higher barriers to access useful information

For further investigation

• Would Industry lead censorship could crowd-out or crowd-in government intervention that could enforce vetting across publishers and in other initiatives?

• Would insulating the riskiest dual use publications from exposure to a broader pool of researchers accessing conventional publishers have unintended consequences?

Implementation

How would vetters, whether a regulatory agency or an independent initiative, screen papers? In the case of DNA synthesis, which does not account for all biosecurity relevant dual use research, a minimalistic approach is for vetters to utilise an encrypted database of riskier sequences of DNA , as proposed by MIT’s Prof Kevin Esvelt.

However, dual use control at the publisher level would presumably not be restricted to DNA synthesis, it would include such things as studies of remote areas at the human-animal boundary.

Next steps

Esvelt is in dialogue with the Nuclear Threat Initiative who are coordinating higher level conversation in this area. If the publishers you mentioned aren’t already part of that dialogue, the best next steps may be to connect Nuclear Threat Initiative folks with those academic publishers. But, I don’t think that should mean this initiative shouldn’t proceed in parallel. I think there is merit in taking some action now in this space because the conversation that the Nuclear Threat Initiative and co are kindling is a slow, multilateral process—screening DNA synthesis orders is not legally required by any national government at this stage.

Time sensitivity

The cost of DNA synthesis is declining and the fixed costs of filtering could grow as fraction of the cost, therefore the viability of a voluntary screening model could its highest right now.

Participation

I’m interested in being involved, but don’t know that much about academic publishing or technical genomics stuff so probably not a fit to be a (solo) project lead. Do know about management, health policy, public administration, stakeholder engagement, communications etc

• Edits based on feedback from LessWrong and the EA Forum:

EDITS:
- Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.

UPDATED ESTIMATES (lower bound, estimate, upper bound):

TECHNICAL
- CHAI: 10-30-60 → 5-25-50
- FHI: 10-10-40 → 5-10-30
- MIRI: 10-15-30 → 5-10-20

NON-TECHNICAL
- CSER: 5-5-10 → 2-5-15
- Delete BERI from the list of non-technical research organizations
- Delete SERI from the list of non-technical research organizations
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)
- FLI: 5-5-20 → 3-5-15
- Epoch: 5-10-15 → 2-4-10

• More minor suggestions:

• OpenAI non-technical: there are more than 5.

• AI Impacts non-technical: there are exactly 5.

• I would have said Epoch is at least 5 FTEs (disagreeing with Mauricio).

• Better estimating the number of independent technical researchers seems pretty important and tractable.

• Edit: updated OpenAI from 5 to 10.

From their website, AI Impacts currently has 2 researchers and 2 support staff (the current total estimate is 3).

The current estimate for Epoch is 4 which is similar to most estimates here.

I’m trying to come up with a more accurate estimate for independent researchers and ‘Other’ researchers.

• Thanks for your work everyone! I am excited to see this develop!

• “EA has a strong cultural bias in favor of believing arbitrary problems are solvable”.

I think you’re pointing to a real phenomenon here (though I might not call it an “optimism bias”—EAs also tend to be unusually pessimistic about some things).

I have pretty strong disagreements with a lot of the more concrete points in the post though, I’ve tried to focus on the most important ones below.

Conclusion One: Pursuing the basic plan entailed in premises 1-4 saves, in expectation, at least 4.8 million lives (800,000 * 0.06 * 0.1 * 0.1).

(I think you may have missed the factor of 0.01, the relative risk reduction you postulated? I get 8 billion * 0.06 * 0.01 * 0.1 * 0.1 = 48,000. So AI safety would look worse by a factor of 100 compared to your numbers.)

But anyway, I strongly disagree with those numbers, and I’m pretty confused as to what kind of model generates them. Specifically, you seem to be extremely confident that we can’t solve AI X-risk (< 110,000 chance if we multiply together the 1% relative reduction with your two 10% chances). On the other hand, you think we’ll most likely be fine by default (94%). So you seem to be saying that there probably isn’t any problem in the first place, but if there is, then we should be extremely certain that it’s basically intractable. This seems weird to me. Why are you so sure that there isn’t a problem which would lead to catastrophe by default, but which could be solved by e.g. 1,000 AI safety researchers working for 10 years? To get to your level of certainty (<1/​10,000 is a lot!), you’d need a very detailed model of AI X-risk IMO, more detailed than I think anyone has written about. A lot of the uncertainty people tend to have about AI X-risk comes specifically from the fact that we’re unsure what the main sources of risk are etc., so it’s unclear how you’d exclude the possibility that there are significant sources of risk that are reasonably easy to address.

As to why I’m not convinced by the argument that leads you to the <1/​10,000 chance: the methodology of “split my claim into a conjunction of subclaims, then assign reasonable-sounding probabilities to each, then multiply” often just doesn’t work well (there are exceptions, but this certainly isn’t one of them IMO). You can get basically arbitrary result by splitting up the claim in different ways, since what probabilities are “reasonable-sounding” isn’t very consistent in humans.

Okay, a longtermist might say. Maybe the odds are really slim that we thread this needle, and then also the subsequent needles required to create an interstellar civilization spanning billions of years. But the value of that scenario is so high that if you shut up and multiply, it’s worth putting a lot of resources in that direction.

I can’t speak for all longtermists of course, but that is decidedly not an argument I want to make (and FWIW, my impression is that this is not the key objection most longtermists would raise). If you convinced me that our chances of preventing an AI existential catastrophe were <1/​10,000, and that additionally we’d very likely die in a few centuries anyway (not sure just how likely you think that is?), then I would probably throw the expected value calculations out the window and start from scratch trying to figure out what’s important. Basically for exactly the reasons you mention: at some point this starts feeling like a Pascal’s mugging, and that seems fishy and confusing.

But I think the actual chances we prevent an AI existential catastrophe are way higher than 110,000 (more like 110 in terms of the order of magnitude). And I think conditioned on that, our chances of surviving for billions of years are pretty decent (very spontaneous take: >=50%). Those feel like cruxes to me way more than whether we should blindly do expected value calculations with tiny probabilities, because my probabilities aren’t tiny.

Scenario Two: Same as scenario one, but there’s a black hole/​alien invasion/​unstoppable asteroid/​solar flare/​some other astronomical event we don’t know about yet that unavoidably destroys the planet in the next millennium or two. (I don’t think this scenario is likely, but it is possible.)

I agree it’s possible in a very weak sense, but I think we can say something stronger about just how unlikely this is (over the next millennium or two): Nothing like this has happened over the past 65 million years (where I’m counting the asteroid back then as “unstoppable” even though I think we could stop that soon after AGI). So unless you think that alien invasions are reasonably likely to happen soon (but were’t likely before we sent out radio waves, for example), this scenario seems to be firmly in the “not really worth thinking about” category.

This may seem really nitpicky, but I think it’s important when we talk about how likely it is that we’ll continue living for billions of years. You give several scenarios for how things could go badly, but it would be just as easy to list scenarios for how things could go well. Listing very unlikely scenarios, especially just on one side, actively makes our impression of the overall probabilities worse.

• Ah yeah, you’re right—I think basically I put in the percent rather than the probability. So it would indeed be very expensive to be competitive with AMF. Though so is everything else, so that’s not hugely surprising.

As for the numbers, yeah, it does just strike me as really, really unlikely that we can solve AI x-risk right now. 110,000 does feel about right to me. I certainly wouldn’t expect everyone else to agree though! I think some people would put the odds much higher, and others (like Tyler Cowen maybe?) would put them a bit lower. Probably the 1% step is the step I’m least confident in—wouldn’t surprise me if the (hard to find, hard to execute) solutions that are findable would reduce risk significantly more.

EDIT: tried to fix the math and switched the “relative risk reduction term” to 10%. I feel like among findable, executable interventions there’s probably a lot of variance, and it’s plausible some of the best ones do reduce risk by 10% or so. And 1/​1000 feels about as plausible as 1/​10000 to me. So, somewhere in there.

• Thank you for the effort you’ve put into this.

“adjust the university chapters and transform them into a student ambassador program to work alongside long-term community organizers.”

I feel like there are some good ideas in here that most people won’t find because they’re buried deep within the text. I’d also like to see you 1) elaborate on your action proposals further 2) rank them in terms of potential impact, 3) discuss them in light of existing opportunities and data/​research. I think that might spark more discussion.

For example, for virtual coworking and exploration there is a more casual EA gathertown as well as CEA’s standardized virtual programs. I’m wondering how your proposals differ from them or could be combined with them or how they could be improved in your view. It seems like you say there’s too much centralization but then also not enough to standardize all the platforms.

• Hi Peter, thanks for your comment! The purpose of this manifesto is that EA’s core team needs to do the work to fix the community. The community can’t do it themselves. And, the community needs to recognize its shortcoming and how to work together to mediate these issues.....if they even believe these are issues. If I had won the contest, maybe I would be inclined to further elaborate and prioritize, but I didn’t, and unless I’m hired as a community consultant, I won’t be doing such unpaid labor. :)

Ideally my proposals should be combined and tested to see which ones work best. Many of my proposals don’t really exist or have any long-term foresight as to facing upcoming challenges that I’ve addressed. If I were to add anything to this I would have made a point that there is an inaccurate centralization, and the centralization needs to be recalibrated.

• Good question, and I think this is definitely healthy discussion. In general, money is a sensitive issue, and I would encourage all parties to show nuance, this includes (but not limited to) when: “judging” someone’s salary, when asking for a salary, and when granting a salary.

Two steelmen for decent chunky grants 1) Bounded loss and unbounded wins—while theoretically salaries could be cut in half, impact could easily be 10-100x. I.e. the focus should be opportunity cost and not expenditure 2) many smart people in ea, and the people granting, may have previously been earning decent significant salaries as programmers/​executives/​consultants. You and I may see 80k USD as a lot of money, but its pretty normal for developers in Cali to earn hundreds of USD. Therefore, expecting people to earn 50k a year may effectively be asking them to donate 75% of their income.

And 2 steelmen for keeping salary low − 1) this is a movement about charity, helping others, and donating. We put a lot of effort and time into building health communities around these principles built on a heavy basis of trust. It’s important to feel like people are in it for the right reason, and high salaries can jeopardise that. 2) it’s pretty easy to justify a high salary with some of the above reasoning, perhaps too easy. As a community builder myself, it seems totally plausible we could attract people that are a poor fit for ea by being too relaxed around the money pedal.

For my own personal opinion, I think it’s far too easy to ignore opportunity cost, and concentrate on short term expenditure and salary. However, I can very much imagine myself leaving the community if I salaries became too inflated. And I am likely to feel less aligned with others who require large salaries (just being honest here). Looking at recent posted receipts, I don’t see anything that catches my eye in a bad way, although it could be said to be unfair that some community builders will be working 3x harder on a volunteer basis than other community builders on a competitive salary. I think this partially reflects the incentives which produced a world we currently live in (I.e. largely unaltruistic).

Whilst I find the arguments for working hard, and concentrating on impact, rather than earning little, pretty compelling—it’s worth pointing out that there’s some fantastic work coming out of Charity Entrepreneurship charities (who’s employees generally earn little) , so it’s not clear the tradeoff is always present.

Lastly, I would say its likely that I’ve made tradeoffs with my own salary, which have likely significantly negatively effected my social impact. I suspect this is easy to do, and would encourage people to avoid failing into this trap.

• HI Diana, lots of interesting stuff here! not claiming to have read the whole thing but a few things that come to mind

Our interpretation of community is too linear, and we need to shift our ideals of community from organizer-centric to a more holistic approach

Who’s ‘we’ in this that constitutes the ‘our’?

The introductory paragraphs of Section 2 - I’m afraid I don’t know what/​who you’re talking about 😃 That’s not to say you aren’t addressing a real phenomenon, it’s just that I lack the context to understand what issues you’re addressing

You use a lot of passive voice, e.g. “Agency is what should be taught and internalized throughout any education”—hard for me to slog through these sentences as a reader

OK will read more and comment more later I hope!

• Hi Seth, thanks for your comment! I ran out of time while I was writing and didn’t properly context switch for Sections 1-3. This was more in a vague societal sense and not through the EA lens. I am writing a book and Sections 1-3 were taken from that. Great feedback though that I should narrow the context.

Hope you have more feedback!

• [ ]
[deleted]
• I think this project seems valuable for three main reasons:

1. It would provide a test of dual-use research (DUR) policies and policy implementation, improving future efforts to limit the publication of DUR. Indeed, if the project succeeds, it is likely to provide a blueprint for future action that can be easily replicated with journals and other preprint servers. If it fails, it will yield valuable information about what our blindspots are and what types of projects we should pursue in the future. As far as I know, the EA biosecurity community has not yet tried working directly with publishers.

2. It would normalize responsible publication (as opposed to maximal openness) and raise awareness of dual-use concerns within the scientific community, making future efforts to limit the publication of DUR more likely to succeed.

3. In some (very rare) scenarios, it might actually limit the spread of dangerous information. Journals tend to have better review processes than preprint servers, so it is possible to imagine a particularly dangerous piece of research (e.g., the creation of a new superbug) being denied publication in a journal only to find that it has spread widely over the internet after being submitted to arXiv.org.

Additionally, it is clear that servers like arXiv.org are highly resource-constrained (see https://​​www.scientificamerican.com/​​article/​​arxiv-org-reaches-a-milestone-and-a-reckoning/​​).

With that said, striking the right balance between openness and security is difficult, and this project might generate significant backlash or have unintended consequences. I wrote up some additional thoughts about this project after discussing it with Yonatan, so please reach out to me if you would like to read more about what this project might look like and why it might or might not be a good idea.

• I really like this point, and it sparks a few questions, confusions and thoughts I have about community building, some of which are:

(1) I believe what you are pointing at can be a good distinction between the tasks of national/​regional groups vs. local groups at universities and in cities—the more local groups catering towards creating a healthy community (for example through the activities you suggested), and the bigger groups catering towards helping/​”making sure” the community serves humanity in addition to itself.

(2) What do we actually mean with community building and what is the difference between a community building and movement building (and is one of them preferred)?

(3) Is what you’re bringing up an example of Goodhart’s law (this question may say more about my understanding of the law than about community building)?

(4) (related to above points) I wonder if we weigh the intrinsic value of (the) EA community(s) too little, compared to its/​their instrumental value, much like treating rest as a means to be, say, more productive, will make it harder to get proper rest (not sure if this is an actual thing, but I’m pretty sure there’s a named concept for it which I have forgotten), and also missing out on the intrinsic value, of course! Making the community healthy would be, I guess, more of catering to the community for the community’s sake, whilst also (probably) making it more likely for it/​its members to have a greater impact.

Perhaps some of these points have already been adressed in previous posts on community building—would love to get tips on this!

I hope this doesn’t interfere with the intent of your post; I really felt like sharing my associations! Please let me know if anything was irrelevant. Would love to hear if you have any thoughts on mine.

• Trying to collect some comments that seem relevant:

# Daniel Greene said:

+1 to [doing things at arXiv]. arXiv could play a big role in contributing to a norm around not publishing dual use bio research. There are challenges of screening large numbers of papers, but they can be met. See here for an example from ASM. bioRxiv may or may not be screening already, but they aren’t sharing information about their practices. It would be helpful if they were more vocal about the importance of not publishing dangerous information.

ASB seems to agree:

See Daniel Greene’s comment about creating better norms around publishing dangerous information (he beat me to it!).

# Tessa said:

Relatedly, an area where I think arXiv could have a huge impact (in both biosecurity and AI) would be setting standards for easy-to-implement manged access to algorithms and datasets.

This is something called for in Biosecurity in an Age of Open Science:

Given the misuse potential of research objects like code, datasets, and protocols, approaches for risk mitigation are needed. Across digital research objects, there appears to be a trend towards increased modularisation, i.e., sharing information in dedicated, purpose built repositories, in contrast to supplementary materials. This modularisation may allow differential access to research products according to the risk that they represent. Curated repositories with greater access control could be used that allow reuse and verification when full public disclosure of a research object is inadvisable. Such repositories are already critical for life sciences that deal with personally identifiable information.

This sort of idea also appears in New ideas for mitigating biotechnology misuse under responsible access to genetic sequences and in Dual use of artificial-intelligence-powered drug discovery as a proposal for managing risks from algorithmically designed toxins.

• [ ]
[deleted]