An existential risk is the risk of an existential catastrophe, i.e. one that threatens the destruction of humanity’s longterm potential.[1][2] Existential risks include natural risks such as those posed by asteroids or supervolcanoes as well as anthropogenic risks like mishaps resulting from synthetic biology or artificial intelligence.
A number of authors have argued that existential risks are especially important because the long-run future of humanity matters a great deal.[1][3][4][5] Many believe that there is no intrinsic moral difference between the importance of a life today and one in a hundred years. However, there may be many more people in the future than there are now. They argue, therefore, that it is overwhelmingly important to preserve that potential, even if the risks to humanity are small.
One objection to this argument is that people have a special responsibility to other people currently alive that they do not have to people who have not yet been born.[6] Another objection is that, although it would in principle be important to manage, the risks are currently so unlikely and poorly understood that existential risk reduction is less cost-effective than work on other promising areas.
Recommendations
In The Precipice: Existential Risk and the Future of Humanity, Toby Ord offers several policy and research recommendations for handling existential risks:[7]
Explore options for new international institutions aimed at reducing existential risk, both incremental and revolutionary.
Investigate possibilities for making the deliberate or reckless imposition of human extinction risk an international crime.
Investigate possibilities for bringing the representation of future generations into national and international democratic institutions.
Each major world power should have an appointed senior government position responsible for registering and responding to existential risks that can be realistically foreseen in the next 20 years.
Find the major existential risk factors and security factors—both in terms of absolute size and in the cost-effectiveness of marginal changes.
Target efforts at reducing the likelihood of military conflicts between the US, Russia and China.
Improve horizon-scanning for unforeseen and emerging risks.
Investigate food substitutes in case of extreme and lasting reduction in the world’s ability to supply food.
Develop better theoretical and practical tools for assessing risks with extremely high stakes that are either unprecedented or thought to have extremely low probability.
Improve our understanding of the chance civilization will recover after a global collapse, what might prevent this, and how to improve the odds.
Develop our thinking about grand strategy for humanity.
Develop our understanding of the ethics of existential risk and valuing the longterm future.
Further reading
Bostrom, Nick (2002) Existential risks: analyzing human extinction scenarios and related hazards, Journal of Evolution and Technology, vol. 9.
A paper surveying a wide range of non-extinction existential risks.
Bostrom, Nick (2013) Existential risk prevention as global priority, Global Policy, vol. 4, pp. 15–31.
Matheny, Jason Gaverick (2007) Reducing the risk of human extinction, Risk Analysis, vol. 27, pp. 1335–1344.
A paper exploring the cost-effectiveness of extinction risk reduction.
Ord, Toby (2020) The Precipice: Existential Risk and the Future of Humanity, London: Bloomsbury Publishing.
Ord, Toby (2020) Existential risks to humanity in Pedro Conceição (ed.) The 2020 Human Development Report: The Next Frontier: Human Development and the Anthropocene, New York: United Nations Development Programme, pp. 106–111.
Related entries
civilizational collapse | criticism of longtermism and existential risk studies | dystopia | estimation of existential risks | ethics of existential risk | existential catastrophe | existential risk factor | existential security | global catastrophic risk | hinge of history | longtermism | Toby Ord | rationality community | Russell–Einstein Manifesto | s-risk
- ^
Bostrom, Nick (2012) Frequently asked questions, Existential Risk: Threats to Humanity’s Future (updated 2013).
- ^
Ord, Toby (2020) The Precipice: Existential Risk and the Future of Humanity, London: Bloomsbury Publishing.
- ^
Beckstead, Nick (2013) On the Overwhelming Importance of Shaping the Far Future, PhD thesis, Rutgers University.
- ^
Bostrom, Nick (2013) Existential risk prevention as global priority, Global Policy, vol. 4, pp. 15–31.
- ^
Greaves, Hilary & William Macaskill (2019) The case for strong longtermism, GPI working paper No. 7-2019, Working paper Global Priorities Institute, Oxford University.
- ^
Roberts, M. A. (2009) The nonidentity problem, Stanford Encyclopedia of Philosophy, July 21 (updated 1 December 2020).
- ^
Ord, Toby (2020) The Precipice: Existential Risk and the Future of Humanity, London: Bloomsbury Publishing, pp. 280–281.
Related, also with some relevant discussion in the comments: https://forum.effectivealtruism.org/posts/sEnkD8sHP6pZztFc2/fanatical-eas-should-support-very-weird-projects
Thanks for sharing this. While I’m less on board with the idea that EA folks should have been able to detect the fraud in advance than some people here, “charity badly burned by fraudster” happens often enough to be an ascertainable risk, and “crypto wealth explodes, sometimes with at least an odor of foul play” is a regular occurence. So I don’t think this was as unforeseeable as the black swan label would suggest.
Any definitive “you aren’t getting any money back” statement is a while off, and SBF/FTX will probably be oldish news by then. I think the more likely media focus points will be his likely indictment, plea, trial, and/or sentencing.
I’ve had vaguely similar thoughts about youth and levels professional experience, but you have articulated them much better than I have. Thanks for writing this.
I’d be very happy to see a “EA managers” Slack (or some other forum/conversation space/community), and I would be very happy to join.
I don’t think the idea of a charitable fund is especially new? More: https://forum.effectivealtruism.org/posts/9Axdtzusq9wSKixEZ/historical-notes-on-charitable-funds
This post hit at a good topic and gave it good nuance, minus the part about “society lead by the elite” and “self obvious statements”
Working on a forum post about animals and longtermism. I have an outline on a google doc and would love to have collaborators or just people to give feedback about the content.
This is a great question. I think the answer depends on the type of storage you’re doing.
If you have a totally static lump of data that you want to encode in a harddrive and not touch for a billion years, I think the challenge is mostly in designing a type of storage unit that won’t age. Digital error correction won’t help if your whole magnetism-based harddrive loses its magnetism. I’m not sure how hard this is.
But I think more realistically, you want to use a type of hardware that you regularly use, regularly service, and where you can copy the information to a new harddrive when one is about to fail. So I’ll answer the question in that context.
As an error rate, let’s use the failure rate of 3.7e-9 per byte per month ~= 1.5e-11 per bit per day from this stack overflow reply. (It’s for RAM, which I think is more volatile than e.g. SSD storage, and certainly not optimised for stability, so you could probably get that down a lot.)
Let’s use the following as an error correction method: Each bit is represented by N bits; for any computation the computer does, it will use the majority vote of the N bits; and once per day,[1] each bit is reset to the majority vote of its group of bits.
If so...
for N=1, the probability that a bit is stable for 1e9 years is ~exp(-1.5e-11*365*1e9)=0.4%. Yikes!
for N=3, the probability that 2 bit flips happen in a single day is ~3*(1.5e-11)^2 and so the probability that a group of bits is stable for 1e9 years is ~exp(-3*(1.5e-11)^2*365*1e9)=1-2e-10. Much better, but there will probably still be a million errors in that petabyte of data.
for N=5, the probability that 3 bit flips happen in a single day is ~(5 choose 2)*(1.5e-11)^3 and so the probability that the whole petabyte of data is safe for 1e9 years is ~99.99%. And so on this scheme, it seems that 5 petabytes of storage is enough to make 1 petabyte stable for a billion years.
Based on the discussion here, I think the errors in doing the majority-voting calculations are negligible compared to the cosmic ray calculations. At least if you do it cleverly so that you don’t get too many correlations and ruin your redundance (which there are ways to do according to results on error correcting computations — though I’m not sure if they might require some fixed amount of extra storage space to do this, in which case you might need N somewhat greater than 5).
Now this scheme requires that you have a functioning civilization that can provide electricity for the computer, that can replace the hardware when it starts failing, and stuff — but that’s all things that we wanted to have anyway. And any essential component of that civilization can run on similarly error-corrected hardware.
And to account for larger-scale problems than cosmic rays (e.g. local earthquake throws harddrive to the ground and shatters it, or you accidentally erase a file when you were supposed to make a copy of it), you’d probably want backup copies of the petabyte on different places across the Earth, which you replaced each time something happened to one of them. If there’s an 0.1% chance of that happening in any one day (corresponding to once/3 years, which seems like an overestimate if you’re careful), and you immediately notice it and replace the copy within a day, and you have 5 copies in total, the probability that one of them keeps working at all times is ~exp(-(0.001)^5*365*1e9)~=99.96%. So combined with the previous 5, that’d be a multiple of 5*5=25.
This felt enlightening. I’ll add a link to this comment from the doc.
Using a day here rather than an hour or a month isn’t super-motivated. If you reset things very frequently, you might interfere with normal use of the computer, and errors in the resetting-operation might start to dominate the errors from cosmic rays. But I think a day should be above the threshold where that’s much of an issue.
Just want to say that I love this format:
Clear title
Summary
Clear introduction to the concept
Clear description of the relevance and limitations of the concept
Application of the concept accompanied by clearly marked recommendations
Thanks! I used that format because it was easy for me to write. I’m glad to see that it improves the reading experience too.
You say that like it is a bad thing. I would suggest reading [Against Empathy](https://en.wikipedia.org/wiki/Against_Empathy#Empathy_versus_Compassion) to understand why it is a very good thing.
We need less emotion-driven decisions and more rational decisions, that is the whole point of EA.
GiveWell has identified several new giving opportunities! They’re just not listed under “top charities” at the moment. See info on their aims for next year here: https://blog.givewell.org/2022/11/23/giving-recommendations-2022/
Thanks for writing all of this Ben. I agree with everything you have said and like your ideas. I also think that we should:
nudge more EAs to get research etc management experience in industry before returning to work in new EA organisations.
have a management consultancy type org or network for providing fractional management oversight/advice to new organisations. For instance, 10 hours a week to advise how to set up a research team or to sit in on supervision meetings and pass on best practices.
have a norm that managers at places like Rethink occassionally do placements at other research organisations to pass on their knowledge and best practice
have some people who are involved across many established and new research orgs (maybe as funders) and incentivised to understand and speed up the dissemination of collective best practice (e.g., via talk with CEOs/founders or writing up what they have learned)
I love the donation live-streams on your website!
I of course agree that we should take into account the size of the future. I somewhat disagree with this:
Do you have examples in mind? I can think of a couple related to anthropics, but their decision-relevance is unclear.
No matter what the universe is like, or whether we’re in a simulation, or whatever, averting x-risk seems roughly equivalent to increasing future option value, which seems roughly equivalent to being able to make the most of the universe, whatever it’s like.
Mostly no, I think: an AI not capable of taking over without descriptions of AI takeover isn’t much more threatening after it reads a few descriptions of AI takeover. (In fact, most such descriptions are not optimized for realism, so I think it’s not even clear that AI reading such descriptions is net negative! The marginal description from this community is probably reasonably realistic, but lacks details on the hard steps, because we don’t know how to do them.) That said, if you think of an excellent tactic to help an AI take over the world, publishing it could cause a rogue AI to learn about it.
I endorse all of this, but would add that the AI in the last sentence “learning about it” still wouldn’t actually matter, and hence it’s not “mostly no” but rather just “no”.
Thanks for writing this!
Your pre-screening steps look a little different to mine, which could be a regional thing given we were both in the process at around the same time.
Your guide is excellent! (And I have added a link to it from my much more hurried post :))
Markus Amalthea Magnuson at https://altruistic.agency/
I would argue that the majority of software products are aiming at making a positive impact, and my guess would be that the products that have the biggest impact are wholly unrelated to EA. Having said that, here are a few that seem in line with the spirit of the question:
Spark Wave
QURI
Computation Democracy Project
Various other prediction markets, like Manifold
There are people who think that building any kind of ML-intensive product is not a good thing, even if the model isn’t SOTA, and so might disagree that what Ought are doing is good. I neither endorse nor disendorse this opinion, but if you disagree with it then you might also like co:here
Toby—interesting essay. But I’m struggling to find any rational or emotive force in your argument that ‘strong longtermism tells us to look at the set of possible theories about the world, pick the one in which the future is largest, and, if it is large enough, act as if that theory were true’
The problem is that this leads to a couple of weird edge cases.
First, if we live in a ‘quantum multiverse’, in which there are quadrillions of time-lines branching off every microsecond into new universes, then the future is very very large indeed, but any decisions we make to influence it seem irrelevant, insofar as we’d make any possible decision in some branching time-line.
Second, the largest possible futures seem associated more with infinite religious afterlives than with scientifically plausible theories. Should ‘strong longtermists’ simply adopt Christian metaphysics, on the assumption that an infinite afterlife in heaven would be really cool, compared to any atheist metaphysics?
I’d welcome any thoughts about these examples.
My opinionated and annotated summary / distillation of SBF’s account of the FTX crisis based on recent articles and interviews.
I don’t come here often, and don’t plan to write something verbose whatsoever. I have one simple thing to say, speaking as a neuroscientist—why on earth would you think something like the number of neurons in a brain is a good proxy for moral weight?
What age group has the most neurons in humans? Babies. It’s babies. The ones you constantly have to tell what is right and what is wrong? To stop hitting their siblings, being selfish, etc? This is not a nuanced conversation, it is one that, if you subscribe to scientific materialism, is quickly and efficiently done away with.
To utilize logic in parallel with the way it’s been done here so far : in humans, decreasing neural count is correlated with increased moral intelligence (incl. the development of empathy, moral reasoning, etc.)
(tldr: We might not be psychologically capable of handling highly emotionally compelling portrayals of our largest-scope, most important cause areas.)
Eleni & Luis—this is a fascinating and thoughtful post, and raises some insightful pros and cons of EA adopting more mainstream emotionally appealing marketing & PR methods.
As someone who’s worked in psychology for almost 40 years, and done a fair amount of consulting with market research companies, ad agencies, and consumer product companies, I support the general idea of some EAs becoming a bit more familiar with the psychology of persuasion, marketing, and influence.
However, I want to point out another possible downside of turning our rational interest in tractable, neglected, scope-sensitive problems into more emotionally compelling images and narratives.
The key problem is that a lot of EA deals with such large-scope problems (e.g. animal suffering, nuclear war, pandemics, longtermist cosmic stakes) that any emotionally impactful, truly compelling, highly memorable presentation of these problems could be extremely depressing, psychologically paralyzing, and traumatically damaging. It would be like being trapped in the ‘Total Perspective Vortex’ in the Hitchhiker’s Guide to the Galaxy series.
I think the current EA messaging/influence strategies embody a tacit understanding that we’re willing to engage with large-scope problems that would seem utterly overwhelming if we really felt the true suffering impact of the problems we’re fighting against, all day, every day.
Other charities have the luxury of presenting their relatively small-scale cause areas using emotionally compelling narratives, precisely because ordinary people can handle the scale of those problems without losing their sanity. For example, the Make a Wish Foundation highlights the suffering of individual kids with terminal illnesses; that’s sad, tragic, and poignant, but emotionally engaging with their plight isn’t psychologically crippling.
By contrast, any truly emotionally compelling portrayals I’ve seen of truly large-scope, high-stakes, global-scale or cosmic-scale EA cause areas might be so traumatizing to most people that they’d become counter-productive. Just as our brains didn’t evolve to reason clearly about how to reduce suffering beyond the scale of prehistoric hominid clans, our brains might not be prepared to handle emotionally compelling presentations of suffering beyond that scale.
Consider some specific examples.
Wild animal suffering: In my ‘Psychology of Effective Altruism’ class, I’ve learned that I simply can’t assign students the Brian Tomasik essay on wild animal suffering, because it can induce serious depression, extreme anxiety, and even panic attacks. It’s just too emotionally compelling, and the scale of the problem is too overwhelming. Similar considerations apply for sharing with students any truly compelling depictions of animal suffering under factory farming. (Vegan activists have learned from bitter experience that there’s an optimal degree of factory farming gore that they can show to non-vegans to convince them to give up meat, and that degree is non-zero, but it’s not very high.)
S-risks: the Black Mirror episode ‘White Christmas’, which depicts a future in which uploaded criminals are tormented in virtual hells for many subjective millennia, is simply too distressing for most people to handle; it haunts their dreams for weeks. I still wish I’d never seen it. (The Iain M. Banks novel ‘Surface Detail’ (2010) raises similar issues.) The S-risks of virtual hells are very important to consider, but it might be most productive to consider those issues at a somewhat abstract, rational level, rather than a truly visceral level.
Nuclear war: Most people have seen science fiction movie depictions of global thermonuclear war, which induce varying levels of fear, horror, disgust, and dread. But very few movies, TV series, books, etc really try to capture the full scope and scale of the impact on a nuclear holocaust on billions of people. Any such attempt would simply be too depressing and horrifying to engage with. (Consider also movies like The Road (2009) that try to do an unflinchingly realistic portrayal of post-apocalyptic life—they’re often praised by critics, but rarely re-watched by ordinary folks.)
In short, I think a distinctive strength of EA is that we can set aside the highly emotive portrayals of the largest-scope cause areas that we rationally know are most important. We can keep the suffering impacts of these cause areas off to the side, in our peripheral vision, so to speak, with some degree of awareness—but without the paralyzing level of emotional response that we’d feel if they were always front and center in our imaginations.
I would caveat this point in two ways.
First, there can still be crucial roles for positive, inspirational, optimistic portrayals of success in solving various large-scope problems—e.g. how awesome life would be if we solved longevity, achieved nuclear security, promoted happier animal lives, and secured a great long-term future for our descendants. I think Nick Bostrom’s Letter from Utopia is a good example of this genre. We need more such fictions and inspiring tales of success. Engaging with our positive emotions can be great; relying on negative emotions elicited by accounts of mass suffering can be much trickier.
Second, at the outreach level of recruiting talent and money, some slightly more emotionally compelling narratives and influence methods could be useful. We just have to pitch them very carefully—eliciting an optimal degree of concern that’s somewhere in the middle between apathetic indifference and paralyzing horror. And discovering those optimal degrees of concern, as the original post mentioned, is very much a matter for empirical psychological research, rather than armchair reasoning.
This was really well written! I appreciate the concise and to the point writing style, as well as a summary at the top.
Regarding the arguments, I think they make sense to me. Although this is where the whole discussion of longtermism does tend to stay pretty abstract, if we can’t actually put real numbers on it.
For ex, in the spirit of your example—does working on AI safety at MIRI prevent extinction, while assuming a sufficiently great future compared to, say, working on AI capabilities at OpenAI? (That is, maybe a misaligned AI can cause a greater future?)
I don’t think it’s actually possible to do a real calculations in this case, and so we make the (reasonable) base assumption that a future with alligned AI is better than a future with a misaligned AI, and go from there.
Maybe I am overly biased against longtermism either way, but in this example it seems to me like the problem you mention isnt really a real-world worry, but only really a theoretically possible pascals mugging.
Having said that I still think it is a good argument against strong longtermism
Yes!I’m agree with you!
What are some negative and positive sentiments about OpenAI within the EA community?
I know OpenPhil made them a 30m grant in 2017, but Holden no longer seems to be on the board and a bunch of people left to create Anthropic. Whats up with that?
Social media systems are notorious for finding galaxy-brained ways to detect unsanctioned research and researchers, and thwart their research without their knowledge, e.g. by feeding the researchers false information that strongly appears to be accurate (this has even been known to happen with sanctioned researchers as well). I’m not saying this is the wrong way to look at things in this particular situation, but in general it is a losing strategy for research.
We’ll still need to wait and see, because odds are high that the scope of this phenomenon is being mismeasured by orders of magnitude, in either direction. Both scenarios are worth investment, especially extreme-seeming versions of the scenarios.
How does GiveWell’s All Grants Fund compare to EA Funds’s Global Health and Development Fund? My understanding is that both funds are looking primarily at global health interventions and have a greater tolerance for risk than GiveWell’s Top Charities Fund. Are there major differences between the funds?
You’re correct about all of this. I developed a fellowship program to help orgs specifically with upskilling and having the support they need to do well. I believe that the 3 critical ingredients to running an org successfully are: a) having the right knowledge b) having peer support and c) having a mentor and accountability. My personal mission is to help orgs succeed. You can find out more information on my website, or shoot me a PM / email.
I’m also working on developing an organization to consolidate all the org support resources—I’ve done this in the small business sector, and am applying the principles to EA. Would love to connect with anyone who wants to be a part of it.
Hi!
I’m Minh, current intern at Nonlinear. Since this thread is clearly picking up again, I’d like to provide my perspective as someone who’s worked at Nonlinear, and on the Emergency Fund. First, context:
All this is my own experience and opinion. The fact that I had overall positive experiences does not invalidate someone else’s negative experience. I admit it’s … not a good look to have such a high rate of strong negative sentiment.
I’ve been working with Nonlinear for ~3 months. About 80% of my interaction is with Drew Spartz [1]and Luca (the other intern). The other 20% is Kat who I’ve had one 2-hour meeting with when planning the Nonlinear Emergency Fund and text correspondence/coordination as Nonlinear processes applications. I have never interacted with Emerson.
What I can do is provide additional context. So far, we’ve received ~40 applications for the Nonlinear Emergency Fund. Kat has been personally following up on requests, coordinating with external funders and handling relevant documents for dozens of grantees. To my knowledge, ~15 of the most time-sensitive requests have been approved. The responsibility is kind of insane, since improper follow-up/delay has immediate consequences on the grantee’s end. About 8 requests indicated they needed to hear back in less than a week, and the shortest was less than three days.
Motivations behind the Emergency Fund
Keep in mind that none of this is something Nonlinear actually has to do. There’s practically no direct impact to us, and the emergency funding we give out are, to my knowledge, unconditional. I did see one grant that stipulated additional non-emergency funding contingent on producing policy-actionable research, but that doesn’t benefit Nonlinear either.
During the 2-hour call when Kat, Luca and I discussed the Emergency Fund, the 2 main focuses were:
Prioritising immediate financial/personal emergencies of grantees
Prioritising counterfactual impact on EA work This means bridge-funding projects/work that would otherwise have to be stopped as grantees have to cease their work. I won’t give names, but let’s just say I was very surprised to see projects I’d used before/applied to.
At no point did I hear ulterior motives. And frankly, I feel like launching an emergency fund of your own money, when you don’t have to, isn’t the kind of thought process a selfish opportunist jumps at.
Implications
Also, keep in mind that, at the time:
Nonlinear also lost $250k in funding from FTX Future Fund. I have been told some of the team also had larger personal funds on FTX. If anything, FTX is paying Nonlinear if clawbacks occur.
Kat was entirely focused on the community. Video calling someone on a treadmill while she earnestly details plans to distribute six figures in emergency funding, right after a third of EA funding vanished overnight, is a very surreal experience.
Kat had no reassurance that any other funders would also step up for any of the grantees. Hence the explicit assumption was that Nonlinear’s founders would self-fund the grantees out of their own pockets. I distinctly recall her saying “If we need to fund them, that’s a good thing to do.”
Conclusion
I’ve told this story because:
This thread about the Nonlinear Emergency Fund has gotten wildly off tangent, so I might as well update the community with positive news. Hi!
I wanted to provide a contrasting perspective. The team I’ve interacted with has been talking about helping the EA community this entire time, actively coordinating and willingly paid out of pocket for people directly. While this doesn’t invalidate or excuse alleged mistreatment, I just figured my perspective would be significantly belief-updating to people who have only read accusations of morally unscrupulous behaviour for the past month.
Drew is great btw, but the accusations so far aren’t directed at him.
Good to know Drew treats you well, though you’re right that the allegations so far aren’t directed at him.
Given your impressions of Kat are largely based off a 2 hour call, I don’t know if I’d consider this “significantly belief-updating” compared to other claims on this thread, though I’m glad you shared your experience.
Out of curiosity, did anyone ask you, or Luca to make comments on this post, or was this completely unprompted (aside from seeing the recent forum comments?
There’s a pretty major difference here between EA and most religions/ideologies.
In EA, the thing we want to do is to have an impact on the world. Thus, sequestering oneself is not a reasonable way to pursue EA, unless done for a temporary period.
An extreme Christian may be perfectly happy spending their life in a monastery, spending twelve hours a day praying to God, deepening their relationship with Him, and talking to nobody. Serving God is the point.
An extreme Buddhist may be perfectly happy spending their life in a monastery, spending twelve hours a day meditating in silence, seeking nirvana, and talking to nobody. Seeking enlightenment is the point.
An extreme EA would not be pursuing their beliefs effectively by being in a monastery, spending twelve hours a day thinking deeply about EA philosophy, and talking to nobody. Seeking moral truths is not the point. Maybe they could do this for a few months, but if they do nothing with their understanding, it is useless—there is no direct impact from understanding EA philosophy really well. You have to apply it in reality.
Unlike Christianity and Buddhism, which emphasise connection to spiritual things, EA is about actually going out and doing things to affect the world. Sequestering an EA off from the world is thus not likely to be something that EA finds satisfying, and doesn’t seem like something they would agree to.
Does anyone have a recommendation for a resource to read on dependency injection?
I’m considering whether to add the pattern to the ForumMagnum codebase. My current state is that I have a tentative grasp of the concept and roughly the pros and cons, but not really a practical grasp of it, or of how it’s played out in a typescript codebase.
I notice that I’m confused.
I read your previous post. Nobody seemed to disagree with your prediction that a misaligned AI could do this, but everybody seemed to disagree that this was a problem such that fixing it was worth the large costs in coordination.
Thus, I don’t really understand how this is supposed to be an update for those who disagreed with you. Could you elaborate on why you think this information would change people’s minds?
For those who don’t want to follow links to a previous post and read the comments, the counterargument as I understand it (and derived, independently, before reading the comments) is:
For this to be a threat, we would need an AGI that was
- Misaligned
- Capable enough to do significant damage if it had access to our safety plans
- Not capable enough to do a similar amount of damage without access to our safety plans
I see the line between 2 and 3 to be very narrow. I expect almost any misaligned AI capable of doing significant damage using our plans to also be capable of doing significant damage without needing them.
By contrast, the cost of not posting our plans online is a likely drastic reduction of effectiveness of the AI alignment field, both coordinating among existing members and bringing new members in. While the threat that Peter talks about is real, it seems that we are in much more danger by slowing down our alignment progress than we are by giving AI’s access to our notes.
Thank you so much for the clarification, Jay! It is extremely fair and valuable.
The underlying question is: does the increase in the amount of AI safety plans resulting from coordinating on the Internet outweigh the decrease in secrecy value of the plans in EV? If the former effect is larger, then we should continue the status-quo strategy. If the latter effect is larger, then we should consider keeping safety plans secret (especially those whose value lies primarily in secrecy, such as safety plans relevant to monitoring).
The disagreeing commenters generally argued that the former effect is larger, and therefore we should continue the status-quo strategy. This is likely because their estimate of the latter effect was quite small and perhaps far-into-the-future.
I think ChatGPT provides evidence that the latter should be a larger concern than many people’s prior. Even current-scale models are capable of nontrivial analysis about how specific safety plans can be exploited, and even how specific alignment researchers’ idiosyncrasies can be exploited for deceptive misalignment.
I am uncertain about whether the line between 2 and 3 will be narrow. I think the argument of the line between 2 and 3 being narrow often assumes fast takeoff, but I think there is a strong empirical case that takeoff will be slow and constrained by scaling, which suggests the line between 2 and 3 might be larger than one might think. But I think this is a scientific question that we should continue to probe and reduce our uncertainty about!
Great list, thanks for sharing!
Animal Advocacy Careers also has a Job Board.
If you don’t mind me plugging my own org, Vegan Hacktivists specializes in building software for other animal advocacy/protection orgs, along with offering tech, design and advisory services, all for free.
Will keep an eye out on this thread, would love to learn about other organizations using technology as a catalyst for good!
The vegan hacktivists sounds really cool and would love to help out! What kind of work/impact do the projects yall do have? Is it mainly with other ea animal orgs?
I think we should not make compromises that decrease impact/dollar to benefit Americans. Americans are not more valuable than people from Poland or Pakistan or Kenya.
I agree that Americans are not more valuable than any other people, but EA orgs like GiveDirectly have made the choice to support impoverished Americans because they are based in the US and can reach them more easily.
I am skeptical that that’s the reason for it. I haven’t seen numbers on this but I would be shocked if the cost of delivery of any intervention to Americans is less than delivery in developing countries, because operating costs for any intervention are higher in the US—hiring people to identify recipients, for example, would be an order of magnitude more expensive than doing the same in Kenya.
I suspect the real reason, which is maybe worth centering in this post, is that the pool of money that people are potentially willing to donate to Americans is a) very large, b) not fungible with money they would otherwise donate to global poverty.
Hi,
Field studies are outside of the scope of Reslab, right?
I guess it depends on what exactly you’re referring to — but probably, yes.
Very good article! Many EA orgs are start-ups so it’s natural for them to start with a relatively inexperienced team. However, it’s normal for start-ups to bring in senior people at the top as they grow rather than hiring from the bottom, which is what most EA orgs seem to. The best CEO for a 5-person organisation is rarely the same as the best CEO for a 100-person organisation. CEOs need to get better at recognising their own weaknesses and Boards need to get better at transitioning founders out of the CEO role. In my opinion, EA orgs also need to increase the weight of experience and decrease the weight of moral philosophy / being in certain cliques.
Would one possible solution to some of these problems be to hire much more outside EA? Move “familiarity with EA” to the “bonus” part of the job requirements, and instead look for experienced managers as a main criterion?
Hi Nathan,
Thanks for writing this!
I only saw your post now, but I had made a draft for a question related to the karma system about 1.5 months ago. Feel free to have a look.
Interestingly, the estimates I got for the correlation coefficient between karma and impact based on this analysis from Nuño are pretty similar to the 0.47 you obtained for the relationship between the inflation-adjusted karma and ranking of the posts of the Decade Review.
I’ve been studying religions a lot and I have the impression that monasteries don’t exist because the less fanatic members want to shut off the more fanatic members from rest of society so they don’t cause harm. I think monasteries exist because religious people really believe in the tenets of their religion and think that this is the best way for some of them to follow their religion and satisfy their spiritual needs. But maybe I’m just naive.
Thanks for sharing! The speakers on the podcast might not have had the time to make detailed arguments, but I find their arguments here pretty uncompelling. For example:
They claim that “many belief systems they have a way of segregating and limiting the impact of the most hardcore believers.” But (at least from skimming) their evidence for this seems to be just the example of monastic traditions.
A speaker claims that “the leaders who take ideas seriously don’t necessarily have a great track record.” But they just provide a few cherry-picked (and dubious) examples, which is a pretty unreliable way of assessing a track record.
Counting Putin a “man of ideas” because he made a speech with lots of historical references—while ignoring the many better leaders who’ve also made history-laden speeches—looks like especially egregious cherry-picking.
So I think, although their conclusions are plausible, these arguments don’t pass enough of an initial sanity check to be worth lots of our attention.
I share this impression of the actual data points being used feeling pretty flimsy
Suppose the Speculation Grantors will consider a grant that is extremely risky. Suppose that grant has a 10% chance to be evaluated in the next round as far more beneficial than all the rest of the Speculation Grants of that round combined; and yet, the grant is net-negative (due to its potential to cause extreme accidental harm, which is not unlikely for extremely-impactful anthropogenic x-risk interventions).
If in your implementation of “impact futures” the “fd_value” of an impact certificate cannot be negative, then to the extent that a Speculation Grantor acts in a way that is aligned with maximizing their expected speculation grant budget, they may bet on that 10% chance and fund the risky intervention. Especially if they think that all the other Speculation Grantors will not fund it, thereby allowing their own budget to receive most of the “settlement_budget” (which seemingly exacerbates the unilateralist’s curse problem).
I co-authored a post about the general risk of impact markets incentivizing risky, net-negative projects. Please consider taking a look at it if you haven’t already.
I think this is perfectly good to try, but I’m personally skeptical that it will end up being especially useful. My sense is that right now, there isn’t a shortage of frontpage content on the forum. Rather, there seems to often be a shortage of deep reading, engagement, and discussion when someone writes a long object-level post. I would be interested to see initiatives aimed at fostering that kind of deeper engagement with content, rather than at trying to get more frontpage posts.
I think it’s important to kindly point out that white people, let alone people of any racial or ethnic or gender background, are not a monolith.
Diversity is more than skin deep. It concerns values, possessed expertise/skills/knowledge, epistemic systems, philosophical and ontological beliefs, spiritual and educational backgrounds, health and disability statuses, age, sexual orientation, occupations, goals, dietary preferences, socioeconomic factors, life experiences.
Arguably many of these categories, particularly ones concerning ethical or epistemic matters alongside cause prioritization, are far more important than what a person looks like or is perceived to be from the outside.
EA should aspire to be accommodating to people of all ethnic and racial backgrounds. Certainly. It should seek to understand why there is a disparity in membership for certain racial classifications relative to US/UK population values or %s and seek to rectify it.
But it is difficult to defend a tendency to equate people who are white or groups with membership that happens to be majority white with a ‘lack of diversity’.
That simply isn’t the case when all categories that constitute a diverse community are considered, not just skin tone or gender.
Truly, if all the white persons in EA thought and acted the same, had the same diet, had the exact same life experiences, age, sexual orientations, causes, held the same or similar occupations, shared the same values, health or disability statuses, epistemic systems, philosophical and ontological beliefs, religious and spiritual backgrounds, educational backgrounds, personal knowledge and skills, and socioeconomic backgrounds then I would say EA has a diversity problem.
But that isn’t simply the case. Every EA is different and unique in all these respects. You are too. An individual person who is fully unlike any other. That’s what makes us diverse. Not a blanketing reduction of people to skin tone or race that missed out on all else that makes us human :)
Thank you for posting this! I’ve been frustrated with the EA movement’s cautiousness around media outreach for a while. I think that the overwhelmingly negative press coverage in recent weeks can be attributed in part to us not doing enough media outreach prior to the FTX collapse. And it was pointed out back in July that the top Google Search result for “longtermism” was a Torres hit piece.
I understand and agree with the view that media outreach should be done by specialists—ideally, people who deeply understand EA and know how to talk to the media. But Will MacAskill and Toby Ord aren’t the only people with those qualifications! There’s no reason they need to be the public face of all of EA—they represent one faction out of at least three. EA is a general concept that’s compatible with a range of moral and empirical worldviews—we should be showcasing that epistemic diversity, and one way to do that is by empowering an ideologically diverse group of public figures and media specialists to speak on the movement’s behalf. It would be harder for people to criticize EA as a concept if they knew how broad it was.
Perhaps more EA orgs—like GiveWell, ACE, and FHI—should have their own publicity arms that operate independently of CEA and promote their views to the public, instead of expecting CEA or a handful of public figures like MacAskill to do the heavy lifting.
I think GiveWell (at least, and maybe ACE too) is already very independent of CEA. The fact that they also haven’t done mass media outreach is probably a result of an independent assessment that it isn’t particularly in their interests to do so. They do have significant marketing and media outreach, e.g. I’ve seen YouTube ads, and I know they’re on podcasts sometimes, so I feel safe guessing that media exposure they haven’t had is because they haven’t pursued it rather than because they don’t have the expertise to do so.
More spending and effort placed into publicity arms makes sense. Less cohesion and coordination is a hard sell though, that’s more points of failure at best, and at worst risks exploitation/playing both sides by clever outsiders, or inter-org conflict/retaliation that is triggered by accident instead of deliberately.
There needs to at least be an apparatus for negotiation.
Great points.
There’s an unfortunate dynamic which has occurred around discussions of longtermism outside EA. Within EA, we have a debate about whether it’s better to donate to nearterm vs longterm charities. A lot of critical outsider discussion on longtermism ends up taking the nearterm side of our internal debate: “Those terrible longtermists want you to fund speculative Silicon Valley projects instead of giving to the world’s poorest!”
But for people outside EA, nearterm charity vs longterm charity is generally the wrong counterfactual. Most people outside EA don’t give 10% of their earnings to any effective charity. Most AI work outside EA is focused on making money or producing “cool” results, not mitigating disaster or planning for the long-term benefit of humanity.
Practically all EAs agree people should give 10% of their earnings to effective developing-world charities instead of 1% to ineffective developed-world ones. And practically all EAs agree that AI development should be done with significantly more thought and care. (I think even Émile Torres may agree on that! Could someone ask?)
It’s unfortunate that the internal nearterm vs longterm debate gets so much coverage, given that what we agree on is way more action-relevant to outsiders.
In any case, I mention this because it could play into your “ideologically diverse group of public figures” point somehow. Your idea seems interesting, but I also don’t like the idea of amplifying internal debates further. I would love to see public statements like “Even though I have cause prioritization disagreements with Person X, y’all should really do as they suggest!” And acquiring a norm of using the media to gain leverage in internal debates seems pretty bad.
Yeah, it’s the narcissism of small differences. If we’re gonna emphasize our diversity more, we should also emphasize our unity. The narrative could be “EA is a framework for how to apply morality, and it’s compatible with several moral systems.”
So, proposing that we give everyone equal voting power gives those on the forum with more voting power an incentive to lessen mine (by downvoting this). So how about this: we make the agreement karma democratic. That way we can see what people actually agree or disagree on and since it doesn’t affect karma we can make it democratic without angering those with disproportionate voting power.
Remember the anti-work subreddit moderator’s disastrous unsanctioned interview? Let that be a cautionary tale of how interacting with the media can go. It should be self evident that only CEA sanctioned individuals should be allowed to speak to the media.
That mod was as central to r/antiwork as it was possible to be: https://tracingwoodgrains.medium.com/r-antiwork-a-tragedy-of-sanewashing-and-social-gentrification-56298af1c1a7
The anti-work interview was kind of horrible, but not because Ford misrepresented what the sub was about, or wasn’t the right person to speak on its behalf, or wasn’t sanctioned to speak to the media. Doreen Ford built the sub. There was no one better placed to do that interview or to approve anyone else to do it.
What the interview actually did was reveal what the sub was actually about to the recent flood of newcomers. They thought they were there to demand better rights for workers or something, but r/antiwork actually existed to oppose the concept of work itself.
You see these things sometimes—a group gets a lot of new entrants who haven’t really internalised the values of the group, and then there’s some kind of drama between the new blood and the old guard over the group’s core beliefs. There was a much lower-key thing on r/wallstreetbets back when GameStop stock shot up in value—a bunch of people flooded in wanting to stick it to the greedy hedge funds. This was all good for a while while they were all posting about how GME was going to the moon etc. But as time went on, you saw a disconnect—newbies saw Martin Shkreli a.k.a the Pharma Bro as emblematic of the kind of greedy capitalist they hated, while to the old guard Shkreli was a hero (they made him a mod).
The antiwork drama isn’t a lesson about the perils of media engagement—it’s an example of a community partially losing its values due to rapid growth.
I didn’t know about this story, so for people who are curious: https://www.independent.co.uk/news/world/americas/anti-work-reddit-abolishwork-fired-b2002606.html
Before this gets out of hand—I think the request referred to was literally me writing down a random idea to get the ball rolling.
I also strongly agree! I think this an important topic that needs to be discussed more frequently within this community. I’m curious whether most EA participants are on the same page that the lack of demographic diversity is harmful to effectiveness.
The EA events I have attended have been much whiter (and more predominantly male) than the general population of my area, and many conversations have had an exclusive, elitist vibe to them. (This is obviously subjective but to me this manifested as people immediately asking people about their credentials, and folks initiating group conversations with narrow intellectual topics that are not inclusive.)
First,
This is useful information to know! Thanks for sharing it. I just want to send good vibes for “if you see something you see is weird, lean towards transparency.”
My personal intuition is that it’s good to have a culture where people are politely, and perhaps lightly, headhunted.
However, there definitely could be more issues here:
It seems bad if some groups/orgs get highly targeted. This can create an uncomfortable situation if not handled well.
It’s hard to be polite. I could easily imagine recruiters being pushy or rude.
I imagine that perhaps in this case at least, some orgs used similar logic to target IDinsight (lots of great talent and training, but work seems less directly aligned for specific EA goals), but didn’t realize how many others were doing it.
I’d recommend messaging the Community Health team at CEA to get a bit of coordination. Or just directly send an email to the various orgs flagging the issue. I imagine they might well be able to find a more reasonable solution.
I’m nervous that readers might conflate “the specific situation of IDInsight, which we still know little about, is justified” with, “it’s generally good to have more headhunting”. I mostly agree with the latter, but can’t say much on the former.
I really want to be in favor of having a less centralized media policy, and do think some level of reform is in-order, but I also think “don’t talk to journalists” is just actually a good and healthy community norm in a similar way that “don’t drink too much” and “don’t smoke” are good community norms, in the sense that I think most journalists are indeed traps, and I think it’s rarely in the self-interest of someone to talk to journalists.
Like, the relationship I want to have to media is not “only the sanctioned leadership can talk to media”, but more “if you talk to media, expect that you might hurt yourself, and maybe some of the people around you”.
I think almost everyone I know who has taken up requests to be interviewed about some community-adjacent thing in the last 10 years has regretted their choice, not because they were punished by the community or something, but because the journalists ended up twisting their words and perspective in a way both felt deeply misrepresentative and gave the interviewee no way to object or correct anything.
So, overall, I am in favor of some kind of change to our media policy, but also continue to think that the honest and true advice for talking to media is “don’t, unless you are willing to put a lot of effort into this”.
I upvoted this post, however I think this section is exaggerrated.
When this was brought up on Twitter, someone brought up a survey for how much people who were involved in events felt journalists accurately characterized them. iirc it was something like 20% substantively/entirely accurate, 60% minor errors but broad gist is true, 20% majorly false.
I couldn’t find the study again and I don’t know how good it was. But at least your comment seems maybe an overestimate.
I appeared on a radio program on behalf of EA London in 2018 and don’t regret it. I thought the coverage was fair to positive.
Here’s another example I think went well, although I don’t know the people involved! https://www.theguardian.com/lifeandstyle/2022/apr/25/how-we-met-i-had-another-date-lined-up-on-tinder-but-i-realised-i-wanted-to-be-with-ben
It sounds like there are two main issues:
When is talking to journalists in the self-interest of an individual or very small org?
When is talking to journalists going to contribute positively to the world?
My gut instinct is that the latter will hold more often than the former, since it develops a wider public discussion, implying that sometimes altruists might want to talk to the media even if they feel it will cast them in a poor light. Over time, as a community we can share our experiences and and collectively decide which publications and individuals we trust to have a conversation with, possibly on which topics or in which broader contexts. One of CEA’s roles could then be to seek to build new such relationships to share with the community, widening our set of options rather than restricting it.
Would be interested in hearing more, like what those interviews were about and whether the interviewed people were mostly from the Bay area and/or part of the rationality community. Could imagine that I wouldn’t want to strongly extrapolate from those experiences to potential media interviews for learning about EA in Germany, for instance.
In 2014 or 2015, several of us in Seattle talked to a journalist who we were told was doing an article on young philanthropists. 3 or 4 people had long interviews with her, and she also took over an EA meeting she’d been invited to observe. When the article came out, it was about how awful young people were for caring about 3rd world poverty instead of the opera.
I also sounded like a goddamn idiot. The journalist asked an absolutely ridiculous question, I worked to answer in a way that wasn’t “I’m sorry, you think what?”, and the quote got used. It accurately reflected my opinion (“no, opera outreach programs aren’t more important than malaria nets”) but I sound stupid because I couldn’t think fast enough to sound smart and not-hostile.
FWIW I remember reading that article and thinking that the net takeaway (from people we want to attract) is neutral or positive towards us. Like if someone doesn’t even believe in the idea of cause prioritization, we are not the right community for them.
Yeah I think this was a relatively gentle introduction to misleading journalists, in that the article’s slant was so obvious and enough people were not on its side that it wasn’t damaging.
Yes I don’t think you sound stupid at all Elizabeth, I think EA comes across reasonably well in the piece and the kind of person who’d be interested in effective giving might Google it because of you.
This seems like the sort of thing where it would really help to have a public database of ‘journalists who we’ve discovered are sucky and journalists who we’ve discovered aren’t sucky’, both as a very mild deterrent and more importantly so future EAs can avoid talking to this particular person.
In terms of understanding the causal effect of talking to journalists, it seems hard to say much in the absence of an RCT.
Someone ought to flip a coin for every interview request, in order to measure (a) the causal effect of accepting an interview on probability of article publication, and (b) the direction of any effects on article accuracy, fairness, and useful critique.
(That was meant as a bit of a joke, but I would honestly be delighted to see a bunch of articles about EA which include sentences like “Person X did not offer any comment because we weren’t assigned to the interview acceptance group in their RCT”. Seems like it sends the right signal to the sort of people we want to attract.)
In any case, until that RCT gets run, maybe it would be worthwhile to compare articles informed by interviews and articles uninformed by interviews side-by-side, and do what we can with the data we have. It’s easy to say “I talked to the journalist and the article was inaccurate”. But claiming that the article ended up worse than it would’ve been in the absence of an interview is harder. (There are also complicating factors: an article with quotes from relevant people may seem more legitimate to readers; no interview might mean no article.)
That might at least be a good way of establishing a lower bound for EV from talking to journalists.
Do you have thoughts about the idea of creating a thread on a site like the EA Forum or Less Wrong where someone takes questions from the media and responds in writing publicly? 3 birds with one stone: written responses can be more considered, public source material discourages misrepresentation, and less need to respond to the same question multiple times.
(This was Wei Dai’s idea for handling journalist questions about Bitcoin.)
I think something like that is a better idea. Or separately, for people to just write up their takes in comments and posts themselves. I’ve been reasonable happy with the outcomes of me doing that during this FTX thing. I think I’ve been quoted in one or two articles, and I think those quotes have been fine.
I agree that public communication is risky, but I think that plenty more people are qualified to do it than just CEA and the movement’s “big three” public intellectuals (MacAskill, Ord, and Singer). My comment here was partly a response to this one.