I agree that in many cases, when bad-faith critics say “EA should not do X” and we stop doing X, the world would actually become worse off.
But making SBF the face of the EA movement was a really bad decision. Especially given that he was unilaterally gambling with the whole EA movement’s credibility.
There are robust lessons to be learned in this saga, which will allow us EAs to course-correct and prevent future catastrophic outcomes (to our movement and in general).
Replicate the Project Ideas Competition for other types of communities than EAs
Research That Can Help Us Improve
People have contributed a lot of really insightful and promising ideas here. Given that “there are no wrong ideas in brainstorming” and that there may be systematic blind spots for effective altruists/longtermists’ paradigm, perhaps doing this broad-idea-crowdsourcing exercise in other types of communities could get us new, potentially promising ideas.
Pipeline for writing books
Effective altruism
It’s plausible that more EAs/longtermists should be writing books on the interesting subjects they are experts in, but they currently do not because of a lack of experience or other types of friction. Crowdsourced resources, networks, and grants may help facilitate this. Books written by EAs would have at least two benefits: (a) dissemination of knowledge, and (b) earning-to-give opportunities (via royalties).
I think we EAs need to increasingly prioritize speaking up about concerns like the ones Habryka mentioned.
Even when positive in-group feelings, the fear of ostracism, and uncertainty/risk aversion internally influences one to not bring up these concerns, we should fight back against this urge because the concerns, if true, will likely grow larger and larger until they blow up.
There is very high EV in course correction before the catastrophic failure point.
Targeted practical statistical training
Economic Growth, Values and Reflective Processes
“Human cognition is characterized by cognitive biases, which systematically lead to errors in judgment: errors that can potentially be catastrophic (e.g., overconfidence as a cause of war). For example, a strong case can be made that Russia’s invasion of Ukraine has been an irrational decision of Putin, a consequence of which is potential nuclear war. Overconfidence is a cause of wars and of underpreparation for catastrophes (e.g., pandemics, as illustrated by the COVID-19 pandemic).
One way to reduce detrimental and potentially catastrophic decisions is to provide people with statistical training that can help empower beneficial decision-making via correct calibration of beliefs. (Statistical training to keep track of the mean past payoff/observation can be helpful in a general sense; see my paper on the evolution of human cognitive biases and implications.) At the moment, statistical training is provided to a very small percentage of people, and most provisions of statistical training are not laser-focused on the improvement of practical learning/decision-making capabilities, but for other indirect goals (e.g., prerequisite for STEM undergraduate majors). It may be helpful to (1) encourage practical, impactful aspects in the provision of statistical training and (2) broaden its provision to a wider segment of people.”
(Quote from my post ’Broadening statistical education”)
Given resource limitations, it may make sense to target the provision of practical statistics training to high-impact decision-makers, such as those in government. An ambitious example is that just as the US president is given a confidential briefing about the nuclear protocols, so too can the president be briefed about statistical reasoning and how to thereby make well-calibrated decisions on behalf of the nation.
Mental health treatment to prevent anthropogenic catastrophic/existential risks
Biorisk and Recovery from Catastrophe
Issues of mental health can be very harmful to the well-being of the self and others. The degree to which this harm can occur can, when combined with technology, even result in catastrophic/existential risks. (The Russian invasion of Ukraine, the cause of which may be the mental state of Putin, can plausibly lead to nuclear war. Another example is engineered pandemics.) Given the disproportionately anthropogenic skew of catastrophic/existential risks, research/funding/advocacy for mental health treatment (general or targeted) may help prevent such risks.
Facilitating relocation
Economic growth, Effective altruism
People are over-averse to moving, even if it moving leads to much better opportunities (e.g., when a volcano destroyed a fraction of nearby houses, their inhabitants who were forced to move ended up better off on earnings and education, conditional on being young; see this paper). Research and incentivization can help reduce this over-aversion.
It is plausible that even EAs underconsider relocation.If so, it means a lot of impactful value may be achieved by convincing and facilitating EAs’ relocation to high-impact career opportunities.
Movement-building targeted at existential-risk-relevant fields’ international scientific communities
Biorisk, Artificial Intelligence
Scientists of the Manhattan Project built the first nuclear bombs, the development and use of which normalized nuclear proliferation. Contrast this with bioweapons, which in principle could also have been normalized if not for the advocacy of scientists like Matthew Meselson, which led to a lasting international agreement to not develop bioweapons (Biological Weapons Convention).
Targeted efforts to build the movement of reducing catastrophic/existential risks (and longtermism in general) specifically in the international scientific communities of fields that are highly relevant to certain existential risks, whose lasting cooperation would be crucial for the non-realization of these risks, could potentially be very impactful. Some potential approaches include funding of fellowships/grants/collaboration opportunities, creating scientific societies/conferences, and organizing advocacy/outreach/petitions.
Optimal 90-second pitches for EA/longtermism
Effective altruism
Longtermism is nuanced; a full discussion requires a large amount of time. It is possible that more people than currently may be interested in learning more about the movement if they are presented with a short but compelling pitch that is suited for the quickness of many people’s lifestyle. (I’ve stated a spontaneous and very suboptimal pitch for EA on at least one occasion, which I regret.)
Optimized 90-second or so pitches may potentially help the movement’s outreach. Persuasive pitches (each focused on each of a myriad of topics/angles that the listener may be interested in) can be selected by community contests/focus groups and posted online, both for viewing and for informing movement builders’ efforts.
Thank you so much, Geoffrey, for this compelling argument! I completely agree that a moral backlash against AI is very plausible (and in my estimation, imminent), especially from people whose career paths are or will be indefinitely automated away by AI, and who will not thrive in the new niche of ‘startup founder who only hires AGI employees.’
OpenAI’s mission is to create “highly autonomous systems that outperform humans at most economically valuable work” (their definition of AGI).
I cannot overstate how broadly unpopular this mission is. In fact, in my experience, whenever I told someone occupationally far-removed from AI about OpenAI’s mission, they immediately diagnosed this mission as dystopian. They were also very open-minded about the plausibility of catastrophic and existential dangers in such an AGI-led future, with very few exceptions.
The only bottleneck is that most people currently don’t believe that AGI is imminent. This is, of course, starting to change: for example, with the recent CAIS statement signed by leading AI researchers and notable figures.
We should tirelessly prepare for a future in which AGI leaves large numbers of people jobless, purposeless, and angry. I don’t know all the answers to how we should prepare for this dystopia, but I’m confident that we should prioritize the following two things:
(1) Truthful, frequent, and high-quality communication to angry people. e.g., communication about the true cause of their hardships—AGI companies—so that they don’t blame a scapegoat.
(2) Make preparations for the prevention and resolution of the first (near-catastrophic or catastrophic) AI disaster, as well as for proposing and implementing effective AI-disaster-prevention policies after the first AI disaster.
Reduce meat consumption
Biorisk, Moral circle expansion
Research and efforts to reduce broad meat consumption would help moral circle expansion, pandemic prevention, and climate change mitigation. Perhaps messaging from the pandemic-prevention angle (in addition to the climate change angle and the moral circle expansion angle) may help.
Thank you very much for the constructive criticisms, Max! I appreciate your honest response, and agree with many of your points.
I am in the process of preparing a (hopefully) well-thought-out response to your comment.
Longtermism movement-building/election/appointment efforts, targeted at federal and state governments
Effective altruism
Increasing knowledge of and alignment with longtermism in government by targeted movement-building and facilitating the election/appointment of sympathetic people (and of close friends and family of sympathetic people) could potentially be very impactful. If longtermism/EA becomes a social norm in, say, Congress or the Washington ‘blob’, we could benefit from the stickiness of this social norm.
I strongly agree with the spirit of the reforms being suggested here (although I might have some different opinions on how to implement it). We need large-scale reforms of the EA community’s social norms to prevent future risks to movement-wide credibility.
Strongly agree. The fact that net upvotes are the only concrete metric by which EA forum posts and LessWrong forum posts are judged has indeed been suboptimal for one of EA’s main goals: to reflect on and adapt our previous beliefs based on new evidence. Reforms designed to increase the engagement of controversial posts would be very helpful for our pursuit of this goal. (Disclaimer: Most of my EA forum posts would rank highly on the “controversial” scale, in that many people upvote and many people downvote them, and the top comment is usually critical and has a lot of net upvotes. I think that we EAs need to increasingly prioritize both posting and engaging with controversial arguments that run contrary to status-quo beliefs, even if it’s hard! This is especially true for LessWrong, which arguably doubles as a scientific venue for AI safety research in addition to an EA-adjacent discussion forum.)
Agree, although I think EAs should be more willing to write and engage with controversial arguments non-anonymously as well.
Strongly agree in spirit. While a norm of unconditionally refusing non-anonymous donations above a certain threshold might be too blunt, I do think we need to have better risk-management about tying our EA movement’s credibility to a single charismatic billionaire, or a single charismatic individual in general. Given how important our work is, we probably need better risk-management practices in general. (And we EAs already care earnestly about this! I do think this is a question not of earnest desire but of optimal implementation.) I also think that many billionaires would actually prefer to donate anonymously or less publicly, because they agree with the bulk of but not all of EA’s principles. Realistically, leaving room for case-by-case decision-making seems helpful.
Increasing social norms of moral circle expansion/cooperation
Moral circle expansion
International cooperation on existential risks and other impactful issues is largely downstream of social norms of, for example, whether foreigners are part of one’s moral circle. Research and efforts to encourage social norms of moral circle expansion and cooperation to include out-group members could potentially be very impactful, especially in relevant countries (e.g., US and China) and among relevant decision-makers.
Strongly agree with all of these points.
On point 2: The EA movement urgently needs more earners-to-give, especially now. One lesson that I think is correct, however, is that we should be wary of making any one billionaire donor the face of the EA movement. The downside risk—a loss of credibility for the whole movement due to unknown information about the billionaire donor—is generally too high.