I’ve worked in the public sector for 13 years trying my best to make a positive difference. Governments, of course, have a significant role to play in solving issues like climate change, but I increasingly see pivotal change as something the private sector can deliver. Given that I am limited by my current employer in the activities I can engage in, my support to EA related causes has been limited to informally advising an organization that receives funding from EA donors.
Zed Tarar
Karma: 47
These are all great suggestions! As for my objections to EA as a whole versus a subset, it reminds me a bit of a defense that folks employ whenever a larger organization is criticised. Defenses that one hears from Republicans in the US for example. “It’s not all of us, just a vocal subset!” That might be true, but I think it misses the point. It’s hard to soul-search and introspect as an organization or a movement if we collectively say, “not all-EA” when someone points to the enthusiasm around SBF and ideas like buying up coal mines.
ah yes sorry, here it is: https://link.springer.com/chapter/10.1007/978-3-662-45704-7_2
Love this thoughtful response!
Good feedback—I see the logic of your points, and don’t find faults with any of them.On AIXR as valid and what the response would be, you’re right; I emphasize the practical nature of the policy recommendation because otherwise, the argument can veer into the metaphysical. To use an analogy, if I claim there’s a 10% chance another planet could collide with Earth and destroy the planet in the next decade, you might begrudgingly accept the premise to move the conversation on to the practical aspect of my forecast. Even if that were true, what would my policy intervention look like? Build interstellar lifeboats? Is that feasible in the absence of concrete evidence?
Agree—armchair psychoanalysis isn’t really useful. What is useful is understanding how heuristics and biases work on a population level. If we know that, in general, projects run over budget and take longer than expected, we can adjust our estimates. If we know experts mis-forecast x-risk, we can adjust for that too. That’s far from psychoanalysis.
I don’t really know what the median view on AIXR within EA communities truly is. One thing’s for certain: the public narrative around the issue highly tilts towards the “pause AI” camp and the Yudkowskys out there.On the common sense of X-risk—one of the neat offices that few people know of at the State Department is the Nuclear Risk Reduction Center or NRRC. It’s staffed 24⁄7 and has foreign language designated positions, meaning at least someone in the room speaks Russian, etc. The office is tasked with staying in touch with other nations to reduce the odds of a miscalculation and nuclear war. That makes tons of sense. Thinking about big problems that could end the world makes sense in general—disease, asteroids, etc.
What I find troubling is the propensity to assign odds to distant right-tail events. And then to take the second step of recommending costly and questionable policy recommendations. I don’t think these are EA consensus positions, but they certainly receive outsized attention.
Exactly, the burden of proof lies with those who make the claim.
I hope EA is able to get back to the basics of doing the most real-world good with limited resources rather than utilitarian nonsense of saving trillions of theoretical future humans.
These are all very reasonable positions, and one would struggle to find fault with them.
Personally, I’m glad there are smart folks out there thinking about what sorts of risks we might face in the near future. Biologists have been talking about the next big pandemic for years. It makes sense to think these issues through.
Where I vehemently object is on the policy side. To use the pandemic analogy, it’s the difference between a research-led investigation into future pandemics and a call to ban the use of CRISPR. It’s impractical and, from a policy perspective, questionable.
The conversation around AI within EA is framed as “we need to stop AI progress before we all die.” It seems tough to justify such an extreme policy position.
This is true—it’s the same tactic anti-GMO lobbies, the NRA, NIMBYs, and anti-vaxxers have used. The public as a whole doesn’t need to be anti-AI, even a vocal minority will be enough to swing elections and ensure an unfavorable regulatory environment. If I had to guess, AI would end up like nuclear fission—not worth the hassle, but with no off-ramp, no way to unring the alarm bell.
I think you put it well when you said:
“Some people think that the kinds of risks I’m worried about are far off, farfetched or ridiculous.”
If I made the claim that we had 12 months before all of humanity is wiped by an asteroid, you’d rightly ask me for evidence. Have I picked up a distant rock in space using radio telescopes? Some other tangible proof? Or is it a best-guess, since, hey, it’s technically possible that we could be hit with an asteroid on any given year. Then imagine if I advocate we spend two percent of global GDP preparing for this event.
That’s where the state of AGI fear is—all scenarios depend on wild leaps of faith and successive assumptions that build on each other.
I’ve attempted to put this all in one place with this post.
Let’s look at the history of global bans:
- They don’t work for doping in the Olympics.
- They don’t work for fissile material.
- They don’t prevent luxury goods from entering North Korea.
- They don’t work against cocaine or heroine.
We could go on. And those examples are much easier to implement—there’s global consensus and law enforcement trying to stop the drug trade, but the economics of the sector mean an escalating war with cartels only leads to greater payoffs for new market entrants.
Setting aside practical limitations, we ought to think carefully before weaponizing the power of central governments against private individuals. When we can identify a negative externality, we have some justification to internalize it. No one wants firms polluting rivers or scammers selling tainted milk.Generative AI hasn’t shown externalities that would necessitate something like a global ban.
Trucks: we know what the externalities of a poorly piloted vehicle are. So we minimize those risks by requiring competence.And on a morally advanced society—yes, I’m certain a majority of folks if asked would say they’d like a more moral and ethical world. But that’s not the question—the question is who gets to decide what we can and cannot do? And what criteria are they using to make these decisions? Real risk, as demonstrated by data, or theoretical risk? The latter was used to halt interest in nuclear fission. Should we expect the same for generative AI?
In my view, we ought to show humility in our ability to accurately forecast risk from any one domain beyond the five-year window. There’s little evidence to suggest anyone is better than random chance at making accurate forecasts beyond a certain time horizon.
At the core of SBF’s grift was the notion that stealing money from Canadian pensioners was justified if that money was spent reducing the odds of an x-event. After all, the end of humanity in the next century would eliminate trillions of potential lives, so some short-term suffering today is easily tolerated. Simple utilitarian logic would dictate that we should sacrifice well-being today if we can prove that those resources have a positive expected value.
I think anyone making an extraordinary claim needs equally extraordinary evidence to back it. That doesn’t mean x-risk isn’t theoretically possible.
Let’s put it this way—if I said there was a chance that life on Earth could be wiped out by an asteroid, few would argue against it since we know the base rate of asteroids hitting Earth is non-zero. We could argue about the odds of getting hit by an asteroid in any given year, but we wouldn’t argue over the very notion. And we could similarly argue over the right level of funding to scan deep space for potential Earth-destroyers, though we wouldn’t argue over the merits of the enterprise in the first place.
That is very different than me claiming with confidence that there is a 25% chance humanity perishes from an asteroid in the next 20 years. And along with this claim I recommend we stop all infrastructure projects globally and direct resources to interstellar lifeboats. You’d rightly ask for concrete evidence in the face of this claim.
The latter is what I hear from AGI alarmists.
Thanks for the thoughtful response.
On background extinction rates, rather than go down that rabbit hole, I think my point still stands, any estimation of human extinction needs to be rooted in some historical analysis. Whether that is one in 87,000 of homo sapiens going extinct in any given year as the Nature piece suggests, or something revised up or down from there.
On false dichotomies—I’d set aside individual behavior for a moment and look at the macro picture. We know from political science basics that elites can meaningfully shift public opinion on issues of low salience. According to Pew, we’ve seen a 15-point shift in the general public expressing “more concern than excitement” over AI in the United States. Rarely do we see such a marked shift in opinion on any particular issue in such a divided electorate.
Let’s put it this way—in a literal sense, yes, one could loudly espouse a belief that AI could destroy humanity within a decade and at the same time, advocate for rudimentary red-teaming to keep napalm recipes out of an LLM’s response, but, in practice, this seems to defy common sense and ignores the effect on public opinion.
Imagine we’re engineers at a new electric vehicle company. At an all hands meeting, we discuss one of the biggest issues with the design, the automatic trunk release. We’re afraid people might get their hands caught in it. An engineer pipes up and says, “while we’re talking about flaws, I think there’s a chance that the car might explode and take out a city block.” Now, there’s nothing stopping us from looking at the trunk release and investigating spontaneous combustion, but in practice, I struggle to imagine those processes happening in parallel in a meaningful way.
Coming back to public opinion, we’ve seen what happens when novel technology gains motivated opponents, from nuclear fission to genetic engineering, to geoengineering, to stem cell research, to gain of function research, to autonomous vehicles, and on. Government policy responds to voter sentiment, not elite opinion. And fear of the unknown is a much more powerful driver of behavior than a vague sense of productivity gains. My sense is that if we continue to see elites writing op-eds on how the world will end soon, we’ll see public opinion treat AI like it treats GMO fruits and veg.My default is to assume folks are sincere in their convictions (and data shows most people are)--I should have clarified that line; it was in reference to claims that outfits calling for AI regulation are cynically putting up barriers to entry and on a path to rent-seeking.
On the pause being a bad idea: my point here is that the very conception is foolish at the strategic level, not that it has practical implementation difficulties. First, what would change in six months? And second, why would creating a prisoner’s dilemma lead to better outcomes? It would be like soft drink makers asking for a non-binding pause on advertising—it only works if there’s consensus and an enforcement mechanism that would impose a penalty on defectors; otherwise, it’s even better for me if you stop advertising, and I continue, stealing your market share.
The IAEA and NPT are their own can of worms, but in general, my broader point here is that even a global attempt to limit the spread of nuclear weapons failed. What is the likelihood of imposing a similar regime on a technology that is much simpler to work with? No centrifuges, no radiation, just code and compute power? I struggle to see how creating an IAEA for AI would have a different outcome.
I once worked on a program with DoD to help buy up loose MANPADS in Libya. There’s a linear causal relationship between portable air defense systems and harm. Other ordnance has a similar relationship.
The relationship is tenuous when we move from the world of atoms to bits. I struggle to see how new software could pose novel risks to life and limb. That doesn’t mean developers of self-driving vehicles or autopilot functions in aircraft should ignore safety in their software design, what I’m suggesting is that those considerations are not novel.
If someone advocates that we treat neural networks unlike any other system in existence today, I would imagine the burden of proof would be on them to justify this new approach.
Red-teaming existential risk from AI
Thanks for sharing, Amy!
The core of EA is indeed a net positive for humanity. And yet like many outfits before it, intra-organizational conflicts and power-seeking behavior can damage the entire collective.
In my opinion, within EA, the utilitarian long-termist emphasis, powered by SBF’s ill-gotten gains, has damaged the entire philosophy’s image. The original emphasis of doing good while pursuing an evidence-based approach seems dwarfed by an obsession with ex-risk. An obsession that lacks common sense, let alone a firm epistemic foundation.
Agreed! My comment was as aimed at the absurd conclusions one makes when weighing the tradeoffs we make today against trillions of unborn humans. That logic leads to extreme positions.