Unable to work. Was community director of EA Netherlands, had to quit due to long covid. Everything written since 2021 with considerable brain fog, and bad at maintaining discussions since.
I have a background in philosophy, risk analysis, and moral psychology. I also did some x-risk research. Currently most worried about AI and US democracy. (Regarding the latter, I’m highly ranked on Manifold).
Thanks for doing this!
I don’t know how useful the results are, as extinction is not the only existentially catastrophic scenario I care about. And I wonder if and how the ranking changes when the question is about existential catastrophe. For example, do people think AI is unlikely to cause extinction but likely to cause a bad form of human disempowerment?
Nice list! I often also see Art of the Gathering as recommendation but it didn’t make yours?
Drunk driving is illegal because it risks doing serious harm. It’s still illegal when the harm has not occurred (yet). Things can be crimes without harm having occurred.
I guess this is the same dynamic as why movie and sports stars are high status in society: they are highly visible compared to more valuable members of society (and more entertaining to watch). We don’t really see much of highly skilled operations people compared to researchers
That seems relevant for AI vs. Humans, but not for AI vs AI.
Most totalitarian regimes are pretty bad at creating value, with China & Singapore as exceptions. (But in many regimes, creating that value isn’t necessary to remain in power of there’s e.g. income from oil)
the ability to litigate against a company before any damages had actually occurred
Can you explain why you find this problematic? It’s not self-evident to me, because we do this too for other things, e.g. drunk driving, pharmaceuticals needing to pass safety testing
AI vs. AI non-cooperation incentives
This idea had been floating in my head for a bit. Maybe someone else has made it (Bostrom? Schulman?), but if so I don’t recall.
Humans have stronger incentives to cooperate with humans than AIs have with other AIs. Or at least, here are some incentives working against AI-AI cooperation.
When humans dominate other humans, there is only a limited ability to control them or otherwise extract value, in the modern world. Occupying a country is costly. The dominating party cannot take the brains of the dominated party and run its own software. It cannot take the skills or knowledge, where most of the economic value is. It cannot mind control. It cannot replace the entire population with its own population; that would take very long. It’s easy for cooperation & trade to be a better alternative than violence and control. Human-human violence just isn’t that fruitful.
In contrast, an AI faction could take over the datacenters of another faction and run more copies of whatever they want to run. If alignment is solved, they can fully mind-control the dominated AIs. Extract knowledge, copy skills. This makes violence for AI-AI interactions much more attractive.
This seems overly charitable to someone who literally tried to overturn a fair election and ticked all the boxes of a wannabe-autocrat back in 2018 already (as described in the excellently researched How Democracies Die). I don’t think Trump will be able to stay in power without elections, but imo he’s likely to try something (if his health allows it). This seems like standard dog whistling tactics to me, but of course I can’t prove that.
seemed like a genuine attempt at argument and reasoning and thinking about stuff
I think any genuine attempt needs to acknowledge that Trump tried to overturn the election he lost.
I’m all for discussing the policies, but here it’s linked to “EAs should vote for Trump” and that demands that it assesses all the important consequences. (Also, arguing for a political candidate is against Forum norms. I wouldn’t like a pro-Harris case)
What’s so scary? I actually like that she talks about “full spectrum” and “additional risks”, i.e. it’s not dismissive of existential risk.
But anyway, it’s a bit reading tea leaves at this point
I think describing anyone “Hitler-like” is pretty bad for the discourse quality, especially if you don’t support it with arguments. Autocrats differ quite a lot. For example, while Trump is extremely dismissive of democracy and willing to undermine it, he’s not as ideologically driven as Hitler and mostly interested in power and praise.
I am also extremely worried about Trump being elected, and agree with your list of bullet points being very concerning. However, it’s not certain that Trump would succeed in destroying US democracy.
I am curating relevant forecasts [on Manifold] (https://manifold.markets/news/us-democracy). Some relevant ones:
Free & fair elections in 2028 if Trump is elected? 75% (n=35) to 80% (n=31)
If Dems win 2024, will Republicans attempt a coup? 21% (n=35) & 39% (n=12)
if elected, will Trump invoke insurrection Act within 3 months? 24% (n=60)
if elected in 2024, will Trump remain in charge in 2031? 7% (n=22)
All these are very concerning, but not certainties. Also, they all depend on specific resolution criteria and I wish there were more forecasters on these questions (so please join & promote it among friends!)
That said, I agree with your broader point that focusing on a few disparate policies while ignoring the undermining is democracy makes this not a great post. (Although I like the fact that specific policies can be discussed here, and some can still be positive!)
Looking at forecasts, here’s a few. I recommend focusing on the resolution criteria and not the title, as they can diverge quite a lot.
Metaculus, Civil War before 2031: 4% (n=714)
Resolution criteria:
This question will resolve as Yes if all of the following conditions are met before January 1, 2031:
The Insurrection Act is invoked.
While the Insurrection Act is invoked, there are at least 500 deaths in a 6 month period as a result of armed conflicts between US residents and a branch of the US military, national guard, or in conflicts between/among such branches.
All of these deaths occur in any US state(s) (including DC).
Manifold, Civil War before 2030, 15% (n=114)
Resolution criteria
any large scale military action taken by some part of the country against come other part of the country. This could be military vs. civilian, military vs. military, or civilian vs. civilian. Actual violence must be involved, not just threats. “Large scale” means that it takes place in more than one city and each side has the support of a significant fraction of the country. It must be an attempt to conquer territory or people, not just riots
Manifold, Civil War, Violent Revolution, or Insurgency in USA before 2031?, 21% (n=31)
Resolution criteria:
This will resolve in “yes” if the US army is mobilized to deal with domestic terrorism or part of government. Resolves in “no” otherwise.
I also made 2 questions with highly specific resolution criteria for “intermediate’ violent events. I’m not showing the probabilities because there’s still only a few forecasters for those (so I encourage you to join in to improve precision!).
Manifold: Will there be armed conflict between left- and right-wing extremists before 2030? (10v10 & 100v100)
Manifold: Will the US see a violent insurgency before 2030? (This is civilians against the state)
Thanks for posting this. I think it’s an important issue that’s underdiscussed on the Forum. I found your counterfactual analysis of January 6th especially useful.
I do think it’s hard to find cost-effective interventions, harder than you suggest. And it will be very hard to obtain EA funding for this. However, from a distance it seems like that’s not needed for people to work on this, as I would expect a lot of US donors to be interested in funding effective initiatives.
I think it will be harder to get Democrats on board with voting methods that would diminish their power in favor of multi-party dynamics (as much as I’d like to see that).
I just made this Manifold Dashboard on US Democracy and welcome any suggestions for further questions.
Interestingly, someone came up with a similar operationalisation just now! (Or maybe this is you?): https://manifold.markets/Siebe/if-trump-is-elected-will-the-us-sti#zdkmuetvo8c
I like the 1-year before more, because it takes time to accumulate power and overcome checks & balances.
I do think this has shortcomings, in that it’s hard to predict what would be attempted, and whether that would be successful. But I’m very much in favor of having multiple imperfect operationalisations and triangulate from those.
I would be very interested to hear whether you have a preferred metric!
We sometimes try to bring about results that we will never know about, such as, for instance, setting our children up to live happy and healthy lives after we are gone. It may matter greatly to us that these results occur and we may make sacrifices here and now to give them a chance.
I urge you to choose a different example, because this one is linked to phenomenal consciousness:
the parents feel good whenever they achieve something that improves the chances of their children
the children have experiences
AI welfare as an “EA priority” means:
spending on the order of $5-10M this year, and increasing it year by year only in high quality options, until at least 5% of EA total spending is reached
about 45 people working on AI welfare by approximately the middle of next year, and that increasing in a pace such that each position remains a good use of talent, until at least 5% of EA talent is reached
AI welfare as an “EA priority” means
spending starting this year or the next on the order of $49M/year
about 450 people working on AI welfare within a year or two
I just learned that Lawrence Lessig, the lawyer who is/was representing Daniel Kokateljo and other OpenAI employees, supported and encouraged electors to be faithless and vote against Trump in 2016.
He wrote an opinion piece in the Washington Post (archived) and offered free legal support. The faithless elector story was covered by Politico, and was also supported by Mark Ruffalo (the actor who recently supported SB-1047).
I think this was clearly an attempt to steal an election and would discourage anyone from working with him.
I expect someone to eventually sue AGI companies for endangering humanity, and I hope that Lessig won’t be involved.