Thanks, David. I mostly agree with @Stephen Clare’s points, notwithstanding that I also generally agree with your critique. (The notion of a future dominated by religious fanaticism always brings to mind Frank Herbert’s Dune saga.)
The biggest issue I have is, to echo David, odds of 1 in 30K strike me as far too low.
Looking at Stephen’s math...
But let’s say there’s:
A 10% chance that, at some point, an AI system is invented which gives whoever controls it a decisive edge over their rivals
A 3% chance that a totalitarian state is the first to invent it, or that the first state to invent it becomes totalitarian
A 1% chance that the state is able to use advanced AI to entrench its rule perpetually
That leaves about a 0.3% chance we see a totalitarian regime with unprecedented power (10% x 3%) and a 0.003% (1 in 30,000) chance it’s able to persist in perpetuity.
… I agree with emphasizing the potential for AI value lock-in, but I have a few questions:
Does the eventual emergence of an AI-empowered totalitarian state most likely involve someone to have a “decisive edge” in AI at the outset?
Could an AI-empowered global totalitarian state start as a democracy or even a corporation, rather than being totalitarian at the moment it acquires powerful AI capabilities?
How do we justify the estimation that the conditioned odds of perpetual lock-in are only 1%?
Thanks, David. I mostly agree with @Stephen Clare’s points, notwithstanding that I also generally agree with your critique. (The notion of a future dominated by religious fanaticism always brings to mind Frank Herbert’s Dune saga.)
The biggest issue I have is, to echo David, odds of 1 in 30K strike me as far too low.
Looking at Stephen’s math...
… I agree with emphasizing the potential for AI value lock-in, but I have a few questions:
Does the eventual emergence of an AI-empowered totalitarian state most likely involve someone to have a “decisive edge” in AI at the outset?
Could an AI-empowered global totalitarian state start as a democracy or even a corporation, rather than being totalitarian at the moment it acquires powerful AI capabilities?
How do we justify the estimation that the conditioned odds of perpetual lock-in are only 1%?