My first gut reaction is skepticism that [a “pretty good” scenario] is a likely or stable state.
I certainly agree that Earthly utopia won’t happen; I just wrote that to illustrate how prosaic values would be disastrous in some circumstances. But here are some similar things that I think are very possible:
Scenarios where some choices that are excellent by prosaic standards unintentionally make great futures unlikely or impossible.
Scenarios where the choices that would tend to promote great futures are very weird by prosaic standards and fail to achieve the level of consensus necessary for adoption.
In retrospect, I should have thought and written more about failure scenarios instead of just risk factors for those scenarios. I expect to revise this post, and failure scenarios would be an important addition. For now, here’s my baseline intuition for a “pretty good” future:
After an intelligence explosion, a state controls aligned superintelligence. Political elites
are not familiar with ideas like long reflection and indirect normativity,
do not understand why such ideas are important,
are constrained from pursuing such goals ( or perhaps because opposed factions can veto such ideas), or
do not get to decide what to do with superintelligence because the state’s decisionmaking system is bound by prior decisions about how powerful AI should be used (either directly, by forbidding great uses of AI, or indirectly, by giving decisionmaking power to groups unlikely to choose a great future)
So the state initially uses AI in prosaic ways and, roughly speaking, thinks of AI in prosaic ways. I don’t have a great model of what happens to our cosmic endowment in this scenario, but since we’re at the point where unwise individuals/institutions are empowered, the following all feel possible:
We optimize for something prosaic
We lock in a choice that disallows intentionally optimizing for anything
We enter a stable state in which we do not choose to optimize for anything
I don’t have much to say about Hanson right now, but I’ll note that a future that involves status-seeking humans making decisions about cosmic-scale policy (for more than a transition period to locking in something great) is probably a failure; success looks more like optimizing the universe.
I suspect that a large fraction of people who seriously start thinking about the longterm future of humanity fall into the camp that you consider “people/institutions that currently want great outcomes”
Historically, sure. But I think that’s due to selection: the people who think about the longterm future are mostly rationalist/EA-aligned. I would be very surprised if a similar fraction of a more representative group had the wisdom/humility/whatever to want a great future, much less the background to understand why we even have a “pretty good” future problem.
If this is true, one might suspect that this will become a much stronger faction
I suspect that humans and institutions will converge considerably towards the “making most of our endowment” stance.
This would surprise me. I expect poor discourse (in the US, at least) about how to use powerful AI. In particular, I expect:
The discourse will focus on prosaic issues like privacy and the future of work.
People will assume that the universe-scale future looks like “humans flying around in spaceships” and debate what those humans should do (rather than “superintelligent von Neumann probes optimizing for something” and debate what they should optimize for — much less recognize that we shouldn’t be thinking about what they should optimize for; we should delegate that decision to a better system than current human judgment).
(Also, your comment implies that aligned superintelligence will try to optimize for all humans’ preferences. I would be surprised if this occurs; I expect aligned superintelligence to try to do what its controller says.)
I would be very excited to call to discuss this further. Please PM me if you’re interested.
Thanks for your comments!
I certainly agree that Earthly utopia won’t happen; I just wrote that to illustrate how prosaic values would be disastrous in some circumstances. But here are some similar things that I think are very possible:
Scenarios where some choices that are excellent by prosaic standards unintentionally make great futures unlikely or impossible.
Scenarios where the choices that would tend to promote great futures are very weird by prosaic standards and fail to achieve the level of consensus necessary for adoption.
In retrospect, I should have thought and written more about failure scenarios instead of just risk factors for those scenarios. I expect to revise this post, and failure scenarios would be an important addition. For now, here’s my baseline intuition for a “pretty good” future:
After an intelligence explosion, a state controls aligned superintelligence. Political elites
are not familiar with ideas like long reflection and indirect normativity,
do not understand why such ideas are important,
are constrained from pursuing such goals ( or perhaps because opposed factions can veto such ideas), or
do not get to decide what to do with superintelligence because the state’s decisionmaking system is bound by prior decisions about how powerful AI should be used (either directly, by forbidding great uses of AI, or indirectly, by giving decisionmaking power to groups unlikely to choose a great future)
So the state initially uses AI in prosaic ways and, roughly speaking, thinks of AI in prosaic ways. I don’t have a great model of what happens to our cosmic endowment in this scenario, but since we’re at the point where unwise individuals/institutions are empowered, the following all feel possible:
We optimize for something prosaic
We lock in a choice that disallows intentionally optimizing for anything
We enter a stable state in which we do not choose to optimize for anything
I don’t have much to say about Hanson right now, but I’ll note that a future that involves status-seeking humans making decisions about cosmic-scale policy (for more than a transition period to locking in something great) is probably a failure; success looks more like optimizing the universe.
Historically, sure. But I think that’s due to selection: the people who think about the longterm future are mostly rationalist/EA-aligned. I would be very surprised if a similar fraction of a more representative group had the wisdom/humility/whatever to want a great future, much less the background to understand why we even have a “pretty good” future problem.
This would surprise me. I expect poor discourse (in the US, at least) about how to use powerful AI. In particular, I expect:
The discourse will focus on prosaic issues like privacy and the future of work.
People will assume that the universe-scale future looks like “humans flying around in spaceships” and debate what those humans should do (rather than “superintelligent von Neumann probes optimizing for something” and debate what they should optimize for — much less recognize that we shouldn’t be thinking about what they should optimize for; we should delegate that decision to a better system than current human judgment).
(Also, your comment implies that aligned superintelligence will try to optimize for all humans’ preferences. I would be surprised if this occurs; I expect aligned superintelligence to try to do what its controller says.)
I would be very excited to call to discuss this further. Please PM me if you’re interested.