Personally, my actual best guess now is that there are many possible non-extinction, non-utopia equilibria, but almost all of them have ~0 value compared to the best possible future. I explore this briefly here. So the shape of my beliefs are closer to your first graph (albeit sharper), but (depending on your definition of existential risk), the verbal description of what I would introduce my belief as would fit closer to your introduction of the third graph.
As an intuition pump, imagine if our descendants aimed pretty hard towards making great lives for all sentient beings, but decided (wrongly) to be biological instead of digital. Even if the per-being utility is the same as their (optimal) digital alternatives, you could easily lose 3 OOMs of efficiency just by forcing a constraint of physical substrate. And this is just one possible disappointing future, and there are many other traps like that. The traps are disjunctive and any one of them can make the value of the future be ~0.0% the value of the best possible future.
I meant broad sense existential risk, not just extinction. The first graph is supposed to represent a specific worldview where the relevant form of existential risk is extinction, and extinction is reasonably likely. In particular I had Eliezer Yudkowsky’s views about AI in mind. (But I decided to draw a graph with the transition around 50% rather than his 99% or so, because I thought it would be clearer.) One could certainly draw many more graphs, or change the descriptions of the existing graphs, without representing everyone’s thoughts on the function mapping percentile performance to total realized value.
Thanks for explaining how you think about this issue, I will have to consider that more. My first thought is that I’m not utilitarian enough to say that a universe full of happy biological beings is ~0.0% as good as if they were digital, even conditioning on being biological being the wrong decision. But maybe I would agree on other possible disjunctive traps.
Personally, my actual best guess now is that there are many possible non-extinction, non-utopia equilibria, but almost all of them have ~0 value compared to the best possible future. I explore this briefly here. So the shape of my beliefs are closer to your first graph (albeit sharper), but (depending on your definition of existential risk), the verbal description of what I would introduce my belief as would fit closer to your introduction of the third graph.
As an intuition pump, imagine if our descendants aimed pretty hard towards making great lives for all sentient beings, but decided (wrongly) to be biological instead of digital. Even if the per-being utility is the same as their (optimal) digital alternatives, you could easily lose 3 OOMs of efficiency just by forcing a constraint of physical substrate. And this is just one possible disappointing future, and there are many other traps like that. The traps are disjunctive and any one of them can make the value of the future be ~0.0% the value of the best possible future.
I meant broad sense existential risk, not just extinction. The first graph is supposed to represent a specific worldview where the relevant form of existential risk is extinction, and extinction is reasonably likely. In particular I had Eliezer Yudkowsky’s views about AI in mind. (But I decided to draw a graph with the transition around 50% rather than his 99% or so, because I thought it would be clearer.) One could certainly draw many more graphs, or change the descriptions of the existing graphs, without representing everyone’s thoughts on the function mapping percentile performance to total realized value.
Thanks for explaining how you think about this issue, I will have to consider that more. My first thought is that I’m not utilitarian enough to say that a universe full of happy biological beings is ~0.0% as good as if they were digital, even conditioning on being biological being the wrong decision. But maybe I would agree on other possible disjunctive traps.