I have substantial credence that the future is at least 99% as good as the optimal future.[3] I do not claim much certainty about what the optimal future looks like — my baseline assumption is that it involves increasing and improving consciousness in the universe, but I have little idea whether that would look like many very small minds or a few very big minds
Overall, I don’t really agree with the above quote, depending on what you mean by substantial (edit: it’s also possible I just don’t see the above statement as well defined as I don’t presupose there is a true “value”). The tldr of my disagreement is (1) that superintelligence doesn’t automatically mean there won’t be (internal and interstellar) political struggles and (2) I don’t think superintelligence will “solve ethics”, because I don’t believe there is a correct solution, and, (3) We would need to do everything very quickly to hit the 99% mark.
I’m gonna talk below under the assumption that value = total utility (with an existing weighting map or rule and a clear definition of what utility is) just for the purpose of clarity, but I don’t think it would matter if you substituted another definition for value.
In order to get 99% of the potential utility
One society probably needs to control its entire lightcone. They (who is they? next bullet point) also need to have total utilitarian values. If they don’t control the lightcone, then all other societies in the lightcone need to have similar total utiltarian values also to get to this 99%. For this to be the case there would need to be strong convergent CEVs, but I don’t see why this is at all obvious. If they don’t have utilitarian values then of course it seems extremely unlikely that they are going to create an optimal world.
This dominant society needs to have complete agreement on what value is. Let’s assume that a single society rules the entire light cone and is ASI boosted. We still need all the sub agents to agree on total utilitarianism (or the people who control the resources). Disagreement with each other could cause infighting or just a split value system. So unless you then accept the split value system is what value is, you have already lost way more than 1% of the value. If a single agent rules this entire society, how did they come to power and why do we think they would be especially caring about maxing value or have the correct value.
We would need to do all of this in less than ~1/100 of the time until the world ended. It’s a bit more complicated than this but the intuition should be clear.
Yea you can get around points 1 and 2 by just defining value as whatever the future people figure out with the help of AGI but that seems somewhat noninteresting to me. If you are a moral realist then this is more interesting because it feels less tautological. At the same time moral realism being true would not mean that any society + an agi would agree on it. In fact I still think most wouldn’t, although I have trouble conceiving of what it even means for a moral fact to be correct so my intuitions may be heavily off.
Almost all of the remaining probability to near-zero futures.
This claim is bolder, I think. Even if it seems reasonable to expect a substantial fraction of possible futures to converge to near-optimal, it may seem odd to expect almost all of the rest to be near-zero. But I find it difficult to imagine any other futures.
Overall, I don’t really agree with the above quote, depending on what you mean by substantial (edit: it’s also possible I just don’t see the above statement as well defined as I don’t presupose there is a true “value”). The tldr of my disagreement is (1) that superintelligence doesn’t automatically mean there won’t be (internal and interstellar) political struggles and (2) I don’t think superintelligence will “solve ethics”, because I don’t believe there is a correct solution, and, (3) We would need to do everything very quickly to hit the 99% mark.
I’m gonna talk below under the assumption that value = total utility (with an existing weighting map or rule and a clear definition of what utility is) just for the purpose of clarity, but I don’t think it would matter if you substituted another definition for value.
In order to get 99% of the potential utility
One society probably needs to control its entire lightcone. They (who is they? next bullet point) also need to have total utilitarian values. If they don’t control the lightcone, then all other societies in the lightcone need to have similar total utiltarian values also to get to this 99%. For this to be the case there would need to be strong convergent CEVs, but I don’t see why this is at all obvious. If they don’t have utilitarian values then of course it seems extremely unlikely that they are going to create an optimal world.
This dominant society needs to have complete agreement on what value is. Let’s assume that a single society rules the entire light cone and is ASI boosted. We still need all the sub agents to agree on total utilitarianism (or the people who control the resources). Disagreement with each other could cause infighting or just a split value system. So unless you then accept the split value system is what value is, you have already lost way more than 1% of the value. If a single agent rules this entire society, how did they come to power and why do we think they would be especially caring about maxing value or have the correct value.
We would need to do all of this in less than ~1/100 of the time until the world ended. It’s a bit more complicated than this but the intuition should be clear.
Yea you can get around points 1 and 2 by just defining value as whatever the future people figure out with the help of AGI but that seems somewhat noninteresting to me. If you are a moral realist then this is more interesting because it feels less tautological. At the same time moral realism being true would not mean that any society + an agi would agree on it. In fact I still think most wouldn’t, although I have trouble conceiving of what it even means for a moral fact to be correct so my intuitions may be heavily off.
This seems significantly less bold to me.
Briefly:
Yeah, “optimal future” should mean something like “future where humanity acts optimally.”
Yeah, I expect strong coordination in the future.
Yeah, I expect stuff to happen fast, on the astronomical scale.