(more minor points)
If only a small number of people have power, then it becomes less likely that the correct moral views are represented among that small group, and therefore less likely that we get to a mostly-great future via trade and compromise.
I believe this is correct, but possibly for the wrong reason. If you just have a smaller group of people and they are drawn randomly from the population, yes there is a higher probability that no-one will have the correct moral view. But there is also a higher probability an unusually high fraction of people will have such a view. So a smaller bottleneck just increases the variance. But this is bad in expectation if you think that the value of the future is a concave function of the fraction of world power wielded by people with the correct values, because of trade and compromise. ie if having 10% of power in good hands is less than 10 times as good as 1%, as I understand you believe, then increasing variance by concentrating power is bad. (And of course, there is the further effect of power-seekers on average having worse values.)
In particular, we could model the process of reflection as a series of independent Brownian motions in R2, all starting at the same point at the same time. Then the expected distance of a view from the starting point, and the expected distance between two given views, both increase with the square root of time. The latter expectation is larger by a factor of sqrt(2)â.
The choice of two dimensions is unmotivated, so I donât trust the numbers, but the general effect seems right and would hold directionally even if there are e.g. 10 dimensions that people are going on a random walk through.
The details and mechanisms make sense and are useful for making possibilities more vivid, but I think the very high-level argument seems strong to me and does much of the work:
Social, values, and institutional change is significantly caused by technological, economic, and demographic changes.
After a period of rapid transformation through the intelligence and industrial explosions, and space colonisation, the pace of technological, economic and demographic change will greatly slow.
C. The pace of social, values, and institutional change will greatly slow. Therefore, achieving a good state before the rate of change slows is very valuable.