Agreed on all counts except that I like the concept of a singleton. I’d be interested to hear why you don’t, if you wish to discuss it.
I’m glad you agree! For the sake of controversy, I’ll add that I’m not entirely sure that scenario is out of consideration from an EV point of view, firstly because the exhaust will have a lot of energy and I’m not sure what will happen to it, and secondly because I’m open to a “diminishing returns” model of population ethics where the computational capacity furloughed does not have an overwhelmingly higher value.
On singletons, I think the distinction between “single agent” and “multiple agents” is more of a difference in how we imagine a system than an actual difference. Human civilization is divided into minds with a high level of communication and coordination within each mind and a significantly lower level between minds. This pattern is an accident of evolutionary history and if technological progress continues I doubt it will remain in the distant future, but I also don’t think there will be perfect communication and coordination between the parts of a future civilization either. Even within a single human mind the communication and coordination is imperfect.
Mmm, good point. Perhaps the way to salvage the concept of a singleton is to define it as the opposite of moloch, i.e. a future is ruled by a singleton to the extent that it doesn’t have moloch-like forces causing drift towards outcomes that nobody wants, money being left on the table, etc. Or maybe we could just say a singleton is where outcomes are on or close to the pareto frontier. Idk.