These are excellent comments, and unfortunately they all have the virtue of being perspicuous and true so I don’t have that much to say about them.
I doubt how rare near-best futures are among desired futures is a strong guide to the expected value of the future. At least, you need to know more about e.g. the feasibility of near-best futures; whether deliberative processes and scientific progress converge on an understanding of which futures are near-best, etc.
Is the core idea here that human desires and the values people reach on deliberation come apart? That makes sense, though it also leaves open how much deliberation our descendants will actually do / how much their values will be based on a deliberating process. I guess I’ll just state my view without defending it that after a decade in philosophy I have become pretty pessimistic about convergence happening through deliberation rather than more divergence as more choice points are uncovered and reasoners either think they have a good loss function or just choose not to do backpropagation.
These are excellent comments, and unfortunately they all have the virtue of being perspicuous and true so I don’t have that much to say about them.
Is the core idea here that human desires and the values people reach on deliberation come apart? That makes sense, though it also leaves open how much deliberation our descendants will actually do / how much their values will be based on a deliberating process. I guess I’ll just state my view without defending it that after a decade in philosophy I have become pretty pessimistic about convergence happening through deliberation rather than more divergence as more choice points are uncovered and reasoners either think they have a good loss function or just choose not to do backpropagation.