I don’t actually think outcome 3 is achievable or particularly desirable. You’re basically asking for an AI that relentlessly cuts any non-optimal resource expenditure in favor of more and more strongly optimizing for the “good”. I think the default result of such a process is that it finds some configuration of matter which is more “happy” / “meaningful” / whatever it’s conception of “good” and sacrifices everything that’s not part of such a conception.
I also don’t think our values are shaped like that. I think a single human’s values derive from a multi agent negotiation among a continuous distribution over possible internal sub agents. This means they’re inherently dynamic, constantly changing in response to your changing cognitive environment. It also means that we essentially have a limitless variety of internal values, whose external expression is limited by our finite resources/ capabilities. Restricting the future’s values to a single, limited snapshot of that process just seems… not good.
I don’t actually think outcome 3 is achievable or particularly desirable. You’re basically asking for an AI that relentlessly cuts any non-optimal resource expenditure in favor of more and more strongly optimizing for the “good”. I think the default result of such a process is that it finds some configuration of matter which is more “happy” / “meaningful” / whatever it’s conception of “good” and sacrifices everything that’s not part of such a conception.
I also don’t think our values are shaped like that. I think a single human’s values derive from a multi agent negotiation among a continuous distribution over possible internal sub agents. This means they’re inherently dynamic, constantly changing in response to your changing cognitive environment. It also means that we essentially have a limitless variety of internal values, whose external expression is limited by our finite resources/ capabilities. Restricting the future’s values to a single, limited snapshot of that process just seems… not good.