I feel like this goes against the principle of not leaving your footprint on the future, no?
A major reason that I got into longtermism in the first place is that I’m quite interested in “leaving a footprint” on the future (albeit a good one). In other words, I’m not sure I understand the intuition for why we wouldn’t deliberately try to leave our footprints on the future, if we want to have an impact. But perhaps I’m misunderstanding the nature of this metaphor. Can you elaborate?
I also don’t believe the standard path for AGI is one of moral reflection.
I think it’s worth being more specific about why you think AGI will not do moral reflection? In the post, I carefully consider arguments about whether future AIs will be alien-like and have morally arbitrary goals, in a respect that you seem to be imagining. I think it’s possible that I addressed some of the intuitions behind your argument here.
I guess I felt that a lot of the post was arguing under a frame of utilitarianism which is generally fair I think. When it comes to “not leaving a footprint on the future” what I’m referring to is epistemic humility about the correct moral theories. I’m quite uncertain myself about what is correct when it comes to morality with extra weight on utilitarianism. From this, we should be worried about being wrong and therefore try our best to not lock in whatever we’re currently thinking. (The classic example being if we did this 200 years ago we might still have slaves in the future)
I’m a believer that virtue ethics and deontology are imperfect information approximations of utilitarianism. Like Kant’s categorical imperative is a way of looking at the long-term future and asking, how do we optimise society to be the best that it can be?
I guess a core crux here for me is that it seems like you’re arguing a bit for naive utilitarianism here. I actually don’t really believe the idea that we will have the AGI follow the VNM-axioms that is being fully rational. I think it will be an internal dynamic system that are weighing for different things that it wants and that it won’t fully maximise utility because it won’t be internally aligned. Therefore we need to get it right or we’re going to have weird and idiosyncratic values that are not optimal for the long-term future of the world.
I hope that makes sense, I liked your post in general.
I see. After briefly skimming that post, I think I pretty strongly disagree with just about every major point in it (along with many of its empirical background assumptions), although admittedly I did not spend much time reading through it. If someone thinks that post provides good reasons to doubt the arguments in my post, I’d likely be happy to discuss the specific ideas within it in more detail.
A major reason that I got into longtermism in the first place is that I’m quite interested in “leaving a footprint” on the future (albeit a good one). In other words, I’m not sure I understand the intuition for why we wouldn’t deliberately try to leave our footprints on the future, if we want to have an impact. But perhaps I’m misunderstanding the nature of this metaphor. Can you elaborate?
I think it’s worth being more specific about why you think AGI will not do moral reflection? In the post, I carefully consider arguments about whether future AIs will be alien-like and have morally arbitrary goals, in a respect that you seem to be imagining. I think it’s possible that I addressed some of the intuitions behind your argument here.
I guess I felt that a lot of the post was arguing under a frame of utilitarianism which is generally fair I think. When it comes to “not leaving a footprint on the future” what I’m referring to is epistemic humility about the correct moral theories. I’m quite uncertain myself about what is correct when it comes to morality with extra weight on utilitarianism. From this, we should be worried about being wrong and therefore try our best to not lock in whatever we’re currently thinking. (The classic example being if we did this 200 years ago we might still have slaves in the future)
I’m a believer that virtue ethics and deontology are imperfect information approximations of utilitarianism. Like Kant’s categorical imperative is a way of looking at the long-term future and asking, how do we optimise society to be the best that it can be?
I guess a core crux here for me is that it seems like you’re arguing a bit for naive utilitarianism here. I actually don’t really believe the idea that we will have the AGI follow the VNM-axioms that is being fully rational. I think it will be an internal dynamic system that are weighing for different things that it wants and that it won’t fully maximise utility because it won’t be internally aligned. Therefore we need to get it right or we’re going to have weird and idiosyncratic values that are not optimal for the long-term future of the world.
I hope that makes sense, I liked your post in general.
The “footprints on the future” thing could be referencing this post.
(Edit: to be clear, this link is not an endorsement.)
I see. After briefly skimming that post, I think I pretty strongly disagree with just about every major point in it (along with many of its empirical background assumptions), although admittedly I did not spend much time reading through it. If someone thinks that post provides good reasons to doubt the arguments in my post, I’d likely be happy to discuss the specific ideas within it in more detail.
Yes, I was on my phone, and you can’t link things there easily; that was what I was referring to.