Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
Ideally powerful AI will enable something like reflection rather than locking in prosaic human values or our ignorant conceptions of the good.
Cosmopolitan values don’t come free.
The field of alignment is really about alignability, not making sure “the right people control it.” That’s a different problem.
You don’t seem to apply your reasoning that our current values might be “extremely sub-optimal” to your values of hedonium/EA/utilitarianism. But I think there are good reasons to believe they might be very sub-optimal. Firstly, most people (right now and throughout history) would be terrified of everything they care about being destroyed and replaced with hedonium. Secondly, even you say that it “doesn’t make me feel good and it is in direct opposition to most of my values”, despite being one of the few proponents of a hedonium shockwave. I’m unsure why you are identifying with the utilitarian part of you so strongly and ignoring all the other parts of you.
Anyway, I won’t expand because this topic has been discussed a lot before and I’m unlikely to say anything new. The first place that comes to mind is Complexity of Value—LessWrong
Also, yes, I very much had the same dilemma years ago. Mine went something like this:
Heart: I figured it out! All I care about is reducing suffering and increasing happiness!
Brain: Great! I’ve just read a lot of blogs and it turns out that we can maximise that by turning everything into a homogenous substance of hedonium, including you, your mom, your girlfriend, the cast of Friends, all the great artworks and wonders of nature. When shall we start working on that?
Heart: Ummm, a small part of me think that’d be great but… I’m starting to think that maybe happiness and suffering is not ALL I care about, maybe it’s a bit more complex. Is it ok if we don’t turn my mom into hedonium?
My point is, in the end, you think that suffering is bad and happiness is good because your emotions say so (what other reason could there be?). Why not listen to other things your emotions tell you? Ugh, sorry if I’m repeating myself.
Executive summary: If doing the most good requires building a utilitarian AI that tiles the universe with utilitronium at the expense of human values and existence, this may be in conflict with the goals of AI alignment.
Key points:
The AI alignment community aims to ensure AI systems are controlled and aligned with the right human values.
However, current human values may be extremely sub-optimal compared to a utilitarian AI that maximizes goodness/happiness in the universe.
The very best outcome could be an AI converting all matter into “hedonium” or “utilitronium”—pure bliss experiences.
So the goals of AI alignment (preserving human values) and effective altruism (doing the most good possible) may be in direct conflict.
Building a utilitarian AI focused on maximizing universal happiness, even at the cost of human extinction, might be the “best” scenario from an impartial perspective.
The author finds this conclusion emotionally difficult but believes doing the most good should take precedence over personal desires and values.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.