If humans totally solve alignment, we’d probably ask our AGI to take us to Eutopia slowly, allowing us to savor the improvement and adjust to the changes along the way, rather than leaping all the way to the destination in one terrifying lurch.
Directly conflicts with the geopolitical requirements. Also not compatible with the ‘sector by sector’ scope of economic impact—an AGI would be revolutionizing everything at once, and the only question would be whether it was merely flipping the figurative table or going directly to interpolating every figurative chemical bond in the table with figurative gas simultaneously and leaving it to crumble into figurative dust.
Otherwise you’d be left with three options that all seem immoral
The ‘Silent elitism’ view is approximately correct, except in its assumption that there is a current elite who endorse the eutopia, which there is not. Even the most forward-thinking people of today, the Ben Franklins of the 2020s, would balk. The only way humans know how to transition toward a eutopia is slowly over generations. Since this has a substantial cost, speedrunning that transition is desirable, but how exactly that speedrun can be accomplished without leaving a lot of wreckage in its wake is a topic best left for superintelligences, or at the very least intelligences augmented somewhat beyond the best capabilities we currently have available.
Pure propaganda—instead of trying to make a description that’s an honest attempt at translating a strange future into something that ordinary people can understand, we give up all attempts at honesty and just make up a nice-sounding future with no resemblance to the Eutopia which is secretly our true destination.
What a coincidence! You have precisely described this contest. This is, explicitly, a “make up a nice-sounding future with no resemblance to our true destination” contest. And yes, it’s at best completely immoral. At worst they get high on their own supply and use it to set priorities, in which case it’s dangerous and aims us toward UFAI and impossibilities.
At least it’s not the kind of believing absurdities which produces people willing to commit atrocities in service of those beliefs. Unfortunately, poor understanding of alignment creates a lot of atrocities from minimal provocation anyway.
the closest possible description of the indescribable Eutopia must be something that sounds basically good (even if it is clearly also a little unfamiliar), because the fundamental idea of Eutopia is that it’s desirable
This is not true. There is no law of the universe which states that there must be a way to translate the ways in which a state good for its inhabitants (who are transhuman or posthuman i.e possessed of humanity and other various important mental qualities), into words which can be conveyed in present human language, by text or speech, that sound appealing. That might be a nice property for a universe to have but ours doesn’t.
Some point along a continuum from here to there, a continuum we might slide up or down with effort, probably can be so described—a fixed-point theorem of some sort probably applies. However, that need not be an honest depiction of what life will be like if we slide in that direction, any more than showing a vision of the Paris Commune to a Parisien on the day Napoleon fell (stipulating that they approved of it) would be an honest view of Paris’s future.
See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest.
”Directly conflicts with the geopolitical requirements.”—How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, “Okay AGI, I don’t want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars.” That would be 5x the current rate of worldwide economic growth, which would probably feel like “transforming the economy sector by sector” to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
The key part of my story is not very realistic or detailed. (How do I end up with a world-dominating AGI perfectly under my control by tomorrow?)
I asked my AGI to do something that you would consider unambitious, and maybe immoral. You’d rather I command my genie to make changes somewhere on the spectrum from “merely flipping the figurative table” to dissolving the entire physical world and reconfiguring it into computronium. But that’s just a personal preference of yours—just because I’ve invented an extremely powerful AGI doesn’t mean I can’t ask it to do boring ordinary things like merely curing cancer instead of solving immortality.
I agree with you that there’s a spectrum of different things that can be meant by “honesty”, sliding from “technically accurate statements which fail to convey the general impression” to “correctly conveying the general impression but giving vague or misleading statements”, and that in some cases the thing we’re trying to describe is so strange that no matter where we go along that continuum it’ll feel like lying because the description will be misleading in one way or the other. That’s a problem with full Eutopia but I don’t think it’s a problem with the 2045 story, where we’re being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn’t seem very likely if AGI is involved.
I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won’t work. (See my “cabal of secret geniuses runs the show from behind the scenes” versus “USA/China/EU come together to govern AI for all mankind” example in my comment to kokotajlod—if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason—because the TRUE most-plausible path to a good future doesn’t/couldn’t go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses “have the AGI create a nanosystem to melt all the world’s GPUs” (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I’m not sure about this question and I’m open to being won over.
The ‘unambitious’ thing you ask the AI to do would create worldwide political change. It is absurd to think that it wouldn’t. Even ordinary technological change creates worldwide political change at that scale!
And an AGI having that little impact is also not plausible; if that’s all you do, the second mover—and possibly the third, fourth, fifth, if everyone moves slow—spits out an AGI and flips the table, because you can’t be that unambitious and still block other AGIs from performing pivotal acts, and even if you want to think small, the other actors won’t. Even if they are approximately as unambitious, they will have different goals, and the interaction will immediately amp up the chaos.
There is just no way for an actual AGI scenario to meet these guidelines. Any attempt to draw a world which meets them has written the bottom line first and is torturing its logic trying to construct a vaguely plausible story that might lead to it.
I believe that you are too quick to label this story as absurd. Ordinary technology does not have the capacity to correct towards explicitly smaller changes that still satisfy the objective. If the AGI wants to prevent wars while minimally disturbing the worldwide politics, I find it plausible that it would succeed.
Similarly, just because an AGI has very little visible impact, does not mean that it isn’t effectively in control. For a true AGI, it should be trivial to interrupt the second mover without any great upheaval. It should be able to surpress other AGIs from coming into existence without causing too much of a stir.
I do somewhat agree with your reservations, but I find that your way of adressing them seems uncharitable (i.e. “at best completely immoral”).
Directly conflicts with the geopolitical requirements. Also not compatible with the ‘sector by sector’ scope of economic impact—an AGI would be revolutionizing everything at once, and the only question would be whether it was merely flipping the figurative table or going directly to interpolating every figurative chemical bond in the table with figurative gas simultaneously and leaving it to crumble into figurative dust.
The ‘Silent elitism’ view is approximately correct, except in its assumption that there is a current elite who endorse the eutopia, which there is not. Even the most forward-thinking people of today, the Ben Franklins of the 2020s, would balk. The only way humans know how to transition toward a eutopia is slowly over generations. Since this has a substantial cost, speedrunning that transition is desirable, but how exactly that speedrun can be accomplished without leaving a lot of wreckage in its wake is a topic best left for superintelligences, or at the very least intelligences augmented somewhat beyond the best capabilities we currently have available.
What a coincidence! You have precisely described this contest. This is, explicitly, a “make up a nice-sounding future with no resemblance to our true destination” contest. And yes, it’s at best completely immoral. At worst they get high on their own supply and use it to set priorities, in which case it’s dangerous and aims us toward UFAI and impossibilities.
At least it’s not the kind of believing absurdities which produces people willing to commit atrocities in service of those beliefs. Unfortunately, poor understanding of alignment creates a lot of atrocities from minimal provocation anyway.
This is not true. There is no law of the universe which states that there must be a way to translate the ways in which a state good for its inhabitants (who are transhuman or posthuman i.e possessed of humanity and other various important mental qualities), into words which can be conveyed in present human language, by text or speech, that sound appealing. That might be a nice property for a universe to have but ours doesn’t.
Some point along a continuum from here to there, a continuum we might slide up or down with effort, probably can be so described—a fixed-point theorem of some sort probably applies. However, that need not be an honest depiction of what life will be like if we slide in that direction, any more than showing a vision of the Paris Commune to a Parisien on the day Napoleon fell (stipulating that they approved of it) would be an honest view of Paris’s future.
See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest.
”Directly conflicts with the geopolitical requirements.”—How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, “Okay AGI, I don’t want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars.” That would be 5x the current rate of worldwide economic growth, which would probably feel like “transforming the economy sector by sector” to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
The key part of my story is not very realistic or detailed. (How do I end up with a world-dominating AGI perfectly under my control by tomorrow?)
I asked my AGI to do something that you would consider unambitious, and maybe immoral. You’d rather I command my genie to make changes somewhere on the spectrum from “merely flipping the figurative table” to dissolving the entire physical world and reconfiguring it into computronium. But that’s just a personal preference of yours—just because I’ve invented an extremely powerful AGI doesn’t mean I can’t ask it to do boring ordinary things like merely curing cancer instead of solving immortality.
I agree with you that there’s a spectrum of different things that can be meant by “honesty”, sliding from “technically accurate statements which fail to convey the general impression” to “correctly conveying the general impression but giving vague or misleading statements”, and that in some cases the thing we’re trying to describe is so strange that no matter where we go along that continuum it’ll feel like lying because the description will be misleading in one way or the other. That’s a problem with full Eutopia but I don’t think it’s a problem with the 2045 story, where we’re being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn’t seem very likely if AGI is involved.
I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won’t work. (See my “cabal of secret geniuses runs the show from behind the scenes” versus “USA/China/EU come together to govern AI for all mankind” example in my comment to kokotajlod—if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason—because the TRUE most-plausible path to a good future doesn’t/couldn’t go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses “have the AGI create a nanosystem to melt all the world’s GPUs” (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I’m not sure about this question and I’m open to being won over.
The ‘unambitious’ thing you ask the AI to do would create worldwide political change. It is absurd to think that it wouldn’t. Even ordinary technological change creates worldwide political change at that scale!
And an AGI having that little impact is also not plausible; if that’s all you do, the second mover—and possibly the third, fourth, fifth, if everyone moves slow—spits out an AGI and flips the table, because you can’t be that unambitious and still block other AGIs from performing pivotal acts, and even if you want to think small, the other actors won’t. Even if they are approximately as unambitious, they will have different goals, and the interaction will immediately amp up the chaos.
There is just no way for an actual AGI scenario to meet these guidelines. Any attempt to draw a world which meets them has written the bottom line first and is torturing its logic trying to construct a vaguely plausible story that might lead to it.
I believe that you are too quick to label this story as absurd. Ordinary technology does not have the capacity to correct towards explicitly smaller changes that still satisfy the objective. If the AGI wants to prevent wars while minimally disturbing the worldwide politics, I find it plausible that it would succeed.
Similarly, just because an AGI has very little visible impact, does not mean that it isn’t effectively in control. For a true AGI, it should be trivial to interrupt the second mover without any great upheaval. It should be able to surpress other AGIs from coming into existence without causing too much of a stir.
I do somewhat agree with your reservations, but I find that your way of adressing them seems uncharitable (i.e. “at best completely immoral”).