See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest.
”Directly conflicts with the geopolitical requirements.”—How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, “Okay AGI, I don’t want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars.” That would be 5x the current rate of worldwide economic growth, which would probably feel like “transforming the economy sector by sector” to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
The key part of my story is not very realistic or detailed. (How do I end up with a world-dominating AGI perfectly under my control by tomorrow?)
I asked my AGI to do something that you would consider unambitious, and maybe immoral. You’d rather I command my genie to make changes somewhere on the spectrum from “merely flipping the figurative table” to dissolving the entire physical world and reconfiguring it into computronium. But that’s just a personal preference of yours—just because I’ve invented an extremely powerful AGI doesn’t mean I can’t ask it to do boring ordinary things like merely curing cancer instead of solving immortality.
I agree with you that there’s a spectrum of different things that can be meant by “honesty”, sliding from “technically accurate statements which fail to convey the general impression” to “correctly conveying the general impression but giving vague or misleading statements”, and that in some cases the thing we’re trying to describe is so strange that no matter where we go along that continuum it’ll feel like lying because the description will be misleading in one way or the other. That’s a problem with full Eutopia but I don’t think it’s a problem with the 2045 story, where we’re being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn’t seem very likely if AGI is involved.
I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won’t work. (See my “cabal of secret geniuses runs the show from behind the scenes” versus “USA/China/EU come together to govern AI for all mankind” example in my comment to kokotajlod—if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason—because the TRUE most-plausible path to a good future doesn’t/couldn’t go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses “have the AGI create a nanosystem to melt all the world’s GPUs” (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I’m not sure about this question and I’m open to being won over.
The ‘unambitious’ thing you ask the AI to do would create worldwide political change. It is absurd to think that it wouldn’t. Even ordinary technological change creates worldwide political change at that scale!
And an AGI having that little impact is also not plausible; if that’s all you do, the second mover—and possibly the third, fourth, fifth, if everyone moves slow—spits out an AGI and flips the table, because you can’t be that unambitious and still block other AGIs from performing pivotal acts, and even if you want to think small, the other actors won’t. Even if they are approximately as unambitious, they will have different goals, and the interaction will immediately amp up the chaos.
There is just no way for an actual AGI scenario to meet these guidelines. Any attempt to draw a world which meets them has written the bottom line first and is torturing its logic trying to construct a vaguely plausible story that might lead to it.
I believe that you are too quick to label this story as absurd. Ordinary technology does not have the capacity to correct towards explicitly smaller changes that still satisfy the objective. If the AGI wants to prevent wars while minimally disturbing the worldwide politics, I find it plausible that it would succeed.
Similarly, just because an AGI has very little visible impact, does not mean that it isn’t effectively in control. For a true AGI, it should be trivial to interrupt the second mover without any great upheaval. It should be able to surpress other AGIs from coming into existence without causing too much of a stir.
I do somewhat agree with your reservations, but I find that your way of adressing them seems uncharitable (i.e. “at best completely immoral”).
See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest.
”Directly conflicts with the geopolitical requirements.”—How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, “Okay AGI, I don’t want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars.” That would be 5x the current rate of worldwide economic growth, which would probably feel like “transforming the economy sector by sector” to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
The key part of my story is not very realistic or detailed. (How do I end up with a world-dominating AGI perfectly under my control by tomorrow?)
I asked my AGI to do something that you would consider unambitious, and maybe immoral. You’d rather I command my genie to make changes somewhere on the spectrum from “merely flipping the figurative table” to dissolving the entire physical world and reconfiguring it into computronium. But that’s just a personal preference of yours—just because I’ve invented an extremely powerful AGI doesn’t mean I can’t ask it to do boring ordinary things like merely curing cancer instead of solving immortality.
I agree with you that there’s a spectrum of different things that can be meant by “honesty”, sliding from “technically accurate statements which fail to convey the general impression” to “correctly conveying the general impression but giving vague or misleading statements”, and that in some cases the thing we’re trying to describe is so strange that no matter where we go along that continuum it’ll feel like lying because the description will be misleading in one way or the other. That’s a problem with full Eutopia but I don’t think it’s a problem with the 2045 story, where we’re being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn’t seem very likely if AGI is involved.
I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won’t work. (See my “cabal of secret geniuses runs the show from behind the scenes” versus “USA/China/EU come together to govern AI for all mankind” example in my comment to kokotajlod—if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason—because the TRUE most-plausible path to a good future doesn’t/couldn’t go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses “have the AGI create a nanosystem to melt all the world’s GPUs” (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I’m not sure about this question and I’m open to being won over.
The ‘unambitious’ thing you ask the AI to do would create worldwide political change. It is absurd to think that it wouldn’t. Even ordinary technological change creates worldwide political change at that scale!
And an AGI having that little impact is also not plausible; if that’s all you do, the second mover—and possibly the third, fourth, fifth, if everyone moves slow—spits out an AGI and flips the table, because you can’t be that unambitious and still block other AGIs from performing pivotal acts, and even if you want to think small, the other actors won’t. Even if they are approximately as unambitious, they will have different goals, and the interaction will immediately amp up the chaos.
There is just no way for an actual AGI scenario to meet these guidelines. Any attempt to draw a world which meets them has written the bottom line first and is torturing its logic trying to construct a vaguely plausible story that might lead to it.
I believe that you are too quick to label this story as absurd. Ordinary technology does not have the capacity to correct towards explicitly smaller changes that still satisfy the objective. If the AGI wants to prevent wars while minimally disturbing the worldwide politics, I find it plausible that it would succeed.
Similarly, just because an AGI has very little visible impact, does not mean that it isn’t effectively in control. For a true AGI, it should be trivial to interrupt the second mover without any great upheaval. It should be able to surpress other AGIs from coming into existence without causing too much of a stir.
I do somewhat agree with your reservations, but I find that your way of adressing them seems uncharitable (i.e. “at best completely immoral”).