Encourage people to start thinking about the future in more positive terms.
It is actively harmful for people to start thinking about the future in more positive terms, if those terms are misleading and unrealistic. The contest ground rules frame “positive terms” as being familiar, not just good in the abstract—they cannot be good but scary, as any true good outcome must be. See Eutopia is Scary:
We, in our time, think our life has improved in the last two or three hundred years. Ben Franklin is probably smart and forward-looking enough to agree that life has improved. But if you don’t think Ben Franklin would be amazed, disgusted, and frightened, then I think you far overestimate the “normality” of your own time.
Receive inspiration for our real-world policy efforts and future projects to run / fund.
It is actively harmful to take fictional evidence as inspiration for what projects are worth pursuing. This would be true even if the fiction was not constrained to be unrealistic and unattainable, but this contest is constrained in that way, which makes it much worse.
Identify potential collaborators from outside of our existing network.
Again, a search which is specifically biased to have bad input data is going to be harmful, not helpful.
Update our messaging strategy.
Your explicit goal here is to look for ‘positive’, meaning ‘non-scary’, futures to try to communicate. This is lying—no such future is plausible, and it’s unclear any is even possiblein theory. You say
not enough effort goes into thinking about what a good future with (e.g.) artificial general intelligence could look like
but this is not true. Lots of effort goes into thinking about it. You just don’t like the results, because they’re either low-quality (failing in all the old ways utopias fail) or they are high-quality and therefore appropriately terrifying.
The best result I can picture emerging from this contest is for the people running the contest to realize the utter futility of the approach they were targeting and change tacks entirely. I’m unsure whether I hope that comes with some resignations, because this was a really, spectacularly terribleidea, and that would tend to imply some drastic action in response, but on the other hand I’d hope FLI’s team is capable from learning from its mistakes better than most.
The contest is only about describing 2045, not necessarily a radically alien far-future “Eutopia” end state of human civilization. If humans totally solve alignment, we’d probably ask our AGI to take us to Eutopia slowly, allowing us to savor the improvement and adjust to the changes along the way, rather than leaping all the way to the destination in one terrifying lurch. So I’m thinking there are probably some good ways to answer this prompt.
But let’s engage with the harder question of describing a full Eutopia. If Eutopia is truly good, then surely there must be honest ways of describing it that express why it is good and desirable, even if Eutopia is also scary. Otherwise you’d be left with three options that all seem immoral:
Silent elitism—the rabble will never understand Eutopia, so we simply won’t tell them where we’re taking humanity. They’ll thank us later, when we get there and they realize it’s good.
Pure propaganda—instead of trying to make a description that’s an honest attempt at translating a strange future into something that ordinary people can understand, we give up all attempts at honesty and just make up a nice-sounding future with no resemblance to the Eutopia which is secretly our true destination.
Doomed self-defeating attempts at honesty—if you tell such a scary story about “Eutopia” that nobody would want to live there, then people will react badly to it and they’ll demand to be steered somewhere else. Because of your dedication to always emphasizing the full horror and incomprehensibility, your attempts to persuade people of Eutopia will only serve to move us farther away from it.
It’s impossible to imagine infinity, but if you’re trying to explain how big infinity is, surely it’s better to say “it’s like the number of stars in the night sky”, or “it’s like the number of drops of water in the ocean”, than to say “it’s like the number of apples you can fit in a bucket”. Similarly, the closest possible description of the indescribable Eutopia must be something that sounds basically good (even if it is clearly also a little unfamiliar), because the fundamental idea of Eutopia is that it’s desirable. I don’t think that’s lying, anymore than trying to describe other indescribable things as well as you can is lying.
Yudkowsky’s own essay “Eutopia is Scary” was part of a larger “Fun Theory” sequence about attempting to describe utopias. He mostly described them in a positive light, with the “Eutopia is Scary” article serving as an important, but secondary, honesty-enhancing caveat: “these worlds will be a lot of fun, but keep in mind they’ll also be a little strange”.
I want to second what Czynski said about pure propaganda. Insofar as we believe that the constraints you are imposing are artificial and unrealistic, doesn’t this contest fall into the “pure propaganda” category? I would be enthusiastically in favor of this contest if there weren’t such unrealistic constraints. Or do you think the constraints are actually realistic after all?
I think it’s fine if we have broad leeway to interpret the constraints as we see fit. E.g. “Technology is improving rapidly because, while the AGI already has mature technology, humans have requested that advanced technology be slowly doled out so as not to give us too much shock. So technology actually used by humans is improving rapidly, even though the cutting-edge stuff used by AGI has stagnated. Meanwhile, while the US, EU, and China have no real power (real power lies with the AGI) the AGI follow the wishes of humans and humans still want the US, EU, and China to be important somehow so lots of decisions are delegated to those entities. Also, humans are gradually starting to realize that if you are delegating decisions to old institutions you might as well do so to more institutions than the US, EU, and China, so increasingly decisions are being delegated to African and South American etc. governments rather than US, EU, and China. So in that sense a ‘balance of power between US, EU, and China has been maintained’ and ‘Africa et al are on the rise.’” Would you accept interpretations such as this?
To clarify, I’m not affiliated with FLI, so I’m not the one imposing the constraints, they are. I’m just defending them, because the contest rules seem reasonable enough to me. Here are a couple of thoughts:
Remember that my comment was drawing a distinction between “describing total Eutopia, a full and final state of human existence that might be strange beyond imagining” versus “describing a 2045 AGI scenario where things are looking positive and under-control and not too crazy”. I certainly agree with you that describing a totally transformed Eutopia where the USA and China still exist in exactly their current form is bizarre and contradictory. My point about Eutopia was just that an honest description of something indescribably strange should err towards trying to get across the general feeling (ie, it will be nice) rather than trying to scare people with the weirdness. (Imagine going back in time and horrifying the Founding Fathers by describing how in the present day “everyone sits in front of machines all day!! people eat packaged food from factories!!!”. Shocking the Founders like this seems misleading if the overall progress of science and technology is something they would ultimately be happy about.) Do you agree with that, or do you at least see what I’m saying?
Anyways, on to the more important issue of this actual contest, the 2045 AGI story, and its oddly-specific political requirements:
I agree with you that a positive AGI outcome that fits all these specific details is unlikely.
But I also think that the idea of AGI having a positive outcome at all seems unlikely—right now, if AGI happens, I’m mostly expecting paperclips!
Suppose I think AGI has a 70% chance of going paperclips, and a 30% chance of giving us any kind of positive outcome. Would it be unrealistic for me to write a story about the underdog 30% scenario in which we don’t all die horribly? No, I think that would be a perfectly fine thing to write about.
What if I was writing about a crazy-unlikely, 0.001% scenario? Then I’d be worried that my story might mislead people, by making it seem more likely than it really is. That’s definitely a fair criticism—for example, I might think it was immoral for someone to write a story about “The USA has a communist revolution, but against all odds and despite the many examples of history, few people are hurt and the new government works perfectly and never gets taken over by a bloodthirsty dictator and central planning finally works better than capitalism and it ushers in a new age of peace and prosperity for mankind!”.
But on the other hand, writing a very specific story is a good way to describe a goal that we are trying to hit, even if it’s unlikely. The business plan of every moonshot tech startup was once an unlikely and overly-specific sci-fi story. (“First we’re going to build the world’s first privately-made orbital rocket. Then we’re going to scale it up by 9x, and we’re going to fund it with NASA contracts for ISS cargo delivery. Once we’ve figured out reusability and dominated the world launch market, we’ll make an even bigger rocket, launch a money-printing satellite internet constellation, and use the profits to colonize Mars!”). Similarly, I would look much more kindly on a communist-revolution story if, instead of just fantasizing, it tried to plot out the most peaceful possible path to a new type of government that would really work—trying to tell the most realistic possible story under a set of unrealistic constraints that define our goal. (”..After the constitution had been fully reinterpreted by our revisionist supreme court justices—yes, I know that’ll be tough, but it seems to be the only way, please bear with me—we’ll use a Georgist land tax to fund public services, and citizens will contribute directly to decisionmaking via a cryptographically secured system of liquid democracy...”)
FLI is doing precisely this: choosing a set of unrealistic constraints that define a positive near-term path for civilization that most normal people (not just wild transhumanist LessWrongers) would be happy about. Chinese people wouldn’t be happy about a sci-fi future that incidentally involved a nuclear war in which their entire country was wiped off the map. Most people wouldn’t be happy if they heard that the world was going to be transformed beyond all recognition, with the economy doubling every two months as the world’s mountains and valleys are ripped up and converted to nanomachine supercomputers. Et cetera. FLI isn’t trying to choose something plausible—they’re just trying to choose a goal that everybody can agree on (peace, very fast but not bewilderingly fast economic growth, exciting new technologies to extend lifespan and make life better). Figuring out if there’s any plausible way to get there is our job. That’s the whole point of the contest.
You say: “[This scenario seems so unrealistic that I can only imagine it happening if we first align AGI and then request that it give us a slow ride even though it’s capable of going faster.] …Would you accept interpretations such as this?”
I’m not FLI so it’s not my job to say which interpretations are acceptable, but I’d say you’re already doing exactly the work FLI was looking for! I agree that this scenario is one of the most plausible ways that civilization might end up fulfilling the contest conditions. Here are some other possibilities:
Our AGI paradigm turns out to be really limited for some reason and it doesn’t scale well, so we get near-human AGI that does a lot to boost growth, but nothing really transformative. (It seems very unlikely to me that AGI capabilities would top out at such a convenient level, but who knows.)
Civilization is totally out of control and the alignment problem isn’t solved at all; we’re in the middle of “slow takeoff” towards paperclips by 2050, but the contest timeline ends in 2045 so all we see is things getting nicer and nicer as cool new technologies are invented, and not the horrifying treacherous turn where it all goes wrong. (This seems quite likely to me, but also seems to go strongly against the spirit of the question and would probably be judged harshly for lacking in the “aspirational” department.)
Who is in control of the AGI? Maybe it’s USA/China/EU all cooperating in a spirit of brotherhood to limit the pace of progress to something palatable and non-disorienting (the scenario you described). Or maybe it’s some kind of cabal of secret geniuses controlling things behind the scenes from their headquarters at Deepmind. If you’re more thinking that AGI will be developed all at once and via fast-takeoff (thus giving huge disproportionate power to the first inventors of aligned AGI), you might see the “cabal of secret geniuses” story as more plausible than the version where governments all come together to competently manage AI for the sake of humanity.
See my response to Czynski for more assorted thoughts, although I’ve written so much now that perhaps I could have entered the contest myself by now if I had been writing stories instead of comments! :P
Edited to add that alas, I only just now saw your other comment about “So in order to describe a good future, people will fiddle with the knobs of those important variables so that they are on their conducive-to-good settings rather than their most probable settings. ”. This strikes me as a fair criticism of the contest. (For one, it will bias people towards handwaving over the alignment problem by saying “it turned out to be surprisingly easy”.) I don’t think that’s devastating for the contest, since I think there’s a lot of value in just trying to envision what an agreeable good outcome for humanity looks like. But definitely a fair critique that lines up with the stuff I was saying above—basically, there are both pros and cons to putting $100K of optimization pressure behind getting people to figure out the most plausible optimistic outcome under a set of constraints. (Maybe FLI should run another contest encouraging people to do more Yudkowsky-style brainstorming of how everything could go horribly wrong before we even realize what we were dealing with, just to even things out!)
If humans totally solve alignment, we’d probably ask our AGI to take us to Eutopia slowly, allowing us to savor the improvement and adjust to the changes along the way, rather than leaping all the way to the destination in one terrifying lurch.
Directly conflicts with the geopolitical requirements. Also not compatible with the ‘sector by sector’ scope of economic impact—an AGI would be revolutionizing everything at once, and the only question would be whether it was merely flipping the figurative table or going directly to interpolating every figurative chemical bond in the table with figurative gas simultaneously and leaving it to crumble into figurative dust.
Otherwise you’d be left with three options that all seem immoral
The ‘Silent elitism’ view is approximately correct, except in its assumption that there is a current elite who endorse the eutopia, which there is not. Even the most forward-thinking people of today, the Ben Franklins of the 2020s, would balk. The only way humans know how to transition toward a eutopia is slowly over generations. Since this has a substantial cost, speedrunning that transition is desirable, but how exactly that speedrun can be accomplished without leaving a lot of wreckage in its wake is a topic best left for superintelligences, or at the very least intelligences augmented somewhat beyond the best capabilities we currently have available.
Pure propaganda—instead of trying to make a description that’s an honest attempt at translating a strange future into something that ordinary people can understand, we give up all attempts at honesty and just make up a nice-sounding future with no resemblance to the Eutopia which is secretly our true destination.
What a coincidence! You have precisely described this contest. This is, explicitly, a “make up a nice-sounding future with no resemblance to our true destination” contest. And yes, it’s at best completely immoral. At worst they get high on their own supply and use it to set priorities, in which case it’s dangerous and aims us toward UFAI and impossibilities.
At least it’s not the kind of believing absurdities which produces people willing to commit atrocities in service of those beliefs. Unfortunately, poor understanding of alignment creates a lot of atrocities from minimal provocation anyway.
the closest possible description of the indescribable Eutopia must be something that sounds basically good (even if it is clearly also a little unfamiliar), because the fundamental idea of Eutopia is that it’s desirable
This is not true. There is no law of the universe which states that there must be a way to translate the ways in which a state good for its inhabitants (who are transhuman or posthuman i.e possessed of humanity and other various important mental qualities), into words which can be conveyed in present human language, by text or speech, that sound appealing. That might be a nice property for a universe to have but ours doesn’t.
Some point along a continuum from here to there, a continuum we might slide up or down with effort, probably can be so described—a fixed-point theorem of some sort probably applies. However, that need not be an honest depiction of what life will be like if we slide in that direction, any more than showing a vision of the Paris Commune to a Parisien on the day Napoleon fell (stipulating that they approved of it) would be an honest view of Paris’s future.
See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest.
”Directly conflicts with the geopolitical requirements.”—How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, “Okay AGI, I don’t want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars.” That would be 5x the current rate of worldwide economic growth, which would probably feel like “transforming the economy sector by sector” to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
The key part of my story is not very realistic or detailed. (How do I end up with a world-dominating AGI perfectly under my control by tomorrow?)
I asked my AGI to do something that you would consider unambitious, and maybe immoral. You’d rather I command my genie to make changes somewhere on the spectrum from “merely flipping the figurative table” to dissolving the entire physical world and reconfiguring it into computronium. But that’s just a personal preference of yours—just because I’ve invented an extremely powerful AGI doesn’t mean I can’t ask it to do boring ordinary things like merely curing cancer instead of solving immortality.
I agree with you that there’s a spectrum of different things that can be meant by “honesty”, sliding from “technically accurate statements which fail to convey the general impression” to “correctly conveying the general impression but giving vague or misleading statements”, and that in some cases the thing we’re trying to describe is so strange that no matter where we go along that continuum it’ll feel like lying because the description will be misleading in one way or the other. That’s a problem with full Eutopia but I don’t think it’s a problem with the 2045 story, where we’re being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn’t seem very likely if AGI is involved.
I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won’t work. (See my “cabal of secret geniuses runs the show from behind the scenes” versus “USA/China/EU come together to govern AI for all mankind” example in my comment to kokotajlod—if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason—because the TRUE most-plausible path to a good future doesn’t/couldn’t go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses “have the AGI create a nanosystem to melt all the world’s GPUs” (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I’m not sure about this question and I’m open to being won over.
The ‘unambitious’ thing you ask the AI to do would create worldwide political change. It is absurd to think that it wouldn’t. Even ordinary technological change creates worldwide political change at that scale!
And an AGI having that little impact is also not plausible; if that’s all you do, the second mover—and possibly the third, fourth, fifth, if everyone moves slow—spits out an AGI and flips the table, because you can’t be that unambitious and still block other AGIs from performing pivotal acts, and even if you want to think small, the other actors won’t. Even if they are approximately as unambitious, they will have different goals, and the interaction will immediately amp up the chaos.
There is just no way for an actual AGI scenario to meet these guidelines. Any attempt to draw a world which meets them has written the bottom line first and is torturing its logic trying to construct a vaguely plausible story that might lead to it.
I believe that you are too quick to label this story as absurd. Ordinary technology does not have the capacity to correct towards explicitly smaller changes that still satisfy the objective. If the AGI wants to prevent wars while minimally disturbing the worldwide politics, I find it plausible that it would succeed.
Similarly, just because an AGI has very little visible impact, does not mean that it isn’t effectively in control. For a true AGI, it should be trivial to interrupt the second mover without any great upheaval. It should be able to surpress other AGIs from coming into existence without causing too much of a stir.
I do somewhat agree with your reservations, but I find that your way of adressing them seems uncharitable (i.e. “at best completely immoral”).
These goals are not good goals.
Encourage people to start thinking about the future in more positive terms.
It is actively harmful for people to start thinking about the future in more positive terms, if those terms are misleading and unrealistic. The contest ground rules frame “positive terms” as being familiar, not just good in the abstract—they cannot be good but scary, as any true good outcome must be. See Eutopia is Scary:
Receive inspiration for our real-world policy efforts and future projects to run / fund.
It is actively harmful to take fictional evidence as inspiration for what projects are worth pursuing. This would be true even if the fiction was not constrained to be unrealistic and unattainable, but this contest is constrained in that way, which makes it much worse.
Identify potential collaborators from outside of our existing network.
Again, a search which is specifically biased to have bad input data is going to be harmful, not helpful.
Update our messaging strategy.
Your explicit goal here is to look for ‘positive’, meaning ‘non-scary’, futures to try to communicate. This is lying—no such future is plausible, and it’s unclear any is even possible in theory. You say
but this is not true. Lots of effort goes into thinking about it. You just don’t like the results, because they’re either low-quality (failing in all the old ways utopias fail) or they are high-quality and therefore appropriately terrifying.
The best result I can picture emerging from this contest is for the people running the contest to realize the utter futility of the approach they were targeting and change tacks entirely. I’m unsure whether I hope that comes with some resignations, because this was a really, spectacularly terrible idea, and that would tend to imply some drastic action in response, but on the other hand I’d hope FLI’s team is capable from learning from its mistakes better than most.
The contest is only about describing 2045, not necessarily a radically alien far-future “Eutopia” end state of human civilization. If humans totally solve alignment, we’d probably ask our AGI to take us to Eutopia slowly, allowing us to savor the improvement and adjust to the changes along the way, rather than leaping all the way to the destination in one terrifying lurch. So I’m thinking there are probably some good ways to answer this prompt.
But let’s engage with the harder question of describing a full Eutopia. If Eutopia is truly good, then surely there must be honest ways of describing it that express why it is good and desirable, even if Eutopia is also scary. Otherwise you’d be left with three options that all seem immoral:
Silent elitism—the rabble will never understand Eutopia, so we simply won’t tell them where we’re taking humanity. They’ll thank us later, when we get there and they realize it’s good.
Pure propaganda—instead of trying to make a description that’s an honest attempt at translating a strange future into something that ordinary people can understand, we give up all attempts at honesty and just make up a nice-sounding future with no resemblance to the Eutopia which is secretly our true destination.
Doomed self-defeating attempts at honesty—if you tell such a scary story about “Eutopia” that nobody would want to live there, then people will react badly to it and they’ll demand to be steered somewhere else. Because of your dedication to always emphasizing the full horror and incomprehensibility, your attempts to persuade people of Eutopia will only serve to move us farther away from it.
It’s impossible to imagine infinity, but if you’re trying to explain how big infinity is, surely it’s better to say “it’s like the number of stars in the night sky”, or “it’s like the number of drops of water in the ocean”, than to say “it’s like the number of apples you can fit in a bucket”. Similarly, the closest possible description of the indescribable Eutopia must be something that sounds basically good (even if it is clearly also a little unfamiliar), because the fundamental idea of Eutopia is that it’s desirable. I don’t think that’s lying, anymore than trying to describe other indescribable things as well as you can is lying.
Yudkowsky’s own essay “Eutopia is Scary” was part of a larger “Fun Theory” sequence about attempting to describe utopias. He mostly described them in a positive light, with the “Eutopia is Scary” article serving as an important, but secondary, honesty-enhancing caveat: “these worlds will be a lot of fun, but keep in mind they’ll also be a little strange”.
I want to second what Czynski said about pure propaganda. Insofar as we believe that the constraints you are imposing are artificial and unrealistic, doesn’t this contest fall into the “pure propaganda” category? I would be enthusiastically in favor of this contest if there weren’t such unrealistic constraints. Or do you think the constraints are actually realistic after all?
I think it’s fine if we have broad leeway to interpret the constraints as we see fit. E.g. “Technology is improving rapidly because, while the AGI already has mature technology, humans have requested that advanced technology be slowly doled out so as not to give us too much shock. So technology actually used by humans is improving rapidly, even though the cutting-edge stuff used by AGI has stagnated. Meanwhile, while the US, EU, and China have no real power (real power lies with the AGI) the AGI follow the wishes of humans and humans still want the US, EU, and China to be important somehow so lots of decisions are delegated to those entities. Also, humans are gradually starting to realize that if you are delegating decisions to old institutions you might as well do so to more institutions than the US, EU, and China, so increasingly decisions are being delegated to African and South American etc. governments rather than US, EU, and China. So in that sense a ‘balance of power between US, EU, and China has been maintained’ and ‘Africa et al are on the rise.’” Would you accept interpretations such as this?
To clarify, I’m not affiliated with FLI, so I’m not the one imposing the constraints, they are. I’m just defending them, because the contest rules seem reasonable enough to me. Here are a couple of thoughts:
Remember that my comment was drawing a distinction between “describing total Eutopia, a full and final state of human existence that might be strange beyond imagining” versus “describing a 2045 AGI scenario where things are looking positive and under-control and not too crazy”. I certainly agree with you that describing a totally transformed Eutopia where the USA and China still exist in exactly their current form is bizarre and contradictory. My point about Eutopia was just that an honest description of something indescribably strange should err towards trying to get across the general feeling (ie, it will be nice) rather than trying to scare people with the weirdness. (Imagine going back in time and horrifying the Founding Fathers by describing how in the present day “everyone sits in front of machines all day!! people eat packaged food from factories!!!”. Shocking the Founders like this seems misleading if the overall progress of science and technology is something they would ultimately be happy about.) Do you agree with that, or do you at least see what I’m saying?
Anyways, on to the more important issue of this actual contest, the 2045 AGI story, and its oddly-specific political requirements:
I agree with you that a positive AGI outcome that fits all these specific details is unlikely.
But I also think that the idea of AGI having a positive outcome at all seems unlikely—right now, if AGI happens, I’m mostly expecting paperclips!
Suppose I think AGI has a 70% chance of going paperclips, and a 30% chance of giving us any kind of positive outcome. Would it be unrealistic for me to write a story about the underdog 30% scenario in which we don’t all die horribly? No, I think that would be a perfectly fine thing to write about.
What if I was writing about a crazy-unlikely, 0.001% scenario? Then I’d be worried that my story might mislead people, by making it seem more likely than it really is. That’s definitely a fair criticism—for example, I might think it was immoral for someone to write a story about “The USA has a communist revolution, but against all odds and despite the many examples of history, few people are hurt and the new government works perfectly and never gets taken over by a bloodthirsty dictator and central planning finally works better than capitalism and it ushers in a new age of peace and prosperity for mankind!”.
But on the other hand, writing a very specific story is a good way to describe a goal that we are trying to hit, even if it’s unlikely. The business plan of every moonshot tech startup was once an unlikely and overly-specific sci-fi story. (“First we’re going to build the world’s first privately-made orbital rocket. Then we’re going to scale it up by 9x, and we’re going to fund it with NASA contracts for ISS cargo delivery. Once we’ve figured out reusability and dominated the world launch market, we’ll make an even bigger rocket, launch a money-printing satellite internet constellation, and use the profits to colonize Mars!”). Similarly, I would look much more kindly on a communist-revolution story if, instead of just fantasizing, it tried to plot out the most peaceful possible path to a new type of government that would really work—trying to tell the most realistic possible story under a set of unrealistic constraints that define our goal. (”..After the constitution had been fully reinterpreted by our revisionist supreme court justices—yes, I know that’ll be tough, but it seems to be the only way, please bear with me—we’ll use a Georgist land tax to fund public services, and citizens will contribute directly to decisionmaking via a cryptographically secured system of liquid democracy...”)
FLI is doing precisely this: choosing a set of unrealistic constraints that define a positive near-term path for civilization that most normal people (not just wild transhumanist LessWrongers) would be happy about. Chinese people wouldn’t be happy about a sci-fi future that incidentally involved a nuclear war in which their entire country was wiped off the map. Most people wouldn’t be happy if they heard that the world was going to be transformed beyond all recognition, with the economy doubling every two months as the world’s mountains and valleys are ripped up and converted to nanomachine supercomputers. Et cetera. FLI isn’t trying to choose something plausible—they’re just trying to choose a goal that everybody can agree on (peace, very fast but not bewilderingly fast economic growth, exciting new technologies to extend lifespan and make life better). Figuring out if there’s any plausible way to get there is our job. That’s the whole point of the contest.
You say: “[This scenario seems so unrealistic that I can only imagine it happening if we first align AGI and then request that it give us a slow ride even though it’s capable of going faster.] …Would you accept interpretations such as this?”
I’m not FLI so it’s not my job to say which interpretations are acceptable, but I’d say you’re already doing exactly the work FLI was looking for! I agree that this scenario is one of the most plausible ways that civilization might end up fulfilling the contest conditions. Here are some other possibilities:
Our AGI paradigm turns out to be really limited for some reason and it doesn’t scale well, so we get near-human AGI that does a lot to boost growth, but nothing really transformative. (It seems very unlikely to me that AGI capabilities would top out at such a convenient level, but who knows.)
Civilization is totally out of control and the alignment problem isn’t solved at all; we’re in the middle of “slow takeoff” towards paperclips by 2050, but the contest timeline ends in 2045 so all we see is things getting nicer and nicer as cool new technologies are invented, and not the horrifying treacherous turn where it all goes wrong. (This seems quite likely to me, but also seems to go strongly against the spirit of the question and would probably be judged harshly for lacking in the “aspirational” department.)
Who is in control of the AGI? Maybe it’s USA/China/EU all cooperating in a spirit of brotherhood to limit the pace of progress to something palatable and non-disorienting (the scenario you described). Or maybe it’s some kind of cabal of secret geniuses controlling things behind the scenes from their headquarters at Deepmind. If you’re more thinking that AGI will be developed all at once and via fast-takeoff (thus giving huge disproportionate power to the first inventors of aligned AGI), you might see the “cabal of secret geniuses” story as more plausible than the version where governments all come together to competently manage AI for the sake of humanity.
See my response to Czynski for more assorted thoughts, although I’ve written so much now that perhaps I could have entered the contest myself by now if I had been writing stories instead of comments! :P
Edited to add that alas, I only just now saw your other comment about “So in order to describe a good future, people will fiddle with the knobs of those important variables so that they are on their conducive-to-good settings rather than their most probable settings. ”. This strikes me as a fair criticism of the contest. (For one, it will bias people towards handwaving over the alignment problem by saying “it turned out to be surprisingly easy”.) I don’t think that’s devastating for the contest, since I think there’s a lot of value in just trying to envision what an agreeable good outcome for humanity looks like. But definitely a fair critique that lines up with the stuff I was saying above—basically, there are both pros and cons to putting $100K of optimization pressure behind getting people to figure out the most plausible optimistic outcome under a set of constraints. (Maybe FLI should run another contest encouraging people to do more Yudkowsky-style brainstorming of how everything could go horribly wrong before we even realize what we were dealing with, just to even things out!)
Thanks for this thoughtful and detailed response. I think we are basically on the same page now. I agree with your point about Eutopia vs. 2045.
Even if you don’t speak for FLI, I (at least somewhat) do, and agree with most of what you say here — thanks for taking the time and effort to say it!
I’ll also add that — again — we envisage this contest as just step 1 in a bigger program, which will include other sets of constraints.
Directly conflicts with the geopolitical requirements. Also not compatible with the ‘sector by sector’ scope of economic impact—an AGI would be revolutionizing everything at once, and the only question would be whether it was merely flipping the figurative table or going directly to interpolating every figurative chemical bond in the table with figurative gas simultaneously and leaving it to crumble into figurative dust.
The ‘Silent elitism’ view is approximately correct, except in its assumption that there is a current elite who endorse the eutopia, which there is not. Even the most forward-thinking people of today, the Ben Franklins of the 2020s, would balk. The only way humans know how to transition toward a eutopia is slowly over generations. Since this has a substantial cost, speedrunning that transition is desirable, but how exactly that speedrun can be accomplished without leaving a lot of wreckage in its wake is a topic best left for superintelligences, or at the very least intelligences augmented somewhat beyond the best capabilities we currently have available.
What a coincidence! You have precisely described this contest. This is, explicitly, a “make up a nice-sounding future with no resemblance to our true destination” contest. And yes, it’s at best completely immoral. At worst they get high on their own supply and use it to set priorities, in which case it’s dangerous and aims us toward UFAI and impossibilities.
At least it’s not the kind of believing absurdities which produces people willing to commit atrocities in service of those beliefs. Unfortunately, poor understanding of alignment creates a lot of atrocities from minimal provocation anyway.
This is not true. There is no law of the universe which states that there must be a way to translate the ways in which a state good for its inhabitants (who are transhuman or posthuman i.e possessed of humanity and other various important mental qualities), into words which can be conveyed in present human language, by text or speech, that sound appealing. That might be a nice property for a universe to have but ours doesn’t.
Some point along a continuum from here to there, a continuum we might slide up or down with effort, probably can be so described—a fixed-point theorem of some sort probably applies. However, that need not be an honest depiction of what life will be like if we slide in that direction, any more than showing a vision of the Paris Commune to a Parisien on the day Napoleon fell (stipulating that they approved of it) would be an honest view of Paris’s future.
See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest.
”Directly conflicts with the geopolitical requirements.”—How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, “Okay AGI, I don’t want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars.” That would be 5x the current rate of worldwide economic growth, which would probably feel like “transforming the economy sector by sector” to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
The key part of my story is not very realistic or detailed. (How do I end up with a world-dominating AGI perfectly under my control by tomorrow?)
I asked my AGI to do something that you would consider unambitious, and maybe immoral. You’d rather I command my genie to make changes somewhere on the spectrum from “merely flipping the figurative table” to dissolving the entire physical world and reconfiguring it into computronium. But that’s just a personal preference of yours—just because I’ve invented an extremely powerful AGI doesn’t mean I can’t ask it to do boring ordinary things like merely curing cancer instead of solving immortality.
I agree with you that there’s a spectrum of different things that can be meant by “honesty”, sliding from “technically accurate statements which fail to convey the general impression” to “correctly conveying the general impression but giving vague or misleading statements”, and that in some cases the thing we’re trying to describe is so strange that no matter where we go along that continuum it’ll feel like lying because the description will be misleading in one way or the other. That’s a problem with full Eutopia but I don’t think it’s a problem with the 2045 story, where we’re being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn’t seem very likely if AGI is involved.
I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won’t work. (See my “cabal of secret geniuses runs the show from behind the scenes” versus “USA/China/EU come together to govern AI for all mankind” example in my comment to kokotajlod—if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason—because the TRUE most-plausible path to a good future doesn’t/couldn’t go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses “have the AGI create a nanosystem to melt all the world’s GPUs” (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I’m not sure about this question and I’m open to being won over.
The ‘unambitious’ thing you ask the AI to do would create worldwide political change. It is absurd to think that it wouldn’t. Even ordinary technological change creates worldwide political change at that scale!
And an AGI having that little impact is also not plausible; if that’s all you do, the second mover—and possibly the third, fourth, fifth, if everyone moves slow—spits out an AGI and flips the table, because you can’t be that unambitious and still block other AGIs from performing pivotal acts, and even if you want to think small, the other actors won’t. Even if they are approximately as unambitious, they will have different goals, and the interaction will immediately amp up the chaos.
There is just no way for an actual AGI scenario to meet these guidelines. Any attempt to draw a world which meets them has written the bottom line first and is torturing its logic trying to construct a vaguely plausible story that might lead to it.
I believe that you are too quick to label this story as absurd. Ordinary technology does not have the capacity to correct towards explicitly smaller changes that still satisfy the objective. If the AGI wants to prevent wars while minimally disturbing the worldwide politics, I find it plausible that it would succeed.
Similarly, just because an AGI has very little visible impact, does not mean that it isn’t effectively in control. For a true AGI, it should be trivial to interrupt the second mover without any great upheaval. It should be able to surpress other AGIs from coming into existence without causing too much of a stir.
I do somewhat agree with your reservations, but I find that your way of adressing them seems uncharitable (i.e. “at best completely immoral”).