I’d like to understand in more detail how this analogy breaks down.
I think the important disanalogy is that once you’ve created a safe AGI of sufficient power, you win. (Because it’s an AGI, so it can go around doing powerful AGI stuff – other projects could be controlled or purchased, etc.)
It’s not for sure the case that first-past-the-post will be the end-of-the-day winner, but being first-past-the-post is probably a big advantage. Bostrom has some discussion of this in the multipolar / singleton section of Superintelligence, if I recall correctly.
Drexler’s Comprehensive AI Services is an alternative framing for what we mean by AGI. Probably relevant here, though I haven’t engaged closely with it yet.
OK, this is what I modeled AI alignment folks as believing. But doesn’t the idea of first-past-the-post-is-the-winner rely on a “hard takeoff” scenario? This is a view I associate with Eliezer. But Paul in the podcast says that he thinks a gradual takeoff is more likely, and envisions a smooth gradient of AI capability such that human-level AI comes into existence in a world where slightly stupider AIs already exist.
The relevant passage:
and in particular, when someone develops human level AI, it’s not going to emerge in a world like the world of today where we can say that indeed, having human level AI today would give you a decisive strategic advantage. Instead, it will emerge in a world which is already much, much crazier than the world of today, where having a human AI gives you some more modest advantage.
So I get why you would drop everything and race to be the first to build an aligned AGI if you’re Eliezer. But if you’re Paul, I’m not sure why you would do this, since you think it will only give you a modest advantage.
(Also, if the idea is to build your AGI first and then use it to stop everyone else from building their AGIs—I feel like that second part of the plan should be fronted a bit more! “I’m doing research to ensure AI does what we tell it to” is quite a different proposition from “I’m doing research to ensure AI does what we tell it to, so that I can build an AI and tell it to conquer the world for me.”)
… why you would drop everything and race to be the first to build an aligned AGI if you’re Eliezer. But if you’re Paul, I’m not sure why you would do this, since you think it will only give you a modest advantage.
Good point. Maybe another thing here is that under Paul’s view, working on AGI / AI alignment now increases the probability that the whole AI development ecosystem heads in a good direction. (Prestigious + safe AI work increases the incentives for others to do safe AI work, so that they appear responsible.)
Speculative: perhaps the motivation for a lot of OpenAI’s AI development work is to increase its clout in the field, so that other research groups take the AI alignment stuff seriously. Also sucking up talented researchers to increase the overall proportion of AI researchers that are working in a group that takes safety seriously.
I think the important disanalogy is that once you’ve created a safe AGI of sufficient power, you win. (Because it’s an AGI, so it can go around doing powerful AGI stuff – other projects could be controlled or purchased, etc.)
It’s not for sure the case that first-past-the-post will be the end-of-the-day winner, but being first-past-the-post is probably a big advantage. Bostrom has some discussion of this in the multipolar / singleton section of Superintelligence, if I recall correctly.
Drexler’s Comprehensive AI Services is an alternative framing for what we mean by AGI. Probably relevant here, though I haven’t engaged closely with it yet.
OK, this is what I modeled AI alignment folks as believing. But doesn’t the idea of first-past-the-post-is-the-winner rely on a “hard takeoff” scenario? This is a view I associate with Eliezer. But Paul in the podcast says that he thinks a gradual takeoff is more likely, and envisions a smooth gradient of AI capability such that human-level AI comes into existence in a world where slightly stupider AIs already exist.
The relevant passage:
So I get why you would drop everything and race to be the first to build an aligned AGI if you’re Eliezer. But if you’re Paul, I’m not sure why you would do this, since you think it will only give you a modest advantage.
(Also, if the idea is to build your AGI first and then use it to stop everyone else from building their AGIs—I feel like that second part of the plan should be fronted a bit more! “I’m doing research to ensure AI does what we tell it to” is quite a different proposition from “I’m doing research to ensure AI does what we tell it to, so that I can build an AI and tell it to conquer the world for me.”)
Would probably incur a lot of bad PR.
Good point. Maybe another thing here is that under Paul’s view, working on AGI / AI alignment now increases the probability that the whole AI development ecosystem heads in a good direction. (Prestigious + safe AI work increases the incentives for others to do safe AI work, so that they appear responsible.)
Speculative: perhaps the motivation for a lot of OpenAI’s AI development work is to increase its clout in the field, so that other research groups take the AI alignment stuff seriously. Also sucking up talented researchers to increase the overall proportion of AI researchers that are working in a group that takes safety seriously.