Regarding the multiple stage fallacy, we recognize it’s a risk of a framework like this and go to some lengths explaining why we think our analysis does not suffer from it. (Namely, in the executive summary, the discussion, and the appendix “Why 0.4% might be less confident than it seems.”)
What are the disjunctive alternatives you think our framework misses?
If you disagree with our admittedly imperfect guesses, we kindly ask that you supply your own preferred probabilities (or framework modifications). It’s easier to tear down than build up, and we’d love to hear how you think this analysis can be improved.
Like Matthew, I think your paper is really interesting and impressive.
Some issues I have with the methodology:
Your framework excludes some factors that could cause the overall probability to increase.
For example, I can think of ways that a great power conflict (over Taiwan, say) actually increases the chances of TAI. But your framework doesn’t easily account for this.
You could have factored it in in all or some of the other stages, but I’m not sure you have, and it seems generally like this asymmetry (the “positive” effect of an event is factored into various other stages if at all, but the “negative” effect of the same event is estimated on its own conjunctive stage) will tend to give lower overall probabilities than it should.
It seems like you sometimes don’t fully condition on preceding propositions.
You calculate a base rate of “10% chance of [depression] in the next 20 years”, and write: “Conditional on being in a world on track toward transformative AGI, we estimate a ~0.5%/yr chance of depression, implying a ~10% chance in the next 20 years.”
But this doesn’t seem like fully conditioning on a world with TAI that is cheap, that can automate ~100% of human tasks, and that can be deployed at scale, and that is relatively unregulated. It seems like once that happens, and when it’s nearly happening (e.g. AIs automate 20% of 2022-tasks), the probability of a severe depression should be way below historical base rates?
Similarly for “We quickly scale up semiconductor manufacturing and electrical generation”, it seems like you don’t fully condition on a world where we have TAI that is cheap, that can automate ~100% of human tasks, and that can operate cheap, high-quality robots, and that can probably be deployed to some fairly wide extent even if not (yet) to actually automate ~all human labour.
Like, your X100 is 100x as cost-effective as the H100, but that doesn’t seem that far off what you’d get from by just projecting the Epoch trend for ML GPU price-performance out 2 decades?
More generally, I think these sorts of things are really hard to get right (i.e. it’s hard to imagine oneself in a conditional world, and estimate probabilities there without anchoring on the present world), and will tend to bias people to smaller overall estimates when using more conjunctive steps.
Your framework excludes some factors that could cause the overall probability to increase.
For example, I can think of ways that a great power conflict (over Taiwan, say) actually increases the chances of TAI. But your framework doesn’t easily account for this.
Totally reasonable to disagree with us on some of these forecasts—they’re rough educated guesses, after all. We welcome others to contribute their own forecasts. I’m curious: What do you think are the rough odds that invasion of Taiwan increases the likelihood of TAGI by 2043?
But this doesn’t seem like fully conditioning on a world with TAI that is cheap, that can automate ~100% of human tasks, and that can be deployed at scale, and that is relatively unregulated.
it seems like you don’t fully condition on a world where we have TAI that is cheap, that can automate ~100% of human tasks, and that can operate cheap, high-quality robots, and that can probably be deployed to some fairly wide extent even if not (yet) to actually automate ~all human labour.
Agree wholeheartedly. In a world with scaled, cheap TAGI, things are going to look wildly different and it will be hard to predict what happens. Change could be a lot faster than what we’re used to, and historical precedent and intuition might be relatively poor guides relative to first principles thinking.
However, we feel somewhat more comfortable with our predictions prior to scaled, cheap AGI. Like, if it takes 3e30 − 3e35 operations to train an early AGI, then I don’t think we can condition on that AGI accelerating us towards construction of the resources needed to generate 3e30 − 3e35 operations. It would be putting the cart before the horse.
What we can (and try to) condition on are potential predecessors to that AGI; e.g., improved narrow AI or expensive human-level AGI. Both of those we have experience with today, which gives us more confidence that we won’t get an insane productivity explosion in the physical construction of fabs and power plants.
We could be wrong, of course, and we’ll find out in 2043.
I’m curious: What do you think are the rough odds that invasion of Taiwan increases the likelihood of TAGI by 2043?
Maybe 20% that it increases the likelihood? Higher if war starts by 2030 or so, and near 0% if it starts in 2041 (but maybe >0% if it starts in 2042?). What number would you put on it, and how would you update your model if that number changed?
However, we feel somewhat more comfortable with our predictions prior to scaled, cheap AGI. Like, if it takes 3e30 − 3e35 operations to train an early AGI, then I don’t think we can condition on that AGI accelerating us towards construction of the resources needed to generate 3e30 − 3e35 operations. It would be putting the cart before the horse.
What we can (and try to) condition on are potential predecessors to that AGI; e.g., improved narrow AI or expensive human-level AGI. Both of those we have experience with today, which gives us more confidence that we won’t get an insane productivity explosion in the physical construction of fabs and power plants.
I think what you’re saying here is, “yes, we condition on such a world, but even in such a world these things won’t be true for all of 2023-2043, but mainly only towards the latter years in that range”. Is that right?
I agree to some extent, but as you wrote, “transformative AGI is a much higher bar than merely massive progress in AI”: I think in a lot of those previous years we’ll still have AI doing lots of work to speed up R&D and carry out lots of other economically useful tasks. Like, we know in this world that we’re headed for AGI in 2043 or even earlier, so we should be seeing really capable and useful AI systems already in 2030 and 2035 and so on.
Maybe you think the progression from today’s systems to potentially-transformative AGI will be discontinuous or something like that, with lots of progress (on algorithms, hardware, robotics, etc.) happening near the end?
I think in a lot of those previous years we’ll still have AI doing lots of work to speed up R&D and carry out lots of other economically useful tasks. Like, we know in this world that we’re headed for AGI in 2043 or even earlier, so we should be seeing really capable and useful AI systems already in 2030 and 2035 and so on.
Maybe you think the progression from today’s systems to potentially-transformative AGI will be discontinuous or something like that, with lots of progress (on algorithms, hardware, robotics, etc.) happening near the end?
No, I actually fully agree with you. I don’t think progress will be discontinuous, and I do think we will see increasingly capable and useful systems by 2030 and 2035 that accelerate rates of progress.
I think where we may differ is that:
I think the acceleration will likely be more “in line” than “out of line” with the exponential acceleration we already see from improving computer tools and specifically LLM computer tools (e.g., GitHub Copilot, GPT-4). Already a software engineer today is many multiples more productive (by some metrics) than a software engineer in the 90s.
I think that tools that, say, cheaply automate half of work, or expensively automate 100% work probably won’t lead to wild, extra orders of magnitude levels of progress. OpenAI has what, 400 employees?
Scenario one: If half their work was automated, ok now those 400 people could do the work of 800 people. That’s great, but honestly I don’t think it’s path-breaking. And sure, that’s only the first order effect. If half the work was automated, we’d of course elastically start spending way much more on the cheap automated half. But on the other hand, there would be diminishing returns, and for every step that becomes free, we just hit bottlenecks in the hard to automate parts. Even in the limit of cheap AGI, those AGIs may be limited by the GPUs they have to experiment on. Labor becoming free just means capital is the constraint.
Scenario two: Or, suppose we have human-cost human-level AGIs. I’m not convinced that would, to first order, change much either. There are millions of smart people on earth who aren’t working on AI research now. We could hire them, but we don’t. We’re not limited by brains. We’re limited by willingness to spend. So even if we invent human-cost human-level brains, it actually doesn’t change much, because that wasn’t the constraint. (Of course, this is massively oversimplifying, and obviously human-cost human-level AGIs would be a bigger deal than human workers because of their ability to be rapidly improved and copied. But I hope it nevertheless conveys why I think AGI will need to get close to transformative levels before growth really explodes.)
Overall where it feels like I’m differing from some folks here is that I think higher levels of AI capability will be needed before we get wild self-improvement takeoff. I don’t think it will be early, because even if we get massive automation due to uneven AI we’ll still be bottlenecked by the things it’s bad at. I acknowledge this is a pretty squishy argument and I find it difficult to quantify and articulate, so I think it’s quite reasonable to disagree with me here. In general though, I think we’ve seen a long history of things being harder to automate than we thought (e.g., self-driving, radiology, etc.). It will be exciting to see what happens!
Already a software engineer today is many multiples more productive (by some metrics) than a software engineer in the 90s.
Do you have any material on this? It sounds plausible to me but I couldn’t find anything with a quick search.
I think that tools that, say, cheaply automate half of work, or expensively automate 100% work probably won’t lead to wild, extra orders of magnitude levels of progress.
Supposing you take “progress” to mean something like GDP per capita or AI capabilities as measured on various benchmarks, I agree that it probably won’t (though I wouldn’t completely rule it out). But also, I don’t think progress would need to jump by OOMs for the chances of a financial crisis large enough to derail transformative AGI to be drastically reduced. (To be clear, I don’t think drastic self-improvement is necessary for this, and I expect to see something more like increasingly sophisticated versions of “we use AI to automate AI research/engineering”.)
I also think it’s pretty likely that, if there is a financial crisis in these worlds, AI progress isn’t noticeably impacted. If you look at papers published in various fields, patent applications, adoption of various IT technologies, numbers of researchers per capita—none of these things seem to slow down in the wake of financial crises. Same thing for AI: I don’t see any derailment from financial crises when looking at model sizes (both in terms of parameters and training compute), dataset sizes or chess program Elo.
Maybe capital expenditure will decrease, and that might only start being really important once SOTA models are extremely expensive, but on the other hand: if there’s anything in these worlds you want to keep investing in it’s probably the technology that’s headed towards full-blown AGI? Maybe I think 1 in 10 financial crises would substantially derail transformative AGI in these worlds, but it seems you think it’s more like 1 in 2.
Scenario one: If half their work was automated, ok now those 400 people could do the work of 800 people. That’s great, but honestly I don’t think it’s path-breaking. And sure, that’s only the first order effect. If half the work was automated, we’d of course elastically start spending way much more on the cheap automated half. But on the other hand, there would be diminishing returns, and for every step that becomes free, we just hit bottlenecks in the hard to automate parts. Even in the limit of cheap AGI, those AGIs may be limited by the GPUs they have to experiment on. Labor becoming free just means capital is the constraint.
Yeah, but why only focus on OAI? In this world we have AIs that cheaply automate half of work. That seems like it would have immense economic value and promise, enough to inspire massive new investments in AI companies.
Scenario two: Or, suppose we have human-cost human-level AGIs. I’m not convinced that would, to first order, change much either. There are millions of smart people on earth who aren’t working on AI research now. We could hire them, but we don’t. We’re not limited by brains. We’re limited by willingness to spend. So even if we invent human-cost human-level brains, it actually doesn’t change much, because that wasn’t the constraint. (Of course, this is massively oversimplifying, and obviously human-cost human-level AGIs would be a bigger deal than human workers because of their ability to be rapidly improved and copied. But I hope it nevertheless conveys why I think AGI will need to get close to transformative levels before growth really explodes.)
Ah, I think we have a crux here. I think that, if you could hire—for the same price as a human—a human-level AGI, that would indeed change things a lot. I’d reckon the AGI would have a 3-4x productivity boost from being able to work 24⁄7, and would be perfectly obedient, wouldn’t be limited to working in a single field, could more easily transfer knowledge to other AIs, could be backed up and/or replicated, wouldn’t need an office or a fun work environment, can be “hired” or “fired” ~instantly without difficulty, etc.
That feels somehow beside the point, though. I think in any such scenario, there’s also going to be very cheap AIs with sub-human intelligence that would have broad economic impact too.
Do you have any material on this? It sounds plausible to me but I couldn’t find anything with a quick search.
Nope, it’s just an unsubstantiated guess based on seeing what small teams can build today vs 30 years ago. Also based on the massive improvement in open-source libraries and tooling compared to then. Today’s developers can work faster at higher levels of abstraction compared to folks back then.
In this world we have AIs that cheaply automate half of work. That seems like it would have immense economic value and promise, enough to inspire massive new investments in AI companies....
Ah, I think we have a crux here. I think that, if you could hire—for the same price as a human—a human-level AGI, that would indeed change things a lot. I’d reckon the AGI would have a 3-4x productivity boost from being able to work 24⁄7, and would be perfectly obedient, wouldn’t be limited to working in a single field, could more easily transfer knowledge to other AIs, could be backed up and/or replicated, wouldn’t need an office or a fun work environment, can be “hired” or “fired” ~instantly without difficulty, etc.
That feels somehow beside the point, though. I think in any such scenario, there’s also going to be very cheap AIs with sub-human intelligence that would have broad economic impact too.
Absolutely agree. AI and AGI will likely provide immense economic value even before the threshold of transformative AGI is crossed.
Still, supposing that AI research today is:
50⁄50 mix of capital and labor
faces diminishing returns
and has elastic demand
...then even a 4x labor productivity boost may not be all that path-breaking when you zoom out enough. Things will speed up, surely, but they might won’t create transformative AGI overnight. Even AGI researchers will need time and compute to do their experiments.
Thanks for the kind words.
Regarding the multiple stage fallacy, we recognize it’s a risk of a framework like this and go to some lengths explaining why we think our analysis does not suffer from it. (Namely, in the executive summary, the discussion, and the appendix “Why 0.4% might be less confident than it seems.”)
What are the disjunctive alternatives you think our framework misses?
Like Matthew, I think your paper is really interesting and impressive.
Some issues I have with the methodology:
Your framework excludes some factors that could cause the overall probability to increase.
For example, I can think of ways that a great power conflict (over Taiwan, say) actually increases the chances of TAI. But your framework doesn’t easily account for this.
You could have factored it in in all or some of the other stages, but I’m not sure you have, and it seems generally like this asymmetry (the “positive” effect of an event is factored into various other stages if at all, but the “negative” effect of the same event is estimated on its own conjunctive stage) will tend to give lower overall probabilities than it should.
It seems like you sometimes don’t fully condition on preceding propositions.
You calculate a base rate of “10% chance of [depression] in
the next 20 years”, and write: “Conditional on being in a world on track toward transformative AGI, we estimate a ~0.5%/yr chance of depression, implying a ~10% chance in the next 20 years.”
But this doesn’t seem like fully conditioning on a world with TAI that is cheap, that can automate ~100% of human tasks, and that can be deployed at scale, and that is relatively unregulated. It seems like once that happens, and when it’s nearly happening (e.g. AIs automate 20% of 2022-tasks), the probability of a severe depression should be way below historical base rates?
Similarly for “We quickly scale up semiconductor manufacturing and
electrical generation”, it seems like you don’t fully condition on a world where we have TAI that is cheap, that can automate ~100% of human tasks, and that can operate cheap, high-quality robots, and that can probably be deployed to some fairly wide extent even if not (yet) to actually automate ~all human labour.
Like, your X100 is 100x as cost-effective as the H100, but that doesn’t seem that far off what you’d get from by just projecting the Epoch trend for ML GPU price-performance out 2 decades?
More generally, I think these sorts of things are really hard to get right (i.e. it’s hard to imagine oneself in a conditional world, and estimate probabilities there without anchoring on the present world), and will tend to bias people to smaller overall estimates when using more conjunctive steps.
Thanks!
Totally reasonable to disagree with us on some of these forecasts—they’re rough educated guesses, after all. We welcome others to contribute their own forecasts. I’m curious: What do you think are the rough odds that invasion of Taiwan increases the likelihood of TAGI by 2043?
Agree wholeheartedly. In a world with scaled, cheap TAGI, things are going to look wildly different and it will be hard to predict what happens. Change could be a lot faster than what we’re used to, and historical precedent and intuition might be relatively poor guides relative to first principles thinking.
However, we feel somewhat more comfortable with our predictions prior to scaled, cheap AGI. Like, if it takes 3e30 − 3e35 operations to train an early AGI, then I don’t think we can condition on that AGI accelerating us towards construction of the resources needed to generate 3e30 − 3e35 operations. It would be putting the cart before the horse.
What we can (and try to) condition on are potential predecessors to that AGI; e.g., improved narrow AI or expensive human-level AGI. Both of those we have experience with today, which gives us more confidence that we won’t get an insane productivity explosion in the physical construction of fabs and power plants.
We could be wrong, of course, and we’ll find out in 2043.
Maybe 20% that it increases the likelihood? Higher if war starts by 2030 or so, and near 0% if it starts in 2041 (but maybe >0% if it starts in 2042?). What number would you put on it, and how would you update your model if that number changed?
I think what you’re saying here is, “yes, we condition on such a world, but even in such a world these things won’t be true for all of 2023-2043, but mainly only towards the latter years in that range”. Is that right?
I agree to some extent, but as you wrote, “transformative AGI is a much higher bar than merely massive progress in AI”: I think in a lot of those previous years we’ll still have AI doing lots of work to speed up R&D and carry out lots of other economically useful tasks. Like, we know in this world that we’re headed for AGI in 2043 or even earlier, so we should be seeing really capable and useful AI systems already in 2030 and 2035 and so on.
Maybe you think the progression from today’s systems to potentially-transformative AGI will be discontinuous or something like that, with lots of progress (on algorithms, hardware, robotics, etc.) happening near the end?
No, I actually fully agree with you. I don’t think progress will be discontinuous, and I do think we will see increasingly capable and useful systems by 2030 and 2035 that accelerate rates of progress.
I think where we may differ is that:
I think the acceleration will likely be more “in line” than “out of line” with the exponential acceleration we already see from improving computer tools and specifically LLM computer tools (e.g., GitHub Copilot, GPT-4). Already a software engineer today is many multiples more productive (by some metrics) than a software engineer in the 90s.
I think that tools that, say, cheaply automate half of work, or expensively automate 100% work probably won’t lead to wild, extra orders of magnitude levels of progress. OpenAI has what, 400 employees?
Scenario one: If half their work was automated, ok now those 400 people could do the work of 800 people. That’s great, but honestly I don’t think it’s path-breaking. And sure, that’s only the first order effect. If half the work was automated, we’d of course elastically start spending way much more on the cheap automated half. But on the other hand, there would be diminishing returns, and for every step that becomes free, we just hit bottlenecks in the hard to automate parts. Even in the limit of cheap AGI, those AGIs may be limited by the GPUs they have to experiment on. Labor becoming free just means capital is the constraint.
Scenario two: Or, suppose we have human-cost human-level AGIs. I’m not convinced that would, to first order, change much either. There are millions of smart people on earth who aren’t working on AI research now. We could hire them, but we don’t. We’re not limited by brains. We’re limited by willingness to spend. So even if we invent human-cost human-level brains, it actually doesn’t change much, because that wasn’t the constraint. (Of course, this is massively oversimplifying, and obviously human-cost human-level AGIs would be a bigger deal than human workers because of their ability to be rapidly improved and copied. But I hope it nevertheless conveys why I think AGI will need to get close to transformative levels before growth really explodes.)
Overall where it feels like I’m differing from some folks here is that I think higher levels of AI capability will be needed before we get wild self-improvement takeoff. I don’t think it will be early, because even if we get massive automation due to uneven AI we’ll still be bottlenecked by the things it’s bad at. I acknowledge this is a pretty squishy argument and I find it difficult to quantify and articulate, so I think it’s quite reasonable to disagree with me here. In general though, I think we’ve seen a long history of things being harder to automate than we thought (e.g., self-driving, radiology, etc.). It will be exciting to see what happens!
Do you have any material on this? It sounds plausible to me but I couldn’t find anything with a quick search.
Supposing you take “progress” to mean something like GDP per capita or AI capabilities as measured on various benchmarks, I agree that it probably won’t (though I wouldn’t completely rule it out). But also, I don’t think progress would need to jump by OOMs for the chances of a financial crisis large enough to derail transformative AGI to be drastically reduced. (To be clear, I don’t think drastic self-improvement is necessary for this, and I expect to see something more like increasingly sophisticated versions of “we use AI to automate AI research/engineering”.)
I also think it’s pretty likely that, if there is a financial crisis in these worlds, AI progress isn’t noticeably impacted. If you look at papers published in various fields, patent applications, adoption of various IT technologies, numbers of researchers per capita—none of these things seem to slow down in the wake of financial crises. Same thing for AI: I don’t see any derailment from financial crises when looking at model sizes (both in terms of parameters and training compute), dataset sizes or chess program Elo.
Maybe capital expenditure will decrease, and that might only start being really important once SOTA models are extremely expensive, but on the other hand: if there’s anything in these worlds you want to keep investing in it’s probably the technology that’s headed towards full-blown AGI? Maybe I think 1 in 10 financial crises would substantially derail transformative AGI in these worlds, but it seems you think it’s more like 1 in 2.
Yeah, but why only focus on OAI? In this world we have AIs that cheaply automate half of work. That seems like it would have immense economic value and promise, enough to inspire massive new investments in AI companies.
Ah, I think we have a crux here. I think that, if you could hire—for the same price as a human—a human-level AGI, that would indeed change things a lot. I’d reckon the AGI would have a 3-4x productivity boost from being able to work 24⁄7, and would be perfectly obedient, wouldn’t be limited to working in a single field, could more easily transfer knowledge to other AIs, could be backed up and/or replicated, wouldn’t need an office or a fun work environment, can be “hired” or “fired” ~instantly without difficulty, etc.
That feels somehow beside the point, though. I think in any such scenario, there’s also going to be very cheap AIs with sub-human intelligence that would have broad economic impact too.
Nope, it’s just an unsubstantiated guess based on seeing what small teams can build today vs 30 years ago. Also based on the massive improvement in open-source libraries and tooling compared to then. Today’s developers can work faster at higher levels of abstraction compared to folks back then.
Absolutely agree. AI and AGI will likely provide immense economic value even before the threshold of transformative AGI is crossed.
Still, supposing that AI research today is:
50⁄50 mix of capital and labor
faces diminishing returns
and has elastic demand
...then even a 4x labor productivity boost may not be all that path-breaking when you zoom out enough. Things will speed up, surely, but they might won’t create transformative AGI overnight. Even AGI researchers will need time and compute to do their experiments.