I haven’t read all of your posts that carefully yet, so I might be misunderstanding something, but generally, it seems to me like this approach has a some “upper bound” modeling assumptions that are then used as an all things considered distribution.
My biggest disagreement is that I think that your distribution over FLOPs [1] required for TAI (pulled from here), is too large.
My reading is that this was generated assuming that we would train TAI primarily via human imitation, which seems really inefficient. There are so many other strategies for training powerful AI systems, and I expect people to transition to using something better than imitation. For example, see the fairly obvious techniques discussed here.
GPT-4 required ~25 log(FLOP)s, and eyeballing it, the mode of this distribution seems to be about ~35 log(FLOP)s, so that is +10 OOMs as the median over GPT-4. The gap between GPT-3 and GPT-4 is ~2 OOMs, so this would imply a median of 5 GPT-sized jumps until TAI. Personally, I think that 2 GPT jumps / +4 OOMs is a pretty reasonable mode for TAI (e.g. the difference between GPT-2 and GPT-4).
In the ‘against very short timelines’ section, it seems like your argument mostly routes through it being computationally difficult to simulate the entire economy with human level AIs, because of inference costs. I agree with this, but think that AIs won’t stay human level for very long, because of AI-driven algorithmic improvements. In 4 year timelines worlds, I don’t expect the economy to be very significantly automated before the point of no return, I instead expect it to look more like faster and faster algorithmic advances. Instead of deploying 10 million human workers dispersed over the entire economy, I think this would look more like deploying 10 million more AGI researchers, and then getting compounding returns on algorithmic progress from there.
But, as I have just argued above, a rapid general acceleration of technological progress from pre-superintelligent AI seems very unlikely in the next few years.
I generally don’t see this argument. Automating the entire economy != automating ML research. It remains quite plausible to me that we reach superintelligence before the economy is 100% automated.
I haven’t read all of your posts that carefully yet, so I might be misunderstanding something, but generally, it seems to me like this approach has a some “upper bound” modeling assumptions that are then used as an all things considered distribution.
I disagree. I’m not using this distribution as an all-things-considered view. My own distribution over the training FLOP for transformative AI is centered around ~10^32 FLOP using 2023 algorithms, with a standard deviation of about 3 OOM.
I think this makes sense after applying a few OOM adjustment downwards to the original distribution, taking into account the fact that it’s an upper bound.
I agree this is confusing because of the timelines model I linked to in the post, but the timelines model I linked to is not actually presenting my own view. I contrasted that model with my all-things-considered view at the end of this post, noting that the model seemed conservative in some respects, but aggressive in other respects (since it doesn’t take into account exogenous delays, or model uncertainty).
GPT-4 required ~25 log(FLOP)s, and eyeballing it, the mode of this distribution seems to be about ~35 log(FLOP)s, so that is +10 OOMs as the median over GPT-4. The gap between GPT-3 and GPT-4 is ~2 OOMs, so this would imply a median of 5 GPT-sized jumps until TAI. Personally, I think that 2 GPT jumps / +4 OOMs is a pretty reasonable mode for TAI (e.g. the difference between GPT-2 and GPT-4).
I’m talking about effective flop, not physical FLOP. Plausibly the difference in effective FLOP between GPT-3 and GPT-4 is more like 3 OOMs rather than 2 OOMs. With that in mind, my median of 10^32 2023 FLOP for training transformative AI doesn’t seem obviously too conservative to me. It would indicate that we’re about 2.3 more jumps the size of GPT-3 --> GPT-4 away from transformative AI, and my intuition seems quite in line with that.
My own distribution over the training FLOP for transformative AI is centered around ~10^32 FLOP using 2023 algorithms, with a standard deviation of about 3 OOM.
Thanks for the numbers!
For comparison, takeoffspeeds.com has an aggressive monte-carlo (with a median of 10^31 training FLOP) that yields a median of 2033.7 for 100% automation — and a p(TAI < 2030) of ~28%. That 28% is pretty radically different from your 2%. Do you know your biggest disagreements with that model?
The 1 OOM difference in training FLOP presumably doesn’t explain that much. (Although maybe it’s more, because takeoffspeeds.com talks about “AGI” and you talk about “TAI”. On the other hand, maybe your bar for “transformative” is lower than 100% automation.)
Some related responses to stuff in your post:
The most likely cause of such a sudden acceleration seems to be that pre-superintelligent systems could accelerate technological progress. But, as I have just argued above, a rapid general acceleration of technological progress from pre-superintelligent AI seems very unlikely in the next few years.
You argued that AI labor would be small in comparison to all of human labor, if we got really good software in the next 4 years. But if we had recently gotten such insane gains in ML-capabilities, people would want to vastly increase investment in ML-research (and hardware production) relative to everything else in the world. Normally, labor spent on ML research would lag behind, because it takes a long time to teach a large number of humans the requisite skills. But for each skill, you’d only need to figure out how to teach AI about it once, and then all 10 million AIs would be able to do it. (There would certainly be some lag, here, too. Your posts says “lag for AI will likely be more than a year”, which I’m sympathetic to, but there’s time for that.)
When I google “total number of ml researchers”, the largest number I see is 300k and I think the real answer is <100k. So I don’t think a huge acceleration in AI-relevant technological progress before 2030 is out of the question.
(I think it’s plausible we should actually be thinking about the best ML researchers rather than just counting up the total number. But I don’t think it’d be crazy for AIs to meet that bar in the hypothetical you paint. Given the parallelizability of AI, it’s both the case that (i) it’s worth spending much more effort on teaching skills to AIs, and (ii) that it’s possible for AIs to spend much more effective time on learning.)
Also, if the AIs are bottlenecked by motor skills, humans can do that part. When automating small parts of the total economy (like ML research or hardware production), there’s room to get more humans into those industries to do all the necessary physical tasks. (And at the point when AI cognitive output is large compared to the entire human workforce, you can get a big boost in total world output by having humans switch into just doing manual labor, directed by AIs.)
However, my unconditional view is somewhat different. After considering all potential delays (including regulation, which I think is likely to be substantial) and model uncertainty, my overall median TAI timeline is somewhere between 20-30 years from now, with a long tail extending many decades into the future.
I can see how stuff like regulation would feature in many worlds, but it seems high variance and like it should allow for a significant probability of ~no delay.
Also, my intuition is that 2% is small enough in the relevant context that model uncertainty should push it up rather than down.
For comparison, takeoffspeeds.com has an aggressive monte-carlo (with a median of 10^31 training FLOP) that yields a median of 2033.7 for 100% automation — and a p(TAI < 2030) of ~28%. That 28% is pretty radically different from your 2%. Do you know your biggest disagreements with that model?
I think my biggest disagreement with the takeoff speeds model is just that it’s conditional on things like: no coordinated delays, regulation, or exogenous events like war, and doesn’t take into account model uncertainty. My other big argument here is that I just think robots aren’t very impressive right now, and it’s hard to see them going from being unimpressive to extremely impressive in just a few short years. 2030 is very soon. Imagining a even a ~4 year delay due to all of these factors produces a very different distribution.
Also, as you note, “takeoffspeeds.com talks about “AGI” and you talk about “TAI”. I think transformative AI is a lower bar than 100% automation. The model itself says they added “an extra OOM to account for TAI being a lower bar than full automation (AGI).” Notably, if you put in 10^33 2022 FLOP into the takeoff model (and keep in mind that I was talking about 2023 FLOP), it produces a median year of >30% GWP growth of about 2032, which isn’t too far from what I said in the post:
Assuming no substantial delays or large disasters such as war in the meantime, I believe that TAI will probably arrive within about 15 years
I added about four years to this 2032 timeline due to robots, which I think is reasonable even given your considerations about how we don’t have to automate everything—we just need to automate the bottlenecks to producing more semiconductor fabs. But you could be right that I’m still being too conservative.
I think my biggest disagreement with the takeoff speeds model is just that it’s conditional on things like: no coordinated delays, regulation, or exogenous events like war, and doesn’t take into account model uncertainty.
Cool, I thought that was most of the explanation for the difference in the median. But I thought it shouldn’t be enough to explain the 14x difference between 28% and 2% by 2030, because I think there should be a ≥20% chance that there are no significant coordinated delays, regulation, or relevant exogenous events if AI goes wild in the next 7 years. (And that model uncertainty should work to increase rather than decrease the probability, here.)
If you think robotics would definitely be necessary, then I can see how that would be significant.
But I think it’s possible that we get a software-only singularity. Or more broadly, simultaneously having (i) AI improving algorithms (...improving AIs), (ii) a large fraction of the world’s fab-capacity redirected to AI chips, and (iii) AIs helping with late-stage hardware stuff like chip-design. (I agree that it takes a long time to build new fabs.) This would simultaneously explain why robotics aren’t necessary (before we have crazy good AI) and decrease the probability of regulatory delays, since the AIs would just need to be deployed inside a few companies. (I can see how regulation would by-default slow down some kinds of broad deployment, but it seems super unclear whether there will be regulation put in place to slow down R&D and internal deployment.)
Update: I changed the probability distribution in the post slightly in line with your criticism. The new distribution is almost exactly the same, except that I think it portrays a more realistic picture of short timelines. The p(TAI < 2030) is now 5% [eta: now 18%], rather than 2%.
Cool, I thought that was most of the explanation for the difference in the median. But I thought it shouldn’t be enough to explain the 14x difference between 28% and 2% by 2030
That’s reasonable. I think I probably should have put more like 3-6% credence before 2030. I should note that it’s a bit difficult to tune the Metaculus distributions to produce exactly what you want, and the distribution shouldn’t be seen as an exact representation of my beliefs.
Sorry, that was very poor wording. I meant that 2023 FLOP is probably about equal to 2 2022 FLOP, due to continued algorithmic progress. I’ll reword the comment you replied to.
Incidentally, as its central estimate for algorithmic improvement, the takeoff speeds model uses AI and Efficiency’s ~1.7x per year, and then halves it to ~1.3x per year (because todays’ algorithmic progress might not generalize to TAI). If you’re at 2x per year, then you should maybe increase the “returns to software” from 1.25 to ~3.5, which would cut the model’s timelines by something like 3 years. (More on longer timelines, less on shorter timelines.)
I haven’t read all of your posts that carefully yet, so I might be misunderstanding something, but generally, it seems to me like this approach has a some “upper bound” modeling assumptions that are then used as an all things considered distribution.
My biggest disagreement is that I think that your distribution over FLOPs [1] required for TAI (pulled from here), is too large.
My reading is that this was generated assuming that we would train TAI primarily via human imitation, which seems really inefficient. There are so many other strategies for training powerful AI systems, and I expect people to transition to using something better than imitation. For example, see the fairly obvious techniques discussed here.
GPT-4 required ~25 log(FLOP)s, and eyeballing it, the mode of this distribution seems to be about ~35 log(FLOP)s, so that is +10 OOMs as the median over GPT-4. The gap between GPT-3 and GPT-4 is ~2 OOMs, so this would imply a median of 5 GPT-sized jumps until TAI. Personally, I think that 2 GPT jumps / +4 OOMs is a pretty reasonable mode for TAI (e.g. the difference between GPT-2 and GPT-4).
In the ‘against very short timelines’ section, it seems like your argument mostly routes through it being computationally difficult to simulate the entire economy with human level AIs, because of inference costs. I agree with this, but think that AIs won’t stay human level for very long, because of AI-driven algorithmic improvements. In 4 year timelines worlds, I don’t expect the economy to be very significantly automated before the point of no return, I instead expect it to look more like faster and faster algorithmic advances. Instead of deploying 10 million human workers dispersed over the entire economy, I think this would look more like deploying 10 million more AGI researchers, and then getting compounding returns on algorithmic progress from there.
I generally don’t see this argument. Automating the entire economy != automating ML research. It remains quite plausible to me that we reach superintelligence before the economy is 100% automated.
I’m assuming that this is something like 2023-effective flops (i.e. baking in algorithmic progress, let me know if I’m wrong about this).
I disagree. I’m not using this distribution as an all-things-considered view. My own distribution over the training FLOP for transformative AI is centered around ~10^32 FLOP using 2023 algorithms, with a standard deviation of about 3 OOM.
I think this makes sense after applying a few OOM adjustment downwards to the original distribution, taking into account the fact that it’s an upper bound.
I agree this is confusing because of the timelines model I linked to in the post, but the timelines model I linked to is not actually presenting my own view. I contrasted that model with my all-things-considered view at the end of this post, noting that the model seemed conservative in some respects, but aggressive in other respects (since it doesn’t take into account exogenous delays, or model uncertainty).
I’m talking about effective flop, not physical FLOP. Plausibly the difference in effective FLOP between GPT-3 and GPT-4 is more like 3 OOMs rather than 2 OOMs. With that in mind, my median of 10^32 2023 FLOP for training transformative AI doesn’t seem obviously too conservative to me. It would indicate that we’re about 2.3 more jumps the size of GPT-3 --> GPT-4 away from transformative AI, and my intuition seems quite in line with that.
Thanks for the numbers!
For comparison, takeoffspeeds.com has an aggressive monte-carlo (with a median of 10^31 training FLOP) that yields a median of 2033.7 for 100% automation — and a p(TAI < 2030) of ~28%. That 28% is pretty radically different from your 2%. Do you know your biggest disagreements with that model?
The 1 OOM difference in training FLOP presumably doesn’t explain that much. (Although maybe it’s more, because takeoffspeeds.com talks about “AGI” and you talk about “TAI”. On the other hand, maybe your bar for “transformative” is lower than 100% automation.)
Some related responses to stuff in your post:
You argued that AI labor would be small in comparison to all of human labor, if we got really good software in the next 4 years. But if we had recently gotten such insane gains in ML-capabilities, people would want to vastly increase investment in ML-research (and hardware production) relative to everything else in the world. Normally, labor spent on ML research would lag behind, because it takes a long time to teach a large number of humans the requisite skills. But for each skill, you’d only need to figure out how to teach AI about it once, and then all 10 million AIs would be able to do it. (There would certainly be some lag, here, too. Your posts says “lag for AI will likely be more than a year”, which I’m sympathetic to, but there’s time for that.)
When I google “total number of ml researchers”, the largest number I see is 300k and I think the real answer is <100k. So I don’t think a huge acceleration in AI-relevant technological progress before 2030 is out of the question.
(I think it’s plausible we should actually be thinking about the best ML researchers rather than just counting up the total number. But I don’t think it’d be crazy for AIs to meet that bar in the hypothetical you paint. Given the parallelizability of AI, it’s both the case that (i) it’s worth spending much more effort on teaching skills to AIs, and (ii) that it’s possible for AIs to spend much more effective time on learning.)
Mostly not ML research.
Also, if the AIs are bottlenecked by motor skills, humans can do that part. When automating small parts of the total economy (like ML research or hardware production), there’s room to get more humans into those industries to do all the necessary physical tasks. (And at the point when AI cognitive output is large compared to the entire human workforce, you can get a big boost in total world output by having humans switch into just doing manual labor, directed by AIs.)
I can see how stuff like regulation would feature in many worlds, but it seems high variance and like it should allow for a significant probability of ~no delay.
Also, my intuition is that 2% is small enough in the relevant context that model uncertainty should push it up rather than down.
I think my biggest disagreement with the takeoff speeds model is just that it’s conditional on things like: no coordinated delays, regulation, or exogenous events like war, and doesn’t take into account model uncertainty. My other big argument here is that I just think robots aren’t very impressive right now, and it’s hard to see them going from being unimpressive to extremely impressive in just a few short years. 2030 is very soon. Imagining a even a ~4 year delay due to all of these factors produces a very different distribution.
Also, as you note, “takeoffspeeds.com talks about “AGI” and you talk about “TAI”. I think transformative AI is a lower bar than 100% automation. The model itself says they added “an extra OOM to account for TAI being a lower bar than full automation (AGI).” Notably, if you put in 10^33 2022 FLOP into the takeoff model (and keep in mind that I was talking about 2023 FLOP), it produces a median year of >30% GWP growth of about 2032, which isn’t too far from what I said in the post:
I added about four years to this 2032 timeline due to robots, which I think is reasonable even given your considerations about how we don’t have to automate everything—we just need to automate the bottlenecks to producing more semiconductor fabs. But you could be right that I’m still being too conservative.
Cool, I thought that was most of the explanation for the difference in the median. But I thought it shouldn’t be enough to explain the 14x difference between 28% and 2% by 2030, because I think there should be a ≥20% chance that there are no significant coordinated delays, regulation, or relevant exogenous events if AI goes wild in the next 7 years. (And that model uncertainty should work to increase rather than decrease the probability, here.)
If you think robotics would definitely be necessary, then I can see how that would be significant.
But I think it’s possible that we get a software-only singularity. Or more broadly, simultaneously having (i) AI improving algorithms (...improving AIs), (ii) a large fraction of the world’s fab-capacity redirected to AI chips, and (iii) AIs helping with late-stage hardware stuff like chip-design. (I agree that it takes a long time to build new fabs.) This would simultaneously explain why robotics aren’t necessary (before we have crazy good AI) and decrease the probability of regulatory delays, since the AIs would just need to be deployed inside a few companies. (I can see how regulation would by-default slow down some kinds of broad deployment, but it seems super unclear whether there will be regulation put in place to slow down R&D and internal deployment.)
Update: I changed the probability distribution in the post slightly in line with your criticism. The new distribution is almost exactly the same, except that I think it portrays a more realistic picture of short timelines. The p(TAI < 2030) is now 5% [eta: now 18%], rather than 2%.
That’s reasonable. I think I probably should have put more like 3-6% credence before 2030. I should note that it’s a bit difficult to tune the Metaculus distributions to produce exactly what you want, and the distribution shouldn’t be seen as an exact representation of my beliefs.
I don’t understand this. Why would there be a 2x speedup in algorithmic progress?
Sorry, that was very poor wording. I meant that 2023 FLOP is probably about equal to 2 2022 FLOP, due to continued algorithmic progress. I’ll reword the comment you replied to.
Nice, gotcha.
Incidentally, as its central estimate for algorithmic improvement, the takeoff speeds model uses AI and Efficiency’s ~1.7x per year, and then halves it to ~1.3x per year (because todays’ algorithmic progress might not generalize to TAI). If you’re at 2x per year, then you should maybe increase the “returns to software” from 1.25 to ~3.5, which would cut the model’s timelines by something like 3 years. (More on longer timelines, less on shorter timelines.)