“Refuted” feels overly strong to me. The essay says that market participants don’t think TAGI is coming, and those market participants have strong financial incentive to be correct, which feels unambiguously correct to me. So either TAGI isn’t coming soon, or else a lot of people with a lot of money on the line are wrong. They might well be wrong, but their stance is certainly some form of evidence, and evidence in the direction of no TAGI. Certainly the evidence isn’t bulletproof, condsidering the recent mispricings of NVIDIA and other semi stocks.
In my own essay, I elaborated on the same point using prices set by more-informed insiders: e.g., valuations and hiring by Anthropic/DeepMind/etc., which also seem to imply that TAGI isn’t coming soon. If they have a 10% chance of capturing 10% of the value for 10 years of doubling the world economy, that’s like $10T. And yet investment expenditures and hiring and valuations are nowhere near that scale. The fact that Google has more people working on ads than TAGI implies that they think TAGI is far off. (Or, more accurately, that marginal investments would not accelerate TAGI timelines or market share.)
Ted Sanders
Transformative AGI by 2043 is <1% likely
What can superintelligent ANI tell us about superintelligent AGI?
Excellent comment; thank you for engaging in such detail. I’ll respond piece by piece. I’ll also try to highlight the things you think we believe but don’t actually believe.
Section 1: Likelihood of AGI algorithms
“Can you say what exactly you are assigning a 60% probability to, and why it’s getting multiplied with ten other factors? Are you saying that there is a 40% chance that by 2043 AI algorithms couldn’t yield AGI no matter how much serial time and compute they had available? (It seems surprising to claim that even by 2023!) Presumably not that, but what exactly are you giving a 60% chance?
Yes, we assign a 40% chance that we don’t have AI algorithms by 2043 capable of learning to do nearly any human task with realistic amounts of time and compute. Some things we probably agree on:
Progress has been promising and investment is rising.
Obviously the development of AI that can do AI research more cheaply than humans could be a huge accelerant, with the magnitude depending on the value-to-cost ratio. Already GPT-4 is accelerating my own software productivity, and future models over the next twenty years will no doubt be leagues better (as well as more efficient).
Obviously slow progress in the past is not great evidence of slow progress in the future, as any exponential curve shows.
But as we discuss in the essay, 20 years is not a long time, much easier problems are taking longer, and there’s a long track record of AI scientists being overconfident about the pace of progress (counterbalanced, to be sure, by folks on the other side who are overconfident about things that would not be achieved and subsequently were). These factors give us pause, so while agree it’s likely we’ll have algorithms for AGI by 2043, we’re not certain of it, which is why we forecast 60%. We think forecasts higher than 60% are completely reasonable, but we personally struggle to justify anything near 100%.
Incidentally, I’m puzzled by your comment and others that suggest we might already have algorithms for AGI in 2023. Perhaps we’re making different implicit assumptions of realistic compute vs infinite compute, or something else. To me, it feels clear we don’t have the algorithms and data for AGI at present.
(ETA: after reading later sections more carefully I think you might be saying 60% chance that our software is about as good as nature’s, and maybe implicitly assuming there is a ~0% chance of being significantly better than that or building TAI without that? I’m not sure if that’s right though, if so it’s a huge point of methodological disagreement. I’ll return to this point later.)”
Lastly, no, we emphatically do not assume a ~0% chance that AGI will be smarter than nature’s brains. That feels like a ridiculous and overconfident thing to believe, and it pains me that we gave this impression. Already GPT-4 is smarter than me in ways, and as time goes on, the number of ways AI is smarter than me will undoubtedly grow.
Section 2: Likelihood of fast reinforcement training
10 years is sufficient for humans to learn most physical skills from scratch, and you are talking about 20 year timelines. So why is the serial time for learning even a candidate blocker?
Agree—if we had AGI today, this would not be a blocker. This becomes a greater and greater blocker the later AGI is developed. E.g., if AGI is developed in 2038, we’d have only 4 years to train it to do nearly every human task. So this factor is heavily entangled with the timeline on which AGI is developed.
(And obviously the development of AGI is not going to a clean line passed on a particular year, but the idea is the same even applied to AGI systems developed gradually and unevenly.)
We can easily run tens of thousands of copies of AI systems in parallel. Existing RL is massively parallelizable. Human evolution gives no evidence about the difficulty of parallelizing learning in this way. Based on observations of human learning it seems extremely likely to me that parallelization 10,000 fold can reduce serial time by at least 10x (which is all that is needed). Extrapolations of existing RL algorithms seem to suggest serial requirements more like 10,000 episodes, with almost all of the compute used to run a massive number of episodes in parallel, which would be 1 year even for a 1-hour task. It seems hard to construct physical tasks that don’t provide rich feedback after even shorter horizons than 1 hour (and therefore suitable for a gradient descent step given enough parallel samples) so this seems pretty conservative.
Agree on nearly everything here. I think the crux on which we differ is that we think interaction with the real world will be a substantial bottleneck (and therefore being able to run 10,000 parallel copies may not save us).
As I mentioned to Zach below:With AlphaZero in particular, fast reinforcement training is possible because (a) the game state can be efficiently modeled by a computer and (b) the reward can be efficiently computed by a computer.
In contrast, look at a task like self-driving. Despite massive investment, our self-driving AIs are learning more slowly than human teenagers. Part of the reason for this is that conditions (a) and (b) no longer hold. First, our simulations of reality are imperfect, and therefore fleets must be deployed to drive millions of miles. Second, calculating reward functions (i.e., “this action causes a collision”) is expensive and typically requires human supervision (e.g., test drivers, labelers), as the actual reward (e.g., a real-life collision) is even more expensive to acquire. This bottleneck of expensive feedback is partly why we can’t just throw more GPUs at the problem and learn self-driving overnight in the way we can with Go.
To recap, we can of course parallelize a million self-driving car AIs and have them drive billions of miles in simulation. But that only works to the extent that (a) our simulations reflect reality and (b) we have the compute resources to do so. And so real self-driving car companies are spending billions on fleets and human supervision in order to gather the necessary data. In general, if an AGI cannot easily and cheaply simulate reality, it will have to learn from real-world interactions. And to the extent that it needs to learn from interactions with the consequences of its earlier actions, that training will need to be sequential.
This isn’t super relevant to my mainline view, since in fact I think AI is much worse at learning quickly than humans and will likely be transformative way before reaching parity.
Agreed. Our expectation is that early AGIs will be expensive and uneven. If they end up being incredibly sample efficient, then this task will be much easier than we’ve forecasted.
In general, I’m pretty open to updating higher here. I don’t think there are any insurmountable barriers here; but a sense that this will both be hard to do (as self-driving illustrates) as well as unlikely to be done (as all sorts of tasks not currently automated illustrate). My coauthor is a bit more negative on this factor than me and may chime in with his own thoughts later.
I personally struggle to imagine how an AlphaZero-like algorithm would learn to become the world’s best swim instructor via massively parallelized reinforcement learning on children, but that may well be a failure of my imagination. Certainly one route is massively parallelized RL to become excellent at AI R&D, then massively parallelized RL to become excellent at many tasks, and then quickly transferring that understanding to teaching children to swim, without any children ever drowning.
Section 3: Operating costs
Here, I think you ascribe many beliefs to us which we do not hold, and I apologize for not being clearer. I’ll start by emphasizing what we do not believe.
Overall you seem to think it is possible that AI will be as effective as brains but unlikely to be much better.
We do not believe this.
AI is already vastly better than human brains at some tasks, and the number of tasks on which AI is superhuman will rise with time. We do expect that early AGIs will be expensive and uneven, as all earliest versions of a technology are. And then they will improve from there.
just to be clear you are saying that [GPT-3] should behave like a brain with ~1000 neurons.
We do not believe this.
Your biological analysis seems to hinge on the assertion that precise simulation of neurons is necessary to get similar levels of computational utility
We do not believe this.
build computers that operate at the Landauer limit (as you are apparently confident the brain does)
We do not believe this. We do not believe that brains operate at the Landauer limit, nor do we believe computers will operate at this limit by 2043.
Incidentally, I studied the Landauer limit deeply during my physics PhD and could write an essay on the many ways it’s misinterpreted, but will save that for another day. :)
It’s like asking about the probability that a sum of 5 normal distributions will be above the mean, and estimating it’s 1/2^5 because each of 5 normal distributions needs to be above its mean.
We do not believe this.
To multiply these probabilities together, one cannot multiply their unconditional expectations; rather, one must multiply their cascading conditional probabilities. You may disagree with our probabilities, but our framework specifically addresses this point. Our unconditional probabilities are far lower for some of these events, because we believe they will be rapidly accelerated conditional on progress in AGI.
Forecasting credentials
Your personal forecasting successes seem like a big part of the evidence for that, so it might be helpful to understand what kinds of predictions were involved and how methodologically analogous they are. Superficially it looks like the SciCast technology forecasting tournament is by far the most relevant; is there a pointer to the list of questions (other info like participants and list of predictions would also be awesome if available)? Or do you think one of the other items is more relevant?
Honestly, I wouldn’t put too much weight on my forecasting success. It’s mostly a mix of common sense, time invested, and luck. I do think it reflects a decent mental model of how the world works, which leads to decent calibration for what’s 3% likely vs 30% likely. The main reason I mention it in the paper is just to help folks realize that we’re not wackos predicting 1% because we “really feel” confident. In many other situations (e.g., election forecasting, sports betting, etc.) I often find myself on the humble and uncertain side of the fence, trying to warn people that the world is more complicated and unpredictable than their gut is telling them. Even here, I consider our component forecasts quite uncertain, ranging from 16% to 95%. It’s precisely our uncertainty about the future which leads to a small product of 0.4%. (From my point of view, you are staking out a much higher confidence position in asserting that AGI algorithms is very likely and that rapid self-improvement is very likely.)
As for SciCast, here’s at least one publication that resulted from the project: https://ieeexplore.ieee.org/abstract/document/7266786
Example questions (from memory) included:What will be the highest reported efficiency of a perovskite photovoltaic cell by date X
What will be the volume of deployed solar in the USA by date X
At the Brazil World Cup, how far will the paraplegic exoskeleton kick the ball for the opening kickoff
Will Amazon offer drone delivery by date X
Will physicists discover Y by date X
Most forecasts related to scientific discoveries and technological inventions and had timescales of months to years.
Conclusion
From your comment, I think the biggest crux between us is the rate of AI self-improvement. If the rate is lower, the world may look like what we’re envisioning. If the rate is higher, progress may take off in a way not well predicted by current trends, and the world may look more like what you’re envisioning. This causes our conditional probabilities to look too low and too independent, from your point of view. Do you think that’s a fair assessment?
Lastly, can I kindly ask what your cascading conditional probabilities would be in our framework? (Let’s hold the framework constant for this question, even if you prefer another.)
If you disagree with our admittedly imperfect guesses, we kindly ask that you supply your own preferred probabilities (or framework modifications). It’s easier to tear down than build up, and we’d love to hear how you think this analysis can be improved.
Here’s a hypothesis:
The base case / historical precedent for existential AI risk is:
- AGI has never been developed
- ASI has never been developed
- Existentially deadly technology has never been developed (I don’t count nuclear war or engineered pandemics, as they’ll likely leave survivors)
- Highly deadly technology (>1M deaths) has never been cheap and easily copied
- We’ve never had supply chains so fully automated end-to-end that they could become self-sufficient with enough intelligence
- We’ve never had technology so networked that it could all be taken over by a strong enough hacker
Therefore, if you’re in the skeptic camp, you don’t have to make as much of an argument about specific scenarios where many things happen. You can just wave your arms and say it’s never happened before because it’s really hard and rare, as supported by the historical record.
In contrast, if you’re in the concerned camp, you’re making more of a positive claim about an imminent departure from historical precedent, so the burden of proof is on you. You have to present some compelling model or principles for explaining why the future is going to be different from the past.
Therefore, I think the concerned camp relying on theoretical arguments with multiple steps of logic might be a structural side effect of them having to argue against the historical precedent, rather than any innate preference for that type of argument.
Maybe another related question: can you make a forecast about human-level self-driving (e.g. similar accident rates vs speed tradeoffs to a tourist driving in a random US city) and explain its correlation with your forecast about human-level AI overall?
Here are my forecasts of self-driving from 2018: https://www.tedsanders.com/on-self-driving-cars/
Five years later, I’m pretty happy with how my forecasts are looking. I predicted:100% that self-driving is solvable (looks correct)
90% that self-driving cars will not be available for sale by 2025 (looks correct)
90% that self-driving cars will debut as taxis years before sale to individuals (looks correct)
Rollout will be slow and done city-by-city, starting in the US (looks correct)
Today I regularly take Cruises around SF and it seems decently likely that self-driving taxis are on track to be widely deployed across the USA by 2030. Feels pretty probable, but still plenty of ways that it could be delayed or heterogenous (e.g., regulation, stalling progress, unit economics).
Plus, even wide robotaxi deployment doesn’t mean human taxi drivers are rendered obsolete. Seems very plausible we operate for many many years with a mixed fleet, where AI taxis with high fixed cost and low marginal cost serve baseload taxi demand while human taxis with lower fixed cost but higher marginal cost serve peak evening and commuter demand. In general it seems likely that as AI gets better we will see more complementary mixing and matching where AIs and humans are partnered to take advantage of their comparative advantages.
What counts as human-level is a bit fuzzier: is that human-level crash rates? human-level costs? human-level ability to deal with weird long tail situations?
On the specific question of random tourist vs self-driving vehicle in a new city, I predict that today the tourist is better (>99%) and that by 2030 I’d still give the edge to the tourist (75%), acknowledging that the closer it gets the more the details of the metric begin to matter.
Overall there’s some connection between self-driving progress and my AGI forecasts.
If most cars and trucks are not self-driving by 2043, then it seems likely that we haven’t achieved human-level human-cost AGI, and that would be a strong negative update.
If self-driving taxis still struggle with crowded parking lots and rare situations by 2030, that would a negative update.
If self-driving taxis are widely deployed across the US by 2030, but personal self-driving vehicles and self-driving taxis across Earth remain rare, that would be in line with my current worldview. Neutral update.
If self-driving improves and by 2030 becomes superhuman across a wide variety of driving metrics (i.e., not just crash rates, which can be maximized at the expense of route choice, speed, etc.), then that would be a positive update.
If self-driving improves by 2030 to the point that it can quickly learn to drive in new areas with new signage or new rules (i.e., we see a rapid expansion across all countries), that would be unexpected and a strong positive update on AGI.
ETA: let me try to make the point about self-driving cars more sharply. You seem to think there’s a <15% chance that by 2043 we can do what a human brain can do even using 1e17 flops (a 60% chance of “having the algorithms” and a 20% chance of being 3 OOMs better than 1e20 flops). Driving uses quite a lot of the functions that human brains are well-adapted to perform—perception, prediction, planning, control. If we call it one tenth of a brain, that’s 1e16 flops. Whereas I think existing self-driving cars use closer to 1e14 flops. So shouldn’t you be pretty much shocked if self-driving cars could be made to work using any amount of data with so little computing hardware? How can you be making meaningful updates from the fact that they don’t?
Excellent argument. Maybe we should update on that, though I find myself resistant. Part of my instinctive justification against updating is that current self-driving AIs, even if they achieve human-ish level crash rates, are still very sub-human in terms of:
Time to learn to drive
Ability to deal with complex situations like crowded parking lots
Ability to deal with rare situations like a firefighter waving you in a particular direction near a burning car
Ability to deal with custom signage
Ability to deal with unmapped routes
Ability to explain why they did what they did
Ability to reason about things related to driving
Ability to deal with heavy snow, fog, inclement weather
It feels quite plausible to me that these abilities could “cost” orders of magnitude of compute. I really don’t know.
Edit: Or, you could make the same argument about walking. E.g., maybe it engages 10% of our brain in terms of spatial modeling and prediction. But then there are all sorts of animals with much smaller brains that are still able to walk, right? So maybe navigation, at a crude level of ability, really needs much less than 10% of human intelligence. After all, we can sleepwalk, but we cannot sleep-reason. :)
Thank you for the clarification. To me, it is not 100.0% guaranteed that AGIs will be able to rapidly parallelize all learning and it is not 100.0% guaranteed that we’ll have enough chips by 2043. Therefore, I think it helps to assign probabilities to them. If you are 100.0% confident in their likelihood of occurrence, then you can of course remove those factors. We personally find it difficult to be so confident about the future.
I agree that the success of AlphaZero and GPT-4 are promising notes, but I don’t think they imply a 100.0% likelihood that AGI, whatever it looks like, will learn just as fast on every task.
With AlphaZero in particular, fast reinforcement training is possible because (a) the game state can be efficiently modeled by a computer and (b) the reward can be efficiently computed by a computer.
In contrast, look at a task like self-driving. Despite massive investment, our self-driving AIs are learning more slowly than human teenagers. Part of the reason for this is that conditions (a) and (b) no longer hold. First, our simulations of reality are imperfect, and therefore fleets must be deployed to drive millions of miles. Second, calculating reward functions (i.e., “this action causes a collision”) is expensive and typically requires human supervision (e.g., test drivers, labelers), as the actual reward (e.g., a real-life collision) is even more expensive to acquire. This bottleneck of expensive feedback is partly why we can’t just throw more GPUs at the problem and learn self-driving overnight in the way we can with Go.
Because computers are taking longer than humans to learn how to drive, despite billions invested and vast datasets, it feels plausible to us (i.e., more than 0% likely) that early AGIs will also take as long as humans to learn some tasks, particularly if those tasks cannot afford to spend billions on data acquisition (e.g., swim instructor).
In conclusion, I think it’s totally reasonable to be more optimistic than we are that fast reinforcement learning on nearly all tasks will be solved for AGI by 2043. But I’d caution against presuming a 100.0% probability, which, to me, is what removing this factor from the framework would imply.
Agreed. AGI can have great influence in the world just by dispatching humans.
But by the definition of transformative AGI that we use—i.e., that AGI is able to do nearly all human jobs—I don’t think it’s fair to equate “doing a job” with “hiring someone else to do the job.” To me, It would be a little silly to say “all human work has been automated” and only mean “the CEO is an AGI, but yeah everyone still has to go to work.”
Of course, if you don’t think robotics is necessary for transformative AGI, then you are welcome to remove the factor (or equivalently set it to 100%). In that case, our prediction would still be <1%.
I think approximately 1 in 10,000 chance of extinction for each new GPT would be acceptable given the benefits of AI. This is approximately my guess for GPT-5, so I think if we could release that model and then pause, I’d be okay with that.
To me, this is wild. 1⁄10,000 * 8 billion people = 800,000 current lives lost in expectation, not even counting future lives. If you think GPT-5 is worth 800k+ human lives, you must have high expectations. :)
One small clarification: the skeptical group was not all superforecasters. There were two domain experts as well. I was one of them.
I’m sympathetic to David’s point here. Even though the skeptic camp was selected for their skepticism, I think we still get some information from the fact that many hours of research and debate didn’t move their opinions. I think there are plausible alternative worlds where the skeptics come in with low probabilities (by construction), but update upward by a few points after deeper engagement reveals holes in their early thinking.
Sorry for seeming disingenuous. :(
(I think I will stop posting here for a while.)
Are you saying that e.g. a war between China and Taiwan makes it impossible to build AGI? Or that serial time requirements make AGI impossible? Or that scaling chips means AGI is impossible?
C’mon Paul—please extend some principle of charity here. :)
You have repeatedly ascribed silly, impossible beliefs to us and I don’t know why (to be fair, in this particular case you’re just asking, not ascribing). Genuinely, man, I feel bad that our writing has either (a) given the impression that we believe such things or (b) given the impression that we’re the type of people who’d believe such things.
Like, are these sincere questions? Is your mental model of us that there’s a genuine uncertainty over whether we’ll say “Yes, a war precludes AGI” vs “No, a war does preclude AGI.”To make it clear: No, of course a war between China and Taiwan does not make it impossible to build AGI by 2043. As our essay explicitly says.
Some things can go wrong and you can still get AGI by 2043. If you want to argue you can’t build AGI if something goes wrong, that’s a whole different story. So multiplying probabilities (even conditional probabilities) for none of these things happening doesn’t seem right.
To make it clear: our forecasts are not the odds of wars, pandemics, and depressions not occurring. They are the odds of wars, pandemics, and depressions not delaying AGI beyond 2043. Most wars, most pandemics, and most depressions will not delay AGI beyond 2043, we think. Our methodology is to forecast only the most severe events, and then assume a good fraction won’t delay AGI. As our essay explicitly says.
We probably forecast higher odds of delay than you, because our low likelihoods of TAGI mean that TAGI, if developed, is likeliest to be developed nearer to the end of the period, without many years of slack. If TAGI is easy, and can be developed early or with plenty of slack, then it becomes much harder for these types of events to derail TAGI.
Agreed. Factors like AI progress, inference costs, and manufacturing scale needed are massively correlated. We discuss this in the paper. Our unconditional independent forecast of semiconductor production would be much, much lower than our conditional forecast of 46%, for example.
“I think this framework is bad and the probabilities are far too low.”
Setting aside assessments of the probabilities (which are addressed in the paper), what do you think is bad about the framework? How would you suggest we improve it?
Confidence intervals over probabilities don’t make much sense to me. The probability itself is already the confidence interval over the binary domain [event happens, event doesn’t happen].
I guess to me the idea of confidence intervals over probabilities implies two different kinds of probabilities. E.g., a reducible flavor and an irreducible flavor. I don’t see what a two-tiered system of probability adds, exactly.
I think you make a good argument and I’m open to changing my mind. I’m certainly no expert on visual processing in the human brain. Let me flesh out some of my thoughts here.
On whether this framework would have yielded bad forecasts for self-driving:
When we guess that brains use 1e20-1e21 FLOPS, and therefore that early AGIs might need 1e16-1e25, we’re not making a claim about AGIs in general, or the most efficient AGI possible, but AGIs by 2043. We expect early AGIs to be horribly inefficient by later standards, and AGIs to get rapidly more efficient over time. AGI in 2035 will be less efficient than AGI in 2042 which will be less efficient than AGI in 2080.
With that clarification, let’s try to apply our logic to self-driving to see whether it bears weight.
Supposing that self-driving needs 1% of human brainpower, or 1e18-1e19 FLOPS, and then similarly widen our uncertainty to 1e14-1e23 FLOPS, it might say yes, we’d be surprised but not stunned at 1e14 FLOPS being enough to drive (10% → 100%). But, and I know my reasoning is motivated here, that actually seems kind of reasonable? Like, for the first decade and change of trying, 1e14 FLOPS actually was not enough to drive. Even now, it’s beginning to be enough to drive, but still is wildly less sample efficient than human drivers and wildly worse at generalizing than human drivers. So it feels like if in 2010 we predicted self-driving would take 1e14-1e23 FLOPS, and then a time traveler from the future told us that actually it was 1e14 FLOPS, but it would take 13 years to get there, and actually would still be subhuman, then honestly that doesn’t feel too shocking. It was the low end of the range, took many years, and still didn’t quite match human performance.
No doubt with more time and more training 1e14 FLOPS will become more and more capable. Just as we have little doubt that with more time AGIs will require fewer and fewer FLOPS to achieve human performance.
So as I reflect on this framework applied to the test case of self-driving, I come away thinking (a) it actually would have made reasonable predictions and (b) in hindsight I think we should have spent some pages modeling rates of AGI progress, as (obviously) AGI needs are not a fixed target but will decline rapidly over time.
In sum, it’s not obvious to me that this logic would have generated bad predictions for self-driving, and so I’m still unconvinced that we’ve made a big blunder here.
On whether driving takes 1% of the human brain:
I’m going to go way outside my lane here, no pun intended, and I welcome comments from those more informed.
From my very brief Googling, sources say that something like half the brain is involved in vision, though that includes vision+motor, vision+attention, vision+spatial, etc. Going out on a limb, it seems if one knew that fact alone, they might feel reasonable saying that if we can solve vision, then AGI will only need twice as much compute, as vision is half of our brainpower. But it feels like we’ve made massive progress in vision (AlexNet, self-driving, etc.) without being on the cusp of AGI. Somehow even though the brain does lots of vision, these visual identification benchmarks feel far short of AGI.
My feeling is that what makes the human brain superior to today’s AI is its ability to generalize from few examples. To illustrate, suppose you take an AI and a human and you train them over and over to drive a route from A to B. With 1e14 FLOPS and enough training, the AI may be able to eventually outperform the human. But now test the AI and human on route C to D. The human will be wildly better, as their pre-trained world model will have prevented overfitting to the features of route AB. The je ne sais quoi of our intelligence seems to be that we are better able to build models of the world that allow us to more easily reason and generalize to new situations.
To me, getting 1e14 FLOPS to drive a route AB and nothing else is a monumentally different challenge than getting 1e14 FLOPS to drive a route AB, plus any other hypothetical route you throw at it. The first is narrow intelligence. The second is general intelligence. So if we discover that an AI can self-drive route AB with only 1e14 FLOPS, should it be a giant update for how much compute AGI will need? I think it depends: if it’s a narrow brittle overfit AI, then no. If it’s a general (in the sense of general to any road) robust AI, then yes, it should be a big update.
So where does self-driving lay? Well, obviously self-driving cars with 1e14 FLOPS are able to generalize well to all sorts of pre-mapped and pre-simulated routes in a city. But at the same time, they appear to generalize pretty poorly overall—I wouldn’t expect a Waymo vehicle to generalize well to a new city, let alone a new country.
Summarizing, I think the impressiveness of only needing 1e14 FLOPS to drive is really dependent on how well that driving generalizes. If it can generalize as well as human drivers, yes, that’s a big update that AGI may need less 1e20-1e21 FLOPS. But today’s self-driving doesn’t cross that bar for me. It’s quite brittle, and so I am less inclined to update.
Put another way, perhaps brittle self-driving takes 0.1% of human brainpower and robust self-driving takes 10% of human brainpower. Or something like that, who knows. It’s really not clear to me that self-driving in general needs 1% of the human brain, if that self-driving generalizes very poorly to new situations.
Lastly, there is typically going to be a tradeoff between training compute and inference compute. If 1e14 FLOPS is enough to self-drive, but only with millions of miles driven and billions of miles simulated, that should far be less impressive than 1e14 FLOPS being enough to drive after only 100 miles of training. For all I know, it may be possible to get a good driver that only uses 1e12 FLOPS if we train it on even more compute and data; e.g., a billion miles of driving and a trillion miles of simulation. But even if that loosely extrapolates to AGI, it’s only useful if we can afford the “brain equivalent” of a billion miles of training. If that’s too costly, then the existence proof of a 1e12 FLOPS driver may be moot in terms of analogizing AGI inference costs. It’s a pretty interesting question and we certainly could have added more modeling and discussion to our already lengthy essay.
it seems difficult to justify a less than 10% probability that there will be really strong pressures to develop AGI
I agree there will be really strong pressures to develop AGI. Already, many research groups are investing billions today (e.g., Google DeepMind, OpenAI, Anthropic). I’d assign 100% probability to this rather than <10%. I guess it depends on how many billions of dollars of investment qualify as “strong pressures.”
it seems similarly difficult to justify a less than 10% success probability given such an effort and what we now know
Well, our essay is an attempt to forecast the likelihood of success, given what we know.
If you disagree with our estimates, would you care to supply your own? What conditional probabilities do you believe that result in a 10%+ chance of TAGI by 2043?
As I asked in the post:If you disagree with our admittedly imperfect guesses, we kindly ask that you supply your own preferred probabilities (or framework modifications). It’s easier to tear down than build up, and we’d love to hear how you think this analysis can be improved.
Thanks! We agree that a common mistake by forecasters is to equate low probability of derailment with negligible probability of derailment. The future is hard to predict, and we think it’s worth taking tail risks seriously.
it seems weird to assume that TAI is infeasible without [TSMC]?
We do not assume TAI is infeasible without TSMC. That would be a terrible reasoning error, and I apologize for giving you that impression.
What we assume is that losing TSMC would likely delay TAI by a handful of years, as it would take:
Time for NVIDIA to bid on capacity from Samsung
Time for Samsung to figure out to what extent it could get out of prior contracted commitments
Time for NVIDIA and Samsung engineers to retune GPU designs for Samsung’s fab design rules
Time to manufacture the masks and put the GPUs into production
Time to iron out early manufacturing and yield issues
Time to build new fabs to absorb the tsunami of demand from TSMC customers (like Apple) and scale up to NVIDIA’s original TSMC volumes
And on top of this, there would massive geopolitical uncertainty that would slow things like investments into new fabs, as companies wonder whether the conflict will escalate or evaporate (both of which massively change the investment case).
What this might look like in reality will also depend on how close we are to transformative AGI.Two example scenarios:
Today, for example, NVIDIA is probably not going to outbid Apple (Apple makes ~$10B in PROFIT per month, which would evaporate if they were starved of chips).Or, imagine it’s 2035 and NVIDIA is worth $10T and the semiconductor industry has been building fabs left and right to fuel the impending AGI boom. In such a world, where NVIDIA is the world’s biggest chip designer, it may already dominate manufacturing on both Samsung and TSMC, meaning that if TSMC goes down, it cannot shift production to Samsung—because it already has production on Samsung.
In any case, we fully agree TAI is feasible without TSMC. But we think losing TSMC delays things by a few years, and if TAI is likely to come in the final 10 or 5 years of this period, then a few years might have a 50⁄50 shot of delaying things beyond 2043.
Great comment. We didn’t explicitly allocate probability to those scenarios, and if you do, you end up with much higher numbers. Very reasonable to do so.
Congrats to the winners, readers, and writers!
Two big surprises for me:
(1) It seems like 5⁄6 of the essays are about AI risk, and not TAGI by 2043. I thought there were going to be 3 winners on each topic, but perhaps that was never stated in the rules. Rereading, it just says there would be two 1st places, two 2nd places, and two 3rd places. Seems the judges were more interested in (or persuaded by) arguments on AI safety & alignment, rather than TAGI within 20 years. A bit disappointing for everyone who wrote on the second topic. If the judges were more interested in safety & alignment forecasting, that would have been nice to know ahead of time.
(2) I’m also surprised that the Dissolving AI Risk paper was chosen. (No disrespect intended; it was clearly a thoughtful piece.)
To me, it makes perfect sense to dissolve the Fermi paradox by pointing out that the expected # of alien civilizations is a very different quantity than the probability of 0 alien civilizations. It’s logically possible to have both a high expectation and a high probability of 0.
But it makes almost no sense to me to dissolve probabilities by factoring them into probabilities of probabilities, and then take the geometric mean of that distribution. Taking the geometric mean of subprobabilities feels like a sleight of hand to end up with a lower number than what you started with, with zero new information added in the process. I feel like I must have missed the main point, so I’ll reread the paper.
Edit: After re-reading, it makes more sense to me. The paper takes the geometric means of odds ratios in order to aggregate survey entries. It doesn’t take the geometric mean of probabilities, and it doesn’t slice up probabilities arbitrarily (as they are the distribution over surveyed forecasters).
Edit2: As Jaime says below, the greater error is assuming independence of each stage. The original discussion got quite nerd-sniped by the geometric averaging, which is a bit of a shame, as there’s a lot more to the piece to discuss and debate.