That’s fair, this was some inference that is probably not justified.
To spell it out: you think brains are as effective as 1e20-1e21 flops. Iclaimed that humans use more than 1% of their brain when driving (e.g. our visual system is large and this seems like a typical task that engages the whole utility of the visual system during the high-stakes situations that dominate performance), but you didn’t say this. I concluded (but you certainly didn’t say) that a human-level algorithm for driving would not have much chance of succeeding using 1e14 flops.
I think you make a good argument and I’m open to changing my mind. I’m certainly no expert on visual processing in the human brain. Let me flesh out some of my thoughts here.
On whether this framework would have yielded bad forecasts for self-driving:
When we guess that brains use 1e20-1e21 FLOPS, and therefore that early AGIs might need 1e16-1e25, we’re not making a claim about AGIs in general, or the most efficient AGI possible, but AGIs by 2043. We expect early AGIs to be horribly inefficient by later standards, and AGIs to get rapidly more efficient over time. AGI in 2035 will be less efficient than AGI in 2042 which will be less efficient than AGI in 2080.
With that clarification, let’s try to apply our logic to self-driving to see whether it bears weight.
Supposing that self-driving needs 1% of human brainpower, or 1e18-1e19 FLOPS, and then similarly widen our uncertainty to 1e14-1e23 FLOPS, it might say yes, we’d be surprised but not stunned at 1e14 FLOPS being enough to drive (10% → 100%). But, and I know my reasoning is motivated here, that actually seems kind of reasonable? Like, for the first decade and change of trying, 1e14 FLOPS actually was not enough to drive. Even now, it’s beginning to be enough to drive, but still is wildly less sample efficient than human drivers and wildly worse at generalizing than human drivers. So it feels like if in 2010 we predicted self-driving would take 1e14-1e23 FLOPS, and then a time traveler from the future told us that actually it was 1e14 FLOPS, but it would take 13 years to get there, and actually would still be subhuman, then honestly that doesn’t feel too shocking. It was the low end of the range, took many years, and still didn’t quite match human performance.
No doubt with more time and more training 1e14 FLOPS will become more and more capable. Just as we have little doubt that with more time AGIs will require fewer and fewer FLOPS to achieve human performance.
So as I reflect on this framework applied to the test case of self-driving, I come away thinking (a) it actually would have made reasonable predictions and (b) in hindsight I think we should have spent some pages modeling rates of AGI progress, as (obviously) AGI needs are not a fixed target but will decline rapidly over time.
In sum, it’s not obvious to me that this logic would have generated bad predictions for self-driving, and so I’m still unconvinced that we’ve made a big blunder here.
On whether driving takes 1% of the human brain:
I’m going to go way outside my lane here, no pun intended, and I welcome comments from those more informed.
From my very brief Googling, sources say that something like half the brain is involved in vision, though that includes vision+motor, vision+attention, vision+spatial, etc. Going out on a limb, it seems if one knew that fact alone, they might feel reasonable saying that if we can solve vision, then AGI will only need twice as much compute, as vision is half of our brainpower. But it feels like we’ve made massive progress in vision (AlexNet, self-driving, etc.) without being on the cusp of AGI. Somehow even though the brain does lots of vision, these visual identification benchmarks feel far short of AGI.
My feeling is that what makes the human brain superior to today’s AI is its ability to generalize from few examples. To illustrate, suppose you take an AI and a human and you train them over and over to drive a route from A to B. With 1e14 FLOPS and enough training, the AI may be able to eventually outperform the human. But now test the AI and human on route C to D. The human will be wildly better, as their pre-trained world model will have prevented overfitting to the features of route AB. The je ne sais quoi of our intelligence seems to be that we are better able to build models of the world that allow us to more easily reason and generalize to new situations.
To me, getting 1e14 FLOPS to drive a route AB and nothing else is a monumentally different challenge than getting 1e14 FLOPS to drive a route AB, plus any other hypothetical route you throw at it. The first is narrow intelligence. The second is general intelligence. So if we discover that an AI can self-drive route AB with only 1e14 FLOPS, should it be a giant update for how much compute AGI will need? I think it depends: if it’s a narrow brittle overfit AI, then no. If it’s a general (in the sense of general to any road) robust AI, then yes, it should be a big update.
So where does self-driving lay? Well, obviously self-driving cars with 1e14 FLOPS are able to generalize well to all sorts of pre-mapped and pre-simulated routes in a city. But at the same time, they appear to generalize pretty poorly overall—I wouldn’t expect a Waymo vehicle to generalize well to a new city, let alone a new country.
Summarizing, I think the impressiveness of only needing 1e14 FLOPS to drive is really dependent on how well that driving generalizes. If it can generalize as well as human drivers, yes, that’s a big update that AGI may need less 1e20-1e21 FLOPS. But today’s self-driving doesn’t cross that bar for me. It’s quite brittle, and so I am less inclined to update.
Put another way, perhaps brittle self-driving takes 0.1% of human brainpower and robust self-driving takes 10% of human brainpower. Or something like that, who knows. It’s really not clear to me that self-driving in general needs 1% of the human brain, if that self-driving generalizes very poorly to new situations.
Lastly, there is typically going to be a tradeoff between training compute and inference compute. If 1e14 FLOPS is enough to self-drive, but only with millions of miles driven and billions of miles simulated, that should far be less impressive than 1e14 FLOPS being enough to drive after only 100 miles of training. For all I know, it may be possible to get a good driver that only uses 1e12 FLOPS if we train it on even more compute and data; e.g., a billion miles of driving and a trillion miles of simulation. But even if that loosely extrapolates to AGI, it’s only useful if we can afford the “brain equivalent” of a billion miles of training. If that’s too costly, then the existence proof of a 1e12 FLOPS driver may be moot in terms of analogizing AGI inference costs. It’s a pretty interesting question and we certainly could have added more modeling and discussion to our already lengthy essay.
This is not a claim we’ve made.
That’s fair, this was some inference that is probably not justified.
To spell it out: you think brains are as effective as 1e20-1e21 flops. I claimed that humans use more than 1% of their brain when driving (e.g. our visual system is large and this seems like a typical task that engages the whole utility of the visual system during the high-stakes situations that dominate performance), but you didn’t say this. I concluded (but you certainly didn’t say) that a human-level algorithm for driving would not have much chance of succeeding using 1e14 flops.
I think you make a good argument and I’m open to changing my mind. I’m certainly no expert on visual processing in the human brain. Let me flesh out some of my thoughts here.
On whether this framework would have yielded bad forecasts for self-driving:
When we guess that brains use 1e20-1e21 FLOPS, and therefore that early AGIs might need 1e16-1e25, we’re not making a claim about AGIs in general, or the most efficient AGI possible, but AGIs by 2043. We expect early AGIs to be horribly inefficient by later standards, and AGIs to get rapidly more efficient over time. AGI in 2035 will be less efficient than AGI in 2042 which will be less efficient than AGI in 2080.
With that clarification, let’s try to apply our logic to self-driving to see whether it bears weight.
Supposing that self-driving needs 1% of human brainpower, or 1e18-1e19 FLOPS, and then similarly widen our uncertainty to 1e14-1e23 FLOPS, it might say yes, we’d be surprised but not stunned at 1e14 FLOPS being enough to drive (10% → 100%). But, and I know my reasoning is motivated here, that actually seems kind of reasonable? Like, for the first decade and change of trying, 1e14 FLOPS actually was not enough to drive. Even now, it’s beginning to be enough to drive, but still is wildly less sample efficient than human drivers and wildly worse at generalizing than human drivers. So it feels like if in 2010 we predicted self-driving would take 1e14-1e23 FLOPS, and then a time traveler from the future told us that actually it was 1e14 FLOPS, but it would take 13 years to get there, and actually would still be subhuman, then honestly that doesn’t feel too shocking. It was the low end of the range, took many years, and still didn’t quite match human performance.
No doubt with more time and more training 1e14 FLOPS will become more and more capable. Just as we have little doubt that with more time AGIs will require fewer and fewer FLOPS to achieve human performance.
So as I reflect on this framework applied to the test case of self-driving, I come away thinking (a) it actually would have made reasonable predictions and (b) in hindsight I think we should have spent some pages modeling rates of AGI progress, as (obviously) AGI needs are not a fixed target but will decline rapidly over time.
In sum, it’s not obvious to me that this logic would have generated bad predictions for self-driving, and so I’m still unconvinced that we’ve made a big blunder here.
On whether driving takes 1% of the human brain:
I’m going to go way outside my lane here, no pun intended, and I welcome comments from those more informed.
From my very brief Googling, sources say that something like half the brain is involved in vision, though that includes vision+motor, vision+attention, vision+spatial, etc. Going out on a limb, it seems if one knew that fact alone, they might feel reasonable saying that if we can solve vision, then AGI will only need twice as much compute, as vision is half of our brainpower. But it feels like we’ve made massive progress in vision (AlexNet, self-driving, etc.) without being on the cusp of AGI. Somehow even though the brain does lots of vision, these visual identification benchmarks feel far short of AGI.
My feeling is that what makes the human brain superior to today’s AI is its ability to generalize from few examples. To illustrate, suppose you take an AI and a human and you train them over and over to drive a route from A to B. With 1e14 FLOPS and enough training, the AI may be able to eventually outperform the human. But now test the AI and human on route C to D. The human will be wildly better, as their pre-trained world model will have prevented overfitting to the features of route AB. The je ne sais quoi of our intelligence seems to be that we are better able to build models of the world that allow us to more easily reason and generalize to new situations.
To me, getting 1e14 FLOPS to drive a route AB and nothing else is a monumentally different challenge than getting 1e14 FLOPS to drive a route AB, plus any other hypothetical route you throw at it. The first is narrow intelligence. The second is general intelligence. So if we discover that an AI can self-drive route AB with only 1e14 FLOPS, should it be a giant update for how much compute AGI will need? I think it depends: if it’s a narrow brittle overfit AI, then no. If it’s a general (in the sense of general to any road) robust AI, then yes, it should be a big update.
So where does self-driving lay? Well, obviously self-driving cars with 1e14 FLOPS are able to generalize well to all sorts of pre-mapped and pre-simulated routes in a city. But at the same time, they appear to generalize pretty poorly overall—I wouldn’t expect a Waymo vehicle to generalize well to a new city, let alone a new country.
Summarizing, I think the impressiveness of only needing 1e14 FLOPS to drive is really dependent on how well that driving generalizes. If it can generalize as well as human drivers, yes, that’s a big update that AGI may need less 1e20-1e21 FLOPS. But today’s self-driving doesn’t cross that bar for me. It’s quite brittle, and so I am less inclined to update.
Put another way, perhaps brittle self-driving takes 0.1% of human brainpower and robust self-driving takes 10% of human brainpower. Or something like that, who knows. It’s really not clear to me that self-driving in general needs 1% of the human brain, if that self-driving generalizes very poorly to new situations.
Lastly, there is typically going to be a tradeoff between training compute and inference compute. If 1e14 FLOPS is enough to self-drive, but only with millions of miles driven and billions of miles simulated, that should far be less impressive than 1e14 FLOPS being enough to drive after only 100 miles of training. For all I know, it may be possible to get a good driver that only uses 1e12 FLOPS if we train it on even more compute and data; e.g., a billion miles of driving and a trillion miles of simulation. But even if that loosely extrapolates to AGI, it’s only useful if we can afford the “brain equivalent” of a billion miles of training. If that’s too costly, then the existence proof of a 1e12 FLOPS driver may be moot in terms of analogizing AGI inference costs. It’s a pretty interesting question and we certainly could have added more modeling and discussion to our already lengthy essay.