Hi Ajeya! I”m a huge fan of your timelines report, it’s by far the best thing out there on the topic as far as I know. Whenever people ask me to explain my timelines, I say “It’s like Ajeya’s, except...”
My question is, how important do you think it is for someone like me to do timelines research, compared to other kinds of research (e.g. takeoff speeds, alignment, acausal trade...)
I sometimes think that even if I managed to convince everyone to shift from median 2050 to median 2032 (an obviously unlikely scenario!), it still wouldn’t matter much because people’s decisions about what to work on are mostly driven by considerations of tractability, neglectedness, personal fit, importance, etc. and even that timelines difference would be a relatively minor consideration. On the other hand, intuitively it does feel like the difference between 2050 and 2032 is a big deal and that people who believe one when the other is true will probably make big strategic mistakes.
Bonus question: Murphyjitsu: Conditional on TAI being built in 2025, what happened? (i.e. how was it built, what parts of your model were wrong, what do the next 5 years look like, what do the 5 years after 2025 look like?)
Thanks so much, that’s great to hear! I’ll answer your first question in this comment and leave a separate reply for your Murphyjitsu question.
First of all, I definitely agree that the difference between 2050 and 2032 is a big deal and worth getting to the bottom of; it would make a difference to Open Phil’s prioritization (and internally we’re trying to do projects that could convince us of timelines significantly shorter than in my report). You may be right that it could have a counterintuitively small impact on many individual people’s career choices, for the reasons you say, but I think many others (especially early career people) would and should change their actions substantially.
I think there are roughly three types of reasons why Bob might disagree with Alice about a bottom line conclusion like TAI timelines, which correspond to three types of research or discourse contributions Bob could make in this space:
1. Disagreements can come from Bob knowing more facts than Alice about a key parameter, which can allow Bob to make “straightforward corrections” to Alice’s proposed value for that parameter. E.g., “You didn’t think much about hardware, but I did a solid research project into hardware and I think experts would agree that because of optical computing progress will be faster than you assumed; changing to the better values makes timelines shorter.” If Bob does a good enough job with this empirical investigation, Alice will often just say “Great, thanks!” and adopt Bob’s number. 2. Disagreements can come from Bob modeling out a part of the world in more mechanistic detail that Alice fudged or simplified, which can allow Bob to propose a better structure than Alice’s model. E.g., “You agree that earlier AI systems can generate revenue which can be reinvested into AI research but you didn’t explicitly model that and just made a guess about spending trajectory; I’ll show that accounting for this properly would make timelines shorter.” Alice may feel some hesitance adopting Bob’s model wholesale here, because Alice’s model may fudge/elide one thing in an overly-conservative direction which she feels is counterbalanced by fudging/eliding another thing in an overly-aggressive direction, but it will often be tractable to argue that the new model is better and Alice will often be happy to adopt it (perhaps changing some other fudged parameters a little to preserve intuitions that seemed important to her). 3. Finally, disagreements can come from differences in intuition about the subjective weight different considerations should get when coming up with values for the more debatable parameters (such as the different biological anchor hypotheses). It’s more difficult for Bob to make a contribution toward changing Alice’s bottom line here, because a lot of the action is in hard-to-access mental alchemy going on in Alice and Bob’s minds when they make difficult judgment calls. Bob can try to reframe things, offer intuition pumps, trace disagreements about one topic back to a deeper disagreement about another topic and argue about that, and so on, but he should expect it to be slow going and expect Alice to be pretty hard to move.
In my experience, most large and persistent disagreements between people about big-picture questions like TAI timelines or the magnitude of risk from AI are mostly the third kind of disagreement, and these disagreements can be entangled with dozens of other differences in background assumptions / outlook / worldview. My sense is that your most major disagreements with me fall into the third category: you think that I’m overweighting the hypothesis that we’d need to do meta-learning in which the “inner loop” takes a long subjective time; you may also think that I’m underweighting the possibility of sudden takeoff or overweighting the efficiency of markets in a certain way, which leads me to lend too much credence to considerations like “Well if the low end of the compute range is actually right, we should probably be seeing more economic impact from the slightly-smaller AI systems right now.” If you were to change my mind on this, it might not even be from doing “timelines research”: maybe you do “takeoff speeds research” that convinces me to take sudden takeoff more seriously, which in turn causes me to take shorter timelines (which would imply more sudden takeoff) more seriously.
I’d say tackling category 3 disagreements is high risk and effort but has the possibility of high reward, and tackling category 1 disagreements is lower risk and effort with more moderate reward. My subjective impression is that EAs tend to under-invest in tackling categories 1 and 2 because they perceive category 3 as where the real action is—in some sense they’re right about that, but they may underestimate how hard it’ll be to change people’s minds there. For example, changing someone’s minds about a category 3 disagreement often greatly benefits from having a lot of face time with them, which isn’t very scalable, and arguments may be more particular to individuals: what finally convinces Alice may not be moving to Charlie.
I think one potential way to get at a category 3 disagreement about a long-term forecast is by proposing bets about nearer-term forecasts, although I think this is often a lot harder than it sounds, because people are sensitive to the possibility of “losing on a technicality”: they were right about the big picture but wrong about how that big picture actually translates to a near-term prediction. Even making short-term bets often benefits from having a lot of face time to hash out the terms.
It occurred to me that another way to try to move someone on complicated category 3 disagreements might be to put together a well-constructed survey of a population that the person is inclined to defer to. This approach is definitely still tricky: you’d have to convince the person that the relevant population was provided with the strongest arguments for that person’s view in addition to your counterarguments, and that the individuals surveyed were thinking about it reasonably hard. But if done well, it could be pretty powerful.
Thanks, this was a surprisingly helpful answer, and I had high expectations!
This is updating me somewhat towards doing more blog posts of the sort that I’ve been doing. As it happens, I have a draft of one that is very much Category 3, let me know if you are interested in giving comments!
Your sense of why we disagree is pretty accurate, I think. The only thing I’d add is that I do think we should update downwards on low-end compute scenarios because of market efficiency considerations, just not as strongly as you perhaps, and moreover I also think that we should update upwards for various reasons (the surprising recent sucesses of deep learning, the fact that big corporations are investing heavily-by-historical-standards in AI, the fact that various experts think they are close to achieving AGI) and the upwards update mostly cancels out the downwards update IMO.
If I condition on your report being wrong in an important way (either in its numerical predictions, or via conceptual flaws) and think about how we might figure that out today, it seems like two salient possibilities are inside-view arguments and outside-view arguments.
The former are things like “this explicit assumption in your model is wrong”. E.g. I count my concern about the infeasibility of building AGI using algorithms available in 2020 as an inside-view argument.
The latter are arguments that, based on the general difficulty of forecasting the future, there’s probably some upcoming paradigm shift or crucial consideration which will have a big effect on your conclusions (even if nobody currently knows what it will be).
Are you more worried about the inside-view arguments of current ML researchers, or outside-view arguments?
I generally spend most of my energy looking for inside-view considerations that might be wrong, because they are more likely to suggest a particular directional update (although I’m not focused only on inside view arguments specifically from ML researchers, and place a lot of weight on inside view arguments from generalists too).
It’s often hard to incorporate the most outside-view considerations into bottom line estimates, because it’s not clear what their implication should be. For example, the outside-view argument “it’s difficult to forecast the future and you should be very uncertain” may imply spreading probability out more widely, but that would involve assigning higher probabilities to TAI very soon, which is in tension with another outside view argument along the lines of “Predicting something extraordinary will happen very soon has a bad track record.”
Shouldn’t a combination of those two heuristics lead to spreading out the probability but with somewhat more probability mass on the longer-term rather than the shorter term?
That’s fair, and I do try to think about this sort of thing when choosing e.g. how wide to make my probability distributions and where to center them; I often make them wider than feels reasonable to me. I didn’t mean to imply that I explicitly avoid incorporating such outside view considerations, just that returns to further thinking about them are often lower by their nature (since they’re often about unkown-unkowns).
True. My main concern here is the lamppost issue (looking under the lamppost because that’s where the light is). If the unknown unknowns affect the probability distribution, then personally I’d prefer to incorporate that or at least explicitly acknowledge it. Not a critique—I think you do acknowledge it—but just a comment.
I sometimes think that even if I managed to convince everyone to shift from median 2050 to median 2032 (an obviously unlikely scenario!)
Just in case any readers would misinterpret that statement: I’m pretty sure that what Daniel is saying is unlikely is not that TAI will be built in 2032, but rather that he would be able to convince to shift their median to that date. I think Daniel’s median for when TAI will be built is indeed somewhere around 2032 or perhaps sooner. (I think that based on conversations around October and this post. Daniel can of course correct me if I’m wrong!)
(Maybe no readers would’ve misinterpreted Daniel anyway and this is just a weird comment...)
Yep, my current median is something like 2032. It fluctuates depending on how I estimate it, sometimes I adjust it up or down a bit based on how I’m feeling in the moment and recent updates, etc.
Murphyjitsu: Conditional on TAI being built in 2025, what happened? (i.e. how was it built, what parts of your model were wrong, what do the next 5 years look like, what do the 5 years after 2025 look like?)
On the object level, I think it would probably turn out to be the case that a) I was wrong about horizon length and something more like ~1 token was sufficient, b) I was wrong about model size and something more like ~10T parameter was sufficient. On a deeper level, it would mean I was wrong about the plausibility of ultra-sudden takeoff and shouldn’t have placed as much weight as I did on the observation that AI isn’t generating a lot of annual revenue right now and its value-added seems to have been increasing relatively smoothly so far.
I would guess that the model looks like a scaled-up predictive model (natural language and/or code), perhaps combined with simple planning or search. Maybe a coding model rapidly trains more-powerful successors in a pretty classically Bostromian / Yudkowskian way.
Since this is a pretty Bostromian scenario, and I haven’t thought deeply about those scenarios, I would default to guessing that the world after looks fairly Bostromian, with risks involving the AI forcibly taking control of most of the world’s resources, and the positive scenario involving cooperatively using the AI to prevent other x-risks (including risks from other AI projects).
Re why AI isn’t generating much revenue—have you considered the productivity paradox? It’s historically normal that productivity slows down before steeply increasing when a new general purpose technologies arrives.
the observation that AI isn’t generating a lot of annual revenue right now
Not sure how relevant, but I saw that Gwern seems to think this comes from a bottleneck of people who can apply AI, not from current AI being insufficient:
But how absurd—to a first approximation, ML/DL has been applied to 𝘯𝘰𝘵𝘩𝘪𝘯𝘨 thus far. We’re 𝘵𝘩𝘢𝘵 bottlenecked on coders!
And the lack of coders may rapidly disappear soon-ish, right? At least in Germany studying ML seems very popular since a couple of years now.
In some sense I agree with gwern that the reason ML hasn’t generated a lot of value is because people haven’t put in the work (both coding and otherwise) needed to roll it out to different domains, but (I think unlike gwern) the main inference I make from that that it wouldn’t have been hugely profitable to put in the work to create ML-based applications (or else more people would have been diverted from other coding tasks to the task of rolling out ML applications).
I mostly agree with that with the further caveat that I tend to think the low value reflects not that ML is useless but the inertia of a local optima where the gains from automation are low because so little else is automated and vice-versa (“automation as colonization wave”). This is part of why, I think, we see the broader macroeconomic trends like big tech productivity pulling away: many organizations are just too incompetent to meaningful restructure themselves or their activities to take full advantage. Software is surprisingly hard from a social and organizational point of view, and ML more so. A recent example is coronavirus/remote-work: it turns out that remote is in fact totally doable for all sorts of things people swore it couldn’t work for—at least when you have a deadly global pandemic solving the coordination problem...
As for my specific tweet, I wasn’t talking about making $$$ but just doing cool projects and research. People should be a little more imaginative about applications. Lots of people angst about how they can possibly compete with OA or GB or DM, but the reality is, as crowded as specific research topics like ‘yet another efficient Transformer variant’ may be, as soon as you add on a single qualifier like, ‘DRL for dairy herd management’ or ‘for anime’, you suddenly have the entire field to yourself. There’s a big lag between what you see on Arxiv and what’s out in the field. Even DL from 5 years ago, like CNNs, can be used for all sorts of things which they are not at present. (Making money or capturing value is, of course, an entirely different question; as fun as This Anime Does Not Exist may be, there’s not really any good way to extract money. So it’s a good thing we don’t do it for the money.)
Ah yeah, that makes sense—I agree that a lot of the reason for low commercialization is local optima, and also agree that there are lots of cool/fun applications that are left undone right now.
Hi Ajeya! I”m a huge fan of your timelines report, it’s by far the best thing out there on the topic as far as I know. Whenever people ask me to explain my timelines, I say “It’s like Ajeya’s, except...”
My question is, how important do you think it is for someone like me to do timelines research, compared to other kinds of research (e.g. takeoff speeds, alignment, acausal trade...)
I sometimes think that even if I managed to convince everyone to shift from median 2050 to median 2032 (an obviously unlikely scenario!), it still wouldn’t matter much because people’s decisions about what to work on are mostly driven by considerations of tractability, neglectedness, personal fit, importance, etc. and even that timelines difference would be a relatively minor consideration. On the other hand, intuitively it does feel like the difference between 2050 and 2032 is a big deal and that people who believe one when the other is true will probably make big strategic mistakes.
Bonus question: Murphyjitsu: Conditional on TAI being built in 2025, what happened? (i.e. how was it built, what parts of your model were wrong, what do the next 5 years look like, what do the 5 years after 2025 look like?)
Thanks so much, that’s great to hear! I’ll answer your first question in this comment and leave a separate reply for your Murphyjitsu question.
First of all, I definitely agree that the difference between 2050 and 2032 is a big deal and worth getting to the bottom of; it would make a difference to Open Phil’s prioritization (and internally we’re trying to do projects that could convince us of timelines significantly shorter than in my report). You may be right that it could have a counterintuitively small impact on many individual people’s career choices, for the reasons you say, but I think many others (especially early career people) would and should change their actions substantially.
I think there are roughly three types of reasons why Bob might disagree with Alice about a bottom line conclusion like TAI timelines, which correspond to three types of research or discourse contributions Bob could make in this space:
1. Disagreements can come from Bob knowing more facts than Alice about a key parameter, which can allow Bob to make “straightforward corrections” to Alice’s proposed value for that parameter. E.g., “You didn’t think much about hardware, but I did a solid research project into hardware and I think experts would agree that because of optical computing progress will be faster than you assumed; changing to the better values makes timelines shorter.” If Bob does a good enough job with this empirical investigation, Alice will often just say “Great, thanks!” and adopt Bob’s number.
2. Disagreements can come from Bob modeling out a part of the world in more mechanistic detail that Alice fudged or simplified, which can allow Bob to propose a better structure than Alice’s model. E.g., “You agree that earlier AI systems can generate revenue which can be reinvested into AI research but you didn’t explicitly model that and just made a guess about spending trajectory; I’ll show that accounting for this properly would make timelines shorter.” Alice may feel some hesitance adopting Bob’s model wholesale here, because Alice’s model may fudge/elide one thing in an overly-conservative direction which she feels is counterbalanced by fudging/eliding another thing in an overly-aggressive direction, but it will often be tractable to argue that the new model is better and Alice will often be happy to adopt it (perhaps changing some other fudged parameters a little to preserve intuitions that seemed important to her).
3. Finally, disagreements can come from differences in intuition about the subjective weight different considerations should get when coming up with values for the more debatable parameters (such as the different biological anchor hypotheses). It’s more difficult for Bob to make a contribution toward changing Alice’s bottom line here, because a lot of the action is in hard-to-access mental alchemy going on in Alice and Bob’s minds when they make difficult judgment calls. Bob can try to reframe things, offer intuition pumps, trace disagreements about one topic back to a deeper disagreement about another topic and argue about that, and so on, but he should expect it to be slow going and expect Alice to be pretty hard to move.
In my experience, most large and persistent disagreements between people about big-picture questions like TAI timelines or the magnitude of risk from AI are mostly the third kind of disagreement, and these disagreements can be entangled with dozens of other differences in background assumptions / outlook / worldview. My sense is that your most major disagreements with me fall into the third category: you think that I’m overweighting the hypothesis that we’d need to do meta-learning in which the “inner loop” takes a long subjective time; you may also think that I’m underweighting the possibility of sudden takeoff or overweighting the efficiency of markets in a certain way, which leads me to lend too much credence to considerations like “Well if the low end of the compute range is actually right, we should probably be seeing more economic impact from the slightly-smaller AI systems right now.” If you were to change my mind on this, it might not even be from doing “timelines research”: maybe you do “takeoff speeds research” that convinces me to take sudden takeoff more seriously, which in turn causes me to take shorter timelines (which would imply more sudden takeoff) more seriously.
I’d say tackling category 3 disagreements is high risk and effort but has the possibility of high reward, and tackling category 1 disagreements is lower risk and effort with more moderate reward. My subjective impression is that EAs tend to under-invest in tackling categories 1 and 2 because they perceive category 3 as where the real action is—in some sense they’re right about that, but they may underestimate how hard it’ll be to change people’s minds there. For example, changing someone’s minds about a category 3 disagreement often greatly benefits from having a lot of face time with them, which isn’t very scalable, and arguments may be more particular to individuals: what finally convinces Alice may not be moving to Charlie.
I think one potential way to get at a category 3 disagreement about a long-term forecast is by proposing bets about nearer-term forecasts, although I think this is often a lot harder than it sounds, because people are sensitive to the possibility of “losing on a technicality”: they were right about the big picture but wrong about how that big picture actually translates to a near-term prediction. Even making short-term bets often benefits from having a lot of face time to hash out the terms.
It occurred to me that another way to try to move someone on complicated category 3 disagreements might be to put together a well-constructed survey of a population that the person is inclined to defer to. This approach is definitely still tricky: you’d have to convince the person that the relevant population was provided with the strongest arguments for that person’s view in addition to your counterarguments, and that the individuals surveyed were thinking about it reasonably hard. But if done well, it could be pretty powerful.
Thanks, this was a surprisingly helpful answer, and I had high expectations!
This is updating me somewhat towards doing more blog posts of the sort that I’ve been doing. As it happens, I have a draft of one that is very much Category 3, let me know if you are interested in giving comments!
Your sense of why we disagree is pretty accurate, I think. The only thing I’d add is that I do think we should update downwards on low-end compute scenarios because of market efficiency considerations, just not as strongly as you perhaps, and moreover I also think that we should update upwards for various reasons (the surprising recent sucesses of deep learning, the fact that big corporations are investing heavily-by-historical-standards in AI, the fact that various experts think they are close to achieving AGI) and the upwards update mostly cancels out the downwards update IMO.
Update: The draft I mentioned is now a post!
An extension of Daniel’s bonus question:
If I condition on your report being wrong in an important way (either in its numerical predictions, or via conceptual flaws) and think about how we might figure that out today, it seems like two salient possibilities are inside-view arguments and outside-view arguments.
The former are things like “this explicit assumption in your model is wrong”. E.g. I count my concern about the infeasibility of building AGI using algorithms available in 2020 as an inside-view argument.
The latter are arguments that, based on the general difficulty of forecasting the future, there’s probably some upcoming paradigm shift or crucial consideration which will have a big effect on your conclusions (even if nobody currently knows what it will be).
Are you more worried about the inside-view arguments of current ML researchers, or outside-view arguments?
I generally spend most of my energy looking for inside-view considerations that might be wrong, because they are more likely to suggest a particular directional update (although I’m not focused only on inside view arguments specifically from ML researchers, and place a lot of weight on inside view arguments from generalists too).
It’s often hard to incorporate the most outside-view considerations into bottom line estimates, because it’s not clear what their implication should be. For example, the outside-view argument “it’s difficult to forecast the future and you should be very uncertain” may imply spreading probability out more widely, but that would involve assigning higher probabilities to TAI very soon, which is in tension with another outside view argument along the lines of “Predicting something extraordinary will happen very soon has a bad track record.”
Shouldn’t a combination of those two heuristics lead to spreading out the probability but with somewhat more probability mass on the longer-term rather than the shorter term?
That’s fair, and I do try to think about this sort of thing when choosing e.g. how wide to make my probability distributions and where to center them; I often make them wider than feels reasonable to me. I didn’t mean to imply that I explicitly avoid incorporating such outside view considerations, just that returns to further thinking about them are often lower by their nature (since they’re often about unkown-unkowns).
True. My main concern here is the lamppost issue (looking under the lamppost because that’s where the light is). If the unknown unknowns affect the probability distribution, then personally I’d prefer to incorporate that or at least explicitly acknowledge it. Not a critique—I think you do acknowledge it—but just a comment.
Just in case any readers would misinterpret that statement: I’m pretty sure that what Daniel is saying is unlikely is not that TAI will be built in 2032, but rather that he would be able to convince to shift their median to that date. I think Daniel’s median for when TAI will be built is indeed somewhere around 2032 or perhaps sooner. (I think that based on conversations around October and this post. Daniel can of course correct me if I’m wrong!)
(Maybe no readers would’ve misinterpreted Daniel anyway and this is just a weird comment...)
Yep, my current median is something like 2032. It fluctuates depending on how I estimate it, sometimes I adjust it up or down a bit based on how I’m feeling in the moment and recent updates, etc.
On the object level, I think it would probably turn out to be the case that a) I was wrong about horizon length and something more like ~1 token was sufficient, b) I was wrong about model size and something more like ~10T parameter was sufficient. On a deeper level, it would mean I was wrong about the plausibility of ultra-sudden takeoff and shouldn’t have placed as much weight as I did on the observation that AI isn’t generating a lot of annual revenue right now and its value-added seems to have been increasing relatively smoothly so far.
I would guess that the model looks like a scaled-up predictive model (natural language and/or code), perhaps combined with simple planning or search. Maybe a coding model rapidly trains more-powerful successors in a pretty classically Bostromian / Yudkowskian way.
Since this is a pretty Bostromian scenario, and I haven’t thought deeply about those scenarios, I would default to guessing that the world after looks fairly Bostromian, with risks involving the AI forcibly taking control of most of the world’s resources, and the positive scenario involving cooperatively using the AI to prevent other x-risks (including risks from other AI projects).
Re why AI isn’t generating much revenue—have you considered the productivity paradox? It’s historically normal that productivity slows down before steeply increasing when a new general purpose technologies arrives.
See “Why Future Technological Progress Is Consistent with Low Current Productivity Growth” in “Artificial Intelligence and the Modern Productivity Paradox”
Not sure how relevant, but I saw that Gwern seems to think this comes from a bottleneck of people who can apply AI, not from current AI being insufficient:
And the lack of coders may rapidly disappear soon-ish, right? At least in Germany studying ML seems very popular since a couple of years now.
In some sense I agree with gwern that the reason ML hasn’t generated a lot of value is because people haven’t put in the work (both coding and otherwise) needed to roll it out to different domains, but (I think unlike gwern) the main inference I make from that that it wouldn’t have been hugely profitable to put in the work to create ML-based applications (or else more people would have been diverted from other coding tasks to the task of rolling out ML applications).
I mostly agree with that with the further caveat that I tend to think the low value reflects not that ML is useless but the inertia of a local optima where the gains from automation are low because so little else is automated and vice-versa (“automation as colonization wave”). This is part of why, I think, we see the broader macroeconomic trends like big tech productivity pulling away: many organizations are just too incompetent to meaningful restructure themselves or their activities to take full advantage. Software is surprisingly hard from a social and organizational point of view, and ML more so. A recent example is coronavirus/remote-work: it turns out that remote is in fact totally doable for all sorts of things people swore it couldn’t work for—at least when you have a deadly global pandemic solving the coordination problem...
As for my specific tweet, I wasn’t talking about making $$$ but just doing cool projects and research. People should be a little more imaginative about applications. Lots of people angst about how they can possibly compete with OA or GB or DM, but the reality is, as crowded as specific research topics like ‘yet another efficient Transformer variant’ may be, as soon as you add on a single qualifier like, ‘DRL for dairy herd management’ or ‘for anime’, you suddenly have the entire field to yourself. There’s a big lag between what you see on Arxiv and what’s out in the field. Even DL from 5 years ago, like CNNs, can be used for all sorts of things which they are not at present. (Making money or capturing value is, of course, an entirely different question; as fun as This Anime Does Not Exist may be, there’s not really any good way to extract money. So it’s a good thing we don’t do it for the money.)
Ah yeah, that makes sense—I agree that a lot of the reason for low commercialization is local optima, and also agree that there are lots of cool/fun applications that are left undone right now.