The animation is so cute! I really enjoyed the details.
Without this video I would’ve either read Karnofsky’s series at least a few months from now, or not at all.
That said, the reason I hadn’t read it in the first place is that I expected it to be very uncovincing—and that’s exactly what I felt about the content of the video. It glosses over things like:
How people and governments would react to changes
Why the thought that AGI is theoretically possible should make us expect it from the current paradigm (my impression is that most researchers don’t expect that, and that’s why their survey answers are so volatile with slight changes in phrasing)
How known limits, like abundance of physical resources, might affect the changes beyond the near term explosion
What operative conclusion can be drawn from the “importance” of this century. If it turned out to be only the 17th most important century, would that affect our choices?
Why the thought that AGI is theoretically possible should make us expect it from the current paradigm (my impression is that most researchers don’t expect that, and that’s why their survey answers are so volatile with slight changes in phrasing)
The argument I most commonly hear that it is “too aggressive” is along the lines of: “There’s no reason to think that a modern-methods-based AI can learn everything a human does, using trial-and-error training—no matter how big the model is and how much training it does. Human brains can reason in unique ways, unmatched and unmatchable by any AI unless we come up with fundamentally new approaches to AI.” This kind of argument is often accompanied by saying that AI systems don’t “truly understand” what they’re reasoning about, and/or that they are merely imitating human reasoning through pattern recognition.
I think this may turn out to be correct, but I wouldn’t bet on it. A full discussion of why is outside the scope of this post, but in brief:
I am unconvinced that there is a deep or stable distinction between “pattern recognition” and “true understanding” (this Slate Star Codex piece makes this point). “True understanding” might just be what really good pattern recognition looks like. Part of my thinking here is an intuition that even when people (including myself) superficially appear to “understand” something, their reasoning often (I’d even say usually) breaks down when considering an unfamiliar context. In other words, I think what we think of as “true understanding” is more of an ideal than a reality.
I feel underwhelmed with the track record of those who have made this sort of argument—I don’t feel they have been able to pinpoint what “true reasoning” looks like, such that they could make robust predictions about what would prove difficult for AI systems. (For example, see this discussion of Gary Marcus’s latest critique of GPT3, and similar discussion on Astral Codex Ten).
“Some breakthroughs / fundamental advances are needed” might be true. But for Bio Anchors to be overly aggressive, it isn’t enough that some breakthroughs are needed; the breakthroughs needed have to be more than what AI scientists are capable of in the coming decades, the time frame over which Bio Anchors forecasts transformative AI. It seems hard to be confident that things will play out this way—especially because:
Even moderate advances in AI systems could bring more talent and funding into the field (as is already happening8).
If money, talent and processing power are plentiful, and progress toward PASTA is primarily held up by some particular weakness of how AI systems are designed and trained, a sustained attempt by researchers to fix this weakness could work. When we’re talking about multi-decade timelines, that might be plenty of time for researchers to find whatever is missing from today’s techniques.
I think more generally, even if AGI is not developed via the current paradigm, it is still a useful exercise to predict when we could in principle develop AGI via deep learning. That’s because, even if some even more efficient paradigm takes over in the coming years, that could make AGI arrive even sooner, rather than later, than we expect.
I’ll note that don’t think any of his arguments are good:
It’s easy to discount “true understanding” as an alternative. But I don’t see why “Pattern matching isn’t enough” translates to “True understanding is needed” and not just to “Something else which we can’t pinpoint is needed”.
Which is why I’m way more convinced by Gary Marcus’ examples than by e.g. Scott Alexander. I don’t think they need to be able to describe “true understanding” to demonstrate that current AI is far from human capabilities.
I also don’t really see what makes the track record of those who do think it’s possible with the current paradigm any more impressive.
Breakthroughs may take less than the model predict. They may also take more—for example if much much better knowledge of the human brain proves needed. Or if other advances if the field are tied together with them.
even if some even more efficient paradigm takes over in the coming years, that could make AGI arrive even sooner, rather than later, than we expect.
Only if it comes before the “due date”.
I’ll clarify that I do expect some form of transformative AI this century, and that I am worried about safety, and I’m actually looking for work in the area! But I’m trying to red-team other people who wrote about this because I want to distill the (unclear) reasons I should actually expect this from my deference to high status figures in the movement.
Which is why I’m way more convinced by Gary Marcus’ examples than by e.g. Scott Alexander. I don’t think they need to be able to describe “true understanding” to demonstrate that current AI is far from human capabilities.
My impression is that this debate is mostly people talking past each other. Gary Marcus will often say something to the effect of, “Current systems are not able to do X”. The other side will respond with, “But current systems will be able to do X relatively soon.” People will act like these statements contradict, but they do not.
I recently asked Gary Marcus to name a set of concrete tasks he thinks deep learning systems won’t be able to do in the near-term future. Along with Ernie Davis, he replied with a set of mostly vague and difficult to operationalize tasks, collectively constituting AGI, which he thought won’t happen by the end of 2029 (with no probability attached).
While I can forgive people for being a bit vague, I’m not impressed by the examples Gary Marcus offered. All of the tasks seem like the type of thing that could easily be conquered by deep learning if given enough trial and error, even if the 2029 deadline is too aggressive. I have yet to see anyone—either Gary Marcus, or anyone else—name a credible, specific reason why deep learning will fail in the coming decades. Why exactly, for example, do we think that it will stop short of being able to write books (when it can already write essays), or it will stop short of being able to write 10,000 lines of code (when it can already write 30 lines of code)?
Now, some critiques of deep learning seem right: it’s currently too data-hungry, and very costly to run large training runs, for example. But of course, these objections only tell us that there might be some even more efficient paradigm that brings us AGI sooner. It’s not a good reason to expect AGI to be centuries away.
What operative conclusion can be drawn from the “importance” of this century. If it turned out to be only the 17th most important century, would that affect our choices?
One major implication is that we should spend our altruistic and charity money now, rather than putting it into a fund and investing it, to be spent much later. The main alternative to this view is the view taken by the Patient Philanthropy Project, which invests money until such time that there is an unusually good opportunity.
I’m not sure that follows. We need X money this century and Y money that other century. Can we really expect to know which century will be more important or how much we’ll need, and how much we’ll be able to save in the future?
What I mean is I think it’s straightforward that we need to save some money for emergencies—but as there are no “older humanities” to learn from, it’s impossibly hard to forecast how much to save and when to spend it, and even then you only do it by always thinking bad things are still to come, so no time is the “most important”.
Being as honest as I can:
The animation is so cute! I really enjoyed the details.
Without this video I would’ve either read Karnofsky’s series at least a few months from now, or not at all.
That said, the reason I hadn’t read it in the first place is that I expected it to be very uncovincing—and that’s exactly what I felt about the content of the video. It glosses over things like:
How people and governments would react to changes
Why the thought that AGI is theoretically possible should make us expect it from the current paradigm (my impression is that most researchers don’t expect that, and that’s why their survey answers are so volatile with slight changes in phrasing)
How known limits, like abundance of physical resources, might affect the changes beyond the near term explosion
What operative conclusion can be drawn from the “importance” of this century. If it turned out to be only the 17th most important century, would that affect our choices?
etc.
Holden Karnofsky does discuss this objection in his blog post sequence,
I think more generally, even if AGI is not developed via the current paradigm, it is still a useful exercise to predict when we could in principle develop AGI via deep learning. That’s because, even if some even more efficient paradigm takes over in the coming years, that could make AGI arrive even sooner, rather than later, than we expect.
Thanks.
I’ll note that don’t think any of his arguments are good:
It’s easy to discount “true understanding” as an alternative. But I don’t see why “Pattern matching isn’t enough” translates to “True understanding is needed” and not just to “Something else which we can’t pinpoint is needed”.
Which is why I’m way more convinced by Gary Marcus’ examples than by e.g. Scott Alexander. I don’t think they need to be able to describe “true understanding” to demonstrate that current AI is far from human capabilities.
I also don’t really see what makes the track record of those who do think it’s possible with the current paradigm any more impressive.
Breakthroughs may take less than the model predict. They may also take more—for example if much much better knowledge of the human brain proves needed. Or if other advances if the field are tied together with them.
Only if it comes before the “due date”.
I’ll clarify that I do expect some form of transformative AI this century, and that I am worried about safety, and I’m actually looking for work in the area! But I’m trying to red-team other people who wrote about this because I want to distill the (unclear) reasons I should actually expect this from my deference to high status figures in the movement.
My impression is that this debate is mostly people talking past each other. Gary Marcus will often say something to the effect of, “Current systems are not able to do X”. The other side will respond with, “But current systems will be able to do X relatively soon.” People will act like these statements contradict, but they do not.
I recently asked Gary Marcus to name a set of concrete tasks he thinks deep learning systems won’t be able to do in the near-term future. Along with Ernie Davis, he replied with a set of mostly vague and difficult to operationalize tasks, collectively constituting AGI, which he thought won’t happen by the end of 2029 (with no probability attached).
While I can forgive people for being a bit vague, I’m not impressed by the examples Gary Marcus offered. All of the tasks seem like the type of thing that could easily be conquered by deep learning if given enough trial and error, even if the 2029 deadline is too aggressive. I have yet to see anyone—either Gary Marcus, or anyone else—name a credible, specific reason why deep learning will fail in the coming decades. Why exactly, for example, do we think that it will stop short of being able to write books (when it can already write essays), or it will stop short of being able to write 10,000 lines of code (when it can already write 30 lines of code)?
Now, some critiques of deep learning seem right: it’s currently too data-hungry, and very costly to run large training runs, for example. But of course, these objections only tell us that there might be some even more efficient paradigm that brings us AGI sooner. It’s not a good reason to expect AGI to be centuries away.
One major implication is that we should spend our altruistic and charity money now, rather than putting it into a fund and investing it, to be spent much later. The main alternative to this view is the view taken by the Patient Philanthropy Project, which invests money until such time that there is an unusually good opportunity.
I’m not sure that follows. We need X money this century and Y money that other century. Can we really expect to know which century will be more important or how much we’ll need, and how much we’ll be able to save in the future?
What I mean is I think it’s straightforward that we need to save some money for emergencies—but as there are no “older humanities” to learn from, it’s impossibly hard to forecast how much to save and when to spend it, and even then you only do it by always thinking bad things are still to come, so no time is the “most important”.