This is an excellent essay, superbly reasoned and articulated. In particular, I think you did a fantastic job teasing out the distinction between intelligence as the ability to learn vs. intelligence as the ability to complete some task. Competing definitions of intelligence cause fundamental conceptual problems with accounts of AGI and our path to it. I thought it was extremely perceptive when you pointed out how âSituational Awarenessâ implicitly uses one definition when it supports the narrative and another definition when that supports the narrative, and this leads to the narrative contradicting itself. Well done!
I think your technical criticisms are extremely strong. Leopold Aschenbrennerâs weird conversion of human brainpower into LLMs tokens/âminute and his completely unrigorous and non-credible claims about unhobbling are quintessential examples of how so much of the discourse about the prospects of near-term AGI is based on hazy, sloppy reasoning. In these instances, he is truly just making stuff up.
Your criticism of the OOMs graphs is funny. These graphs are more diagrams or sketches used for rhetorical purposes rather than real graphs in the sense of representing real numbers gleaned empirically. In principle, it might be okay in some contexts to sketch a made-up graph just to illustrate a concept (I have done this and been very clear this is all Iâm doing), but a) we should be cautious about the use of graphs giving an argument or a narrative the impression of being scientific or evidence-based when it isnât and b) if an argument or narrative relies on made-up graphs, that shows the weakness in the argument/ânarrative. The ambiguity about what the graphs are even measuring is funny. You wouldnât just write âhundredsâ or âthousandsâ on a graph (hundreds or thousands of what?), you would note what the unit of measurement is. The same should apply regardless of what the numbers are or whether the graph is logarithmic.
Itâs implied that whatever is increasing by OOMs will lead to a commensurate amount of capabilities or intelligence, but that does not necessarily follow and needs an argument. For example, what if scaling compute leads to steeply diminishing returns? What if there is simply no path from increasing the compute used in narrow AI to AGI? (In that case, you might as well be scaling the OOMs of compute used to mine Bitcoin, for all that itâs going to get you to AGI.) Measurements of algorithmic efficiency are tied to specific definitions of performance, and that should be spelled out. How do those definitions of performance relate to AGI? What benchmarks or tests are we using and why are those the important ones for understanding the prospects of near-term AGI?
I also find your analysis of the conspiracy theory-like rhetoric on point. Rhetoric that provokes fear or excitement, or that gives readers the sense that theyâre accessing secret knowledge, can blunt peopleâs desire to dig into the technical arguments. Given that the flaws in the technical arguments are so glaring yet âSituational Awarenessâ is still influential in certain circles, should I worry thatâs whatâs happening?
This is an excellent essay, superbly reasoned and articulated. In particular, I think you did a fantastic job teasing out the distinction between intelligence as the ability to learn vs. intelligence as the ability to complete some task. Competing definitions of intelligence cause fundamental conceptual problems with accounts of AGI and our path to it. I thought it was extremely perceptive when you pointed out how âSituational Awarenessâ implicitly uses one definition when it supports the narrative and another definition when that supports the narrative, and this leads to the narrative contradicting itself. Well done!
I think your technical criticisms are extremely strong. Leopold Aschenbrennerâs weird conversion of human brainpower into LLMs tokens/âminute and his completely unrigorous and non-credible claims about unhobbling are quintessential examples of how so much of the discourse about the prospects of near-term AGI is based on hazy, sloppy reasoning. In these instances, he is truly just making stuff up.
Your criticism of the OOMs graphs is funny. These graphs are more diagrams or sketches used for rhetorical purposes rather than real graphs in the sense of representing real numbers gleaned empirically. In principle, it might be okay in some contexts to sketch a made-up graph just to illustrate a concept (I have done this and been very clear this is all Iâm doing), but a) we should be cautious about the use of graphs giving an argument or a narrative the impression of being scientific or evidence-based when it isnât and b) if an argument or narrative relies on made-up graphs, that shows the weakness in the argument/ânarrative. The ambiguity about what the graphs are even measuring is funny. You wouldnât just write âhundredsâ or âthousandsâ on a graph (hundreds or thousands of what?), you would note what the unit of measurement is. The same should apply regardless of what the numbers are or whether the graph is logarithmic.
Itâs implied that whatever is increasing by OOMs will lead to a commensurate amount of capabilities or intelligence, but that does not necessarily follow and needs an argument. For example, what if scaling compute leads to steeply diminishing returns? What if there is simply no path from increasing the compute used in narrow AI to AGI? (In that case, you might as well be scaling the OOMs of compute used to mine Bitcoin, for all that itâs going to get you to AGI.) Measurements of algorithmic efficiency are tied to specific definitions of performance, and that should be spelled out. How do those definitions of performance relate to AGI? What benchmarks or tests are we using and why are those the important ones for understanding the prospects of near-term AGI?
I also find your analysis of the conspiracy theory-like rhetoric on point. Rhetoric that provokes fear or excitement, or that gives readers the sense that theyâre accessing secret knowledge, can blunt peopleâs desire to dig into the technical arguments. Given that the flaws in the technical arguments are so glaring yet âSituational Awarenessâ is still influential in certain circles, should I worry thatâs whatâs happening?