What percentage chance would you put on an imminent alien invasion and what amount of resources would say is rational to allocate for defending against it? The astrophysicist Avi Loeb at Harvard is warning that there is a 30-40% chance interstellar alien probes have entered our solar system and pose a threat to human civilization. (This is discussed at length in the Hank Green video I linked and embedded in the post.)
It’s possible we should start investing in R&D now that can defend against advanced autonomous space-based technologies that might be used by a hostile alien intelligence. Even if you think there’s only a 1% chance of this happening, it justifies some investment.
And most of what EAs are working on is determining whether we’re in that world
As I see it, this isn’t happening — or just barely. Everything flows from the belief that AGI is imminent, or at least that’s there’s a very significant, very realistic chance (10%+) that it’s imminent, and whether that’s true or not is barely ever questioned.
Extraordinary claims require extraordinary evidence. Most of the evidence cited by the EA community is akin to pseudoscience — Leopold Aschenbrenner’s “Situational Awareness” fabricates graphs and data; AI 2027 is a complete fabrication and the authors openly admit that, but it’s not foregrounded enough in the presentation such that it’s misleading. Most stuff is just people reasoning philosophically based on hunches. (And in these philosophical discussions, people in EA even invent their own philosophical terms like “epistemics” and “truth-seeking” that have no agreed-upon definition that anyone has written down — and which don’t come from academic philosophy or any other academic field.)
It’s very easy to make science come to whatever conclusion you want when you can make up data or treat personal subjective guesses plucked from thin air as data. Very little EA “research” on AGI would clear the standards for publication in a reputable peer-reviewed journal, and the research that would clear that standard (and has occasionally passed it, in fact) tends to make much more narrow, conservative, caveated claims than the beliefs that people in EA actually hold and act on. The claims that people in EA are using to guide the movement are not scientifically defensible. If they were, they would be publishable.
There is a long history of an anti-scientific undercurrent in the EA community. People in EA are frequently disdainful of scientific expertise. Eliezer Yudkowsky and Nate Soares seem to call for the rejection of the “whole field of science” in their new book, which is a theme Yudkowsky has promoted in his writing for many years.
The overarching theme of my critique is that the EA approach to the near-term AGI question is unscientific and anti-scientific. The treatment of the question of whether it’s possible, realistic, or likely in the first place is unscientific and anti-scientific. It isn’t an easy out to invoke the precautionary principle because the discourse/”research” on what to do to prepare for AGI is also unscientific and anti-scientific. In some cases, it seems incoherent.
If AGI will require new technology and new science, and we don’t yet know what that technology or science is, then it’s highly suspect to claim that we can do something meaningful to prepare for AGI now. Preparation depends on specific assumptions about the unknown science and technology that can’t be predicted in advance. The number of possibilities is far too large to prepare for them all, and most of them we probably can’t even imagine.
Your picture of EA work on AGI preparation is inaccurate to the extent I don’t think you made a serious effort to understand the space you’re criticizing. Most of the work looks like METR benchmarking, model card/RSP policy (companies should test new models for dangerous capabilities a propose mitigations/make safety cases), mech interp, compute monitoring/export controls research, and trying to test for undesirable behavior in current models.
Other people do make forecasts that rely on philosophical priors, but those forecasts are extrapolating and responding to the evidence being generated. You’re welcome to argue that their priors are wrong or that they’re overconfident, but comparing this to preparing for an alien invasion based on Oumuamua is bad faith. We understand the physics of space travel well enough to confidently put a very low prior on alien invasion. One thing basically everyone in the AI debate agrees on is that we do not understand where the limits of progress are as data reflecting continued progress continues to flow.
Your accusation of bad faith is incorrect. You shouldn’t be so quick to throw the term “bad faith” around (it means something specific and serious, involving deception or dishonesty) just because you disagree with something — that’s a bad habit that closes you off to different perspectives.
I think it’s an entirely apt analogy. We do not have an argument from the laws of physics that shows Avi Loeb is wrong about the possible imminent threat from aliens, or the probability of it. The most convincing argument against Loeb’s conclusions is about the epistemology of science. That same argument applies, mutatis mutandis, to near-term AGI discourse.
With the work you mentioned, there is often an ambiguity involved. To the extent it’s scientifically defensible, it’s mostly not about AGI. To the extent it’s about AGI, it’s mostly not scientifically defensible.
For example, the famous METR graph about the time horizons of tasks AI systems can complete 80% of the time is probably perfectly fine if you only take it for what it is, which is a fairly narrow, heavily caveated series of measurements of current AI systems on artificially simplified benchmark tasks. That’s scientifically defensible, but it’s not about AGI.
When people outside of METR make an inference from this graph to conclusions about imminent AGI, that is not scientifically defensible. This is not a complaint about METR’s research — which is not directly about AGI (at least not in this case) — but about the interpretation of it by people outside of METR to draw conclusions the research does not support. That interpretation is just a hand-wavy philosophical argument, not a scientifically defensible piece of research.
Just to be clear, this is not a criticism of METR, but a criticism of people who misinterpret their work and ignore the caveats that people at METR themselves give.
I suppose it’s worth asking: what evidence, scientific or otherwise, would convince you that this all has been a mistake? That the belief in a significant probability of near-term AGI actually wasn’t well-supported after all?
I can give many possible answers to the opposite question, such as (weighted out of 5 in terms of how important they would be to me deciding that I was wrong):
Profitable applications of LLMs or other AI tools that justify current investment levels (3/5)
Evidence of significant progress on fundamental research problems such as generalization, data inefficiency, hierarchical planning, continual learning, reliability, and so on (5/5)
Any company such as Waymo or Tesla solving Level 4 or 5 autonomy without a human in the loop and without other things that make the problem artificially easy (4/5)
Profitable and impressive new applications of humanoid robots in real world applications (4/5)
Any sort of significant credible evidence of a major increase in AI capabilities, such as LLMs being able to autonomously and independently come up with new correct ideas in science, technology, engineering, medicine, philosophy, economics, psychology, etc. (not as a tool for human researchers to more easily search the research literature or anything along those lines, but doing the actual creative intellectual act itself) (5/5)
A pure reinforcement learning agent learning to play StarCraft II at an above-average level without first bootstrapping via imitation learning, using no more experience to learn this than AlphaStar (3/5)
My list is very similar to yours. I believe items 1, 2, 3, 4, and 5 have already been achieved to substantial degrees and we continue to see progress in the relevant areas on a quarterly basis. I don’t know about the status of 6.
For clarity on item 1, AI company revenues in 2025 are on track to cover 2024 costs, so on a product basis, AI models are profitable; it’s the cost of new models that pull annual figures into the red. I think this will stop being true soon, but that’s my speculation, not evidence, so I remain open that scaling will continue to make progress towards AGI, potentially soon.
What percentage chance would you put on an imminent alien invasion and what amount of resources would say is rational to allocate for defending against it? The astrophysicist Avi Loeb at Harvard is warning that there is a 30-40% chance interstellar alien probes have entered our solar system and pose a threat to human civilization. (This is discussed at length in the Hank Green video I linked and embedded in the post.)
It’s possible we should start investing in R&D now that can defend against advanced autonomous space-based technologies that might be used by a hostile alien intelligence. Even if you think there’s only a 1% chance of this happening, it justifies some investment.
As I see it, this isn’t happening — or just barely. Everything flows from the belief that AGI is imminent, or at least that’s there’s a very significant, very realistic chance (10%+) that it’s imminent, and whether that’s true or not is barely ever questioned.
Extraordinary claims require extraordinary evidence. Most of the evidence cited by the EA community is akin to pseudoscience — Leopold Aschenbrenner’s “Situational Awareness” fabricates graphs and data; AI 2027 is a complete fabrication and the authors openly admit that, but it’s not foregrounded enough in the presentation such that it’s misleading. Most stuff is just people reasoning philosophically based on hunches. (And in these philosophical discussions, people in EA even invent their own philosophical terms like “epistemics” and “truth-seeking” that have no agreed-upon definition that anyone has written down — and which don’t come from academic philosophy or any other academic field.)
It’s very easy to make science come to whatever conclusion you want when you can make up data or treat personal subjective guesses plucked from thin air as data. Very little EA “research” on AGI would clear the standards for publication in a reputable peer-reviewed journal, and the research that would clear that standard (and has occasionally passed it, in fact) tends to make much more narrow, conservative, caveated claims than the beliefs that people in EA actually hold and act on. The claims that people in EA are using to guide the movement are not scientifically defensible. If they were, they would be publishable.
There is a long history of an anti-scientific undercurrent in the EA community. People in EA are frequently disdainful of scientific expertise. Eliezer Yudkowsky and Nate Soares seem to call for the rejection of the “whole field of science” in their new book, which is a theme Yudkowsky has promoted in his writing for many years.
The overarching theme of my critique is that the EA approach to the near-term AGI question is unscientific and anti-scientific. The treatment of the question of whether it’s possible, realistic, or likely in the first place is unscientific and anti-scientific. It isn’t an easy out to invoke the precautionary principle because the discourse/”research” on what to do to prepare for AGI is also unscientific and anti-scientific. In some cases, it seems incoherent.
If AGI will require new technology and new science, and we don’t yet know what that technology or science is, then it’s highly suspect to claim that we can do something meaningful to prepare for AGI now. Preparation depends on specific assumptions about the unknown science and technology that can’t be predicted in advance. The number of possibilities is far too large to prepare for them all, and most of them we probably can’t even imagine.
Your picture of EA work on AGI preparation is inaccurate to the extent I don’t think you made a serious effort to understand the space you’re criticizing. Most of the work looks like METR benchmarking, model card/RSP policy (companies should test new models for dangerous capabilities a propose mitigations/make safety cases), mech interp, compute monitoring/export controls research, and trying to test for undesirable behavior in current models.
Other people do make forecasts that rely on philosophical priors, but those forecasts are extrapolating and responding to the evidence being generated. You’re welcome to argue that their priors are wrong or that they’re overconfident, but comparing this to preparing for an alien invasion based on Oumuamua is bad faith. We understand the physics of space travel well enough to confidently put a very low prior on alien invasion. One thing basically everyone in the AI debate agrees on is that we do not understand where the limits of progress are as data reflecting continued progress continues to flow.
Your accusation of bad faith is incorrect. You shouldn’t be so quick to throw the term “bad faith” around (it means something specific and serious, involving deception or dishonesty) just because you disagree with something — that’s a bad habit that closes you off to different perspectives.
I think it’s an entirely apt analogy. We do not have an argument from the laws of physics that shows Avi Loeb is wrong about the possible imminent threat from aliens, or the probability of it. The most convincing argument against Loeb’s conclusions is about the epistemology of science. That same argument applies, mutatis mutandis, to near-term AGI discourse.
With the work you mentioned, there is often an ambiguity involved. To the extent it’s scientifically defensible, it’s mostly not about AGI. To the extent it’s about AGI, it’s mostly not scientifically defensible.
For example, the famous METR graph about the time horizons of tasks AI systems can complete 80% of the time is probably perfectly fine if you only take it for what it is, which is a fairly narrow, heavily caveated series of measurements of current AI systems on artificially simplified benchmark tasks. That’s scientifically defensible, but it’s not about AGI.
When people outside of METR make an inference from this graph to conclusions about imminent AGI, that is not scientifically defensible. This is not a complaint about METR’s research — which is not directly about AGI (at least not in this case) — but about the interpretation of it by people outside of METR to draw conclusions the research does not support. That interpretation is just a hand-wavy philosophical argument, not a scientifically defensible piece of research.
Just to be clear, this is not a criticism of METR, but a criticism of people who misinterpret their work and ignore the caveats that people at METR themselves give.
I suppose it’s worth asking: what evidence, scientific or otherwise, would convince you that this all has been a mistake? That the belief in a significant probability of near-term AGI actually wasn’t well-supported after all?
I can give many possible answers to the opposite question, such as (weighted out of 5 in terms of how important they would be to me deciding that I was wrong):
Profitable applications of LLMs or other AI tools that justify current investment levels (3/5)
Evidence of significant progress on fundamental research problems such as generalization, data inefficiency, hierarchical planning, continual learning, reliability, and so on (5/5)
Any company such as Waymo or Tesla solving Level 4 or 5 autonomy without a human in the loop and without other things that make the problem artificially easy (4/5)
Profitable and impressive new applications of humanoid robots in real world applications (4/5)
Any sort of significant credible evidence of a major increase in AI capabilities, such as LLMs being able to autonomously and independently come up with new correct ideas in science, technology, engineering, medicine, philosophy, economics, psychology, etc. (not as a tool for human researchers to more easily search the research literature or anything along those lines, but doing the actual creative intellectual act itself) (5/5)
A pure reinforcement learning agent learning to play StarCraft II at an above-average level without first bootstrapping via imitation learning, using no more experience to learn this than AlphaStar (3/5)
My list is very similar to yours. I believe items 1, 2, 3, 4, and 5 have already been achieved to substantial degrees and we continue to see progress in the relevant areas on a quarterly basis. I don’t know about the status of 6.
For clarity on item 1, AI company revenues in 2025 are on track to cover 2024 costs, so on a product basis, AI models are profitable; it’s the cost of new models that pull annual figures into the red. I think this will stop being true soon, but that’s my speculation, not evidence, so I remain open that scaling will continue to make progress towards AGI, potentially soon.