fergusq

Karma: 283

fergusq 4 Mar 2026 15:48 UTC
1 point
0 ∶ 4
on: How much of a post are you comfortable for AI to write?
I don’t really see why anyone would use AI other than maybe translate phrases if one does not know English (or the language they are writing) well.
Language models add unnecessary fluff to the text that takes space away from actual content. I’d ask the following question: is the text written by the AI longer than the prompt? If the answer is yes, then please do not use that AI text. All extra text written by the model is fluff and meaningless.
Language models are also really bad at writing. They tend to overuse stylistic devices to the point that the text is heavy and difficult to read. They split the text into unnatural chapters, tend to add more subheadings than necessary, overuse tables and lists, and in general write horribly. Some people claim that AI helps people who are bad at writing to be able to take part in discussions, and that argument really makes no sense since AI cannot write either and I’d rather read bad English than fluff. The only exception to this is using AI for translation, since that is obviously a real blocker.
Lastly, there is the question of effort. To put an effort into writing is a proof that someone cares about this enough to write it down, so I should also care about reading it. If I recognize that a text is AI written, my motivation to go through it drops immediately. In a world of constant spam and sensory – and literary – overload, my time is valuable, and effort is the currency that gets me to read your text.

fergusq 30 Jan 2026 15:43 UTC
5 points
0 ∶ 1
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
Your argument is very similar to creationist and other pseudoscientific/conspiracy theory-style arguments.
A creationist might argue that the existence of life, humanity, and other complex phenomena is “evidence” for intelligent design. If we allow this to count as “limited” evidence (or whatever term we choose to use), it is possible to follow through a Pascal’s wager-style argument and posit that this “evidence”, even if it has high uncertainty, is enough to merit an action.
It is always possible to come up with “evidence” for any claim. In evidence-based decision making, we must set a bar for evidence. Otherwise, the word “evidence” would lose it’s meaning, and we’d be wasting our resources considering every piece of knowledge there exists as “evidence”.
You could have moderate-quality studies that withstand scutiny
If the studies withstand scrutiny, then they are high-quality studies. Of course, it is possible that the study has multiple conclusions, and some of them are undermined by scrutiny and some are not, or that there are errors that do not undermine the conclusions. These studies can of course be used as evidence. I used “high-quality” as the opposite of “low-quality”, and splitting hair about “moderate-quality” is uninteresting.
you could have preliminary studies which are suggestive but which haven’t been around long enough for scrutiny to percolate up
This is a good basis when, e.g., funding new research, as confirming and replicating recent studies is an important part of science. In this case, it doesn’t matter that much if the study’s conclusions end up being true or false, as confirming either way is valuable. Researching interesting things is good, and even bad studies are evidence that the topic is interesting. But they are not evidence that should be used for other kind of decision-making.
The fact that you are using the word “evidence” in this way is causing you to misinterpret the quoted statement.
You are again splitting a hair about the meanings of words. The important thing is that they are advocating for making decisions without sufficient evidence, which is something I oppose. Their report is long and contains many AI risks, some of which (like deepfakes) have high-quality studies behind them, while others (like X-risks) do not. As a whole, the report “has some evidence” that there are risks associated with AI. So they talk about “limited evidence”. What is important is that they imply this “limited evidence” is not sufficient for making decisions.
But “limited evidence” isn’t the same as near-zero evidence
Splitting a hair. You can call your evidence limited evidence if you want. It won’t get you a free pass that your argument should be considered. If it has too much uncertainty or doesn’t withstand scrutiny, it shouldn’t be taken in as evidence. Otherwise we end up in the creationist situation.

fergusq 29 Jan 2026 21:47 UTC
1 point
0 ∶ 0
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
If you believe that evidence that does not withstand scrutiny (that is, evidence that does not meet basic quality standards, contains major methodological errors, is statistically insignificant, is based on fallacious reasoning, or any other reason why the evidence is scrutinized) is evidence that we should use, then you are advocating for pseudoscience. The expected value of benefits based on such evidence is near zero.
I’m sorry if criticizing pseudoscience is frustrating, but that kind of thinking has no place in rational decision-making.
Your summary of the quoted text is inaccurate. You claim that this is an arguement that evidence is not something that in inherently required, but the quote says no such thing. Instead, it references “a large body of scientific evidence” and “stronger evidence” vs “limited evidence”. This quote essential makes the same arguement I do above. How can we square the differences in these interpretations?
The quoted text implies that the evidence would not be sufficient under normal circumstances, hence the “evidence dilemma”. If the amount of evidence was sufficient, there would be no question about what is the correct action. While the text washes its hands from making the actual decision to rely on insufficient evidence, it clearly considers this as a serious possibility, which is not something that I believe anyone should advocate.
You are splitting hairs about the difference between “no evidence” and “limited evidence”. The report considers a multitude of different AI risks, some of which have more evidence and some of which have less. What is important is that they bring up the idea that policy should be made without proper evidence.

fergusq 29 Jan 2026 20:18 UTC
1 point
0 ∶ 0
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
The “scientific” phrasing frustrates me because I feel like it is often used to suggest high rigor without actually demonstrating that such rigor actually applies to a give situation, and because I feel like it is used to exclude certain categories of evidence when those categories are relevant, even if they are less strong compared to other kinds of evidence. I think we should weigh all relevant evidence, not exclude cetain pieces because they aren’t scientific enough.
Again, you are attacking me because of the word “scientific” instead of attacking my arguments. As I have many, many times said, studies should be weighted based on their content and the scrutiny it receives. To oppose the word “science” just because of the word itself is silly. Your idea that works are arbitrarily sorted to “scientific” and “non-scientific” based on “style points” instead of assessing their merits is just wrong and a straw-man argument.
I don’t think my argument leads to this conclusion. I’m just saying that AI risk has some evidence behind it, even if it isn’t the most rigorous evidence! That’s why I’m being such a stickler about this! If it were true that AI risk has actually zero evidence then of course I wouldn’t buy it! But I don’t think there actually is zero evidence even if AI risk advocates sometimes overestimate the strength of the evidence.
Where have I ever claimed that there is no evidence worth considering? In the start of my post, I write:
What unites many of these statements is the thorough lack of any evidence.
There are some studies that are rigorously conducted that provide some meager evidence. Not really enough to justify any EA intervention. But instead of referring to these studies, people use stuff like narrative arguments and ad-hoc models, which have approximately zero evidential value. That is the point of my post.
What about these statements makes you think that I don’t believe uncertainty affects decision making? It seems like I say that it does affect decision making in my comment.
If you believe this, I don’t understand where you disagree with me, other than you weird opposition to the word “scientific”.

fergusq 29 Jan 2026 17:05 UTC
1 point
0 ∶ 0
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
I think many people in the EA community in fact have this view. Do you think those people should still prefer GHD because AI is off limits due to not being “scientific”? I would consider this to be “for style points”, and disagree with this approach.
It seems you have an issue with the word “scientific” and are constructing a straw-man argument around it. This has nothing to do with “style points”. As I have already explained, by scientific I only refer to high-quality studies that withstand scrutiny. If a study doesn’t, then it’s value as evidence is heavily discounted, as the probability of the conclusions of the study being right despite methodological errors, failures to replicate it, etc. is lower than if the study does not have these issues. If a study hasn’t been scrutinized at all, it is likely bad, because the amount of bad research is greater than the amount of good research (for example, if we look at the rejection rates of journals/conferences), and lack of scrutiny implies lack of credibility as researchers do not take the study seriously enough to scrutinize it.
The conclusion that cause A is preferable to cause B involves the uncertainty about both causes. Even if cause A has more rigorous evidence than cause B, that doesn’t mean the conclusion that benefits(A) > benefits(B) is similarly rigorous.
Yet E[benefits(A)] > E[benefits(B)] is a rigorous conclusion, because the uncertainty can be factored into the expected value.
Can I ask why? Do you think AI won’t be a “big deal” in the reasonably near future?
The International AI Safety Report lists many realistic threats (the first one of those is deepfakes, to give an example). Studying and regulating these things is nice, but they are not effective interventions in terms of lives saved etc.
I’m really at a loss here. If your argument is taken literally, I can convince you to fund anything, since I can give you highly uncertain arguments for almost everything. I cannot believe this is really your stance. You must agree with me that uncertainty affects decision making. It only seems that the word “scientific” bothers you for some reason, which I cannot really understand either. Do you believe that methodological errors are not important? That statistical significance is not required? That replicability does matter? To object to the idea that these issues cause uncertainty is absurd.

fergusq 29 Jan 2026 14:57 UTC
1 point
0 ∶ 1
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
But the reason is that using evidence will improve the quality of the decision, not for “style points” so-to-speak.
No one has ever claimed that evidence should be collected for “style points”.
We can’t just magically have more rigorous evidence, we have to make decisions and allocate resources in order to get that evidence.
Fortunately, AI research has a plenty of funding right now (without any EA money), so in principle getting evidence should not be an issue. I am not against research, I am a proponent of it.
Doing nothing and sticking with the status quo is also a decision that can have important consequences. [...] If we lack scientific evidence, then that policy decision won’t be evidence-based even if we do nothing.
Sticking with status quo is often the best decision. When deciding how to use funds efficiently, you have to consider the opportunity cost of using those funds to something that has a certain positive benefit. And that alternative action is evidence-based. Thus, the dichotomy between “acting on AI without evidence” and “doing nothing without evidence” is false, the options are actually “acting on AI without evidence” and “acting on another cause area with evidence”.
If the estimated value of using the money for AI is below the benefit of the alternative, we should not use it for AI and instead stick to the status quo on that matter. Most AI interventions are not tractable, and due to this their actual utility might even be negative.
Do you think you would be more open to some types of AI policy if the case for those policies didn’t rely on the emergence of “AGI”?
Yes, there are several types of AI policy I support. However, I don’t think they are important cause areas for EA.

fergusq 29 Jan 2026 12:40 UTC
17 points
5 ∶ 2
in reply to: NickLaing’s comment on: Unfalsifiable stories of doom
While motivated reasoning is certainly something to look out for, the substance of the argument should also be taken into account. I believe that the main point of this post, that Yudkowsky and Soares’s book is full of narrative arguments and unfalsifiable hypotheses mostly unsupported by references to external evidence, is obviously true. As you yourself say, OP’s arguments are reasonable. On that background, this kind of attack from you seems unjustified, and I’d like to hear what parts/viewpoints/narratives/conclusions of the post are motivated reasoning in your estimation.
I do agree that motivated reasoning is common with the proponents of AI adoption. As an example, I think the white paper Sparks of Artificial General Intelligence: Early experiments with GPT-4 by Microsoft is clearly a piece of advertising masquerading as a scientific paper. Microsoft has a lot to benefit from the commercial success of its partner company OpenAI, and the conclusions it suggests are almost certainly colored by this. Same could be said about many of OpenAI’s own white papers. But this does not mean that the examples or experiments they showcase are wrong per se (even if cherry-picked), or that there is no real information in them. Their results merely need to be read with the skeptical lenses.

fergusq 29 Jan 2026 9:55 UTC
9 points
0 ∶ 0
in reply to: mjkerrison🔸️’s comment on: Beware of non-evidence-based argumentation
Thank you for your criticism.
The point of this post is not to address specific issues of AI 2027, but narrative arguments and ad-hoc models in general. AI 2027 contains both, and thus exemplifies them well. By choosing a model not based on reference literature, and thus established consensus, the authors risk incorporating their own biases and assumptions into the model. This risk is present in all ad-hoc, models, not just AI 2027, which is why all ad-hoc models should be met with strong skepticism until supported by wider consensus.
You make a good observation that the criticisms of AI 2027 do not form an “academic consensus” either. This is because AI 2027 itself is not an academic publication, nor has it been a topic of any major academic discussion. It is possible for non-academic works to contain valuable contributions – as I say in an above comment, peer-review is not magic. Furthermore, even new and original models that were “ad-hoc” when first published can be good. However, the lack of wider adoption of this model suggests scientists suggests it is not viewed as a solid foundation to build on. But this lack does not, of course, explain why this is the case, so I have included links to commentary by other people in the EA community that describe the concrete issues in their model. Again, these issues are not the main point of my post, and are only provided for the reader’s convenience.
You also don’t actually point to any specific or characteristic issues from those 2 blog posts in that paragraph, instead appealing to heuristics, concept handles, and accusations. I would honestly describe that paragraph as a narrative argument.
A narrative argument presents the argument in a form of a story, like AI 2027′s science fiction scenario or the parables in Yudkowsky and Soares’s book. I’m not sure what part of my text you characterize as a story, could you elaborate on that?

fergusq 29 Jan 2026 8:32 UTC
3 points
1 ∶ 0
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
In my post, I referred to the concept of “evidence-based policy making”. In this context, evidence refers specifically to rigorous, scientific evidence, as opposed to intuitions, unsubstantiated beliefs and anecdotes. Scientific evidence, as I said, referring to high-quality studies corroborated by other studies. And, as I emphasize the point of evidence mismatch, using a study that concludes something as evidence for something else is a fallacy.
The idea that current progress in AI can be taken as evidence for AGI, which in some sense is the most extreme progress in AI imaginable, incomparable to current progress, is an extraordinary claim that requires extraordinary evidence. People arguing for this are mostly basing their argument on their intuition and guesses, yet they often demand drastic actions over their beliefs. We, as the EA community, should make decisions based on evidence. Currently, people are providing substantial funding to the “AI cause” based on arguments that do not meet the bar of evidence-based policy, and I think that is something that should and must be criticized.

fergusq 29 Jan 2026 0:35 UTC
6 points
1 ∶ 1
in reply to: TFD’s comment on: Beware of non-evidence-based argumentation
The purpose of peer-review is to make sure that the publication has no obvious errors and meets some basic standards of publication. I have been a peer-reviewer myself, and what I have seen is that the general quality of stuff sent to computer science conferences is low. Peer-review removes the most blatantly bad papers. To a layperson who doesn’t know the field and who cannot judge the quality of studies, it is safest to stick to peer-reviewed papers.
But it has never been suggested that peer-review somehow magically separates good evidence from bad evidence. In my work, I often refer to arXiv papers that are not peer-reviewed, but which I believe are methodologically sound and present valuable contributions. On the other hand, I know that conferences and journals often publish papers even with grave methodological errors or lack of statistical understanding.
Ultimately, the real test of a study is the criticism it receives after its publication, not peer-review. If researchers in the field think that the study is good and build their research on it, it is much more credible evidence than a study that is disproved by studies that come after it. One should never rely on a single study alone.
In case of METR’s study, their methodological errors do not preclude that their conclusions are correct. I think what they are trying to do is interesting and worth of research. I’d love to see other researchers attempt to replicate the study while improving on methodology, and if they succeed in having similar results, providing evidence for METR’s conclusions. So far, we haven’t seen this (or at least I am not aware of). Although even in that case, the problem of evidence mismatch stays, and we should be careful not to draw those conclusions to far.
What links here?
- Beware of non-evidence-based argumentation by fergusq (28 Jan 2026 21:23 UTC; 16 points)

fergusq 24 Jan 2026 4:38 UTC
3 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Experts’ AI timelines are longer than you have been told?
Your bet proposal talks about the Metaculus question “resolving non-ambiguously”. Since the question is about the duration of time between the “weak AGI” and “superintelligent AI”, it is possible that it cannot be resolved “non-ambiguously” due to the definition of weak AGI being ambiguous even if SAI is invented. This might discourage people who believe in short SAI timelines from accepting the bet.

fergusq 24 Jan 2026 3:38 UTC
3 points
0 ∶ 0
in reply to: David Mathers🔸’s comment on: The Basic Argument for AI Safety
I guess markets are efficient most of the time, but stock market bubbles do exist and are common even, which goes against the efficient market hypothesis. I believe it is a debated topic in economics and I don’t know what the current consensus regarding it is.
My own experience points to the direction that there is an AI bubble, as cases like Lovable indicate that investors are overvaluing companies. I cannot explain their valuation, other than that investors bet on things they do not understand. As I mentioned, anecdotally this seems to often be the case.

fergusq 23 Jan 2026 16:11 UTC
11 points
0 ∶ 0
in reply to: David Mathers🔸’s comment on: The Basic Argument for AI Safety
I’m not an economist, but the general consensus among the economists I have spoken to is that different kinds of bubbles (such as the dot-com bubble) are commonplace and natural, and even large companies make stupid mistakes that affect their stock hugely.
Anecdotally, there are a lot of small companies that are clearly overvalued, such as the Swedish startup Lovable, which recently reached the valuation of $6.6 billion. It is insane for a startup whose only product is a wrapper for another company’s LLM in a space where every AI lab has their own coding tool. If people are willing to invest money in that, I’d assume they’d invest even more in larger companies that actually do have a product, even if it is overhyped. Overvaluation leads to a cycle where the company must keep overpromising to avoid a market correction, which in turn leads to even more overvaluation.
Again, I’m not an economist so all that is said with the caveat that I might misunderstand some market mechanisms. But what I am is an AI researcher. I would be more than willing to believe that the investors are right, if they provided evidence. But my experience talking with people who are investing seriously in AI is that they don’t understand the tech at all. At my day-job, talking to the management that wants AI to do things it cannot do and allocating resources for hopeless projects. At investor-sponsored events, where people are basically handing off money to any company that seems reasonable enough (e.g., employing qualified people, having experience in doing business) regardless of the AI project they are proposing. I know a person who got bought for millions due to having a good-sounding idea even though they have no product, no clients and no research backing up their idea. Some people are just happy to get rid of their money.
There do exist reasonable investors too, but there is not too much they can do about the valuations of private companies. And even though in theory markets allow things such as shorting that can correct overvaluations, these instruments are very risky and I think these people are more likely to just stay away of AI companies as a whole than attempt to short a bubble when they don’t know when it crashes.
“despite those benchmarks not really being related to AGI in any way.” This is your judgment, but clearly it is not the judgment of some of the world’s leading scientific experts in the area. (Though there may well be other experts who agree with you.)
Are you sure? Even in that report, they carefully avoid using the term “AGI” and instead refer to “general-purpose AI systems”, a legal term used in the EU AI Act and generally thought to refer to current LLMs. Although both terms contain the word “general”, they mean very different things, which is something that the authors mention as well^[1].
They also quite straight say that they do not have any strong evidence. According to them, “[t]he pace and unpredictability of advancements in general-purpose AI pose an ‘evidence dilemma’ for policymakers.” (This PDF, page 14). They continue that due to the “rapid an unexpected” advancements, policymakers will have to make decisions “without having a large body of scientific evidence available”. They admit that “pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary.” Still, they claim that “waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible”.
This type of text using wordings such as “impending risk” even though they do not have strong evidence adds an artificial layer of urgency to the issue that is not based on facts.
Even their “weak evidence” is not very good. They list several risks, but of us the most important is the risk they call “loss of control”, under which they put X-risks. They reference several highly speculative papers on this issue that on a cursory reading make several mistakes and have generally a quite low quality. Going through them would be worth of a whole article, but as an example, the paper by Dung (2024) argues that “given that no one has identified an important capacity which does not improve with scaling, the currently best supported hypothesis is arguably that further scaling will bring about AGI.” That is just wrong: Based on a recent survey, 76% of AI experts believe that scaling is not enough. The most important missing capability is continual learning, which is a feature current algorithms simply lack that cannot be scaled. Dung does mention some objections to his argument, but omits the most obvious ones that he should have been aware of. This is not the kind of argument we should take even as weak evidence.
Going back to the International AI Safety Report, it seems that Bengio et al. know that their evidence is not enough, and they have carefully written their report so that it doesn’t claim anything that is technically incorrect, but overall gives a much more alarmist tone than justified by their findings. If they really had better evidence, they would say so more clearly.
1. ^
  “General-purpose AI is not to be confused with ‘artificial general intelligence’ (AGI). The term AGI lacks a universal definition but is typically used to refer to a potential future AI that equals or surpasses human performance on all or almost all cognitive tasks. By contrast, several of today’s AI models and systems already meet the criteria for counting as general-purpose AI as defined in this report.” (This PDF, page 27).

fergusq 22 Jan 2026 22:50 UTC
7 points
1 ∶ 0
in reply to: David Mathers🔸’s comment on: The Basic Argument for AI Safety
I believe that you are underestimating just how strong incentives OpenAI etc. have to lie about AGI. For them, it is an existential question, as there is a real chance of them becoming bankrupt if they do not deliver. This means that we should expect them to always say that AGI is close regardless of their true beliefs, because no CEO is ever going to make public claims that could risk the whole company.
Even in case of companies such as Microsoft and Google that would not fail if there is no AGI, saying out loud that there won’t be AGI would possibly crash their stocks. They will likely maintain the illusion as long as they can.
I will also push back a little on relying too much on views of individual researchers such as Hinton or Bengio, which would be much more credible if they dared to present any evidence for their claims. See, for instance, this report from October 2025 by Bengio, Hinton, and others. It fails to provide any good evidence for progress in capabilities required for general intelligence, mainly focusing on how AI systems are better at some benchmarks, despite those benchmarks not really being related to AGI in any way. Instead, the report admits that while “AI systems continue to improve on most standardised evaluations,” they “show lower success rates on more realistic workplace tasks”, hinting that even the benchmark progress is fluff to at least some degree.
If even their own report doesn’t find any progress towards AGI, what is the basis for their short timelines? I think we are right to require more evidence before using their opinion as a basis for EA interventions or funding.

fergusq 21 Jan 2026 21:45 UTC
5 points
0 ∶ 1
in reply to: Vasco Grilo🔸’s comment on: Experts’ AI timelines are longer than you have been told?
You are right, I mistook which Metaculus question you linked to. However, it seems that even that question is somewhat ill-defined due to referencing the “weak AGI” definition of the other question. For that reason alone, I wouldn’t bet large sums of money on it. But it is not as problematic as the weak AGI question itself.

fergusq 20 Jan 2026 0:37 UTC
9 points
1 ∶ 0
on: Experts’ AI timelines are longer than you have been told?
I recently wrote a comment criticizing the Metaculus definition of AGI. I don’t think it is a good starting point for bets as I believe that a system that is clearly not a general intelligence could pass all conditions listed in the question. There is a risk that either the question is resolved based on a non-AGI system that technically fulfills the Metaculus definition (favoring people with shorter timelines), or that the question is never resolved due to its ambiguity (favoring those with longer timelines).

fergusq 12 Jan 2026 18:34 UTC
7 points
0 ∶ 0
in reply to: RationalHippy’s comment on: Reasons for my negative feelings towards the AI risk discussion
Sorry for answering late.
My opinions are mostly the same. Last years have seen mostly incremental improvements in AI capabilities, with no development on areas I believe are crucial for AGI, such as considerably more efficient training algorithms and introspection. The current trend of using exponentially more compute without seeing the same increase in capabilities (outside of few exceptions such as coding^[1]) is a demonstration of our lack of development: algorithmic development should enable us to achieve more with less compute, which is not what we are seeing^[2].
There are many groups taking AI risk seriously. This enforces my opinion that AI risk is not neglected. Since I also believe it is not tractable, it makes a poor choice for interventions. I believe this to be true regardless of what probability we assign for achieving AGI in near future.
I might write a longer follow-up post later that goes through these in more detail.
1. ^
  Mathematics and coding are examples of skills that can be automatically validated to some extent, enabling us to train them without a training corpus. However, most skills are not like this, and we are not seeing improvements on those areas. Since one of my research areas in computational creativity, one example where progress is lacking noticeably is creative writing. Creativity has indeed seemed to even taken a step backwards in case of some models. This is due to lack of suitable training material and the impossibility of automatically valuating creative text. Human-created corpora are expensive and we’ve ran out of them. I believe strong creativity is one of the key areas required to achieve AGI, and we are not seeing progress there.
2. ^
  There are some algorithmic improvements increasing efficiency, but most of them are kind of incremental development that gives small gains but not a breakthrough that would be required.

fergusq 12 Jan 2026 12:06 UTC
12 points
1 ∶ 0
in reply to: David Mathers🔸’s comment on: The Basic Argument for AI Safety
Thanks for your answer.
other smart people disagree
I’m generally against this sort of appeal to authority. While I’m open to hear the arguments of smart people, we should evaluate those arguments themselves and not the people giving them. So far, I’ve heard no argument that would change my opinion on this matter.
You seem to make a similar argument in your other comment:
[...] But when I ask myself what evidence I have for “there are not >20 similar sized jumps before AGI” I come up short. I don’t necessarily think the burden of proof here is actually on people arguing that the chance of AGI in the next decade is non-negligible though: it’s a goal of some serious people within the relevant science [...]
Again, I think just because there are serious people with this goal, that doesn’t mean it is a justified belief. As you say yourself, you can’t find evidence for your view. Extraordinary claims require extraordinary evidence, and the burden of proof lays on the person making the claim. That some serious/smart people believe in it is not enough evidence.
1%
I want to stress that even if we gave AGI/ASI a 1% probability in the next decade, my other point that AI safety work is not tractable and not neglected still stands, and it is thus not a good intervention that people in EA should focus on.

fergusq 10 Jan 2026 13:01 UTC
11 points
0 ∶ 0
in reply to: Richard Y Chappell🔸’s comment on: The Basic Argument for AI Safety
It is true that Lovelace and Menabrea should have assumed a credible chance of rapid progress. Who knows, maybe if they had had the right resources and people, we could have had computers much earlier than we ultimately had.
But when talking about ASI, we are not just talking about rapid progress, we are talking about the most extreme progress imaginable. Extraordinary claims require extraordinary evidence, and so forth. We do not know what breakthroughs ASI requires, nor do we know how far we are from it.
It all comes down to the question of whether the current tech is relevant for ASI or not. In my estimation, it is not – something else entirely is required. The probability for us discovering that something else just now is low.

fergusq 9 Jan 2026 3:02 UTC
14 points
1 ∶ 0
in reply to: Richard Y Chappell🔸’s comment on: The Basic Argument for AI Safety
While it might feel to you that AI progress has been rapid in the past decade, most innovations behind it such as neural networks, gradient descent, backpropagation, and the concept of language models are very old innovations. The only major innovation in the past decade is the Transformer architecture from 2017, and almost everything else is just incremental progress and scaling on larger models and datasets. Thus, the pace of AI architecture development is very slow and the idea that a groundbreaking new AGI architecture will surface has a low probability.
ASI is the ultimate form of AI and in some sense computer science as a whole. Claiming that we will reach it just because we’ve just got started in computer science seems premature, akin to claiming that physics is soon solved just because we’ve made so much progress recently. Science (and AI in particular) is often compared to an infinite ladder: you can take as many steps as you like, and there will still be infinite steps ahead. I don’t believe there are literally infinite steps to ASI, but assuming there must be only a few steps ahead just because there are a lot of steps behind is a fallacy.
I was recently reading Ada Lovelace’s “Translator’s Notes” from 1843, and came across this timeless quote (emphasis original):
It is desirable to guard against the possibility of exaggerated ideas that might arise as to the powers of the Analytical Engine. In considering any new subject, there is frequently a tendency, first, to overrate what we find to be already interesting or remarkable; and, secondly, by a sort of natural reaction, to undervalue the true state of the case, when we do discover that our notions have surpassed those that were really tenable.
This is a comment to the text by Luigi Menabrea she was translating, in which he was hyping that the “conceptions of intelligence” could be encoded into the instructions of the Analytical Engine^[1]. Having a much better technical understanding of the machine than Menabrea, Lovelace was skeptical of his ideas and urged him to calm down.
The rest of their discussion is much more focused on concrete programs the machine could execute, but this short quote stroke me as very familiar of our current discussion. There existed (some level of) scientific discussion of artificial intelligence in 1840s, and their talking points seem so similar to ours, with some hyping and others being skeptical!
From the perspective of Lovelace and Menabrea, computer science progressed incredibly fast. Babbage’s Analytical Engine was a schematic for a working computer that was much better than earlier plans such as the Differential Engine. Designing complex programs became possible. I can feel their excitement while reading their texts. But even despite this, it took a hundred years until ENIAC, the first general purpose digital computer, was built in 1945. The fact that a field progresses fast in its early days does not mean much when predicting its progress in future.
1. ^
  The quote she was commenting: “Considered under the most general point of view, the essential object of the machine being to calculate, according to the laws dictated to it, the values of numerical coefficients which it is then to distribute appropriately on the columns which represent the variables, it follows that the interpretation of formulae and of results is beyond its province, unless indeed this very interpretation be itself susceptible to expression by means of the symbols which the machine employs. Thus, although it is not itself the being that reflects, it may yet be considered as the being which executes the conceptions of intelligence.”
What links here?
- fergusq's comment on Beware of non-evidence-based argumentation by fergusq (29 Jan 2026 8:32 UTC; 3 points)