That’s marketing.
Anthropic and OpenAI have been quite successful at promoting the view that their models are very capable or even dangerous. By exaggerating the risks of these models they create a reputation as morally conscious, safety-oriented companies that nevertheless have the best models that everyone should be aware of. This gets the sympathies of investors and AI safety people alike.
But of course they must sell their models to get their money back.
I often hear the suggestion that people should short stock when they don’t believe in a company. I don’t think that it is a very good piece of advice.
Shorting is notoriously difficult and carries the possibility of unlimited loss. Even if you believe that a stock will crash, small errors such as timing the crash one year too early or using a miscalculated stop order to stop shorting too early can lead to massive losses. Determining the actual risk involved is not very straightforward. Shorting is often accompanied by risk-managing tactics such as hedging that also require careful considerations.
Due to all this, even if you believe a stock is going to crash, shorting is usually a bad idea. Basically, the skillset required to short properly is different from the skills used to predict whether or not AI will succeed commercially or not. An expert in AI can do the latter prediction, but the former requires you to also be specialized in complex investments. If you are “only” an AI expert, bets offer a much more managed risk profile.
In the context of my post, the term “narrative argument” was really clearly used to refer to fictional stories embedded in a text as part of its argument, such as AI2027′s short story or the parables in Y&S’s book. My text simply does not have that.
By completely misunderstanding even basic concepts such as “narrative argument”, your comment was just a snarky jab and had little value. It didn’t comment the actual content of my text, instead, and largely unjustified by any real analysis, it tried to “get back at me” by blaming me for things I blame others for, like non-evidence based argumentation and narrative arguments.
Since I find it difficult to believe that you really don’t understand what a narrative argument is, I reasonably count a high probability for you simply acting in bad faith.
I don’t know if “snitching” you is allowed or not, and I apologize for the moderators if it’s not, but I think using examples of bad behavior is relevant in discussions of what is allowed on the forum.
I feel that there could be greater transparency in how these moderation decisions are done. In my opinion, although Yarrow has been quite confrontational at times, they haven’t really done anything that would necessitate such a punishment. While the forum is allowed to moderate its users in any way it likes, one shouldn’t be rate-limited just because they have disagreements, especially if their opinions go against the mainstream opinion on the forum. This puts the neutrality of the forum moderation at question.
Personally, I have pretty bad experiences writing on this forum. I get highly-upvoted comments from people who seemingly haven’t even read what I wrote and instead throw in bad-faith arguments, which I find borderline insulting and certainly unwelcoming. See, for example, this comment that was given as a reply to my text, where I pointed out that many sources use narrative arguments (stories, parables, etc.) instead of technical evidence. The commenter accused me of writing a narrative argument… which is simply not true, as my text was not written in a form of a story. Either they didn’t read at all what I wrote or intentionally misunderstood it. And that is the most upvoted comment!
In general, the quality of discussion on this forum is very low. I feel it is pretty harsh to say this, but it is my experience and it strongly discourages me from participating at all. In this atmosphere, rate-limiting Yarrow appears not as an impartial decision, since many other, much more problematic comments appear in the forum all the time.
I don’t really see why anyone would use AI other than maybe translate phrases if one does not know English (or the language they are writing) well.
Language models add unnecessary fluff to the text that takes space away from actual content. I’d ask the following question: is the text written by the AI longer than the prompt? If the answer is yes, then please do not use that AI text. All extra text written by the model is fluff and meaningless.
Language models are also really bad at writing. They tend to overuse stylistic devices to the point that the text is heavy and difficult to read. They split the text into unnatural chapters, tend to add more subheadings than necessary, overuse tables and lists, and in general write horribly. Some people claim that AI helps people who are bad at writing to be able to take part in discussions, and that argument really makes no sense since AI cannot write either and I’d rather read bad English than fluff. The only exception to this is using AI for translation, since that is obviously a real blocker.
Lastly, there is the question of effort. To put an effort into writing is a proof that someone cares about this enough to write it down, so I should also care about reading it. If I recognize that a text is AI written, my motivation to go through it drops immediately. In a world of constant spam and sensory – and literary – overload, my time is valuable, and effort is the currency that gets me to read your text.
Your argument is very similar to creationist and other pseudoscientific/conspiracy theory-style arguments.
A creationist might argue that the existence of life, humanity, and other complex phenomena is “evidence” for intelligent design. If we allow this to count as “limited” evidence (or whatever term we choose to use), it is possible to follow through a Pascal’s wager-style argument and posit that this “evidence”, even if it has high uncertainty, is enough to merit an action.
It is always possible to come up with “evidence” for any claim. In evidence-based decision making, we must set a bar for evidence. Otherwise, the word “evidence” would lose it’s meaning, and we’d be wasting our resources considering every piece of knowledge there exists as “evidence”.
You could have moderate-quality studies that withstand scutiny
If the studies withstand scrutiny, then they are high-quality studies. Of course, it is possible that the study has multiple conclusions, and some of them are undermined by scrutiny and some are not, or that there are errors that do not undermine the conclusions. These studies can of course be used as evidence. I used “high-quality” as the opposite of “low-quality”, and splitting hair about “moderate-quality” is uninteresting.
you could have preliminary studies which are suggestive but which haven’t been around long enough for scrutiny to percolate up
This is a good basis when, e.g., funding new research, as confirming and replicating recent studies is an important part of science. In this case, it doesn’t matter that much if the study’s conclusions end up being true or false, as confirming either way is valuable. Researching interesting things is good, and even bad studies are evidence that the topic is interesting. But they are not evidence that should be used for other kind of decision-making.
The fact that you are using the word “evidence” in this way is causing you to misinterpret the quoted statement.
You are again splitting a hair about the meanings of words. The important thing is that they are advocating for making decisions without sufficient evidence, which is something I oppose. Their report is long and contains many AI risks, some of which (like deepfakes) have high-quality studies behind them, while others (like X-risks) do not. As a whole, the report “has some evidence” that there are risks associated with AI. So they talk about “limited evidence”. What is important is that they imply this “limited evidence” is not sufficient for making decisions.
But “limited evidence” isn’t the same as near-zero evidence
Splitting a hair. You can call your evidence limited evidence if you want. It won’t get you a free pass that your argument should be considered. If it has too much uncertainty or doesn’t withstand scrutiny, it shouldn’t be taken in as evidence. Otherwise we end up in the creationist situation.
If you believe that evidence that does not withstand scrutiny (that is, evidence that does not meet basic quality standards, contains major methodological errors, is statistically insignificant, is based on fallacious reasoning, or any other reason why the evidence is scrutinized) is evidence that we should use, then you are advocating for pseudoscience. The expected value of benefits based on such evidence is near zero.
I’m sorry if criticizing pseudoscience is frustrating, but that kind of thinking has no place in rational decision-making.
Your summary of the quoted text is inaccurate. You claim that this is an arguement that evidence is not something that in inherently required, but the quote says no such thing. Instead, it references “a large body of scientific evidence” and “stronger evidence” vs “limited evidence”. This quote essential makes the same arguement I do above. How can we square the differences in these interpretations?
The quoted text implies that the evidence would not be sufficient under normal circumstances, hence the “evidence dilemma”. If the amount of evidence was sufficient, there would be no question about what is the correct action. While the text washes its hands from making the actual decision to rely on insufficient evidence, it clearly considers this as a serious possibility, which is not something that I believe anyone should advocate.
You are splitting hairs about the difference between “no evidence” and “limited evidence”. The report considers a multitude of different AI risks, some of which have more evidence and some of which have less. What is important is that they bring up the idea that policy should be made without proper evidence.
The “scientific” phrasing frustrates me because I feel like it is often used to suggest high rigor without actually demonstrating that such rigor actually applies to a give situation, and because I feel like it is used to exclude certain categories of evidence when those categories are relevant, even if they are less strong compared to other kinds of evidence. I think we should weigh all relevant evidence, not exclude cetain pieces because they aren’t scientific enough.
Again, you are attacking me because of the word “scientific” instead of attacking my arguments. As I have many, many times said, studies should be weighted based on their content and the scrutiny it receives. To oppose the word “science” just because of the word itself is silly. Your idea that works are arbitrarily sorted to “scientific” and “non-scientific” based on “style points” instead of assessing their merits is just wrong and a straw-man argument.
I don’t think my argument leads to this conclusion. I’m just saying that AI risk has some evidence behind it, even if it isn’t the most rigorous evidence! That’s why I’m being such a stickler about this! If it were true that AI risk has actually zero evidence then of course I wouldn’t buy it! But I don’t think there actually is zero evidence even if AI risk advocates sometimes overestimate the strength of the evidence.
Where have I ever claimed that there is no evidence worth considering? In the start of my post, I write:
What unites many of these statements is the thorough lack of any evidence.
There are some studies that are rigorously conducted that provide some meager evidence. Not really enough to justify any EA intervention. But instead of referring to these studies, people use stuff like narrative arguments and ad-hoc models, which have approximately zero evidential value. That is the point of my post.
What about these statements makes you think that I don’t believe uncertainty affects decision making? It seems like I say that it does affect decision making in my comment.
If you believe this, I don’t understand where you disagree with me, other than you weird opposition to the word “scientific”.
I think many people in the EA community in fact have this view. Do you think those people should still prefer GHD because AI is off limits due to not being “scientific”? I would consider this to be “for style points”, and disagree with this approach.
It seems you have an issue with the word “scientific” and are constructing a straw-man argument around it. This has nothing to do with “style points”. As I have already explained, by scientific I only refer to high-quality studies that withstand scrutiny. If a study doesn’t, then it’s value as evidence is heavily discounted, as the probability of the conclusions of the study being right despite methodological errors, failures to replicate it, etc. is lower than if the study does not have these issues. If a study hasn’t been scrutinized at all, it is likely bad, because the amount of bad research is greater than the amount of good research (for example, if we look at the rejection rates of journals/conferences), and lack of scrutiny implies lack of credibility as researchers do not take the study seriously enough to scrutinize it.
The conclusion that cause A is preferable to cause B involves the uncertainty about both causes. Even if cause A has more rigorous evidence than cause B, that doesn’t mean the conclusion that benefits(A) > benefits(B) is similarly rigorous.
Yet E[benefits(A)] > E[benefits(B)] is a rigorous conclusion, because the uncertainty can be factored into the expected value.
Can I ask why? Do you think AI won’t be a “big deal” in the reasonably near future?
The International AI Safety Report lists many realistic threats (the first one of those is deepfakes, to give an example). Studying and regulating these things is nice, but they are not effective interventions in terms of lives saved etc.
I’m really at a loss here. If your argument is taken literally, I can convince you to fund anything, since I can give you highly uncertain arguments for almost everything. I cannot believe this is really your stance. You must agree with me that uncertainty affects decision making. It only seems that the word “scientific” bothers you for some reason, which I cannot really understand either. Do you believe that methodological errors are not important? That statistical significance is not required? That replicability does matter? To object to the idea that these issues cause uncertainty is absurd.
But the reason is that using evidence will improve the quality of the decision, not for “style points” so-to-speak.
No one has ever claimed that evidence should be collected for “style points”.
We can’t just magically have more rigorous evidence, we have to make decisions and allocate resources in order to get that evidence.
Fortunately, AI research has a plenty of funding right now (without any EA money), so in principle getting evidence should not be an issue. I am not against research, I am a proponent of it.
Doing nothing and sticking with the status quo is also a decision that can have important consequences. [...] If we lack scientific evidence, then that policy decision won’t be evidence-based even if we do nothing.
Sticking with status quo is often the best decision. When deciding how to use funds efficiently, you have to consider the opportunity cost of using those funds to something that has a certain positive benefit. And that alternative action is evidence-based. Thus, the dichotomy between “acting on AI without evidence” and “doing nothing without evidence” is false, the options are actually “acting on AI without evidence” and “acting on another cause area with evidence”.
If the estimated value of using the money for AI is below the benefit of the alternative, we should not use it for AI and instead stick to the status quo on that matter. Most AI interventions are not tractable, and due to this their actual utility might even be negative.
Do you think you would be more open to some types of AI policy if the case for those policies didn’t rely on the emergence of “AGI”?
Yes, there are several types of AI policy I support. However, I don’t think they are important cause areas for EA.
While motivated reasoning is certainly something to look out for, the substance of the argument should also be taken into account. I believe that the main point of this post, that Yudkowsky and Soares’s book is full of narrative arguments and unfalsifiable hypotheses mostly unsupported by references to external evidence, is obviously true. As you yourself say, OP’s arguments are reasonable. On that background, this kind of attack from you seems unjustified, and I’d like to hear what parts/viewpoints/narratives/conclusions of the post are motivated reasoning in your estimation.
I do agree that motivated reasoning is common with the proponents of AI adoption. As an example, I think the white paper Sparks of Artificial General Intelligence: Early experiments with GPT-4 by Microsoft is clearly a piece of advertising masquerading as a scientific paper. Microsoft has a lot to benefit from the commercial success of its partner company OpenAI, and the conclusions it suggests are almost certainly colored by this. Same could be said about many of OpenAI’s own white papers. But this does not mean that the examples or experiments they showcase are wrong per se (even if cherry-picked), or that there is no real information in them. Their results merely need to be read with the skeptical lenses.
Thank you for your criticism.
The point of this post is not to address specific issues of AI 2027, but narrative arguments and ad-hoc models in general. AI 2027 contains both, and thus exemplifies them well. By choosing a model not based on reference literature, and thus established consensus, the authors risk incorporating their own biases and assumptions into the model. This risk is present in all ad-hoc, models, not just AI 2027, which is why all ad-hoc models should be met with strong skepticism until supported by wider consensus.
You make a good observation that the criticisms of AI 2027 do not form an “academic consensus” either. This is because AI 2027 itself is not an academic publication, nor has it been a topic of any major academic discussion. It is possible for non-academic works to contain valuable contributions – as I say in an above comment, peer-review is not magic. Furthermore, even new and original models that were “ad-hoc” when first published can be good. However, the lack of wider adoption of this model suggests scientists suggests it is not viewed as a solid foundation to build on. But this lack does not, of course, explain why this is the case, so I have included links to commentary by other people in the EA community that describe the concrete issues in their model. Again, these issues are not the main point of my post, and are only provided for the reader’s convenience.
You also don’t actually point to any specific or characteristic issues from those 2 blog posts in that paragraph, instead appealing to heuristics, concept handles, and accusations. I would honestly describe that paragraph as a narrative argument.
A narrative argument presents the argument in a form of a story, like AI 2027′s science fiction scenario or the parables in Yudkowsky and Soares’s book. I’m not sure what part of my text you characterize as a story, could you elaborate on that?
In my post, I referred to the concept of “evidence-based policy making”. In this context, evidence refers specifically to rigorous, scientific evidence, as opposed to intuitions, unsubstantiated beliefs and anecdotes. Scientific evidence, as I said, referring to high-quality studies corroborated by other studies. And, as I emphasize the point of evidence mismatch, using a study that concludes something as evidence for something else is a fallacy.
The idea that current progress in AI can be taken as evidence for AGI, which in some sense is the most extreme progress in AI imaginable, incomparable to current progress, is an extraordinary claim that requires extraordinary evidence. People arguing for this are mostly basing their argument on their intuition and guesses, yet they often demand drastic actions over their beliefs. We, as the EA community, should make decisions based on evidence. Currently, people are providing substantial funding to the “AI cause” based on arguments that do not meet the bar of evidence-based policy, and I think that is something that should and must be criticized.
The purpose of peer-review is to make sure that the publication has no obvious errors and meets some basic standards of publication. I have been a peer-reviewer myself, and what I have seen is that the general quality of stuff sent to computer science conferences is low. Peer-review removes the most blatantly bad papers. To a layperson who doesn’t know the field and who cannot judge the quality of studies, it is safest to stick to peer-reviewed papers.
But it has never been suggested that peer-review somehow magically separates good evidence from bad evidence. In my work, I often refer to arXiv papers that are not peer-reviewed, but which I believe are methodologically sound and present valuable contributions. On the other hand, I know that conferences and journals often publish papers even with grave methodological errors or lack of statistical understanding.
Ultimately, the real test of a study is the criticism it receives after its publication, not peer-review. If researchers in the field think that the study is good and build their research on it, it is much more credible evidence than a study that is disproved by studies that come after it. One should never rely on a single study alone.
In case of METR’s study, their methodological errors do not preclude that their conclusions are correct. I think what they are trying to do is interesting and worth of research. I’d love to see other researchers attempt to replicate the study while improving on methodology, and if they succeed in having similar results, providing evidence for METR’s conclusions. So far, we haven’t seen this (or at least I am not aware of). Although even in that case, the problem of evidence mismatch stays, and we should be careful not to draw those conclusions to far.
Your bet proposal talks about the Metaculus question “resolving non-ambiguously”. Since the question is about the duration of time between the “weak AGI” and “superintelligent AI”, it is possible that it cannot be resolved “non-ambiguously” due to the definition of weak AGI being ambiguous even if SAI is invented. This might discourage people who believe in short SAI timelines from accepting the bet.
I guess markets are efficient most of the time, but stock market bubbles do exist and are common even, which goes against the efficient market hypothesis. I believe it is a debated topic in economics and I don’t know what the current consensus regarding it is.
My own experience points to the direction that there is an AI bubble, as cases like Lovable indicate that investors are overvaluing companies. I cannot explain their valuation, other than that investors bet on things they do not understand. As I mentioned, anecdotally this seems to often be the case.
I’m not an economist, but the general consensus among the economists I have spoken to is that different kinds of bubbles (such as the dot-com bubble) are commonplace and natural, and even large companies make stupid mistakes that affect their stock hugely.
Anecdotally, there are a lot of small companies that are clearly overvalued, such as the Swedish startup Lovable, which recently reached the valuation of $6.6 billion. It is insane for a startup whose only product is a wrapper for another company’s LLM in a space where every AI lab has their own coding tool. If people are willing to invest money in that, I’d assume they’d invest even more in larger companies that actually do have a product, even if it is overhyped. Overvaluation leads to a cycle where the company must keep overpromising to avoid a market correction, which in turn leads to even more overvaluation.
Again, I’m not an economist so all that is said with the caveat that I might misunderstand some market mechanisms. But what I am is an AI researcher. I would be more than willing to believe that the investors are right, if they provided evidence. But my experience talking with people who are investing seriously in AI is that they don’t understand the tech at all. At my day-job, talking to the management that wants AI to do things it cannot do and allocating resources for hopeless projects. At investor-sponsored events, where people are basically handing off money to any company that seems reasonable enough (e.g., employing qualified people, having experience in doing business) regardless of the AI project they are proposing. I know a person who got bought for millions due to having a good-sounding idea even though they have no product, no clients and no research backing up their idea. Some people are just happy to get rid of their money.
There do exist reasonable investors too, but there is not too much they can do about the valuations of private companies. And even though in theory markets allow things such as shorting that can correct overvaluations, these instruments are very risky and I think these people are more likely to just stay away of AI companies as a whole than attempt to short a bubble when they don’t know when it crashes.
“despite those benchmarks not really being related to AGI in any way.” This is your judgment, but clearly it is not the judgment of some of the world’s leading scientific experts in the area. (Though there may well be other experts who agree with you.)
Are you sure? Even in that report, they carefully avoid using the term “AGI” and instead refer to “general-purpose AI systems”, a legal term used in the EU AI Act and generally thought to refer to current LLMs. Although both terms contain the word “general”, they mean very different things, which is something that the authors mention as well[1].
They also quite straight say that they do not have any strong evidence. According to them, “[t]he pace and unpredictability of advancements in general-purpose AI pose an ‘evidence dilemma’ for policymakers.” (This PDF, page 14). They continue that due to the “rapid an unexpected” advancements, policymakers will have to make decisions “without having a large body of scientific evidence available”. They admit that “pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary.” Still, they claim that “waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible”.
This type of text using wordings such as “impending risk” even though they do not have strong evidence adds an artificial layer of urgency to the issue that is not based on facts.
Even their “weak evidence” is not very good. They list several risks, but of us the most important is the risk they call “loss of control”, under which they put X-risks. They reference several highly speculative papers on this issue that on a cursory reading make several mistakes and have generally a quite low quality. Going through them would be worth of a whole article, but as an example, the paper by Dung (2024) argues that “given that no one has identified an important capacity which does not improve with scaling, the currently best supported hypothesis is arguably that further scaling will bring about AGI.” That is just wrong: Based on a recent survey, 76% of AI experts believe that scaling is not enough. The most important missing capability is continual learning, which is a feature current algorithms simply lack that cannot be scaled. Dung does mention some objections to his argument, but omits the most obvious ones that he should have been aware of. This is not the kind of argument we should take even as weak evidence.
Going back to the International AI Safety Report, it seems that Bengio et al. know that their evidence is not enough, and they have carefully written their report so that it doesn’t claim anything that is technically incorrect, but overall gives a much more alarmist tone than justified by their findings. If they really had better evidence, they would say so more clearly.
“General-purpose AI is not to be confused with ‘artificial general intelligence’ (AGI). The term AGI lacks a universal definition but is typically used to refer to a potential future AI that equals or surpasses human performance on all or almost all cognitive tasks. By contrast, several of today’s AI models and systems already meet the criteria for counting as general-purpose AI as defined in this report.” (This PDF, page 27).
I believe that you are underestimating just how strong incentives OpenAI etc. have to lie about AGI. For them, it is an existential question, as there is a real chance of them becoming bankrupt if they do not deliver. This means that we should expect them to always say that AGI is close regardless of their true beliefs, because no CEO is ever going to make public claims that could risk the whole company.
Even in case of companies such as Microsoft and Google that would not fail if there is no AGI, saying out loud that there won’t be AGI would possibly crash their stocks. They will likely maintain the illusion as long as they can.
I will also push back a little on relying too much on views of individual researchers such as Hinton or Bengio, which would be much more credible if they dared to present any evidence for their claims. See, for instance, this report from October 2025 by Bengio, Hinton, and others. It fails to provide any good evidence for progress in capabilities required for general intelligence, mainly focusing on how AI systems are better at some benchmarks, despite those benchmarks not really being related to AGI in any way. Instead, the report admits that while “AI systems continue to improve on most standardised evaluations,” they “show lower success rates on more realistic workplace tasks”, hinting that even the benchmark progress is fluff to at least some degree.
If even their own report doesn’t find any progress towards AGI, what is the basis for their short timelines? I think we are right to require more evidence before using their opinion as a basis for EA interventions or funding.
You are right, I mistook which Metaculus question you linked to. However, it seems that even that question is somewhat ill-defined due to referencing the “weak AGI” definition of the other question. For that reason alone, I wouldn’t bet large sums of money on it. But it is not as problematic as the weak AGI question itself.
There is a limit to how much a for-profit company can care about safety. It is of existential importance to make profit, so you have to sell your product. It’s just how it is.
OpenAI and Anthropic are technically public benefit companies that have other objectives than just profit, but to my understanding they are formulated in a way that makes them not relevant currently, so profit is more important.
I don’t say that there aren’t people in these companies that care about safety, or that safety is completely ignored when developing products, just that there is a limit to how far the companies will consider safety.