Pronouns: she/âher or they/âthem.
I got interested in effective altruism back before it was called effective altruism, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now Iâm trying to figure out where effective altruism can fit into my life these days and what it means to me.
I write on Substack, and used to write on Medium.
The AI revenue growth weâve seen so far is compatible with several different explanations, including an AI investment bubble and narrow AI applications that are economically useful but will not lead to AGI anytime soon. Professional investors and financial analysts are generally split between these two camps. Only a small minority believe in near-term AGI.
Some criticisms of the famous METR time horizons graph:
As you mentioned, some of the problems and limitations of the METR time horizons graph are sometimes (but not always) clearly disclosed by METR employees. However, note the wide difference between the caveated description of what the graph says and the interpretation of the graph as a strong indicator of rapid, exponential improvement in general AI capabilities.
Gary Marcus, a cognitive scientist and AI researcher, and Ernest Davis, a computer scientist and AAAI fellow, co-authored a blog post on the METR graph that looks at how the graph was made and concludes that âattempting to use the graph to make predictions about the capacities of future AI is misguidedâ.
Nathan Witkin, a research writer at NYU Sternâs Tech and Society Lab, published a detailed breakdown of some of the problems with METRâs methodology. He concludes that itâs âimpossible to draw meaningful conclusions from METRâs Long Tasks benchmarkâ and that the METR graph âcontains far too many compounding errors to excuseâ. Witkin calls out a specific tweet from METR, which presents the METR graph in the broad, uncaveated way that itâs often interpreted by believers in near-term AGI. He calls the tweet âan uncontroversial example of misleading science communicationâ. In a response to a comment on that post asking how much we should update our views based on the METR graph, Witkin responded, âto be very clear I am in fact claiming that the proper update is zero.â
Iâm just summarizing the conclusions here, not the substance of the critiques. I recommend that people go and read the critiques to how the authors reach these conclusions.
I guess the point of the expert survey you cited was to explain that it does not support the idea of near-term AGI, right? I was confused because the title and introduction strongly states that the evidence has turned in favour of near-term AGI, but then you say that 2 out of the 4 pieces of evidence you cite do not support the idea of near-term AGI. I think youâre just trying to do a general survey of the evidence, both the convincing and unconvincing evidence, right?
I agree that Bio Anchors is also not convincing evidence of anything, for the reasons explained here.
Something I changed my mind about after looking into both the AI Impacts survey and the Forecasting Researching Instituteâs LEAP survey (as I wrote about here) is that survey results seem to be super sensitive to survey design, even choices in survey design that seem small to the designers, and they donât anticipate having an impact. Iâm not sure these kinds of surveys really matter that much, anyway, but Iâm at least more interested in surveys where the designers are careful about these factors that can bias the results. The effects are not small, either. In one case, the result was 750,000 times higher or lower depending on how the question was posed.
Overall, this post is a bit weird because the title and intro make a super strong claim â the tables have turned! â but then the body doesnât cash the cheque that the title and intro write. The new evidence that has turned the tables on AI skepticism is just AI revenue and the METR graph? So, if you agree that the METR graph has been debunked at this point, then itâs just AI revenue. And what does AI revenue really show? Can narrow AI not make a lot of money? Are you really prepared to defend that claim? Have at it!
Maybe the claim is something really specific, which is that if you take AI revenue growth over the last 3 years and extrapolate the same rate of growth indefinitely, you end up with some ridiculously large number, and for that number to be true, we would need to have something like AGI. But you canât just take any trend and extrapolate it indefinitely. You need to have some explanation of whatâs causing the trend and whether it will continue or not. When you step on the accelerator of your car, extrapolating that trend forward indefinitely means youâll eventually exceed the speed of light. But we donât just extrapolate things forward, we think about cause and effect.
You could look at all sorts of industries (like SaaS) or companies (like Tesla) during a few years when growth is super fast, extrapolate that forever, and conclude that one day they will account for 100% of gross world product and take over the entire world economy. But we assume this wonât happen because we understand what will prevent this from happening, and we also donât know about anything that would cause it to happen. So, will AI revenue increase until the Singularity happens? That depends on the technology. So, what will happen with the technology? Now weâre back to square one! Looking at a chart of AI revenue doesnât settle anything. Will the chart go asymptotic into AI heaven? Or will it level out, or even crash? The answer to that question is not in the chart. Itâs in the world.
Extrapolation of past trends with no causal explanation of why the trend will continue is not empiricism! It is mysticism! It amounts to saying: we donât know whatâs happening or why or how, but, somehow, we know what will happen. This is not science. This is not financial analysis. This is not anything.
A facetious graph from The Economist extrapolating when the first 14-bladed razor will arrive:
My own facetious graph:
(Why do you expect this trend not to continue?)
This is a beautifully written comment, and succinct, and funny, and true.
I would give EA much more grace if its self-image was the same as what I presume the Big Garden Birdwatchâs self-image is. Part of what gets me tilted out of my mind about the EA community is when people express this almost messianic Chosen Ones self-image â which ties into the pseudo-religious aspect you mentioned.
The high-impact, low-probability logic of existential risk is hypnotically alluring. If a 1 in 1 quintillion chance of reducing existential risk is equivalent to 100 human lives, what does that imply in terms of your moral responsibility when discussing existential risk? If you have things to say that could cast doubt on existential risk arguments, should you self-censor and hold your tongue? If you speak out and youâre wrong, it could be the moral equivalent of killing 100 people. Would it be okay to lie? To exaggerate? Why not? Wouldnât you lie or exaggerate to save 100 lives? If the Nazis knocked at your door, wouldnât you lie to save Anne Frank in the attic?
I donât think many people are actually outright lying when it comes to existential risk. But I do think people are self-censoring when it comes to criticism, and I do think people are willing to make excuses for really low-quality products like AI 2027 or 80,000 Hoursâ video on it because anything that builds momentum for existential risk fear is plausibly extremely high in expected value.
Much could be said in response to this comment. Probably the most direct and succinct response is my post âUnsolved research problems on the road to AGIâ.
Largely for the reasons explained in that post, I think AGI is much less than 0.01% likely in the next decade.
How much of a post are you comfortable for AI to write?
I will never let AI write a single sentence! I resent reading AI-generated writing passed off as written by a human, and I would never inflict this upon my readers.
I have found that the most common explanation for why people using AI for writing is a lack of self-confidence. I keep encouraging people to write in their own words and use their own voice because all the flaws of unpolished human writing are vastly preferable to chatbot writing.
Thank you for your supportive comment. I think David Mathers is an exceptionally and commendably valuable contributor to the EA Forum in terms of engaging deeply with the substance of arguments around AI safety and AGI forecasting. David engages in discussions with a high level of reasoning transparency, which I deeply appreciate. It isnât always clear to me why people who fall on the opposite side of debates around AI safety and AGI forecasting believe what they do, and talking to David has helped me understand this better. I would love to have more discussions about these topics with David, or with interlocutors like him. I feel as though there is still much work to be done in bringing the cruxes of these debates into sharp relief.
The EA Forum has a little-used âDialoguesâ feature that I think has some potential. Anyone who would be interested in having a Dialogue on AGI forecasting and/âor AGI safety should send me a private message.
On to the rest of your comment:
I think the current investments in AGI safety will end up being wasted. I think itâs a bit like paying philosophers in the 1920s to think about how to mitigate social media addiction, years before the first proper computer was built, and even before the concept of a Turing machine was formalized. There is simply too much unknown about how AGI might eventually be built.
Conversely, investments in narrow, prosaic âAI safetyâ like making LLM chatbots less likely to give people dangerous medical advice are modestly useful today but will have no applicability to AGI much later on. Other than having the name âAIâ in common and running on computers using probably some sort of connectionist architecture, I donât think todayâs AI systems will have any meaningful resemblance to AGI, if it is eventually created.
I canât remember where â I thought it was maybe in a comment on this post, but apparently not â but I seem to recall someone on the EA Forum saying that MIRI or Yudkowsky deserved credit for correctly predicting the sort of âalignmentâ failures that modern AI systems like LLMs would exhibit. (If anyone remembers the specific comment Iâm thinking of, please let me know.) I want to set the record straight and explain why this is not true.
Reinforcement learning was originally developed in the late 1970s and in the 1980s. Years before the founding of MIRI (originally called the Singularity Institute, and created with a different focus) and before Yudkowsky first wrote about âfriendly AIâ, RL researchers noticed the phenomenon of âreward hackingâ or âspecification gamingâ (although these exact terms were not always used to describe it). One example is found in the 1998 paper âLearning to Drive a Bicycle using Reinforcement Learning and Shapingâ. The authors created a bicycle riding simulation and tasked an RL agent with riding the bicycle toward a target. The RL agent found it could maximize reward by riding in circles around the target (page 6):
We agree with Mataric [Mataric, 1994] that these heterogeneous reinforcement functions have to be designed with great care. In our first experiments we rewarded the agent for driving towards the goal but did not punish it for driving away from it. Consequently the agent drove in circles with a radius of 20â50 meters around the starting point. Such behavior was actually rewarded by the reinforcement functionâŚ
One could cite many more examples like this.[1]
It is possible to mistake an awareness of well-known concepts in a field (such as RL or AI more generally) with prescience or insight. Readers of MIRIâs or Yudkowskyâs writing should be wary of this.
A similar example to the one just given but using real robots made of Lego is found in the 2004 paper âLego Mindstorms Robots as a Platform for Teaching Reinforcement Learningâ. One robot learned to continually drive backwards and forwards along the same stretch of track to maximize reward (page 5):
After some experimentation with the reinforcement signal, the reinforcement learning system was much more successful on the line-following task than on the walking task.
In the initial trials, the reinforcement signal used rewarded the robot with positive reinforcement for any action which led to the robot remaining on the track (measured by applying a threshold to the summed value of the light sensors). As the actions available did not provide an option for staying still, this was expected to lead to the robot moving forward along the path and eventually traversing the circuit. However the learning algorithm discovered that alternating turning left and right allowed the robot to reverse slowly in a straight line, and hence maximal reinforcement could be achieved by travelling along a straight section of line at the beginning of the track, and then reversing back along that same section of track.
Thereâs an expert consensus that tobacco is harmful, and there is a well-documented history of tobacco companies engaging in shady tactics. There is also a well-documented history of government propaganda being misleading and deceptive, and if you asked anyone with relevant expertise â historians, political scientists, media experts, whoever â they would certainly tell you that government propaganda is not reliable.
But just lumping in âAI accelerationist companiesâ with that is not justified. âAI accelerationistâ just means anyone who works on making AI systems more capable who doesnât agree with the AI alignment/âAI safety communityâs peculiar worldview. In practice, that means youâre saying most people with expertise AI are compromised and not worth listening to, but you are willing to listen to this weird random group of people, some of whom like Yudkowsky who have no technical expertise in contemporary AI paradigms (i.e. deep learning and deep reinforcement learning). This seems like a recipe for disaster, like deciding that capitalist economists are all corrupt and that only Marxist philosophers are worth trusting.
A problem with motivated reasoning arguments, when stretched to this extent, is that anyone can accuse anyone over the thinnest pretext. And rather than engaging with peopleâs views and arguments in any serious, substantive way, it just turns into a lot of finger pointing.
Yudkowskyâs gotten paid millions of dollars to prophesize AI doom. Many people have argued that AI safety/âAI alignment narratives benefit the AI companies and their investors. The argument goes like this: Exaggerating the risks of AI exaggerates AIâs capabilities. Exaggerating AIâs capabilities makes the prospective financial value of AI much higher than it really is. Therefore, talking about AI risk or even AI doom is good business.
I would add that exaggerating risk may be a particularly effective way to exaggerate AIâs capabilities. People tend to be skeptical of anything that sounds like pie-in-the-sky hope or optimism. On the other hand, talking about risk sounds serious and intelligent. Notice what goes unsaid: many near-term AGI believers think thereâs a high chance of some unbelievably amazing utopia just on the horizon. How many times have you heard someone imagine that utopia? One? Zero? And how many times have you heard various AI doom or disempowerment stories? Why would no one ever bring up this amazing utopia they think might happen very soon?
Even if youâre very pessimistic and think thereâs a 90% chance of AI doom, a 10% chance of utopia is still pretty damn interesting. And many people are much more optimistic, thinking thereâs around a 1-30% chance of doom, which implies a 70%+ chance of utopia. So, what gives? Whereâs the utopia talk? Even when people talk about the utopian elements of AGI futures, they emphasize the worrying parts: what if intelligent machines produce effectively unlimited wealth, how will we organize the economy? What policies will we need to implement? How will people cope? We need to start worrying about this now! When I think about what would happen if I won the lottery, my mind does not go to worrying about the downsides.
I think the overwhelming majority of people who express views on this topic are true believers. I think they are sincere. I would only be willing to accuse someone of possibly doing something underhanded if, independently, they had a track record of deceptive behaviour. (Sam Altman has such a track record, and generally I donât believe anything he says anymore. I have no way of knowing whatâs sincere, whatâs a lie, and whatâs something heâs convinced himself of because it suits him to believe it.) I think the specific accusation that AI safety/âAI alignment is a deliberate, conscious lie cooked up to juice AI investment is silly. Itâs probably true, though, that people at AI companies have some counterintuitive incentive or bias toward talking up AI doom fears.
However, my general point is that just as itâs silly to accuse AI safety/âalignment people of being shills for AI companies, it also seems silly to me to say that AI companies (or âAI accelerationistâ companies, which is effectively all major AI companies and almost all startups) are the equivalent of tobacco companies, and you shouldnât pay attention to what people at AI companies say about AI. Motivated reasoning accusations made on thin grounds can put you into a deluded bubble (e.g. becoming a Marxist) and I donât think AI is some clear-cut, exceptional case like tobacco or state propaganda where obviously you should ignore the message.
Thereâs a fine line between steelmanning peopleâs views and creating new views that are facially similar to those views but are crucially different from the views those people actually hold. I think what youâre describing is not steelmanning, but developing your own views different from Yudkowsky and Soaresâ â views that they would almost certainly disagree with in strong terms.
I think it would be constructive for you to publish the views you developed after reading Yudkowsky and Soaresâ book. People might find that useful to read. That could give people something interesting to engage with. But if you write that Yudkowsky and Soaresâ claim about alien preferences is wrong, many people will disagree with you (including Yudkowsky and Soares, if they read it). So, itâs important to get very clear on what different people in a discussion are saying and what theyâre not saying. Just to keep everything straight, at least.
I agree the alien preferences thing is not necessarily a crux of AI doom arguments more generally, but it is certainly a crux of Yudkowsky and Soaresâ overall AI doom argument specifically. Yes, you can change their overall argument into some other argument that doesnât depend on the alien preferences thing anymore, but then thatâs no longer their argument, thatâs a different argument.
I agree that Yudkowsky and Soares (and their book) are not fully representative of the AI safety communityâs views, and probably no single text or person (or pair of people) are. I agree that it isnât really reasonable to say that if you can refute Yudkowsky and Soares (or their book), you refute the AI safety communityâs views overall. So, I agree with that critique.
In contrast, if Mechanize succeeds, Matthew Barnett will probably be a billionaire.
If Mechanize succeeds in its long-term goal of âthe automation of all valuable work in the economyâ, then everyone on Earth will be a billionaire.
So, if the best version of Yudkowsky and Soaresâ argument is not the one made in their book, what is the best version? Can you explain how that version of the argument, which they made previously elsewhere, is different than the version in the book?
I canât tell if youâre saying:
a) that the alien preferences thing is not a crux of Yudkowsky and Soaresâ overall argument for AI doom (it seems like it is) or if
b) the version of the specific argument about alien preferences they gave in the book isnât as good as previous versions theyâve given (which is why I asked what version is better) or if
c) youâre saying that Yudkowsky and Soaresâ book overall isnât as good as their previous writings on AI alignment.
I donât know that academic reviewers of Yudkowsky and Soaresâ argument would take a different approach. The book is supposed to be the most up-to-date version of the argument, and one the authors took a lot of care in formulating. It doesnât feel intuitive to go back and look at their earlier writings and compare different version of the argument, which arenât obviously different at first glance. (Will MacAskill and Clara Collier both complained the book wasnât sufficiently different from previous formulations of the argument, i.e. wasnât updated enough in light of advancements in deep learning and deep reinforcement learning over the last decade.) I think an academic reviewer might just trust that Yudkowsky and Soaresâ book is going to be the best thing to read and respond to if they want to engage with their argument.
You might, as an academic, engage in a really close reading of many versions of a similar argument made by Aristotle in different texts, if youâre a scholar of Aristotle, but this level of deep textual analysis doesnât typical apply to contemporary works by lesser-known writers outside academia.
The academic philosopher David Thorstad is writing a blog series in response to the book. I havenât read it yet, so I donât know if he pulls his alternative Yudkowsky and Soares writings other than the book itself. However, I think it would be perfectly fine for him to just focus on the book, and not seek out other texts from the same authors that make the same argument in maybe a better form.
If what youâre saying is that there are multiple independent (and mutually incompatible) arguments for the AI safety communityâs core claims, including ones that Yudkowsky and Soares donât make, then I agree with that. I agree you can criticize that sentence in the Mechanize co-foundersâ essay if you believe Yudkowskyâs views and arguments donât actually unifies (or adequately represents) the views and arguments of the AI safety community overall. Maybe you could point out what those other arguments are and who has formulated them best. Maybe the Mechanize co-founders would write a follow-up piece engaging with those non-Yudkowsky arguments as well, to give a more complete engage with the AI safety communityâs worldview.
I think the claim that Yudkowskyâs views on AI risk are meaningfully influenced by money is very weak.
To be clear, I agree. I also agree with your general point that other factors are often more important than money. Some of these factors include the allure of millennialism, or the allure of any sort of totalizing worldview or âideologyâ.
I was trying to make a general point against accusations of motivated reasoning related to money, at least in this context. If two sets of people are each getting paid to work on opposite sides of an issue, why only accuse one side of motivated reasoning?
This is indicated by the hundreds of comments, tweets, in-person arguments, DMs, and posts from at least 2023 onward in which I expressed skepticism about AI risk arguments and AI pause proposals.
Thanks for describing this history. Evidence of a similar kind lends strong credence to Yudkowsky forming his views independent from the influence of money as well.
My general view is that reasoning is complex, motivation is complex, peopleâs real psychology is complex, and that the forum-like behaviour of accusing someone of engaging in X bias is probably a misguided pop science simplification of the relevant scientific knowledge. For instance, when people engage in distorted thinking, the actual underlying reasoning often seems to be a surprisingly complicated multi-step sequence.
The essay above that you co-wrote is incredibly strong. I was the one who originally sent it to Vasco and, since he is a prolific cross-poster and I donât like to cross-post under my name, encouraged him to cross-post it. Iâm glad more people in the EA community have now read it. I think everyone in the EA community should read it. Itâs regrettable that thereâs only been one object-level comment on the substance of the essay so far, and so many comments about this (to me) relatively uninteresting and unimportant side point about money biasing peopleâs beliefs. I hope more people will comment on the substance of the essay at some point.
I think part of where the angsty energy comes from is that Yudkowsky and Soares are incredibly brazen and insulting when they express their views on AI. For instance, Yudkowsky recently said that people with AGI timelines longer than 30 years are no âsmarter than a potted plantâ. Yudkowsky has publicly said, on at least two occasions, that he believes heâs the smartest person in the world â at least on AI safety and maybe just in general â and thereâs no second place thatâs particularly close. Yudkowsky routinely expresses withering contempt, even for people who are generally âon his sideâ and trying to be helpful. Itâs really hard to engage with this style of âdebateâ (as it were) and not feel incredibly pissed off.
When I was running an EA university group, if anyone had behaved like Yudkowsky routinely behaves, they would have been banned from the group, and Iâm sure the members of my group would have unanimously agreed the behaviour is unacceptable. The same applies to any other in-person group, community, or social circle Iâve been apart of. It would scarcely be more acceptable than a man in an EA group repeatedly telling the women he just met there how hot they are. People generally donât tolerate this kind of thing. I think many people would prefer not to reward this behaviour with attention, but given that Yudkowsky (and Soares) have already successfully gotten a lot of attention, itâs necessary to write replies like this essay (the one above, by the Mechanize co-founders).
Privately, some people in the LessWrong community, where Yudkowsky is deeply revered, have said they find Yudkowskyâs style of engagement unpleasant and regrettable (in stronger words than that). Some have said it publicly. (Soares, too, has been publicly criticized for his demeanor toward people who are âon his sideâ and trying to be helpful, let alone people he disagrees with, or thinks he does.)
I think itâs close to impossible not to feel angsty when engaging with Yudkowsky (and Soares), unless you happen to be one of those people who revere him and treat him as a role model (or, I donât know, youâre a Zen Buddhist master). I agree that itâs regrettable for the debates to become as heated as they often get. I agree it would be more interesting to have intellectual discussions based in civility, mutual respect, curiosity about the other personâs opinion, intellectual generosity, and so on. But if someone isnât willing to play ball, I think youâve gotta either just ignore them, bite your tongue and be artificially polite (in which case some amount of angst will probably still be revealed), or write angry refutations.
I agree with Ben Stewartâs response that this is not a helpful thing to say. You are making some very strange and unintuitive claims. I canât imagine how you would persuade a reasonable, skeptical, well-informed person outside the EA/âLessWrong (or adjacent) bubble that these are credible claims, let alone that they are true. (Even within the EA Forum bubble, it seems like significantly more people disagree with you than agree.)
To pick on just one aspect of this claim: it is my understanding that Yudkowsky has no meaningful technical proficiency with deep learning-based or deep reinforcement learning-based AI systems. In my understanding, Yudkowsky lacks the necessary skills and knowledge to perform the role of an entry-level AI capabilities researcher or engineer at any AI company capable of paying multi-million-dollar salaries. If there is evidence that shows my understanding is mistaken, I would like to see that evidence. Otherwise, I can only conclude that you are mistaken.
I think the claim that an endorsement is worth billions of billions is also wrong, but itâs hard to disprove a claim about what would happen in the event of a strange and unlikely hypothetical. Yudkowsky, Soares, and MIRI have an outsized intellectual influence in the EA community (and obviously on LessWrong). There is some meaningful level of influence on the community of people working in the AI industry in the Bay Area, but itâs much less. Among the sort of people who could make decisions that would realize billions or tens of billions in value, namely the top-level executives at AI companies and investors, the influence seems pretty marginal. I would guess the overwhelming majority of investors either donât know who Yudkowsky and Soares are or do but donât care what their views are. Top-level executives do know who Yudkowsky is, but in every instance Iâve seen, they tend to be politely disdainful or dismissive toward his views on AGI and AI safety.
Anyway, this seems like a regrettably unproductive and unimportant tangent.
Help me understand what youâre saying here. Are you saying that Yudkowsky and Soaresâs argument is just so obviously wrong that itâs almost uninteresting to discuss why itâs wrong? That you find the Mechanize co-founders refutation of the Yudkowsky and Soares argument disappointing because you found that argument so weak to begin with?
If so, Iâm not saying thatâs a wrong view â not at all. But itâs worth noting how controversial that view is in the EA community (and other communities that talk a lot about AGI). Essays like this need to be written because so many people in this community (and others) believe Yudkowsky and Soaresâ argument is correct. If my impression of the EA community is off base and actually thereâs a community consensus that Yudkowsky and Soaresâ argument is wrong, then more people should talk about this, because itâs really hard to get the wrong impression.
I think itâs also worth discussing the question of what if AGI turns out to have generally human-like motivations and psychology. What dangers might it pose? How would it behave? But not every relevant and worthy question can be addressed in a single essay.
We should generally be skeptical of corporations (or even non-profits!) releasing pre-prints that look like scientific papers but might not pass peer review at a scientific journal. We should indeed view such pre-prints as somewhere between research and marketing. OpenAIâs pre-prints or white papers are a good example.
I think itâs hard to claim that a pre-print like Sparks of AGI is insincere (it might be, but how could we support that claim?), but this doesnât undermine the general point. Suppose employees at Microsoft Research wanted to publish a similar report arguing that GPT-4â˛s seeming cognitive capabilities are actually just a bunch of cheap tricks and not sparks of anything. Would Microsoft publish that report? Itâs not just about how financial or job-related incentives shape what you believe (although that is worth thinking about), itâs also about how they shape what you can say out loud. (And, importantly, what you are encouraged to focus on.)
Itâs interesting to claim that money stops being an incentive for people after a certain fixed amount well below $1 million/âyear. Letâs say thatâs true â maybe it is true â then why do we treat people like Sam Altman, Dario Amodei, Elon Musk, and so on as having financial incentives around AI? Are we wrong to do so? (What about AI researchers and engineers who receive multi-million-dollar compensation packages? After the first, say, $5 million, are they free and clear to form unmotivated opinions?)
I think a very similar argument can be made about the Mechanize co-founders. They could make âenoughâ money doing something else â including their previous jobs â even if itâs less money than they might stand to gain from a successful AI capabilities startup. Should we then rule out money as an incentive?
To be clear, I donât claim that Eliezer Yudkowsky, Nate Soares, others at MIRI, or the Mechanize co-founders are unduly motivated by money in forming their beliefs. I have no way of knowing that, and since thereâs no way to know, Iâm willing to give them all the benefit of the doubt. Iâm saying I dislike accusations of motivated reasoning in large part because theyâre so easy to level at people you disagree with, and itâs easy to overlook how the same argument could apply to yourself or people you agree with. Iâm pointing out how a similar accusation could be levelled at Yudkowsky and Soares in order to illustrate this general point, specifically to challenge Nick Laingâs accusation against the Mechanize co-founders above.
I generally think that ideological motivation around AGI is a powerful motivator. I think the psychology around how people form their beliefs on AGI is complex and involves many factors (e.g. millennialist cognitive bias, to name just one).
Iâm not sure how they decide on what salaries to pay themselves. But the reason they have the money to pay themselves those salaries in the first place is that MIRIâs donors believe thereâs a significant chance of AI destroying the world within the next 5-20 years and that MIRI (especially Yudkowsky) is uniquely positioned to prevent this from happening.
People who have radical anti-institutionalist views often take reasonable criticisms of institutions and use them to argue for their preferred radical alternative. There are many reasonable criticisms of liberal democracy; these are eagerly seized on by Marxist-Leninists, anarchists, and right-wing authoritarians to insist that their preferred political system must be better. But of course this conclusion does not necessarily follow from those criticisms, even if the criticisms are sound. The task for the challenger is to support the claim that their preferred system is robustly superior, not simply that liberal democracy is flawed.
The same is true for radical anti-institutionalist views on institutional science (which the LessWrong community often espouses, or at least whenever it suits them). Pointing out legitimate failures in institutional science does not necessarily support the radical anti-institutionalistsâ conclusion that peer-reviewed journals, universities, and government science agencies should be abandoned in favour of blogs, forums, tweets, and self-published reports or pre-prints. On what basis can the anti-institutionalists claim that this is a robustly superior alternative and not a vastly inferior one?
To be clear, I interpret you as making a moderate anti-institutionalist argument, not a radical one. But the problem with the reasoning is the same in either case â which is why Iâm using the radical arguments for illustration. The guardrails in academic publishing sometimes fail, as in the case of research misconduct or in well-intentioned, earnestly conducted research that doesnât replicate as you mentioned. But is this an argument for kicking down all guardrails? Shouldnât it be the opposite? Doesnât this just show us that deeply flawed research can slip under the radar? Shouldnât this underscore the importance of savvy experts doing close, critical readings of research to find flaws? Shouldnât the replication crisis remind of us of the importance of replication (which has always been a cornerstone of institutional science)? Why should the replication crisis be taken as license to give up on institutions and processes that attempt to enforce academic rigour, including replication?
In the case of both AI 2027 and the METR graph, half of the problem is the underlying substance â the methodology, the modelling choices, the data. The other half of the problem is the presentation. Both have been used to make bold, sweeping, confident claims. Academic journals referee both the substance and the presentation of submitted research; they push back on authors trying to use their data or modelling to make conclusions that are insufficiently supported.
In this vein, one of the strongest critiques of AI 2027 is that it is an exercise in judgmental forecasting, in which the authors make intuitive, subjective guesses about the future trajectory of AI research and technology development. Thereâs nothing inherently wrong with a judgmental forecasting exercise, but I donât think the presentation of AI 2027 is clear enough that AI 2027 is nothing more than that. (80,000 Hoursâ video on AI 2027, which is 34 minutes long and was carefully written and produced at a cost of $160,000, doesnât even mention this.)
If AI 2027 had been submitted to a reputable peer-reviewed journal, besides hopefully catching the modelling errors, the reviewers probably would have insisted the authors make it clear from the outset what data the conclusions are based on (i.e. the authorsâ judgmental forecasts) and where that data came from. They would probably also have insisted the conclusions are appropriately moderated and caveated in light of that. But, overall, I think AI 2027 would probably just be unpublishable.
I donât really like accusations of motivated reasoning. The logic you presented cuts both ways.
MIRIâs business model relies on the opposite narrative. MIRI pays Eliezer Yudkowsky $600,000 a year. It pays Nate Soares $235,000 a year. If they suddenly said that the risk of human extinction from AGI or superintelligence is extremely low, in all likelihood that money would dry up and Yudkowsky and Soares would be out of a job.
The financial basis for motivated reasoning is arguably even stronger in MIRIâs case than in Mechanizeâs case. The kind of work MIRI is doing and the kind of experience Yudkowsky and Soares have isnât really transferable to anything else. This means they are dependent on people being scared of enough of AGI to give money to MIRI. On the other hand, the technical skills needed to work on trying to advance the capabilities of current deep learning and reinforcement learning systems are transferable to working on the safety of those same systems. If the Mechanize co-founders wanted to focus on safety rather than capabilities, they could.
Iâm also guessing the Mechanize co-founders decided to start the company after forming their views on AI safety. They were publicly discussing these topics long before Mechanize was founded. (Conversely, Yudkowsky/âMIRIâs current core views on AI were formed roughly around 2005 and have not changed in light of new evidence, such as the technical and commercial success of AI systems based on deep learning and deep reinforcement learning.)
I would consider driving people to delusion and suicide, killing people for self-preservation and even Hitler the man himself to be at least a somewhat âalienâ style of evil.
The Yudkowsky/âSoares/âMIRI argument about AI alignment is specifically that an AGIâs goals and motivations are highly likely to be completely alien from human goals and motivations in a way thatâs highly existentially dangerous. If youâre making an argument to the effect that âhumans can also be misaligned in a way thatâs extremely dangerousâ, I think, at that point, you should acknowledge youâve moved on from the Yudkowsky/âSoares/âMIRI argument (and maybe decided to reject it). Youâre now making a quite distinct argument that needs to be evaluated independently. It may be worth asking what to do about the risk that powerful AI systems will have human-like goals and motivations that are dangerous in the same way that human goals and motivations can be dangerous. But that is a separate premise from what Yudkowsky and Soares are arguing.
Hi Kouadio. Just want to let you know that your comments donât have paragraph breaks between the paragraphs. Maybe you are copying and pasting from another app and the formatting is getting messed up? Iâm just saying this because the text looks like itâs all in one big block and that makes it harder to read. I want to make sure you get a fair shot at saying what you want to say, and fixing this formatting issue will make people more likely to read your comments.