Pronouns: she/âher or they/âthem.
I got interested in effective altruism back before it was called effective altruism, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now Iâm trying to figure out where effective altruism can fit into my life these days and what it means to me.
I write on Substack, and used to write on Medium.
I donât know much about this topic myself, but my understanding is that market efficiency is less about having the objectively correct view (or making the objectively right decision) and more about the difficulty of any individual investor making investments that systematically outperform the market. (An explainer page here helps clarify the concept). So, the concept, I think, is not that the market is always right, but when the market is wrong (e.g. that generative AI is a great investment), youâre probably wrong too. Or, more precisely, that youâre unlikely to be systematically right more often then the market is right, and systematically wrong less often than the market is wrong.
As I understand it, there are differing views among economists on how efficient the market really is. And there is the somewhat paradoxical fact that people disagreeing with the market is part of what makes it as efficient as it is in the first place. For instance, some people worry that the rise of passive investing (e.g. via Vanguard ETFs) will make the market less efficient, since more people are just deferring to the market to make all the calls, and not trying to make calls themselves. If nobody ever tried to beat the market, then the market would become completely inefficient.
There is an analogy here to forecasting, with regard to epistemic deference to other forecasters versus herding that throws out outlier data and makes the aggregate forecast less accurate. If all forecasters just circularly updated until all their individual views were the aggregate view, surely that would be a big mistake. Right?
Do you have a specific forecast for AGI, e.g. a median year or a certain probability within a certain timeframe?
If so, Iâd be curious to know how important AI investment is to that forecast. How much would your forecast change if it turned out the AI industry is in a bubble and the bubble popped, and the valuations of AI-related companies dropped significantly? (Rather than trying to specifically operationalize âbubbleâ, we could just defer the definition of bubble to credible journalists.)
There are a few different reasons youâve cited for credence in near-term AGI â investment in AI companies, the beliefs of certain AI industry leaders (e.g. Sam Altman), the beliefs of certain AI researchers (e.g. Geoffrey Hinton), etc. â and I wonder how significant each of them is. I think each of these different considerations could be spun out into its own lengthy discussion.
I wrote a draft of a comment that addresses several different topics you raised, topic-by-topic, but itâs far too long (2000 words) and Iâll have to put in a lot of work if I want to revise it down to a normal comment length. There are multiple different rabbit holes to go down, like Sam Altmanâs history of lying (which is why the OpenAI Board fired him) or Geoffrey Hintonâs belief that LLMs have near-human-level consciousness.
I feel like going deeper into each individual reason for credence in near-term AGI and figuring out how significant each one is for your overall forecast could be a really interesting discussion. The EA Forum has a little-used feature called Dialogues that could be well-suited for this.
Because timelines are so uncertain (3-15 years?)
In fact, they are much more uncertain than that. AI researchers and superforecasters tend to guess a range between around 20 and 90 years for the median year of AGI.
100 megaseconds (why that unit of measurement?) is 3 years and 2 months. 100 megaseconds from now is March 2029. This is way earlier than the median year of AGI guessed by AI researchers and superforecasters. Itâs even earlier than Metaculus, which is disproportionately used by people who strongly believe in near-term AGI. Metaculus currently says 2033.
(My personal forecast, for whatever itâs worth, is a significantly less than 0.01%, or 1 in 10,000, chance of AGI by the end of 2035 and a ~95% chance that the AI industry is in a bubble that will pop sometime within the next ~5 years.)
[Adapted from this comment.]
Two pieces of evidence commonly cited for near-term AGI are AI 2027 and the METR time horizons graph. AI 2027 is open to multiple independent criticisms, one of which is its use of the METR time horizons graph to forecast near-term AGI or AI capabilities more generally. Using the METR graph to forecast near-term AGI or AI capabilities more generally is not supported by the data and methodology used to make the graph.
Two strong criticisms that apply specifically to the AI 2027 forecast are:
It depends crucially on the subjective intuitions or guesses or the authors. If you donât personally share the authorsâ intuitions, or donât personally trust that the authorsâ intuitions are likely correct, then there is no particular reason to take AI 2027â˛s conclusions seriously.
Credible critics claim that the headline results of the AI 2027 timelines model are largely baked in by the authorsâ modelling decisions, irrespective of what data the model uses. That means, to a large extent, AI 2027â˛s conclusions are not actually determined by the data they use. We already saw with the previous bullet point that the conclusions of AI 2027 are largely a restatement of the authorsâ personal and contestable beliefs. This is another way in which AI 2027â˛s conclusions are, effectively, a restatement of the pre-existing beliefs or assumptions that the authors chose to embed in their timelines model.
AI 2027 is largely based on extrapolating the METR time horizons graph. The following criticisms of the METR time horizons graph therefore extend to AI 2027:
Some of the serious problems and limitations of the METR time horizons graph are sometimes (but not always) clearly disclosed by METR employees. Note the wide difference between the caveated description of what the graph says and the interpretation of the graph as a strong indicator of rapid, exponential improvement in general AI capabilities.
Gary Marcus, a cognitive scientist and AI researcher, and Ernest Davis, a computer scientist and AAAI fellow, co-authored a blog post on the METR graph that looks at how the graph was made and concludes that âattempting to use the graph to make predictions about the capacities of future AI is misguidedâ.
Nathan Witkin, a research writer at NYU Sternâs Tech and Society Lab, published a detailed breakdown of some of the problems with METRâs methodology. He concludes that itâs âimpossible to draw meaningful conclusions from METRâs Long Tasks benchmarkâ and that the METR graph âcontains far too many compounding errors to excuseâ. Witkin calls out a specific tweet from METR, which presents the METR graph in the broad, uncaveated way that the AI 2027 authors interpret it. He calls the tweet âan uncontroversial example of misleading science communicationâ.
Since AI 2027 leans so heavily on this interpretation of the METR graph to make its forecast, it is hard to see how AI 2027 could be credible if its interpretation of the METR graph is not credible.
Itâs worth contrasting AI 2027 and similar forecasts of near-term AGI with expert opinion:
76% of AI experts think it is unlikely or very unlikely that existing approaches to AI, which includes LLMs, will scale to AGI. (See page 66 of the AAAI 2025 survey. See also the preceding two pages about open research challenges in AI â such as continual learning, long-term planning, generalization, and causal reasoning â none of which are about scaling more, or at least not uncontroversially so. If you want an example of a specific, prominent AI researcher who emphasizes the importance of fundamental AI research over scaling, Ilya Sutskever believes that further scaling will be inadequate to get to AGI.)
Expert surveys about AGI timelines are not necessarily reliable, but the AI Impacts survey in late 2023 found that AI researchersâ median year for AGI is 20 to 90 years later than the AI 2027 scenario.
Two overall takeaways:
There are good reasons to be highly skeptical of AI 2027 and the METR time horizons graph as evidence for near-term AGI or for a rapid, exponential increase in general AI capabilities.
Peer review in academic research is designed to catch these sort of flaws prior to publication. This means flaws can be fixed, the claims made can be moderated and properly caveated, or publication can be prevented entirely so that research below a certain threshold of quality or rigour is not given the stamp of approval. (This helps readers know whatâs worth paying attention to and what isnât.) Research published via blogs and self-published reports donât go through academic peer review, and may fall below the standards of academic publishing. In the absence of peer review doing quality control, deeply flawed research, or deeply flawed interpretations of research, may propagate.
Even if there is say, a 30% chance that in the next 5 years even 30-40% of white collar jobs get replaced...that feels like a massive shock to high-income countries way of life, and a major shift in the social contract for the generation of kids who have got into massive debt for university courses only to find substantially reduced market opportunities. That requires substantial state action.
Rather than 30%, I would personally guess the chance is much less than 0.01% (1 in 10,000), possibly less than 0.001% (1 in 100,000) or even 0.0001% (1 in 1 million).
I agree with Huw that thereâs insufficient evidence to say that AI is causing significant or even measurable unemployment right now, and Iâm highly skeptical this will happen anytime soon. Indeed, Iâd personally guess thereâs a ~95% chance thereâs an AI bubble that will pop sometime within the next several years. So far, AI has stubbornly refused to deliver the sort of productivity or automation that has been promised. I think the problem is a fundamental science problem, not something that can be solved with scaling or incremental R&D.
However, letâs imagine a scenario where in a short time, say 5 years, a huge percentage of jobs get automated by AI â automation of white-collar work on a massive scale.[1]
What should governmentsâ response be right now, before this has started to happen, or at least before there is broad agreement it has started to happen? People often talk about this as a gloomy, worrying outcome. I suppose it could turn out to be, but why should that be the default assumption? It would lead to much faster economic growth than developed countries are used to seeing. It might even be record-breaking, unprecedented economic growth. It would be a massive economic windfall. Thatâs a good thing.[2]
To be a bit more specific, when people imagine the sort of AI that is capable of automating white-collar work on the scale youâre describing (30-40% of jobs), they also often imagine wildly high rates of GDP growth, ranging from 10% to 30%.[3][4] The level of growth is supposed to be commensurate with the percentage of labour that AI can automate. I donât know about these specific figures, but the general idea is intuitive.
Surely passing UBI would become much easier once both a) unemployment significantly increased and b) economic growth significantly increased. There would be both a clear problem to address and a windfall of money with which to address it. By analogy, it would have been much harder for governments to pass stimulus bills in January 2020 in anticipation of covid-19. In March 2020, it was much easier, since the emergency was clear. But the covid-19 emergency caused a recession. What if, instead, it had caused an economic boom, and a commensurate increase in government revenues? Surely, then, it would have been even easier to pass stimulus bills.
This is why, even if I accept the AI automation scenario for the sake of argument, I donât worry about the practical, logistical, or fiscal obstacles to enacting UBI in a hurry. Governments can send cheques to people on short notice, as we saw with covid-19. This would presumably be especially true if the government and the economy overall were experiencing a windfall from AI. The sort of administrative bottlenecks we saw in some places during in covid-19 â those could be solved by AI, since weâre stipulating an unlimited supply of digital white-collar workers. Maybe there are further aspects to implementing UBI that would be more complicated than sending people cheques and that couldnât be assisted by AI. What would those be?
The typical concerns raised over UBI are that it would be too expensive, that it would be poorly targeted (i.e. it would be more efficient to run means-tested programs), that it would discourage people from working, and that it would reduce economic growth. None of those apply to this scenario.
If thereâs more complexity in implementing UBI that Iâm not considering, surely in this scenario politicians would quickly become focused on dealing with that complexity, and civil servants (and AI workers) would be assigned to the task. As opposed to something like decarbonizing the economy, UBI seems like it could be implemented relatively quickly and easily, given a sudden emergency that called for it and a sudden windfall of cash. Part of the supposed appeal of UBI is its simplicity relative to means-tested programs like welfare and non-cash-based programs like food stamps and subsidized housing. So, if youâre not satisfied with my answer, maybe you could elaborate on why you think it wouldnât be so easy to figure out UBI in a hurry.
As mentioned up top, I regard this just as an interesting hypothetical, since I think the chance of this actually happening is below 0.01% (1 in 10,000).
Letâs assume, for the sake of argument, that the sort of dire outcomes hypothesized under the heading of AI safety or AI alignment do not occur. Letâs assume that AI is safe and aligned, and that itâs not able to be misused by humans to destroy or take over the world.
Letâs also assume that AI wonât be a monopoly or duopoly or oligopoly but, like today, even open source models that are free to use are a viable alternative to the most cutting-edge proprietary models. Weâll imagine that the pricing power of the AI companies will be put in check by competition from both proprietary and open source models. Sam Altman might become a trillionaire, but only a small fraction of the wealth created by AI will be captured by the AI companies. (As an analogy, think about how much wealth is created by office workers using Windows PCs, and how much of that wealth is captured by Microsoft or by PC manufacturers like HP and Dell, or components manufacturers like Intel.)
Iâm putting these other concerns aside in order to focus on labour automation and technological unemployment, since thatâs the concern you raised.
The specific worries around AI automation people most commonly cite are about wealth distribution, and about people finding purpose and meaning in their lives if thereâs large-scale technological unemployment. Iâll focus on the wealth distribution worry, since the topic is UBI, and your primary concern seems to be economic or material.
Some people are also worried about workers being disempowered if they can be replaced by AI, at the same time that capital owners become much wealthier. If theyâre right to worry about that, then maybe itâs important to consider well in advance of it happening. Maybe workers should act while they still have power and leverage. But itâs a bit of a separate topic, I think, from whether to start implementing UBI now. Maybe UBI would be one of a suite of policies workers would want to enact in advance of large-scale AI automation of labour, but whatâs to prevent UBI from being repealed after workers are disempowered?
For the sake of this discussion, Iâll assume that workers (or former workers) will remain politically empowered, and healthy democracies will remain healthy.
Potlogea, Andrei. âAI and Explosive Growth Redux.â Epoch AI, 20 June 2025, https://ââepoch.ai/ââgradient-updates/ââai-and-explosive-growth-redux.
Davidson, Tom. âCould Advanced AI Drive Explosive Economic Growth?â Coefficient Giving, 25 June 2021, https://ââcoefficientgiving.org/ââresearch/ââcould-advanced-ai-drive-explosive-economic-growth/ââ.
Hence my initial mention of âhigh state capacityâ? But I think itâs fair to call abundance a deregulatory movement overall, in terms of, like⌠some abstract notion of what proportion of economic activity would become more vs less heavily involved with government, under an idealized abundance regime.
I guess it depends what version of abundance youâre talking about. I have in mind the book Abundance as my primary idea of what abundance is, and in that version of abundance, I donât think itâs clear that a politics of abundance would result in less economic activity being heavily involved with government. It might depend how you define that. If laws, regulations, or municipal processes that obstruct construction count as heavy involvement with the government, then that would count for a lot of economic activity, I guess. But if we donât count that and we do count higher state capacity, like more engineers working for the government, then maybe abundance would lead to a bigger government. I donât know.
I think youâre right about why abundance is especially appealing to people of a certain type of political persuasion. A lot of people with more moderate, centrist, technocratic, socially/âculturally less progressive, etc. tendencies have shown a lot of enthusiasm about the abundance label. Iâm not ready to say that they now own the abundance label and abundance just is moderate, centrist, technocratic, etc. If a lot of emos were a fan of my favourite indie rock band, I wouldnât be ready to call it an emo band, even if I were happy for the emosâ support.
There are four reasons I want to deconflate abundance and those other political tendencies:
Itâs intellectually limiting, and at least partially incorrect, to say that abundance is conceptually the same thing as a lot of other independent things that a lot of people who like abundance happen to also like.
I think the coiners and popularizers of abundance deserve a little consideration, and they donât (necessarily, wholeheartedly) agree with those other political tendencies. For instance, Ezra Klein has, to me, been one of the more persuasive proponents of Black Lives Matter for people with a wonk mindset, and has had guests on his podcast from the policy wonk side of BLM to make their case. Klein and Thompson have both expressed limited, tepid support for left-wing economic populist policies, conditional on abundance-style policies also getting enacted.
Iâm personally skeptical of many of the ideas found within those other political tendencies, both on the merits and in terms of whatâs popular or wins elections. (My skepticism has nothing to do with my skepticism of the ideas put forward in the book Abundance, which overall I strongly support and which are orthogonal to the ideas Iâm skeptical of.)
Itâs politically limiting to conflate abundance and these other political tendencies when this isnât intellectually necessary. Maybe moderates enjoy using abundance as a rallying cry for their moderate politics, but conflating abundance and moderate politics makes it a polarized, factional issue and reduces the likelihood of it receiving broad support. I would rather see people try to find common ground on abundance rather than claim it for their faction. Gavin Newsom and Zohran Mamdani are both into abundance, so why canât it have broad appeal? Why try to make it into a factional issue rather than a more inclusive liberal/âleft idea?
Edit: I wrote the above before I saw what you added to your comment. I have a qualm with this:
But, uh, this is the EA Forum, which is in part about describing the world truthfully, not just spinning PR for movements that I happen to admire. And I think itâs an appropriate summary of a complex movement to say that abundance stuff is mostly a center-left, deregulatory, etc movement.
I think it really depends on which version of abundance youâre talking about. If youâre talking about the version in the book Abundance by Ezra Klein and Derek Thompson, or the version that the two authors have more broadly advocated (e.g. on their press tour for the book, or in their writing and podcasts before and after the book was published), then, no, I donât think thatâs an accurate summary of that particular version of abundance.
If youâre referring to the version of abundance advocated by centrists, moderates, and so on, then, okay, it may be accurate to say that version of abundance is centrist, moderate, etc. But I donât want to limit how I define to âabundanceâ to just that version, for the reasons I gave above.
I donât think it makes sense to call it âspinâ or âPRâ to describe an idea in the terms used by the originators of that idea, or in terms that are independently substantively correct, e.g. as supported by examples of progressive supporters of abundance like Mamdani. If your impression of what abundance is comes from centrists, moderates, and so on, then maybe thatâs why you have the impression that abundance simply is centrist, moderate, etc. and that saying otherwise is âuntruthfulâ or âPRâ. There is no âcanonicalâ version of abundance, so to some extent, abundance just means what people who use the term want it to mean. So, that impression of abundance isnât straightforwardly wrong. Itâs just avoidably limited.
Imagine someone complainingâitâs so unfair to describe abundance as a âdemocratâ movement!! Thatâs so off-putting for conservativesâinstead of ostracising them, we should be trying to entice them to adopt these ideas that will be good for the american people! Like Montana and Texas passing great YIMBY laws, Idaho deploying modular nuclear reactors, etc. In lots of ways abundance is totally coherent with conservative goals of efficient government services, human liberty, a focus on economic growth, et cetera!!
To the extent people care what Abundance says in deciding what abundance is, one could quote from the first chapter of the book, specifically the section âA Liberalism That Buildsâ, which explicitly addresses this topic.
This might be a correct description of some people who have adopted the abundance label, but itâs not a correct description of the book Abundance or its authors, Ezra Klein and Derek Thompson, who coined and popularized abundance (or abundance liberalism) as a political term and originated the abundance movement thatâs playing out in U.S. politics right now. Abundance is deregulatory on NIMBY restrictions to building housing and environmental bills that are perversely used to block solar and wind projects. However, it also advocates for the government of California to in-house the engineering of its high-speed rail project rather than try to outsource it to private contractors. There is a chapter on government science funding, of which it is strongly in favour. Abundance is in favour of the government getting out of the way, or deregulating, in some areas, such as housing, but in other areas, itâs in favour of, for lack of a better term, big government.
Others are of course free to read Abundance and run with it any direction they like, even if the authors disagree with it. Nobody owns the abundance label, so people can use it how they like. But I think the framing of abundance as necessarily or inherently moderate, technocratic, or deregulatory is limiting. Thatâs one particular way that some people think about abundance, but not everybody has to think of it that way, and not even the originators of the idea do. The progressive mayor of New York City, Zohran Mamdani, is a fan of Abundance and recently voiced his support for YIMBY housing reform in the state of New York. Abundance is not synonymous with either the moderate or progressive wings of the Democratic Party; itâs a set of ideas that is compatible with either a moderate or progressive political orientation.
Klein and Thompson, and of course Mamdani, are not âunified in their opposition to lefty economic proposalsâ.
I think saying that abundance implies moderate politics or technocracy is not only limiting and at least partially inaccurate, it also encourages progressives to oppose abundance when, as weâve seen in Mamdaniâs case, it is compatible with progressivism and largely independent from and orthogonal to (in the figurative sense) the disagreements between moderates and progressives.
technocratic⌠more moderate
Abundance or abundance liberalism originated with the journalists Ezra Klein and Derek Thompson in their book Abundance (which was my favourite non-fiction book of 2025). Since Klein and Thompson popularized the term abundance in Democratic politics, a number of others have latched onto the term and assigned their own meanings to it. Klein and Thompson themselves do not advocate for the Democratic Party to become more moderate. Maybe some people who have picked up the abundance label do. But Klein, for instance, argues that the Democratic Party needs to allow for ideological diversity based on geography. That is, it should be a party that can include both a mayor like Zohran Mamdani in New York City and a senator like Joe Manchin in West Virginia. Klein and Thompsonâs own views are not more moderate than the Democratic Party currently is or has been recently.
One of the popular criticisms voiced against Abundance following the publication of the book is that economic populist policies poll better than abundance-style policies. Klein and Thompsonâs reply is that these policies are not incompatible, so it would be possible for Democratic politicians to do both. A mistake many people have made in interpreting abundance liberalism is that itâs not a complete theory of politics or a complete political worldview. It doesnât answer every question Democratic politicians need to answer.
Abundance is specifically focused on governments providing people with an abundance of the things they need and expect: housing, infrastructure (e.g., highways and high-speed rail), government administrative services (e.g. timely processing of claims for unemployment benefits), and innovations in science, technology, and medicine that translate into practical applications in the real world. The book offers a diagnosis for why governments, particularly governments run by Democrats, have so often failed to provide people with these things. It also offers ideas for improving Democratsâ governing performance in the aforementioned areas. Other topics are outside of scope, although Klein and Thompson have also expressed their opinions on those topics in places other than their book.
Itâs easy to see how a somewhat complex and subtle argument like this gets simplified into âthe Democratic Party should become more moderate and technocraticâ. But that simplified version misses a lot.
(I wrote a long comment here addressing objections to abundance liberalism and to Coefficient Givingâs work in that area, specifically housing policy reform and metascience.)
The 80,000 Hours video on AI 2027 is missing important caveats. These are the sort of caveats that deserve to emphasized and foregrounded in any discussion of AI 2027.
The three major caveats that should be made about AI 2027 are:
It depends crucially on the subjective intuitions or guesses or the authors. If you donât personally share the authorsâ intuitions, or donât personally trust that the authorsâ intuitions are likely correct, then there is no particular reason to take AI 2027â˛s conclusions seriously. (By the time they finish the 80,000 Hours video, are viewers aware this is the case?)
Credible critics claim that the headline results of the AI 2027 timelines model are largely baked in by the authorsâ modelling decisions, irrespective of what data the model uses. That means, to a large extent, AI 2027â˛s conclusions are not actually determined by the data they use. We already saw with (1) that the conclusions of AI 2027 are largely a restatement of the authorsâ personal and contestable beliefs. This is another way in which AI 2027â˛s conclusions are, effectively, a restatement of the pre-existing beliefs or assumptions that the authors chose to embed in their timelines model.
AI 2027 is largely based on extrapolating the METR time horizons graph, which has serious problems and limitations, some of which are sometimes (but not always) clearly disclosed by METR employees. Gary Marcus, a cognitive scientist and AI researcher, and Ernest Davis, a computer scientist and AAAI fellow, co-authored a blog post on the METR graph that looks at how the graph was made and concludes that âattempting to use the graph to make predictions about the capacities of future AI is misguidedâ. Nathan Witkin, a research writer at NYU Sternâs Tech and Society Lab, published a detailed breakdown of some of the problems with METRâs methodology. He concludes that itâs âimpossible to draw meaningful conclusions from METRâs Long Tasks benchmarkâ and that the METR graph âcontains far too many compounding errors to excuseâ. Witkin calls out a specific tweet from METR, which presents the METR graph in the broad, uncaveated way that the AI 2027 authors interpret it. He calls the tweet âan uncontroversial example of misleading science communicationâ. Since AI 2027 leans so heavily on this interpretation of the METR graph to make its forecast, it is hard to see how AI 2027 could be credible if its interpretation of the METR graph is not credible.
The 80,000 Hours video insinuates that most AI experts agree with the core assumptions or conclusions of AI 2027, but there is evidence to the contrary:
76% of AI experts think it is unlikely or very unlikely that existing approaches to AI, which includes LLMs, will scale to AGI. (See page 66 of the AAAI 2025 survey. See also the preceding two pages about open research challenges in AI â such as continual learning, long-term planning, generalization, and causal reasoning â none of which are about scaling more, or at least not uncontroversially so. If you want an example of a specific, prominent AI researcher who emphasizes the importance of fundamental AI research over scaling, Ilya Sutskever believes that further scaling will be inadequate to get to AGI.)
Expert surveys about AGI timelines are not necessarily reliable, but the AI Impacts survey in late 2023 found that AI researchersâ median year for AGI is 20 to 90 years later than the AI 2027 scenario.
I canât find survey data on this, but I get the impression there is a diversity of opinions among AI experts on how existentially dangerous AGI would be. For example, the Turing Award-winning AI researchers Yann LeCun and Richard Sutton both have a very different perspective on this than the AI 2027 authors. Both have been outspoken in their belief that that the AI safety/âAI alignment communityâs perspective on this topic is misguided.
Iâm not sure if 80,000 Hours is interested in making videos that try to explain complexities like these, but, personally, I would love to see that. I think there is immense importance and value in helping viewers understand how conclusions are reached, particularly when they are radical and contentious. Many viewers would surely disagree with the reasoning behind the conclusions that the 80,000 Hours video presents if it were clearly explained to them.
There is a kernel of truth in this; some version of this argument is a good argument. But the devil is in the details.
If youâre not a scientist or a person with relevant expertise and you feel inclined to disagree with the ~97-99.9% of climate scientists who think anthropogenic climate change is happening, you better adopt a boatload of epistemic humility. In practice, many non-scientists or non-experts disagree with the scientific consensus. Iâm not aware of one example of such a person adopting the appropriate level of epistemic humility.
On the other hand, people invoke this same epistemic humility argument when it comes to fringe stuff like UFOs, ESP, and conspiracy theories. A lot of smart people believe in fringe stuff and, boy howdy, have they ever spent a lot of time researching it and thinking about it.
I think the difference between these two examples of epistemic humility arguments is some form of Carl Sagan-esque scientific skepticism.[1]
The difference also has to do with the idea of expertise, the idea of science (both the scientific method and scientific institutions or communities), and the idea of academia. This goes beyond the general concept of disagreement, or the general concept of some number of people holding an opinion. Climate scientists are an expert community. People who believe in alien UFOs are not an expert community. There is something to be said here about knowledge production and how it happens, which is deeper than just some group of people believing something.
Thereâs lots to be said here about epistemology, and about science or scientific skepticism as an approach to knowledge versus less reliable approaches. For instance, around unfalsifiable claims. The claim that there is a 2% chance that AGI or superintelligence will be developed within the next decade is unfalsifiable. Even if the next decade plays out exactly how someone who assigns a 0.002% (or 1 in 50,000) probability to AGI would expect, a person who assigned a 2% or 20% or even 60% probability could still think they were right to do so. Itâs not exactly a scientific question. Itâs also outside the scope of the sort of forecasting questions that the forecasting research literature can support. This is strongly disanalogous to epistemic deference toward scientific experts.[2]
In my view, itâs not a coincidence that a hugely disproportionate number of people who believe in a high level of existential risk from near-term AGI a) are strongly skeptical of mainstream, institutional science and strongly advocate fringe, dubious, borderline, or minority scientific views, b) believe in fringe stuff like aliens, conspiracy theories, or ESP, or c) have joined a cult.
The rate of cult membership among people who believe in a high level of existential risk from near-term AGI has got to be something like 10,000x higher than the general population. The LessWrong community in the Bay Area is, I think, around 1,000 people (if not less) and it has started half a dozen cults in 17 years. Former members have committed as many murders in that timeframe. Small towns with much larger populations and much longer histories typically have never had a single cult (and very rarely have cults that murder people).
Clearly, the argument for epistemic humility that applies to say, climate scientists, doesnât apply cleanly to this community. If there is strong independent evidence of a community frequently holding extreme, false beliefs, an appeal to their overall rationality doesnât work. (Also, LessWrongâs primary thought leader has confidently made incorrect predictions about the timing of AGI and about the imminent existential dangers from advanced technology in the past â another reason for skepticism.)
The EA community is also directly implicated. One of the aforementioned cults, Leverage Research, ran the Centre for Effective Altruism for a few years, and organized the first EA conferences.
The number of people who believe in Hinduism is vastly larger than the number who believe in near-term AGI: 15% of the world population, 1.2 billion people. Far more university professors, scientists, philosophers, and so on believe in Hinduism than near-term AGI. Should you assign more than a 1% chance to the probability that Hinduism is a correct theory of metaphysics, cosmology, consciousness, and morality? The stakes are also very large in this case.
Changing the simulation hypothesis from a simulation of a world full of people to a simulation of an individual throws the simulation argument out the window. Here is how Sean Carroll articulates the first three steps of the simulation argument:
We can easily imagine creating many simulated civilizations.
Things that are that easy to imagine are likely to happen, at least somewhere in the universe.
Therefore, there are probably many civilizations being simulated within the lifetime of our universe. Enough that there are many more simulated people than people like us.
The simulation argument doesnât apply to you, as an individual. Unless you think that you, personally, are going to create a simulation of a world or an individual â which obviously youâre not.
Changing the simulation hypothesis from a world-scale simulation to an individual-scale simulation also doesnât change the other arguments against the simulation hypothesis:
The bottoming out argument. This is the one from Sean Carroll. Even if we supposed you, personally, were going to create individual-scale simulations in the future, eventually a nesting cascade of such simulations would exhaust available computation in the top-level universe, i.e. the real universe. The bottom-level simulations within which no further simulations are possible would outnumber higher-level ones. The conclusion of the simulation argument contradicts a necessary premise.[1]
The ethical argument. It would be extremely unethical to imprison an individual in a simulation without their consent, especially a simulation with a significant amount of pain and suffering that the simulators are programming in. Would you create an individual-scale simulation even of an unrealistically pleasant life, let alone a life with significant pain and suffering? If we had the technology to do this today, I think it would be illegal. It would be analogous to false imprisonment, kidnapping, torture, or criminal child abuse (since you are creating this person).
The computational waste argument. The amount of computation required to make an individual-scale simulation would require at least as much computation as creating a digital mind in the real universe. In fact, it would require more computation, since you also to have to simulate the whole world around the individual, not just the individual themselves. If the simulators think marginally, they would prefer to use these resources to create a digital mind in the real universe or put them to some other, better use.
If the point of the simulation is to cater it to the individualâs preferences, we should ask:
a) Why isnât this actually happening? Why is there so much unnecessary pain and suffering and unpleasantness in every individualâs life? Why simulate the covid-19 pandemic?
b) Why not cater to the individualâs fundamental and overriding preference not to be in a simulation?
c) Why not put these resources toward any number of superior uses that must surely exist?[2]
Perhaps most importantly, changing the simulation hypothesis from world-scale to individual-scale doesnât change perhaps the most powerful counterargument to the simulation hypothesis:
The unlimited arbitrary, undisprovable hypotheses argument. There is no reason to think the simulation hypothesis makes any more sense or is any more likely to be true than the hypothesis that the world you perceive is an illusion created by an evil demon or a trickster deity like Loki. There are an unlimited number of equally arbitrary and equally unjustified hypotheses of this type that could be generated. In my previous comment, I argued that versions of the simulation hypotheses in which the laws of physics or laws of nature are radically different in the real universe than in the simulation are supernatural hypotheses. Versions of the simulation hypothesis that assume real universe physics is the same as simulation physics suffer from the bottoming out argument and the computational waste argument. So, either way, the simulation hypothesis should be rejected. (Also, whether the simulation has real universe physics or not, the ethical argument applies â another reason to reject it.)
This argument also calls into question why we should think simulation physics is the same as real universe physics, i.e. why we should think the simulation hypothesis makes more sense as a naturalistic hypothesis than a supernatural hypothesis. The simulation hypothesis leans a lot on the idea that humans or post-humans in our hypothetical future will want to create âancestor simulationsâ, i.e. realistic simulations of the simulatorsâ past, which is our present. If there were simulations, why would ancestor simulations be the most common type? Fantasy novels are about equally popular as historical fiction or non-fiction books about history. Would simulations skew toward historical realism significantly more than books currently do? Why not simulate worlds with magic or other supernatural phenomena? (Maybe we should conclude that, since this is more interesting, ghosts probably exist in our simulation. Maybe God is simulated too?) The âancestor simulationâ idea is doing a lot of heavy lifting; itâs not clear that this is in any way a justifiable assumption rather than an arbitrary one. The more I dig into the reasoning behind the simulation hypothesis, the more it feels like Calvinball.[3]
The individual-scale simulation hypothesis also introduces new problems that are unique to it:
Simulation of other minds. If you wanted to build a robot that could perfectly simulate the humans you know best, the underlying software would need to be a digital mind. Since, on the individual-scale simulation hypothesis, you are a digital mind, then the other minds in the simulation â at least the ones you know well â are as real as you are. You could try to argue that these other minds only need to be partially simulated. For example, the mind simulations donât need to be running when you arenât observing or interacting with these people. But then why donât these people report memory gaps? If the answer is that the simulation fills in the gaps with false memories, what process continually generates new false memories? Why would this process be less computationally expensive than just running the simulation normally? (You could also try to say that consciousness is some kind of switch that can be flipped on or off for some simulations but not others. But I canât think of any theory of consciousness this would be compatible with, and itâs a problem for the individual-scale simulation hypothesis if it just starts making stuff up ad hoc to fit the hypothesis.)
If we decide that at least the people you know well must be fully simulated, in the same way you are, then what about the people they know well? What about the people who they know well know well? If everyone in the world is connected through six degrees of separation or fewer, then it seems like individual-scale simulations are actually impossible and all simulations must be world-scale simulations.
Abandoning the simulation of history at large scale. Individual-scale simulations donât provide the same informational value that world-scale simulations might. When people talk about why âancestor simulationsâ would supposedly be valuable or desired, they usually appeal to the notion of simulating historical events on a large scale. This obviously wouldnât apply to individual-scale simulations. To the extent credence toward the simulation hypothesis depends on this, an individual-scale simulation hypothesis may be even less credible than a world-scale simulation hypothesis.
The Wikipedia page on the simulation hypothesis notes that itâs a contemporary twist on a centuries-old if not millennia-old idea. Weâve replaced dreams and evil demons with computers, but the underlying idea is largely the same. The reasons to reject it are largely the same, although the simulation argument has some unique weaknesses. That page is a good resource for finding still more arguments against the simulation hypothesis.[4]
Carroll, who is a physicist and cosmologist, also criticizes the anthropic reasoning of the simulation argument. I recommend reading his post, itâs short and well-written.
You could try to argue that, despite societyâs best efforts, it will be impossible to supress a large number of simulations from being created. Pursuing this line of argument re quires speculating about the specific details of a distant, transhuman or post-human future. Would an individual creating a simulation be more like an individual today operating a meth lab or launching a nuclear ICBM? Iâm not sure we can know the answer to this question. If dangerous or banned technologies canât be controlled, what does this say about existential risk? Will far future, post-human terrorists be able to deploy doomsday devices? If so, that would undermine the simulation argument. (Will post-humans even have the desire to be terrorists, or is that a defect of humanity?)
Related to this are various arguments that the simulation argument is self-defeating. We infer things about the real universe from our perceived universe. We then conclude that our perceived universe is a simulation. But, if it is, this undermines our ability to infer anything about the real universe from our perceived universe. In fact, this undermines the inference that our perceived universe is a simulation within a real universe. So, the simulation argument defeats itself.
In addition to all the above, I would be curious to hear empirical, scientific arguments about the amount of computation that might be required for world-scale simulations, which would be partly applicable to individual-scale simulations. Obviously, our universe canât run a full-scale, one-to-one simulation of our universe with perfect fidelity â that would require more computation, matter, and energy than our universe has. If you only simulate the solar system with perfect fidelity, you can pare that down a lot. You can make other assumptions to pare down the computation required. Itâs much less important than all the arguments and considerations described above, but if we get a better understanding of approximately how difficult or costly a world-scale simulation might be, that could help put some considerations like computational waste in perspective.
Humourously?!!
Thank you so much for this.
Breaking character, if you want to see just how strongly people are biased toward misjudging their own track record, itâs instructive to have friends and family who do retail stock picking (essentially gambling). I have strongly advocated passive ETF investing based on the wealth of research and analysis that supports it. Even when theyâre well-informed about the case for passive ETF investing, people still think they can beat the market. And the thing about going up against the market is, your performance is absolutely quantifiable! In a way that vibes-y predictions about AI arenât. Yet, even being consummately quantifiable, people who pick stocks donât benchmark their performance, or do it selectively when their stocks are up (obviously giving a biased impression), or somehow justify or rationalize or explain away why theyâre actually winning.
I donât trust for a second that someone judging their own performance on informal, selectively remembered, largely subjective AI predictions is doing a fair job. Even when people like Dario Amodei or Ray Kurzweil have publicly made specific AI or technology predictions that turned out to be unambiguously dead wrong, they have subsequently twisted and contorted the truth in order to make themselves right â lied, essentially, or else fooled themselves. I do not trust people to grade their own homework and to have the result be scientific-quality evidence.
The high-impact/âlow-probability reasoning, on the other hand, is completely sound and demonstrates why deep-pocketed donors should give me $1 million/âyear.
2027 is far too conservative. You can clearly see in this graph from the scientific report âSituational Awarenessâ by Leopold Aschenbrenner that AGI was deployed in 2023:
I believe that GPT-5 is already an ASI, but we will need a super-ASI to defend against bad nanobots. The first super-ASI will be Gronk 5, which Elon Musk says is coming out in 2 months.
Just as long as I get my funding⌠The lightcone depends on it...
All the things you mentioned arenât uniquely evidence for the simulation hypothesis but are equally evidence for a number of other hypotheses, such as the existence of a supernatural, personal God who designed and created the universe. (There are endless variations on this hypothesis, and we could come up endless more.)
The fine-tuning argument is a common argument for the existence of a supernatural, personal God. The appearance of fine-tuning supports this conclusion equally as well it supports the simulation hypothesis.
Some young Earth creationists believe that dinosaur fossils and other evidence of an old Earth were intentionally put there by God to test peopleâs faith. You might also think that God tests our faith in other ways, or plays tricks, or gets easily bored, and creates the appearance of a long history or a distant future that isnât really there. (I also think itâs just not true that this is the most interesting point in history.)
Similarly, the book of Genesis says that God created humans in his image. Maybe he didnât create aliens with high-tech civilizations because heâs only interested in beings with high technology made in his image.
It might not be God who is doing this, but in fact an evil demon, as Descartes famously discussed in his Meditations around 400 years ago. Or it could be some kind of trickster deity like Loki who is neither fully good or fully evil. There are endless ideas that would slot in equally well to replace the simulation hypothesis.
You might think the simulation hypothesis is preferable because itâs a naturalistic hypothesis and these are supernatural hypotheses. But this is wrong, the simulation hypothesis is a supernatural hypothesis. If there are simulators, the reality they live in is stipulated to have different fundamental laws of nature, such as the laws of physics, than exist in what we perceive to be the universe. For example, in the simulatorsâ reality, maybe the fundamental relationship between consciousness and physical phenomena such as matter, energy, space, time, and physical forces is such that consciousness can directly, automatically shape physical phenomena to its will. If we observed this happening in our universe, we would describe this as magic or a miracle.
Whether you call them âsimulatorsâ or âGodâ or an âevil demonâ or âLokiâ, and whether you call it a âsimulationâ or an âillusionâ or a âdreamâ, these are just different surface-level labels for substantially the same idea. If you stipulate laws of nature radically other than the ones we believe we have, what youâre talking about is supernatural.
If you try to assume that the physics and other laws of nature in the simulatorsâ reality is the same as in our perceived reality, then the simulation argument runs into a logical self-contradiction, as pointed out by the physicist Sean Carroll. Endlessly nested levels of simulation means computation in the original simulatorsâ reality will run out. Simulations at the bottom of the nested hierarchy, which donât have enough computation to run still more simulations inside them, will outnumber higher-level simulations. Since the simulation argument says, as one of its key premises, that in our perceived reality we will be able to create simulations of worlds or universes filled with many digital minds, but the simulation hypothesis implies this is actually impossible, then the simulation argumentâs conclusion contradicts one of its premises.
There are other strong reasons to reject the simulation argument. Remember that a key premise is that we ourselves or our descendants will want to make simulations. Really? Theyâll want to simulate the Holocaust, malaria, tsunamis, cancer, cluster headaches, car crashes, sudden infant death syndrome, and Guantanamo Bay? Why? On our ethical views today, we would not see this as permissible, but rather the most grievous evil. Why would our descendants feel differently?
Less strongly, computation is abundant in the universe but still finite. Why spend computation on creating digital minds inside simulations when there is always a trade-off between doing that and creating digital minds in our universe, i.e. the real world? If we or our descendants think marginally and hold as one of our highest goals to maximize the number of future lives with a good quality of life, using huge amounts of computation on simulations might be seen as going against that goal. Plus, there are endlessly more things we could do with our finite resource of computation, most we canât imagine today. Where would creating simulations fall on the list?
You can argue that creating simulations would be a small fraction of overall resources. Iâm not sure thatâs actually true; I havenât done the math. But just because something is a small fraction of overall resources doesnât mean it will be likely be done. In an interstellar, transhumanist scenario, our descendants could create a diamond statue of Hatsune Miku the size of the solar system and this would take a tiny percentage of overall resources, but that doesnât mean it will likely happen. The simulation argument specifically claims that making simulations of early 21st century Earth will interest our descendants more than alternative uses of resources. Why? Maybe theyâll be more interested in a million other things.
Overall, the simulation hypothesis is undisprovable but no more credible than an unlimited number of other undisprovable hypotheses. If something seems nuts, it probably is. Initially, you might not be able to point out the specific logical reasons itâs nuts. But thatâs to be expected â the sort of paradoxes and thought experiments that get a lot of attention (that âgo viralâ, so to speak) are the ones that are hard to immediately counterargue.
Philosophy is replete with oddball ideas that are hard to convincingly refute at first blush. The Chinese Room is a prime example. Another random example is the argument that utilitarianism is compatible with slavery. With enough time and attention, refutations may come. I donât think oneâs inability to immediately articulate the logical counterargument is a sign that an oddball idea is correct. Itâs just that thinking takes time and, usually, by the time an oddball idea reaches your desk, itâs proven to be resistant to immediate refutation. So, trust that intuition that something is nuts.
My experience is similar. LLMs are powerful search engines but nearly completely incapable of thinking for themselves. I use these custom instructions for ChatGPT to make it much more useful for my purposes:
When asked for information, focus on citing sources, providing links, and giving direct quotes. Avoid editorializing or doing original synthesis, or giving opinions. Act like a search engine. Act like Google.
There are still limitations:
You still have to manually check the cited links to verify the information yourself.
ChatGPT is, for some reason, really bad at actually linking to the correct webpage itâs quoting from. This wastes time and is frustrating.
ChatGPT is limited to short quotes and often gives even shorter quotes than necessary, which is annoying. It often makes it hard to understand what the quote actually even says, which almost defeats the purpose.
Itâs common for ChatGPT to misunderstand what itâs quoting and take something out of context, or it quotes something inapplicable. This often isnât obvious until you actually go check the source (especially with the truncated quotes). You can get tricked by ChatGPT this way.
Every once in a while, ChatGPT completely fabricates or hallucinates a quote or a source.
The most one-to-one analogy for LLMs in this use case is Google. Google is amazingly useful for finding webpages. But when you Google something (or search on Google Scholar), you get a list of results, many of which are not what youâre looking for, and you have to pick which results to click on. And then, of course, you actually have to read the webpages or PDFs. Google doesnât think for you; itâs just an intermediary between you and the sources.
I call LLMs SuperGoogle because they can do semantic search on hundreds of webpages and PDFs in a few minutes while youâre doing something else. LLMs as search engines is a geniune innovation.
On the other hand, when Iâve asked LLMs to respond to the reasoning or argument in a piece of writing or even just do proofreading, they have given incoherent responses, e.g. making hallucinatory âcorrectionsâ to words or sentences that arenât in the text theyâve been asked to review. Run the same text by the same LLM twice and it will often give the opposite opinion of the reasoning or argument. The output is also often self-contradictory, incoherent, incomprehensibly vague, or absurd.
âCruxâ is its own word; it isnât short for âcrucial considerationâ or anything else. From Merriam-Webster:
In Latin, crux referred literally to an instrument of torture, often a cross or stake, and figuratively to the torture and misery inflicted by means of such an instrument. Crux eventually developed the sense of âa puzzling or difficult problemâ; that was the first meaning that was used when the word entered English in the early 18th century. Later, in the late 19th century, crux began to be used more specifically to refer to an essential point of a legal case that required resolution before the case as a whole could be resolved. Today, the verdict on crux is that it can be used to refer to any important part of a problem or argument, inside or outside of the courtroom.
The EA Forum wiki also notes that a âcrucial considerationâ is a related concept, not a synonym of âcruxâ or a longhand version of the term.
Where do you get that 5-20x figure from?
I recall Elon Musk once said the goal was to get to an average of one intervention per million miles of driving. I think this is based on the statistic of one crash per 500,000 miles on average.
I believe interventions currently happen more than once per 100 miles on average. If so, and if one intervention per million miles is what Tesla is indeed targeting, then Tesla is more than 10,000x off from its goal.
There are other ways of measuring Teslaâs FSD softwareâs performance compared to average human driving performance and getting another number. I am skeptical it would be possible to use real, credible numbers and come to the conclusion that Tesla is currently less than 100x away from human-level driving.
I very much doubt that Hardware 5/âAI5 is going to provide what it takes for Tesla to achieve SAE Level 4â5 autonomy at human-level or better performance, or that Tesla will achieve that goal (in any robust, meaningful sense) within the next 2 years. I still think what I said is true â Tesla, internally, would have evidence of this if it were true (or would be capable of obtaining it), and would be incentivized to show off that evidence.
Andrej Karpathy understands this topic better than almost anyone else in the world, and he is clear that he thinks fully autonomous driving is not solved (at Tesla, Waymo, or elsewhere) and thereâs long way to go still. Thereâs good reason to listen to Karpathy on this.
I also very much doubt that the best AI models in 2 years will be capable of writing 90% of commercial, production code, let alone that this will happen within six months. I think thereâs essentially no chance of this happening in 2026. As far as I can see, there is no good evidence currently available that would suggest this is starting to happen or should be possible soon. Extrapolating from performance on narrow, contrived benchmark tasks to real world performance is just a mistake. And the evidence about real world use of AI for coding does not support this.
Misinformation and clickbait are also common ways to get attention. I wouldnât recommend those tactics, either.
The way that a lot of people get attention online is fundamentally destructive. It gets them clicks and ad revenue, but it doesnât help cause positive change in the world.
I donât think it makes sense to justify manipulative, dishonest, or deceptive tactics like ragebait on the basis that they are good at getting attention. This is taking a business model from social media, which in some cases is arguably like digital cigarettes, and inappropriately applying it to animal advocacy. If the goal is to get people to scroll a lot and show them a lot of ads, sure, copy the tactics used in social media. But that isnât the goal here.
One form of ragebait is when you generate rage at a target other than yourself, but another form is when you bait people into getting angry at you (e.g. by expressing an insincere opinion) because that drives engagement, and engagement gets you paid. Making people angry at you is especially not applicable to animal advocacy.