Matthew_Barnett

Karma: 4,234

Matthew_Barnett May 14, 2025, 3:50 AM
2 points
0 ∶ 0
in reply to: CarlShulman’s comment on: What if we just…didn’t build AGI? An Argument Against Inevitability
Perhaps I overstated some of my claims or was unclear. So let me try to be more clear about my basic thesis. First of all, I agree that in the most basic model of the situation, being slightly ahead of a competitor can be the decisive factor between going bankrupt and making enormous profits. This creates a significant personal incentive to race ahead, even if doing so only marginally increases existential risk overall. As a result, AI labs may end up taking on more risk than they would in the absence of such pressure. More generally, I agree that without competition—whether between states or between AI companies—progress would likely be slower than it currently is.
My main point, however, is that these effects are likely not strong enough to justify the conclusion that the socially optimal pace of AI R&D is meaningfully slower than the current pace we in fact observe. In other words, I’m not convinced that what’s rational from an individual actor’s perspective diverges greatly from what would be rational from a collective or societal standpoint.
This is the central claim underlying my objection: if there is no meaningful difference between what is individually rational and what is collectively rational, then there is little reason to believe we are facing a tragedy-of-the-commons scenario as suggested in the post.
To sketch a more complete argument here, I would like to make two points:
First, while some forces incentivize speeding up AI development, others push in the opposite direction. Measures like export controls, tariffs, and (potentially) future AI regulations can slow down progress. In these cases, the described dynamic flips: the global costs of slowing down are shared, while the political rewards—such as public credit or influence—are concentrated among the policymakers or lobbyists who implement the slowdown.
Second, as I’ve mentioned, a large share of both the risks and benefits of AI accrue directly to those driving its development. This alignment of incentives gives them a reason to avoid reckless acceleration that would dramatically increase risk.
As a testable prediction of my view, we could ask whether AI labs are actively lobbying for slower progress internationally. If they truly preferred collective constraint but felt compelled to move forward individually, we would expect them to support measures that slow everyone down—while personally moving forward as fast as they can in the meantime. However, to my knowledge, such lobbying is not happening. This suggests that labs may not, in fact, collectively prefer significantly slower development.

Matthew_Barnett May 13, 2025, 4:28 AM
3 points
1 ∶ 0
in reply to: Nate Sharpe’s comment on: What if we just…didn’t build AGI? An Argument Against Inevitability
There are psychological pressures that can lead to motivated reasoning on both sides of this issue. On the pro-acceleration side, individuals may be motivated to downplay or dismiss the potential risks and downsides of rapid AI development. On the other side, those advocating for slowing or pausing AI progress may be motivated to dismiss or undervalue the possible benefits and upsides. Because both the risks and the potential rewards of AI are substantial, I don’t see a compelling reason to assume that one side must be much more prone to denial or bias than the other.
At most, I see a simple selection effect: the people most actively pushing for faster AI development are likely those who are least worried about the risks. This could lead to a unilateralist curse, where the least concerned actors push capabilities forward despite a high risk of disaster. But the opposite scenario could also happen, if the most concerned actors are able to slow down progress for everyone else, delaying the benefits of AI unacceptably. Whether you should care more about the first or second scenario depends on your judgement of whether rapid AI progress is good or bad overall.
Ultimately, I think it’s more productive to frame the issue around empirical facts and value judgments: specifically, how much risk rapid AI development actually introduces, and how much value we ought to place on the potential benefits of rapid development. I find this framing more helpful, not only because it identifies the core disagreement between accelerationists and pause advocates, but also because I think it better accounts for the pace of AI development we actually observe in the real world.

Matthew_Barnett May 12, 2025, 10:54 PM
4 points
0 ∶ 1
in reply to: CarlShulman’s comment on: What if we just…didn’t build AGI? An Argument Against Inevitability
If you look at the leaders of major AI companies you see people like Elon Musk and others who are concerned with getting to AGI before others who they distrust and fear. They fear immense power in the hands of rivals with conflicting ideologies or in general.
Musk is a vivid example of the type of dynamic you’re describing, but he’s also fairly unusual in this regard. Sundar Pichai, Satya Nadella, and most other senior execs strike me as more like conventional CEOs: they want market share, profits, and higher margins, but they’re not seeking the kind of hegemonic control that would justify accepting a much higher p(doom). If the dominant motive is ordinary profit maximization rather than paranoid power-seeking, then my original point stands: both the upside (huge profit streams) and the downside (self-annihilation) accrue to the people pushing AI forward, so the private incentives already internalize a large chunk of the social calculus.
Likewise, it’s true that governments often seek hegemonic control over the world in a way that creates destructive arms races, but even in the absence of such motives, there would still be a strong desire among humans to advance most technologies to take advantage of the benefits.
The most important fact here is that AI has an enormous upside: people would still have strong reasons to aggressively seek it to obtain life extension and extraordinary wealth even in the absence of competitive dynamics—unless they were convinced that the risk from pursuing that upside was unacceptably high (which is an epistemic consideration, not a game-theoretic trap).
Power, including the power to coerce or harm others, and relative standing, are more important there than access to advanced medicine or broad prosperity for the competitive dynamics.
You may have misread me here. I’m not claiming that AI labs are motivated by a desire to create broad prosperity. They certainly do care about “power,” but the key question is whether they’re primarily driven by the type of zero-sum, adversarial power-seeking you described. I’m skeptical that this is the dominant motive. Instead, I think ordinary material incentives likely play a larger role.
And the epistemic variation makes it all worse, where the most unconcerned players set a higher baseline risk spontaneously.
The unilateralist’s curse is primarily worrisome when the true value of an initiative is negative; for good projects it usually helps them proceed. Moreover, if good projects can be vetoed (e.g., via regulators), this creates a reverse curse that can block beneficial progress. Ultimately I don’t see a strong argument for a trap that forces AI to advance forward at everyone’s expense. We aren’t really at the mercy of Moloch here. The main story is that other people simply assign (a) lower extinction probabilities or (b) higher weight to the upside potential than EAs tend to. That is a rather ordinary disagreement over facts and values, not a real curse.

Matthew_Barnett May 11, 2025, 11:06 PM
6 points
0 ∶ 3
on: What if we just…didn’t build AGI? An Argument Against Inevitability
This confluence of factors creates a powerful coordination problem. Everyone might privately agree that racing headlong into AGI without robust safety guarantees is madness, but nobody wants to be the one who urges caution while others surge ahead.
I dispute that we’re facing a coordination problem in the sense you described. Chad Jones’ paper is a helpful comparison here to illustrate AI racing dynamics.
His model starts from the observation that the very same actors who might enjoy benefits from faster AI also face the extinction hazard that faster AI could bring. In his social-planner formulation, the idea is to pick an R&D pace that equates the marginal gain in permanent consumption growth with the marginal rise in a one-time extinction probability^[1]; when the two curves cross, that is the point you stop. Nothing in the mechanism lets one party obtain the upside while another bears the downside of existential risk, so the familiar logic of a classic tragedy of the commons—”If I restrain myself, someone else will defect and stick me with the loss”—doesn’t apply. The optimal policy is simply to pick whatever pace of development makes the risk–reward ratio come out favorable.
Why is that a realistic way to view today’s situation? First, extinction risk is highly non-rival: if an unsafe system destroys the world, it wipes out everyone, including the engineers and investors who pushed the system forward. They cannot dump that harm on an outside group the way a factory dumps effluent into a river. Second, the primary benefits—higher incomes and earlier biomedical breakthroughs—are also broadly shared; they are not gated to the single lab that crosses the finish line first. Because both tails of the distribution are so widely spread, each lab’s private calculus already contains a big slice of the social calculus.
Third, empirical incentives inside frontier labs look far more like “pick your preferred trade-off” than “cheat while others cooperate.” Google, Anthropic, OpenAI, and their peers hold billions of dollars in equity that vaporizes if a catastrophic failure occurs; their founders and employees live in the metaphorical blast radius just like everyone else.
So why does it look in practice as though labs are racing? The Jones model suggests the answer is epistemic, not game-theoretic. Different actors slot in different parameter values for how much economic growth matters or how sharply risk rises with capability, and those disagreements lead to divergent optimal policies. That is a dispute over facts and forecasts, not a coordination failure in the classic tragedy-of-the-commons sense, where each player gains by defecting even though all would jointly prefer restraint.
1. ^
  He technically models the problem as a choice of when to stop, rather than strictly picking a pace of R&D. However, for the purpose of this analysis, the difference between these two ways of modeling the problem is largely irrelevant.

Matthew_Barnett Apr 25, 2025, 7:08 PM
4 points
0 ∶ 0
in reply to: Neel Nanda’s comment on: Simplify EA Pitches to “Holy Shit, X-Risk”
When I wrote this post, pauseAI and similar were much less of a thing
I agree, this seems broadly accurate. I suppose I should have clarified that your post was perhaps true at the time, but in my view, has since become false if one counts AI pause as a “core action relevant point” of EA.
I just think if you told people “there’s this new technology that could cause human extinction, or be a really big deal and save many lives and cause an age of wonders, should we be slow and cautious in how we develop it” most people would say yes?
I believe that people’s answers to questions like this are usually highly sensitive to how the issue is framed. If you simply presented them with the exact quote you wrote here, without explaining that “saving many lives” would likely include the lives of their loved ones, such as their elderly relatives, I agree that most would support slowing down development. However, if you instead clarified that continuing development would likely save their own lives and the lives of their family members by curing most types of diseases, and if you also emphasized that the risk of human extinction from continued development is very low (for example, 1-2%), then I think there would be a significantly higher chance that most people would support moving forward with the technology at a reasonably fast pace, though presumably with some form of regulation in place to govern the technology.
One possible response to my argument is to point to survey data that shows most people favor pausing AI. However, while I agree survey data can be useful, I don’t think it provides strong evidence in this case for the claim. This is because most people, when answering survey questions, lack sufficient context and have not spent much time thinking deeply about these complex issues. Their responses are often made without fully understanding the stakes or the relevant information. In contrast, if you look at the behavior of current legislators and government officials who are being advised by scientific experts and given roughly this same information, it does not seem that they are currently strongly in favor of pausing AI development.

Matthew_Barnett Apr 23, 2025, 8:14 PM
6 points
0 ∶ 2
on: Simplify EA Pitches to “Holy Shit, X-Risk”
TL;DR If you believe the key claims of “there is a >=1% chance of AI causing x-risk and >=0.1% chance of bio causing x-risk in my lifetime” this is enough to justify the core action relevant points of EA. This clearly matters under most reasonable moral views and the common discussion of longtermism, future generations and other details of moral philosophy in intro materials is an unnecessary distraction.
I think the central thesis of this post—as I understand it—is false, for the reasons I provided in this comment. [Edit: to be clear, I think this post was perhaps true at the time, but in my view, has since become false if one counts pausing AI as a “core action relevant point” of EA]. To quote myself:
Let’s assume that there’s a 2% chance of AI causing existential risk, and that, optimistically, pausing [AI progress] for a decade would cut this risk in half (rather than barely decreasing it, or even increasing it). This would imply that the total risk would diminish from 2% to 1%.
According to OWID, approximately 63 million people die every year, although this rate is expected to increase, rising to around 74 million in 2035. If we assume that around 68 million people will die per year during the relevant time period, and that they could have been saved by AI-enabled medical progress, then pausing AI for a decade would kill around 680 million people.
This figure is around 8.3% of the current global population, and would constitute a death count higher than the combined death toll from World War 1, World War 2, the Mongol Conquests, the Taiping rebellion, the Transition from Ming to Qing, and the Three Kingdoms Civil war.
(Note that, although we are counting deaths from old age in this case, these deaths are comparable to deaths in war from a years of life lost perspective, if you assume that AI-accelerated medical breakthroughs will likely greatly increase human lifespan.)
From the perspective of an individual human life, a 1% chance of death from AI is significantly lower than a 8.3% chance of death from aging—though obviously in the former case this risk would apply independently of age, and in the latter case, the risk would be concentrated heavily among people who are currently elderly.
Even a briefer pause lasting just two years, while still cutting risk in half, would not survive this basic cost-benefit test. Of course, it’s true that it’s difficult to directly compare the individual personal costs from AI existential risk to the diseases of old age. For example, AI existential risk has the potential to be briefer and less agonizing, which, all else being equal, should push us to favor it. On the other hand, most people might consider death from old age to be preferable since it’s more natural and allows the human species to continue.
Nonetheless, despite these nuances, I think the basic picture that I’m presenting holds up here: under typical assumptions [...] a purely individualistic framing of the costs and benefits of AI pause do not clearly favor pausing, from the perspective of people who currently exist. This fact was noted in Nick Bostrom’s original essay on Astronomical Waste, and more recently, by Chad Jones in his paper on the tradeoffs involved in stopping AI development.

Matthew_Barnett Mar 27, 2025, 9:53 PM
8 points
0 ∶ 0
in reply to: AnonymousTurtle’s comment on: Ozzie Gooen’s Shortform
Using the data cited in your source (the Distributional Financial Accounts (DFA) provided by the Federal Reserve Board of Governors), it seems to me that the growth in the share of wealth held by the top 0.1% has not been very fast in the last 20 years—growing from around 10-11% to around 14% over that period. In my opinion, this is a significant, albeit rather unimportant trend relative to other social shifts in the last 20 years.
Moreover, this data does not include wealth held in social insurance programs (as I pointed out in another comment). If included, this would presumably decrease the magnitude of the trend seen in this plot, especially regarding the declining share of wealth held by the bottom 90%.

Matthew_Barnett Mar 27, 2025, 9:23 PM
4 points
0 ∶ 0
in reply to: Ozzie Gooen’s comment on: Ozzie Gooen’s Shortform
I could sympathize with the frustration, but I feel like I’m being attacked in a way that’s pretty unfair.
Sorry if my previous comment came across as rude or harsh—that wasn’t my intention. I didn’t mean to attack you. I asked those questions to clarify your exact claim because I wanted to understand it fully and potentially challenge it depending on its interpretation. My intent was for constructive disagreement, not criticism of you personally.
I find your other papers you linked in other comments interesting. That said, I don’t see them changing my main argument much.
Your main argument started with and seemed to depend heavily on the idea that inequality has been increasing. If it turns out that this key assumption is literally incorrect, then it seems like that should significantly affect your argument.

Matthew_Barnett Mar 27, 2025, 8:44 PM
6 points
1 ∶ 0
in reply to: titotal’s comment on: Ozzie Gooen’s Shortform
Your point about the top 1%’s rising income share uses pre-tax and transfers data, which can be misleading here because the discussion is specifically about how much income rich people actually control and can redirect towards their desired ends. Post-tax and transfer measures are more informative in this context since they directly reflect the resources individuals genuinely have available after taxes and redistribution. In other words, taxes and transfers matter because they substantially reduce the actual amount of wealth the rich can freely use, donate, or influence society with. Ignoring this gives a distorted picture of how much power or control rich people practically possess, which is central to the original discussion.
Other studies have notably not found meaningful increases in the top 1%’s income share after taxes and transfers are taken into account:

Matthew_Barnett Mar 27, 2025, 8:33 PM
8 points
1 ∶ 1
in reply to: huw’s comment on: Ozzie Gooen’s Shortform
It’s worth noting that much of the reported increase in wealth inequality since 1989 seems to be explained by the rising share of wealth held via social insurance programs. Catherine et al. notes,
Recent influential work finds large increases in inequality in the U.S. based on measures of wealth concentration that notably exclude the value of social insurance programs. This paper shows that top wealth shares have not changed much over the last three decades when Social Security is properly accounted for. This is because Social Security wealth increased substantially from $7 trillion in 1989 to $39 trillion in 2019 and now represents 49% of the wealth of the bottom 90% of the wealth distribution. This finding is robust to potential changes to taxes and benefits in response to system financing concerns.
Since both ordinary private wealth and social insurance programs are similar in that they provide continuous streams of income to people, I think it’s likely misleading to suggest that wealth inequality has gone up meaningfully in recent decades in the United States—at least based on the reported datasets that presently exist.
Social insurance income streams are especially relevant in this context because they directly affect how much real economic power and control people have in practice. Ignoring social insurance thus exaggerates how concentrated real economic power actually is, since it underestimates the resources available to the broader population.
That said, inequality statistics are quite contentious in general given the lack of reliable data on the exact variables we care about, so I’m not highly confident in this picture. Ultimately I’m unsure whether inequality has remained roughly constant over the last few decades in the sense we should care about.

Matthew_Barnett Mar 27, 2025, 6:39 AM
2 points
0 ∶ 0
in reply to: Ozzie Gooen’s comment on: Ozzie Gooen’s Shortform
It seems like recently (say, the last 20 years) inequality has been rising.
When you say inequality has been rising, do you mean income inequality or wealth inequality? What’s your source for this claim?
[Edit: reworded to be less curt and harsh]

Matthew_Barnett Mar 17, 2025, 11:41 PM
2 points
0 ∶ 0
in reply to: Pablo’s comment on: Discussion Thread: Existential Choices Debate Week
I tentatively agree with your statement that,
To me, it seems much more likely that Earth-originating intelligence will go extinct this century than, say, in the 8973th century AD.
That said, I still suspect the absolute probability of total extinction of intelligent life during the 21st century is very low. To be more precise, I’d put this probability at around 1% (to be clear: I recognize other people may not agree that this credence should count as “extremely low” or “very low” in this context). To justify this statement, I would highlight several key factors:
1. Throughout hundreds of millions of years, complex life has demonstrated remarkable resilience. Since the first vertebrates colonized land during the late Devonian period (approximately 375–360 million years ago), no extinction event has ever eradicated all species capable of complex cognition. Even after the most catastrophic mass extinctions, such as the end-Permian extinction and the K-Pg extinction, vertebrates rebounded. Not only did they recover, but they also surpassed their previous levels of ecological dominance and cognitive complexity, as seen in the increasing brain size and adaptability of various species over time.
2. Unlike non-intelligent organisms, intelligent life—starting with humans—possesses advanced planning abilities and an exceptional capacity to adapt to changing environments. Humans have successfully settled in nearly every climate and terrestrial habitat on Earth, from tropical jungles to arid deserts and even Antarctica. This extreme adaptability suggests that intelligent life is less vulnerable to complete extinction compared to other complex life forms.
3. As human civilization has advanced, our species has become increasingly robust against most types of extinction events rather than more fragile. Technological progress has expanded our ability to mitigate threats, whether they come from natural disasters or disease. Our massive global population further reduces the likelihood that any single event could exterminate every last human, while our growing capacity to detect and neutralize threats makes us better equipped to survive crises.
4. History shows that even in cases of large-scale violence and genocide, the goal has almost always been the destruction of rival groups—not the annihilation of all life, including the perpetrators themselves. This suggests that intelligent beings have strong instrumental reasons to avoid total extinction events. Even in scenarios involving genocidal warfare, the likelihood of all intelligent beings willingly or accidentally destroying all life—including their own—seems very low.
5. I have yet to see any compelling evidence that near-term or medium-term technological advancements will introduce a weapon or catastrophe capable of wiping out all forms of intelligent life. While near-term technological risks certainly exist that threaten human life, none currently appear to pose a credible risk of total extinction of intelligent life.
6. Some of the most destructive long-term technologies—such as asteroid manipulation for planetary bombardment—are likely to develop alongside technologies that enhance our ability to survive and expand into space. As our capacity for destruction grows, so too will our ability to establish off-world colonies and secure alternative survival strategies. This suggests that the overall trajectory of intelligent life seems to be toward increasing resilience, not increasing vulnerability.
7. Artificial life could rapidly evolve to become highly resilient to environmental shocks. Future AIs could be designed to be at least as robust as insects—able to survive in a wide range of extreme and unpredictable conditions. Similar to plant seeds, artificial hardware could be engineered to efficiently store and execute complex self-replicating instructions in a highly compact form, enabling them to autonomously colonize diverse environments by utilizing various energy sources, such as solar and thermal energy. Having been engineered rather than evolved naturally, these artificial systems could take advantage of design principles that surpass biological organisms in adaptability. By leveraging a vast array of energy sources and survival strategies, they could likely colonize some of the most extreme and inhospitable environments in our solar system—places that even the most resilient biological life forms on Earth could never inhabit.

Matthew_Barnett Mar 17, 2025, 10:23 PM
2 points
0 ∶ 0
in reply to: Pablo’s comment on: Discussion Thread: Existential Choices Debate Week
In my comment I later specified “in [the] next century” though it’s quite understandable if you missed that. I agree that eventual extinction of Earth-originating intelligent life (including AIs) is likely; however, I don’t currently see a plausible mechanism for this to occur over time horizons that are brief by cosmological standards.
(I just edited the original comment to make this slightly clearer.)

Matthew_Barnett Mar 17, 2025, 8:47 PM
3 points
1 ∶ 1
on: Discussion Thread: Existential Choices Debate Week
In my view, the extinction of all Earth-originating intelligent life (including AIs) seems extremely unlikely over the next several decades. While a longtermist utilitarian framework takes even a 0.01 percentage point reduction in extinction risk quite seriously, there appear to be very few plausible ways that all intelligent life originating from Earth could go extinct in the next century. Ensuring a positive transition to artificial life seems more useful on current margins.

Matthew_Barnett Mar 3, 2025, 8:11 AM
2 points
0 ∶ 0
in reply to: sammyboiz🔸’s comment on: Opinionated take on EA and AI Safety
That makes sense. For what it’s worth, I’m also not convinced that delaying AI is the right choice from a purely utilitarian perspective. I think there are reasonable arguments on both sides. My most recent post touches on this topic, so it might be worth reading for a better understanding of where I stand.
Right now, my stance is to withhold strong judgment on whether accelerating AI is harmful on net from a utilitarian point of view. It’s not that I think a case can’t be made: it’s just I don’t think the existing arguments are decisive enough to justify a firm position. In contrast, the argument that accelerating AI benefits people who currently exist seems significantly more straightforward and compelling to me.
This combination of views leads me to see accelerating AI as a morally acceptable choice (as long as it’s paired with adequate safety measures). Put simply:
- When I consider the well-being of people who currently exist, the case for acceleration appears fairly strong and compelling.
- When I take an impartial utilitarian perspective—one that prioritizes long-term outcomes for all sentient beings—the arguments for delaying AI seem weak and highly uncertain.
Since I give substantial weight to both perspectives, the stronger and clearer case for acceleration (based on the interests of people alive today) outweighs the much weaker and more uncertain case for delay (based on speculative long-term utilitarian concerns) in my view.
Of course, my analysis here doesn’t apply to someone who gives almost no moral weight to the well-being of people alive today—someone who, for instance, would be fine with everyone dying horribly if it meant even a tiny increase in the probability of a better outcome for the galaxy a billion years from now. But in my view, this type of moral calculus, if taken very seriously, seems highly unstable and untethered from practical considerations.
Since I think we have very little reliable insight into what actions today will lead to a genuinely better world millions of years down the line, it seems wise to exercise caution and try to avoid overconfidence about whether delaying AI is good or bad on the basis of its very long-term effects.

Matthew_Barnett Mar 3, 2025, 7:11 AM
2 points
0 ∶ 2
in reply to: sammyboiz🔸’s comment on: Opinionated take on EA and AI Safety
I think it’s extremely careless and condemnable to impose this risk on humanity just because you have personally deemed it acceptable.
I’m not sure I fully understand this criticism. From a moral subjectivist perspective, all moral decisions are ultimately based on what individuals personally deem acceptable. If you’re suggesting that there is an objective moral standard—something external to individual preferences—that we are obligated to follow, then I would understand your point.
That said, I’m personally skeptical that such an objective morality exists. And even if it did, I don’t see why I should necessarily follow it if I could instead act according to my own moral preferences—especially if I find my own preferences to be more humane and sensible than the objective morality.
This would be a deontological nightmare. Who gave AI labs the right to risk the lives of 8 billion people?
I see why a deontologist might find accelerating AI troublesome, especially given their emphasis on act-omission asymmetry—the idea that actively causing harm is worse than merely allowing harm to happen. However, I don’t personally find that distinction very compelling, especially in this context.
I’m also not a deontologist: I approach these questions from a consequentialist perspective. My personal ethics can be described as a mix of personal attachments and broader utilitarian concerns. In other words, I both care about people who currently exist, and more generally about all morally relevant beings. So while I understand why this argument might resonate with others, it doesn’t carry much weight for me.

Matthew_Barnett Mar 3, 2025, 5:18 AM
5 points
1 ∶ 1
in reply to: sammyboiz🔸’s comment on: Opinionated take on EA and AI Safety
I think the benefits of AGI arriving sooner are substantial. Many of my family members, for example, could be spared from death or serious illness if advanced AI accelerates medical progress. However, if AGI is delayed for many years, they will likely die before such breakthroughs occur, leaving me to live without them.
I’m not making a strictly selfish argument here either, since this situation isn’t unique to me—most people have loved ones in similar circumstances. Therefore, speeding up the benefits of AGI would have substantial ethical value from a perspective that values the lives of all humans who are alive today.
A moral point of view in which we give substantial weight to people who exist right now is indeed one of the most common ethical frameworks applied to policy. This may even be the most common mainstream ethical framework, as it’s implicit in most economic and political analysis. So I don’t think I’m proposing a crazy ethical theory here—just an unusual one within EA.
To clarify, I’m not arguing that AI should always be accelerated at any cost. Instead, I think we should carefully balance between pushing for faster progress and ensuring AI safety. If you either (1) believe that p(doom) is low, or (2) doubt that delaying AGI would meaningfully reduce p(doom), then it makes a lot of sense—under many common ethical perspectives—to view Anthropic as a force for good.

Matthew_Barnett Mar 3, 2025, 3:51 AM
9 points
1 ∶ 5
on: Opinionated take on EA and AI Safety
I’m admittedly unusual within the EA community on the issue of AI, but I’ll just give my thoughts on why I don’t think it’s productive to shame people who work at AI companies advancing AI capabilities.
In my view, there are two competing ethical priorities that I think we should try to balance:
1. Making sure that AI is developed safely and responsibly, so that AIs don’t harm humans in the future.
2. Making sure that AI is developed quickly, in order to take advantage of the enormous economic and technological benefits of AI sooner in time. This would, among other things, enable us to save lives by hastening AI-assisted medical progress.
If you believe that AI safety (priority 1) is the only meaningful ethical concern and that accelerating AI progress (priority 2) has little or no value in comparison, then it makes sense why you might view AI companies like Anthropic as harmful. From that perspective, any effort to advance AI capabilities could be seen as inherently trading off against an inviolable goal.
However, if you think—as I do—that both priorities matter substantially, then what companies like Anthropic are doing seems quite positive. They are not simply pushing forward AI development; rather, they are working to advance AI while also trying to ensure that it is developed in a safe and responsible way.
This kind of balancing act isn’t unusual. In most industries, we typically don’t perceive safety and usefulness as inherently opposed to each other. Rather, we usually recognize that both technological progress and safe development are important objectives to push for.

Matthew_Barnett Feb 26, 2025, 8:43 PM
27 points
7 ∶ 1
in reply to: Linch’s comment on: How confident are you that it’s preferable for America to develop AGI before China does?
Personally, I haven’t spent that much time investing this question, but I currently believe it’s very unlikely that the One Child Policy was primarily responsible for demographic collapse.
This may not have been the original intention behind the claim, but in my view, the primary signal I get from the One Child Policy is that the Chinese government has the appetite to regulate what is generally seen as a deeply personal matter—one’s choice to have children. Even if the policy only had minor adverse effects on China’s population trajectory, I find it alarming that the government felt it had the moral and legal authority to restrict people’s freedom in this particular respect. This mirrors my attitudes toward those who advocate for strict anti-abortion policies, and those who advocate for coercive eugenics.
In general, there seems to be a fairly consistent pattern where the Chinese government has less respect for personal freedoms than the United States government. While there are certainly exceptions to this rule, the pattern was recently observed quite clearly during the pandemic, where China imposed what was among the most severe peacetime restrictions on the movement of ordinary citizens that we have observed in recent world history. It is broadly accurate to say that China effectively imprisoned tens of million of its own people without due process. And of course, China is known for restricting free speech and digital privacy to an extent that would be almost inconceivable in the United States.
Personal freedom is just one measure of the quality of governance, but I think it’s quite an important one. While I think the United States is worse than China along some other important axes—for example, I think China has proven to be more cooperative internationally and less of a warmonger in recent decades—I consider the relative lack of respect for personal freedoms in China to be one of the best arguments for preferring United States to “win” any relevant technological arms race. This is partly because I find the possibility of a future world-wide permanent totalitarian regime to be an important source of x-risk, and in my view, China currently seems more likely than the United States to enact such a state.
That said, I still favor a broadly more cooperative approach toward China, seeking win-win compromises rather than aggressively “racing” them through unethical or dangerous means. The United States has its own share of major flaws, and the world is not a zero-sum game: China’s loss is not our gain.

Matthew_Barnett Feb 26, 2025, 8:17 PM
2 points
1 ∶ 1
in reply to: sawyer🔸’s comment on: Stop calling them labs
Unfortunately there’s momentum behind the term “AI lab” in a way that is not true for “AI bananas”. Also, it is unambiguously true that a major part of what these companies do is scientific experimentation, as one would expect in a laboratory—this makes the analogy to “AI bananas” imperfect.