I’m admittedly unusual within the EA community on the issue of AI, but I’ll just give my thoughts on why I don’t think it’s productive to shame people who work at AI companies advancing AI capabilities.
In my view, there are two competing ethical priorities that I think we should try to balance:
Making sure that AI is developed safely and responsibly, so that AIs don’t harm humans in the future.
Making sure that AI is developed quickly, in order to take advantage of the enormous economic and technological benefits of AI sooner in time. This would, among other things, enable us to save lives by hastening AI-assisted medical progress.
If you believe that AI safety (priority 1) is the only meaningful ethical concern and that accelerating AI progress (priority 2) has little or no value in comparison, then it makes sense why you might view AI companies like Anthropic as harmful. From that perspective, any effort to advance AI capabilities could be seen as inherently trading off against an inviolable goal.
However, if you think—as I do—that both priorities matter substantially, then what companies like Anthropic are doing seems quite positive. They are not simply pushing forward AI development; rather, they are working to advance AI while also trying to ensure that it is developed in a safe and responsible way.
This kind of balancing act isn’t unusual. In most industries, we typically don’t perceive safety and usefulness as inherently opposed to each other. Rather, we usually recognize that both technological progress and safe development are important objectives to push for.
Reaping the benefits of AGI later is pretty insignificant in my opinion. If we get aligned AGI utopia, we will have utopia for millions of years. Acceleration by a few years if negligible if it increase p(doom) by >1%.
1% X 1 million utopia years = 10 thousand utopia years (better than 2 utopia years)
Dario gives a 25% p(doom) if I’m not mistaken. He still continues the build the tech that could knowingly bring doom. Dario and Anthropic are pro-acceleration via their messaging and actions according to a LW’er. How is this position coherent?
I don’t think you can name another company that admits to building technology with a >1% chance of killing everyone… besides maybe OpenAI.
Reaping the benefits of AGI later is pretty insignificant in my opinion. If we get aligned AGI utopia, we will have utopia for millions of years. Acceleration by a few years if negligible if it increase p(doom) by >1%.
This is not true depending on what you think AGI utopia will look like. There’s some math outlined in What We Owe the Future about this dilemma i.e. area under the curve of these hypothetical AGI utility functions.
∫∞02(x+1),dx∫∞02x,dx=2
Getting utopia 1 year faster creates a 2x better universe. (hypothetically)
I was reluctant to get into the weeds here but how can anything near this model be possible if 2^300 is around how many atoms there are in the universe and we already have conquered 2^150 of them. At some point, there will likely be no more growing and then there will be millions of stable utopia years.
I think the benefits of AGI arriving sooner are substantial. Many of my family members, for example, could be spared from death or serious illness if advanced AI accelerates medical progress. However, if AGI is delayed for many years, they will likely die before such breakthroughs occur, leaving me to live without them.
I’m not making a strictly selfish argument here either, since this situation isn’t unique to me—most people have loved ones in similar circumstances. Therefore, speeding up the benefits of AGI would have substantial ethical value from a perspective that values the lives of all humans who are alive today.
A moral point of view in which we give substantial weight to people who exist right now is indeed one of the most common ethical frameworks applied to policy. This may even be the most common mainstream ethical framework, as it’s implicit in most economic and political analysis. So I don’t think I’m proposing a crazy ethical theory here—just an unusual one within EA.
To clarify, I’m not arguing that AI should always be accelerated at any cost. Instead, I think we should carefully balance between pushing for faster progress and ensuring AI safety. If you either (1) believe that p(doom) is low, or (2) doubt that delaying AGI would meaningfully reduce p(doom), then it makes a lot of sense—under many common ethical perspectives—to view Anthropic as a force for good.
For the interest of the people today, there is an argument to be made for taking on risk of extinction. However, if this is not a purely utilitarian argument, I think it’s extremely careless and condemnable to impose this risk on humanity just because you have personally deemed it acceptable. This would be a deontological nightmare. Who gave AI labs the right to risk the lives of 8 billion people?
I think it’s extremely careless and condemnable to impose this risk on humanity just because you have personally deemed it acceptable.
I’m not sure I fully understand this criticism. From a moral subjectivist perspective, all moral decisions are ultimately based on what individuals personally deem acceptable. If you’re suggesting that there is an objective moral standard—something external to individual preferences—that we are obligated to follow, then I would understand your point.
That said, I’m personally skeptical that such an objective morality exists. And even if it did, I don’t see why I should necessarily follow it if I could instead act according to my own moral preferences—especially if I find my own preferences to be more humane and sensible than the objective morality.
This would be a deontological nightmare. Who gave AI labs the right to risk the lives of 8 billion people?
I see why a deontologist might find accelerating AI troublesome, especially given their emphasis on act-omission asymmetry—the idea that actively causing harm is worse than merely allowing harm to happen. However, I don’t personally find that distinction very compelling, especially in this context.
I’m also not a deontologist: I approach these questions from a consequentialist perspective. My personal ethics can be described as a mix of personal attachments and broader utilitarian concerns. In other words, I both care about people who currently exist, and more generally about all morally relevant beings. So while I understand why this argument might resonate with others, it doesn’t carry much weight for me.
Oh I see. I was quick to bifurcate between deontology and utilitarianism. I guess I’m less familiar with other branches of consequentialism. Sorry for being unclear in the critique. My whole reply was just centered around being bad deontologically.
That makes sense. For what it’s worth, I’m also not convinced that delaying AI is the right choice from a purely utilitarian perspective. I think there are reasonable arguments on both sides. My most recent post touches on this topic, so it might be worth reading for a better understanding of where I stand.
Right now, my stance is to withhold strong judgment on whether accelerating AI is harmful on net from a utilitarian point of view. It’s not that I think a case can’t be made: it’s just I don’t think the existing arguments are decisive enough to justify a firm position. In contrast, the argument that accelerating AI benefits people who currently exist seems significantly more straightforward and compelling to me.
This combination of views leads me to see accelerating AI as a morally acceptable choice (as long as it’s paired with adequate safety measures). Put simply:
When I consider the well-being of people who currently exist, the case for acceleration appears fairly strong and compelling.
When I take an impartial utilitarian perspective—one that prioritizes long-term outcomes for all sentient beings—the arguments for delaying AI seem weak and highly uncertain.
Since I give substantial weight to both perspectives, the stronger and clearer case for acceleration (based on the interests of people alive today) outweighs the much weaker and more uncertain case for delay (based on speculative long-term utilitarian concerns) in my view.
Of course, my analysis here doesn’t apply to someone who gives almost no moral weight to the well-being of people alive today—someone who, for instance, would be fine with everyone dying horribly if it meant even a tiny increase in the probability of a better outcome for the galaxy a billion years from now. But in my view, this type of moral calculus, if taken very seriously, seems highly unstable and untethered from practical considerations.
Since I think we have very little reliable insight into what actions today will lead to a genuinely better world millions of years down the line, it seems wise to exercise caution and try to avoid overconfidence about whether delaying AI is good or bad on the basis of its very long-term effects.
I’m admittedly unusual within the EA community on the issue of AI, but I’ll just give my thoughts on why I don’t think it’s productive to shame people who work at AI companies advancing AI capabilities.
In my view, there are two competing ethical priorities that I think we should try to balance:
Making sure that AI is developed safely and responsibly, so that AIs don’t harm humans in the future.
Making sure that AI is developed quickly, in order to take advantage of the enormous economic and technological benefits of AI sooner in time. This would, among other things, enable us to save lives by hastening AI-assisted medical progress.
If you believe that AI safety (priority 1) is the only meaningful ethical concern and that accelerating AI progress (priority 2) has little or no value in comparison, then it makes sense why you might view AI companies like Anthropic as harmful. From that perspective, any effort to advance AI capabilities could be seen as inherently trading off against an inviolable goal.
However, if you think—as I do—that both priorities matter substantially, then what companies like Anthropic are doing seems quite positive. They are not simply pushing forward AI development; rather, they are working to advance AI while also trying to ensure that it is developed in a safe and responsible way.
This kind of balancing act isn’t unusual. In most industries, we typically don’t perceive safety and usefulness as inherently opposed to each other. Rather, we usually recognize that both technological progress and safe development are important objectives to push for.
Reaping the benefits of AGI later is pretty insignificant in my opinion. If we get aligned AGI utopia, we will have utopia for millions of years. Acceleration by a few years if negligible if it increase p(doom) by >1%.
1% X 1 million utopia years = 10 thousand utopia years (better than 2 utopia years)
Dario gives a 25% p(doom) if I’m not mistaken. He still continues the build the tech that could knowingly bring doom. Dario and Anthropic are pro-acceleration via their messaging and actions according to a LW’er. How is this position coherent?
I don’t think you can name another company that admits to building technology with a >1% chance of killing everyone… besides maybe OpenAI.
This is not true depending on what you think AGI utopia will look like. There’s some math outlined in What We Owe the Future about this dilemma i.e. area under the curve of these hypothetical AGI utility functions.
∫∞02(x+1),dx∫∞02x,dx=2
Getting utopia 1 year faster creates a 2x better universe. (hypothetically)
I was reluctant to get into the weeds here but how can anything near this model be possible if 2^300 is around how many atoms there are in the universe and we already have conquered 2^150 of them. At some point, there will likely be no more growing and then there will be millions of stable utopia years.
I think the benefits of AGI arriving sooner are substantial. Many of my family members, for example, could be spared from death or serious illness if advanced AI accelerates medical progress. However, if AGI is delayed for many years, they will likely die before such breakthroughs occur, leaving me to live without them.
I’m not making a strictly selfish argument here either, since this situation isn’t unique to me—most people have loved ones in similar circumstances. Therefore, speeding up the benefits of AGI would have substantial ethical value from a perspective that values the lives of all humans who are alive today.
A moral point of view in which we give substantial weight to people who exist right now is indeed one of the most common ethical frameworks applied to policy. This may even be the most common mainstream ethical framework, as it’s implicit in most economic and political analysis. So I don’t think I’m proposing a crazy ethical theory here—just an unusual one within EA.
To clarify, I’m not arguing that AI should always be accelerated at any cost. Instead, I think we should carefully balance between pushing for faster progress and ensuring AI safety. If you either (1) believe that p(doom) is low, or (2) doubt that delaying AGI would meaningfully reduce p(doom), then it makes a lot of sense—under many common ethical perspectives—to view Anthropic as a force for good.
I see your point.
For the interest of the people today, there is an argument to be made for taking on risk of extinction. However, if this is not a purely utilitarian argument, I think it’s extremely careless and condemnable to impose this risk on humanity just because you have personally deemed it acceptable. This would be a deontological nightmare. Who gave AI labs the right to risk the lives of 8 billion people?
I’m not sure I fully understand this criticism. From a moral subjectivist perspective, all moral decisions are ultimately based on what individuals personally deem acceptable. If you’re suggesting that there is an objective moral standard—something external to individual preferences—that we are obligated to follow, then I would understand your point.
That said, I’m personally skeptical that such an objective morality exists. And even if it did, I don’t see why I should necessarily follow it if I could instead act according to my own moral preferences—especially if I find my own preferences to be more humane and sensible than the objective morality.
I see why a deontologist might find accelerating AI troublesome, especially given their emphasis on act-omission asymmetry—the idea that actively causing harm is worse than merely allowing harm to happen. However, I don’t personally find that distinction very compelling, especially in this context.
I’m also not a deontologist: I approach these questions from a consequentialist perspective. My personal ethics can be described as a mix of personal attachments and broader utilitarian concerns. In other words, I both care about people who currently exist, and more generally about all morally relevant beings. So while I understand why this argument might resonate with others, it doesn’t carry much weight for me.
Oh I see. I was quick to bifurcate between deontology and utilitarianism. I guess I’m less familiar with other branches of consequentialism. Sorry for being unclear in the critique. My whole reply was just centered around being bad deontologically.
That makes sense. For what it’s worth, I’m also not convinced that delaying AI is the right choice from a purely utilitarian perspective. I think there are reasonable arguments on both sides. My most recent post touches on this topic, so it might be worth reading for a better understanding of where I stand.
Right now, my stance is to withhold strong judgment on whether accelerating AI is harmful on net from a utilitarian point of view. It’s not that I think a case can’t be made: it’s just I don’t think the existing arguments are decisive enough to justify a firm position. In contrast, the argument that accelerating AI benefits people who currently exist seems significantly more straightforward and compelling to me.
This combination of views leads me to see accelerating AI as a morally acceptable choice (as long as it’s paired with adequate safety measures). Put simply:
When I consider the well-being of people who currently exist, the case for acceleration appears fairly strong and compelling.
When I take an impartial utilitarian perspective—one that prioritizes long-term outcomes for all sentient beings—the arguments for delaying AI seem weak and highly uncertain.
Since I give substantial weight to both perspectives, the stronger and clearer case for acceleration (based on the interests of people alive today) outweighs the much weaker and more uncertain case for delay (based on speculative long-term utilitarian concerns) in my view.
Of course, my analysis here doesn’t apply to someone who gives almost no moral weight to the well-being of people alive today—someone who, for instance, would be fine with everyone dying horribly if it meant even a tiny increase in the probability of a better outcome for the galaxy a billion years from now. But in my view, this type of moral calculus, if taken very seriously, seems highly unstable and untethered from practical considerations.
Since I think we have very little reliable insight into what actions today will lead to a genuinely better world millions of years down the line, it seems wise to exercise caution and try to avoid overconfidence about whether delaying AI is good or bad on the basis of its very long-term effects.