I agree with the main ideas of this post. But I want to flag that, as someone who’s engaged with the AI safety community (outside of the Bay Area) for several years, I don’t recognize this depiction. In my experience, it’s very common to say “even a [1, 5, 10]% chance of AI x-risk justifies taking this very seriously.”
I don’t have time to track down many examples, but just to illustrate:
Neel Nanda’s “Simplify EA Pitches to ‘Holy Shit, X-Risk’”: “If you believe the key claims of “there is a >=1% chance of AI causing x-risk and >=0.1% chance of bio causing x-risk in my lifetime” this is enough to justify the core action relevant points of EA.”
Toby Ord estimated AI x-risk at 10% in The Precipice.
Scott Alexander is sympathetic to libertarianism and put AI x-risk at 20-25% in 2023.
To address another claim: “The Dial of Progress” by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the “Technology Bucket Error” post) and the comments overwhelmingly seem to agree.
I’m sure someone has said that 10-25% x-risk would not be worth addressing due to libertarianism—but I don’t believe I’ve heard this argument, and wasn’t able to find someone making it after a few minutes of searching. (But it’s a hard thing to search for.)
I don’t doubt that Holly is accurately reporting her experiences, and she’s almost certainly engaged more widely than I have with people in AI safety. I wouldn’t be surprised if there are people saying these things in the Bay Area. But I don’t have the impression that they represent the mainstream of the AI safety community in any forum I’m familiar with (Twitter, LW, and definitely EA).
I am not making a case for who should be referred to as “the AI Safety community”, and if you draw a larger circle then you get a lot more people with lower p(doom)s. You still get mostly x-risk people as opposed to other risks and people thinking that it’s only x-risk that justifies intervening, implicitly if not explicitly.
> To address another claim: “The Dial of Progress” by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the “Technology Bucket Error” post) and the comments overwhelmingly seem to agree.
This post is a great example of my point. If you pay attention to the end, Zvi says that the dial thinking is mostly right and is morally sympathetic to it, just that AGI is an exception. I wanted it to mean what you thought it meant :(
Again, just giving my impressions from interacting with AI safety people: it doesn’t seem to me like I get this impression by drawing a larger circle—I don’t recall hearing the types of arguments you allude to even from people I consider “core” to AI safety. I think it would help me understand if you were able to provide some examples? (Although like I said, I found examples either way hard to search for, so I understand if you don’t have any available.)
I still disagree about the Dial post: at the end Zvi says
Seeing highly intelligent thinkers who are otherwise natural partners and allies making a variety of obvious nonsense arguments, in ways that seem immune to correction, in ways that seem designed to prevent humanity from taking action to prevent its own extinction, is extremely frustrating. Even more frustrating is not knowing why it is happening, and responding in unproductive ways.
So my read is that he wants to explain and understand the position as well as possible, so that he can cooperate as effectively as possible with people who take the Dial position. He also agrees on lots of object-level points with the people he’s arguing against. But ultimately actually using the Dial as an argument is “obvious nonsense,” for the same reason the Technology Bucket Error is an error.
I agree with the main ideas of this post. But I want to flag that, as someone who’s engaged with the AI safety community (outside of the Bay Area) for several years, I don’t recognize this depiction. In my experience, it’s very common to say “even a [1, 5, 10]% chance of AI x-risk justifies taking this very seriously.”
I don’t have time to track down many examples, but just to illustrate:
Neel Nanda’s “Simplify EA Pitches to ‘Holy Shit, X-Risk’”: “If you believe the key claims of “there is a >=1% chance of AI causing x-risk and >=0.1% chance of bio causing x-risk in my lifetime” this is enough to justify the core action relevant points of EA.”
Toby Ord estimated AI x-risk at 10% in The Precipice.
Scott Alexander is sympathetic to libertarianism and put AI x-risk at 20-25% in 2023.
To address another claim: “The Dial of Progress” by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the “Technology Bucket Error” post) and the comments overwhelmingly seem to agree.
I’m sure someone has said that 10-25% x-risk would not be worth addressing due to libertarianism—but I don’t believe I’ve heard this argument, and wasn’t able to find someone making it after a few minutes of searching. (But it’s a hard thing to search for.)
I don’t doubt that Holly is accurately reporting her experiences, and she’s almost certainly engaged more widely than I have with people in AI safety. I wouldn’t be surprised if there are people saying these things in the Bay Area. But I don’t have the impression that they represent the mainstream of the AI safety community in any forum I’m familiar with (Twitter, LW, and definitely EA).
I am not making a case for who should be referred to as “the AI Safety community”, and if you draw a larger circle then you get a lot more people with lower p(doom)s. You still get mostly x-risk people as opposed to other risks and people thinking that it’s only x-risk that justifies intervening, implicitly if not explicitly.
> To address another claim: “The Dial of Progress” by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the “Technology Bucket Error” post) and the comments overwhelmingly seem to agree.
This post is a great example of my point. If you pay attention to the end, Zvi says that the dial thinking is mostly right and is morally sympathetic to it, just that AGI is an exception. I wanted it to mean what you thought it meant :(
Again, just giving my impressions from interacting with AI safety people: it doesn’t seem to me like I get this impression by drawing a larger circle—I don’t recall hearing the types of arguments you allude to even from people I consider “core” to AI safety. I think it would help me understand if you were able to provide some examples? (Although like I said, I found examples either way hard to search for, so I understand if you don’t have any available.)
I still disagree about the Dial post: at the end Zvi says
So my read is that he wants to explain and understand the position as well as possible, so that he can cooperate as effectively as possible with people who take the Dial position. He also agrees on lots of object-level points with the people he’s arguing against. But ultimately actually using the Dial as an argument is “obvious nonsense,” for the same reason the Technology Bucket Error is an error.
I was going on my memory of that post and I don’t have the spoons to go through it again, so I’ll take your word for it.