I’m really glad you gave this talk, even as something of a sceptic of AI x-risk. As you say, this shouldn’t be a partisan issue. I would contest one claim though:
> Generally, the more you know about AI, the higher your p(doom), or estimated probability that ASI would doom humanity to imminent extinction.
I don’t see the evidence for this claim, and I keep seeing people in the doomer community uncritically repeat it. AI wouldn’t be progressing if everyone who understood it became convinced it would kill us all.
Ok, one might explain this progress via the unilateralists curse, but when the players include major divisions at Google, Meta and multiple startups with (according to Google) >1000 employees it would be quite a stretch to call them ‘unilateralists’. In fact, I suspect when factoring in people working elsewhere with a deep knowledge of frontier models for their jobs, AI-positive workers substantially outnumber AI doomers. And, since their work actively requires it, they probably have a higher average understanding of AI.
What I see is some high profile AI researchers shifting to higher p-dooms and getting highlighted by the doomer community; but this is explainable by the doomer community being much more active and coordinated than the companies competing to advance the field, and by it just making for better news headlines. The AI-positive community seems to have much less incentive to tout people who shift in the other direction—or who simply review the arguments and remain unconvinced. (shoutout to titotal, who has written some excellent critiques of AI safety arguments)
To be clear, I think there’s a trivial sense in which the claim is true—people who know very little about AI capabilities and learn more about it are likely to shift to being pessimistic about it from having no opinion. But I think the claim that people who are already knowledgeable about the field get more pessimistic as they gain even more knowledge—while plausibly true—needs some actual data to proclaim so boldly.
To be strictly accurate, perhaps I should have said ‘the more you know about AI risks and AI safety, the higher your p(doom)’. I do think that’s an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.
And I might have added that thousands of AI devs employed by AI companies to build AGI/ASI have very strong incentives not to learn about too much about AI risks and AI safety of the sort that EAs have talked about for years, because such knowledge would cause massive cognitive dissonance, ethical self-doubt, regret (as in the case of Geoff Hinton), and/or would handicap their careers and threaten their salaries and equity stakes.
To be strictly accurate, perhaps I should have said ‘the more you know about AI risks and AI safety, the higher your p(doom)’. I do think that’s an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.
That makes it sound like a continuous function when it isn’t really. Sure, people who’ve never or barely thought about it and then proceed to do so are likely to become more concerned—since they have a base of ~0 concern. That doesn’t mean the effect will have the same shape or even same direction for people who have a reasonable initial familiarity with the issue.
Inasmuch as there is such an effect, it’s also hard to separate from reverse causation, where people who are more concerned about such outcomes tend to engage more with the arguments for them.
As for incentives—sure, that’s an effect. I also think the AI safety movement has its own cognitive biases: orgs like MIRI have an operational budget in the 10s if not 100s of millions; people who believe in high p(doom) and short timelines have little reason to present arguments fairly, leading to silly claims like the orthogonality thesis showing far more than it does, or to gross epistemic behaviour by AI safety orgs.
In any case, if the claim is that knowing more makes people have a higher p(doom), then you have to evidence that claim, not argue that they would do if it weren’t for cognitive biases.
Finally if you want to claim that the people working at those orgs don’t actually much about ‘know about AI risks and AI safety’ and so wouldn’t be counterpoints to your revised claim, I think you need to evidence that. The arguments really aren’t that complicated, and they’ve been out there for decades, and more recently shouted loudly by people trying to stop, pause or otherwise impede the work of the people working on AI capabilities—to the point where I find it hard to imagine there’s anyone working on capabilities who doesn’t have some level of familiarity with them (and, I would guess substantially more so than people on the left hand side of the discontinuous function).
I’m really glad you gave this talk, even as something of a sceptic of AI x-risk. As you say, this shouldn’t be a partisan issue. I would contest one claim though:
> Generally, the more you know about AI, the higher your p(doom), or estimated probability that ASI would doom humanity to imminent extinction.
I don’t see the evidence for this claim, and I keep seeing people in the doomer community uncritically repeat it. AI wouldn’t be progressing if everyone who understood it became convinced it would kill us all.
Ok, one might explain this progress via the unilateralists curse, but when the players include major divisions at Google, Meta and multiple startups with (according to Google) >1000 employees it would be quite a stretch to call them ‘unilateralists’. In fact, I suspect when factoring in people working elsewhere with a deep knowledge of frontier models for their jobs, AI-positive workers substantially outnumber AI doomers. And, since their work actively requires it, they probably have a higher average understanding of AI.
What I see is some high profile AI researchers shifting to higher p-dooms and getting highlighted by the doomer community; but this is explainable by the doomer community being much more active and coordinated than the companies competing to advance the field, and by it just making for better news headlines. The AI-positive community seems to have much less incentive to tout people who shift in the other direction—or who simply review the arguments and remain unconvinced. (shoutout to titotal, who has written some excellent critiques of AI safety arguments)
To be clear, I think there’s a trivial sense in which the claim is true—people who know very little about AI capabilities and learn more about it are likely to shift to being pessimistic about it from having no opinion. But I think the claim that people who are already knowledgeable about the field get more pessimistic as they gain even more knowledge—while plausibly true—needs some actual data to proclaim so boldly.
Arepo—thanks for your comment.
To be strictly accurate, perhaps I should have said ‘the more you know about AI risks and AI safety, the higher your p(doom)’. I do think that’s an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.
And I might have added that thousands of AI devs employed by AI companies to build AGI/ASI have very strong incentives not to learn about too much about AI risks and AI safety of the sort that EAs have talked about for years, because such knowledge would cause massive cognitive dissonance, ethical self-doubt, regret (as in the case of Geoff Hinton), and/or would handicap their careers and threaten their salaries and equity stakes.
This seems pretty false. E.g. see this survey.
That makes it sound like a continuous function when it isn’t really. Sure, people who’ve never or barely thought about it and then proceed to do so are likely to become more concerned—since they have a base of ~0 concern. That doesn’t mean the effect will have the same shape or even same direction for people who have a reasonable initial familiarity with the issue.
Inasmuch as there is such an effect, it’s also hard to separate from reverse causation, where people who are more concerned about such outcomes tend to engage more with the arguments for them.
As for incentives—sure, that’s an effect. I also think the AI safety movement has its own cognitive biases: orgs like MIRI have an operational budget in the 10s if not 100s of millions; people who believe in high p(doom) and short timelines have little reason to present arguments fairly, leading to silly claims like the orthogonality thesis showing far more than it does, or to gross epistemic behaviour by AI safety orgs.
In any case, if the claim is that knowing more makes people have a higher p(doom), then you have to evidence that claim, not argue that they would do if it weren’t for cognitive biases.
Finally if you want to claim that the people working at those orgs don’t actually much about ‘know about AI risks and AI safety’ and so wouldn’t be counterpoints to your revised claim, I think you need to evidence that. The arguments really aren’t that complicated, and they’ve been out there for decades, and more recently shouted loudly by people trying to stop, pause or otherwise impede the work of the people working on AI capabilities—to the point where I find it hard to imagine there’s anyone working on capabilities who doesn’t have some level of familiarity with them (and, I would guess substantially more so than people on the left hand side of the discontinuous function).