To be strictly accurate, perhaps I should have said ‘the more you know about AI risks and AI safety, the higher your p(doom)’. I do think that’s an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.
And I might have added that thousands of AI devs employed by AI companies to build AGI/ASI have very strong incentives not to learn about too much about AI risks and AI safety of the sort that EAs have talked about for years, because such knowledge would cause massive cognitive dissonance, ethical self-doubt, regret (as in the case of Geoff Hinton), and/or would handicap their careers and threaten their salaries and equity stakes.
To be strictly accurate, perhaps I should have said ‘the more you know about AI risks and AI safety, the higher your p(doom)’. I do think that’s an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.
That makes it sound like a continuous function when it isn’t really. Sure, people who’ve never or barely thought about it and then proceed to do so are likely to become more concerned—since they have a base of ~0 concern. That doesn’t mean the effect will have the same shape or even same direction for people who have a reasonable initial familiarity with the issue.
Inasmuch as there is such an effect, it’s also hard to separate from reverse causation, where people who are more concerned about such outcomes tend to engage more with the arguments for them.
As for incentives—sure, that’s an effect. I also think the AI safety movement has its own cognitive biases: orgs like MIRI have an operational budget in the 10s if not 100s of millions; people who believe in high p(doom) and short timelines have little reason to present arguments fairly, leading to silly claims like the orthogonality thesis showing far more than it does, or to gross epistemic behaviour by AI safety orgs.
In any case, if the claim is that knowing more makes people have a higher p(doom), then you have to evidence that claim, not argue that they would do if it weren’t for cognitive biases.
Finally if you want to claim that the people working at those orgs don’t actually much about ‘know about AI risks and AI safety’ and so wouldn’t be counterpoints to your revised claim, I think you need to evidence that. The arguments really aren’t that complicated, and they’ve been out there for decades, and more recently shouted loudly by people trying to stop, pause or otherwise impede the work of the people working on AI capabilities—to the point where I find it hard to imagine there’s anyone working on capabilities who doesn’t have some level of familiarity with them (and, I would guess substantially more so than people on the left hand side of the discontinuous function).
Arepo—thanks for your comment.
To be strictly accurate, perhaps I should have said ‘the more you know about AI risks and AI safety, the higher your p(doom)’. I do think that’s an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.
And I might have added that thousands of AI devs employed by AI companies to build AGI/ASI have very strong incentives not to learn about too much about AI risks and AI safety of the sort that EAs have talked about for years, because such knowledge would cause massive cognitive dissonance, ethical self-doubt, regret (as in the case of Geoff Hinton), and/or would handicap their careers and threaten their salaries and equity stakes.
This seems pretty false. E.g. see this survey.
That makes it sound like a continuous function when it isn’t really. Sure, people who’ve never or barely thought about it and then proceed to do so are likely to become more concerned—since they have a base of ~0 concern. That doesn’t mean the effect will have the same shape or even same direction for people who have a reasonable initial familiarity with the issue.
Inasmuch as there is such an effect, it’s also hard to separate from reverse causation, where people who are more concerned about such outcomes tend to engage more with the arguments for them.
As for incentives—sure, that’s an effect. I also think the AI safety movement has its own cognitive biases: orgs like MIRI have an operational budget in the 10s if not 100s of millions; people who believe in high p(doom) and short timelines have little reason to present arguments fairly, leading to silly claims like the orthogonality thesis showing far more than it does, or to gross epistemic behaviour by AI safety orgs.
In any case, if the claim is that knowing more makes people have a higher p(doom), then you have to evidence that claim, not argue that they would do if it weren’t for cognitive biases.
Finally if you want to claim that the people working at those orgs don’t actually much about ‘know about AI risks and AI safety’ and so wouldn’t be counterpoints to your revised claim, I think you need to evidence that. The arguments really aren’t that complicated, and they’ve been out there for decades, and more recently shouted loudly by people trying to stop, pause or otherwise impede the work of the people working on AI capabilities—to the point where I find it hard to imagine there’s anyone working on capabilities who doesn’t have some level of familiarity with them (and, I would guess substantially more so than people on the left hand side of the discontinuous function).