Obvious point, but you could assign significant credence to this being the right take, and still think working on A.I. risk is very good in expectation, given exceptional neglectedness and how bad an A.I. takeover could be. Something feels sleazy and motivated about this line of defence to me, but I find it hard to see where it goes wrong.
Something feels sleazy and motivated about this line of defence to me, but I find it hard to see where it goes wrong.
I’m not sure if we’re picking up on the same notion of sleaziness, and I guess it depends on what you mean by “significant credence”, and “working on A.I risk” but I think it’s hard to imagine someone doing really good mission-critical research work if they come into it from a perspective of “oh I don’t think AI risk is at all an issue but smart people disagree and there’s a small chance that I’m wrong and the EV is higher than working on other issues.” Though I think it’s plausible my psychology is less well-suited to “grim determination” than most people in EA.
(Donations or engineering in comparison seem comparatively much more reasonable).
Just my anecdotal experience, but when I ask a lot of EAs working in or interested in AGI risk why they think it’s a hugely important x-risk, one of the first arguments that comes to people’s minds is some variation on “a lot of smart people [working on AGI risk] are very worried about it”. My model of many people in EA interested in AI safety is that they use this heuristic as a dominant factor in their reasoning — which is perfectly understandable! After all, formulating a view of the magnitude of risk from transformative AI without relying on any such heuristics is extremely hard. But I think this post is a valuable reminder that it’s not particularly good epistemics for lots of people to think like this.
when I ask a lot of EAs working in or interested in AGI risk
Can I ask roughly what work they’re doing? Again I think it makes more sense if you’re earning-to-give or doing engineering work, and less if you’re doing conceptual or strategic research. It also makes sense if you’re interested in it as an avenue to learn more.
Obvious point, but you could assign significant credence to this being the right take, and still think working on A.I. risk is very good in expectation, given exceptional neglectedness and how bad an A.I. takeover could be. Something feels sleazy and motivated about this line of defence to me, but I find it hard to see where it goes wrong.
I’m not sure if we’re picking up on the same notion of sleaziness, and I guess it depends on what you mean by “significant credence”, and “working on A.I risk” but I think it’s hard to imagine someone doing really good mission-critical research work if they come into it from a perspective of “oh I don’t think AI risk is at all an issue but smart people disagree and there’s a small chance that I’m wrong and the EV is higher than working on other issues.” Though I think it’s plausible my psychology is less well-suited to “grim determination” than most people in EA.
(Donations or engineering in comparison seem comparatively much more reasonable).
Just my anecdotal experience, but when I ask a lot of EAs working in or interested in AGI risk why they think it’s a hugely important x-risk, one of the first arguments that comes to people’s minds is some variation on “a lot of smart people [working on AGI risk] are very worried about it”. My model of many people in EA interested in AI safety is that they use this heuristic as a dominant factor in their reasoning — which is perfectly understandable! After all, formulating a view of the magnitude of risk from transformative AI without relying on any such heuristics is extremely hard. But I think this post is a valuable reminder that it’s not particularly good epistemics for lots of people to think like this.
Can I ask roughly what work they’re doing? Again I think it makes more sense if you’re earning-to-give or doing engineering work, and less if you’re doing conceptual or strategic research. It also makes sense if you’re interested in it as an avenue to learn more.