Thanks for this post. As someone who has only recently started exploring the field of AI safety, much of this resonates with my initial impressions. I would be interested to hear the counterpoints from those who have Disagree-voted on this post.
Do you think the “very particular worldview” you describe is found equally among those working on technical AI safety and AI governance/policy? My impression is that policy inherently requires thinking through concrete pathways of how AGI would lead to actual harm as well as greater engagement with people outside of AI safety.
I have also noticed a split between the “superintelligence will kill us all” worldview (which you seem to be describing) and “regardless of whether superintelligence kills us all, AGI/TAI will be very disruptive and we need to manage those risks” (which seemed to be more along the lines of the Will MacAskill post you linked to—especially as he talks about directing people to causes other than technical safety or safety governance). Both of these worldviews seem prominent in EA. I haven’t gotten the impression that the superintelligence worldview is stronger, but perhaps I just haven’t gotten deep enough into AI safety circles yet.
I would be interested to hear the counterpoints from those who have Disagree-voted on this post.
Likewise!
Do you think the “very particular worldview” you describe is found equally among those working on technical AI safety and AI governance/policy? My impression is that policy inherently requires thinking through concrete pathways of how AGI would lead to actual harm as well as greater engagement with people outside of AI safety.
I think they’re quite prevalent regardless. While some people’s roles indeed require them to analyze concrete pathways more than others, the foundation of their analysis is often implicitly built upon this worldview in the first place. The result is that their concrete pathways tend to be centred around some kind of misaligned AGI, just in much more detail. Conversely, someone with a very different worldview who does such an analysis might end up with concrete pathways centred around severe discrimination of marginalized groups.
I have also noticed a split between the “superintelligence will kill us all” worldview (which you seem to be describing) and “regardless of whether superintelligence kills us all, AGI/TAI will be very disruptive and we need to manage those risks” (which seemed to be more along the lines of the Will MacAskill post you linked to—especially as he talks about directing people to causes other than technical safety or safety governance).
There are indeed many different “sub-worldviews”, and I was kind of lumping them all under one big umbrella. To me, the most defining characteristic of this worldview is AI-centrism, and treating the impending AGI as an extremely big deal — not just like any other big deals we have seen before, but this will be unprecedented. Those within this overarching worldview would differ in terms of the details, e.g. will it kill everyone? or will it just lead to gradual disempowerment? are LLMs getting us to AGI? or is it some yet-to-be-discovered architecture? should we focus on getting to AGI safely? or start thinking more about the post-AGI world? I think many people move between these “sub-worldviews” as they see evidences that update their priors, but way fewer people move out of this overarching worldview entirely.
Thanks for this post. As someone who has only recently started exploring the field of AI safety, much of this resonates with my initial impressions. I would be interested to hear the counterpoints from those who have Disagree-voted on this post.
Do you think the “very particular worldview” you describe is found equally among those working on technical AI safety and AI governance/policy? My impression is that policy inherently requires thinking through concrete pathways of how AGI would lead to actual harm as well as greater engagement with people outside of AI safety.
I have also noticed a split between the “superintelligence will kill us all” worldview (which you seem to be describing) and “regardless of whether superintelligence kills us all, AGI/TAI will be very disruptive and we need to manage those risks” (which seemed to be more along the lines of the Will MacAskill post you linked to—especially as he talks about directing people to causes other than technical safety or safety governance). Both of these worldviews seem prominent in EA. I haven’t gotten the impression that the superintelligence worldview is stronger, but perhaps I just haven’t gotten deep enough into AI safety circles yet.
Likewise!
I think they’re quite prevalent regardless. While some people’s roles indeed require them to analyze concrete pathways more than others, the foundation of their analysis is often implicitly built upon this worldview in the first place. The result is that their concrete pathways tend to be centred around some kind of misaligned AGI, just in much more detail. Conversely, someone with a very different worldview who does such an analysis might end up with concrete pathways centred around severe discrimination of marginalized groups.
There are indeed many different “sub-worldviews”, and I was kind of lumping them all under one big umbrella. To me, the most defining characteristic of this worldview is AI-centrism, and treating the impending AGI as an extremely big deal — not just like any other big deals we have seen before, but this will be unprecedented. Those within this overarching worldview would differ in terms of the details, e.g. will it kill everyone? or will it just lead to gradual disempowerment? are LLMs getting us to AGI? or is it some yet-to-be-discovered architecture? should we focus on getting to AGI safely? or start thinking more about the post-AGI world? I think many people move between these “sub-worldviews” as they see evidences that update their priors, but way fewer people move out of this overarching worldview entirely.