Wow very well put. This is the one that scares me the most out of these three, and I think there could be more exploring to be done as to first, how strong an incentive this might be, and then how that incentive can change people’s view on their job and AI
“To work for an AI capabilities company rather than outside (higher salary)”
I know it’s a side note not directly related to the original question, but I would be interested to see data comparing
Safety researchers’ pdoom who work for AI capabilities companies vs. Those who work for independent safety orgs (this might have been done already)
What proportion of AI safety people who started working for capabilities orgs have moved on over time (I would call it defected) to working more on capabilities than alignment.
These are all great points. I was planning to add this into the main post, but I don’t think it ended up in the final draft—so thanks for raising this!
There are also big incentive gradients within longtermism:
To work on AI experiments rather than AI theory (higher salary, better locations, more job security)
To work for a grantmaker rather than a grantee (for job security), and
To work for an AI capabilities company rather than outside (higher salary)
To work in AI instead of other areas (higher salary, topic is shiny)
(Disclosure: I decided to work in biorisk and not AI)
Wow very well put. This is the one that scares me the most out of these three, and I think there could be more exploring to be done as to first, how strong an incentive this might be, and then how that incentive can change people’s view on their job and AI
“To work for an AI capabilities company rather than outside (higher salary)”
I know it’s a side note not directly related to the original question, but I would be interested to see data comparing
Safety researchers’ pdoom who work for AI capabilities companies vs. Those who work for independent safety orgs (this might have been done already)
What proportion of AI safety people who started working for capabilities orgs have moved on over time (I would call it defected) to working more on capabilities than alignment.
These are all great points. I was planning to add this into the main post, but I don’t think it ended up in the final draft—so thanks for raising this!