On the “Worries about bias towards AI and lack of AI expertise” section, can’t you also make the argument that everyone finds AI cool, experts and novices alike?
AI novices also find AI cool, and finally, there is a way for them to get into an AI career, associate with a cool community full of funding opportunities even for novices.
I’m surprised by your reason for being skeptical about AI novices on the grounds that they don’t know enough to be worried. Take a “novice” who has read all the x-risk books, forum posts and podcasts vs an AI expert who’s worked on ML for 15 years. It’s possible that they know the same amount about AI X-risk mitigation, and would perhaps have similar success rate working on some alignment research (which to a great deal involves GPT-3 prompt hacking with near-0 maths).
What’s more, a AI novice might be better off than an AI expert. They might find it easier to navigate the funding landscape, have more time/smaller opportunity cost to go to all the EA events, are less likely to critically argue all the time, and thus may have better opportunities to get involved in grantmaking or get maybe smaller grants themselves. Imagine that two groups wanted to organise an AI camp or event: a group of AI novice undergrads who have been engaged in EA vs a group of AI profs with no EA connections. Who is more likely to get funding?
EA-funded AI safety is actually a pretty sweet deal for an AI novice who gets to do something that’s cool at very little cost.
Consequently, it’s possible to be skeptical of the motivations anyone in AI safety, expert or novice, on the grounds that “isn’t it convenient the best way to save the world is to do cool AI stuff?”
Consequently, it’s possible to be skeptical of the motivations anyone in AI safety, expert or novice, on the grounds that “isn’t it convenient the best way to save the world is to do cool AI stuff?”
Fair point overall, and I’ll edit in a link to this comment in the post. It would be interesting to see data on what percentage of people working AI safety due to EA motivations would likely be working in AI regardless of impact. I’d predict that it’s significant but not a large majority (say, 80% CI of 25-65%).
A few reactions to specific points/claims:
It’s possible that they know the same amount about AI X-risk mitigation, and would perhaps have similar success rate working on some alignment research (which to a great deal involves GPT-3 prompt hacking with near-0 maths).
My understanding is that most alignment research involves either maths or skills similar to ML research/engineering; there is some ~GPT-3 prompt hacking (e.g. this post?) but it seems like <10% of the field?
Imagine that two groups wanted to organise an AI camp or event: a group of AI novice undergrads who have been engaged in EA vs a group of AI profs with no EA connections. Who is more likely to get funding?
I’m not sure about specifically organizing an event, but I’d guess that experienced AI profs with no EA connections but who seemed genuinely interested in reducing AI x-risk would be able to get substantial funding/support for their research.
EA-funded AI safety is actually a pretty sweet deal for an AI novice who gets to do something that’s cool at very little cost.
The field has probably gotten easier to break into over time but I’d guess most people attempting to enter still experience substantial costs, such as rejections and mental health struggles.
On the “Worries about bias towards AI and lack of AI expertise” section, can’t you also make the argument that everyone finds AI cool, experts and novices alike?
AI novices also find AI cool, and finally, there is a way for them to get into an AI career, associate with a cool community full of funding opportunities even for novices.
I’m surprised by your reason for being skeptical about AI novices on the grounds that they don’t know enough to be worried. Take a “novice” who has read all the x-risk books, forum posts and podcasts vs an AI expert who’s worked on ML for 15 years. It’s possible that they know the same amount about AI X-risk mitigation, and would perhaps have similar success rate working on some alignment research (which to a great deal involves GPT-3 prompt hacking with near-0 maths).
What’s more, a AI novice might be better off than an AI expert. They might find it easier to navigate the funding landscape, have more time/smaller opportunity cost to go to all the EA events, are less likely to critically argue all the time, and thus may have better opportunities to get involved in grantmaking or get maybe smaller grants themselves. Imagine that two groups wanted to organise an AI camp or event: a group of AI novice undergrads who have been engaged in EA vs a group of AI profs with no EA connections. Who is more likely to get funding?
EA-funded AI safety is actually a pretty sweet deal for an AI novice who gets to do something that’s cool at very little cost.
Consequently, it’s possible to be skeptical of the motivations anyone in AI safety, expert or novice, on the grounds that “isn’t it convenient the best way to save the world is to do cool AI stuff?”
Fair point overall, and I’ll edit in a link to this comment in the post. It would be interesting to see data on what percentage of people working AI safety due to EA motivations would likely be working in AI regardless of impact. I’d predict that it’s significant but not a large majority (say, 80% CI of 25-65%).
A few reactions to specific points/claims:
My understanding is that most alignment research involves either maths or skills similar to ML research/engineering; there is some ~GPT-3 prompt hacking (e.g. this post?) but it seems like <10% of the field?
I’m not sure about specifically organizing an event, but I’d guess that experienced AI profs with no EA connections but who seemed genuinely interested in reducing AI x-risk would be able to get substantial funding/support for their research.
The field has probably gotten easier to break into over time but I’d guess most people attempting to enter still experience substantial costs, such as rejections and mental health struggles.