There aren’t many people with PhD-level research experience in relevant fields who are focusing on AI safety, so I think it’s a bit early to conclude these skills are “extremely rare” amongst qualified individuals.
AI safety research spans a broad range of areas, but for the more ML-oriented research the skills are, unsurprisingly, not that different from other fields of ML research. There are two main differences I’ve noticed:
In AI safety you often have to turn ill-defined, messy intuitions into formal problem statements before you can start working on them. In other areas of AI, people are more likely to have already formalized the problem for you.
It’s important to be your own harshest critic. This is cultivated in some other fields, such as computer security and (in a different way) in mathematics. But ML tends to encourage a sloppy attitude here.
Both of these I think are fairly easily measurable from looking at someone’s past work and talking to them, though.
Identifying highly capable individuals is indeed hard, but I don’t think this is any more of a problem in AI safety research than in other fields. I’ve been involved in screening in two different industries (financial trading and, more recently, AI research). In both cases there’s always been a lot of guesswork involved, and I don’t get the impression it’s any better in other sectors. If anything I’ve found screening in AI easier: at least you can actually read the person’s work, rather than everything behind behind an NDA (common in many industries).
Identifying highly capable individuals is indeed hard, but I don’t think this is any more of a problem in AI safety research than in other fields.
Quite. I think that my model of Eli was setting the highest standard possible—not merely a good researcher, but a great one, the sort of person who can bring whole new paradigms/subfields into existence (Kahneman & Tversky, Von Neumann, Shannon, Einstein, etc), and then noting that because the tails come apart (aka regressional goodharting), optimising for the normal metrics used in standard hiring practices won’t get you these researchers (I realise that probably wasn’t true for Von Neumann, but I think it was true for all the others).
There aren’t many people with PhD-level research experience in relevant fields who are focusing on AI safety, so I think it’s a bit early to conclude these skills are “extremely rare” amongst qualified individuals.
AI safety research spans a broad range of areas, but for the more ML-oriented research the skills are, unsurprisingly, not that different from other fields of ML research. There are two main differences I’ve noticed:
In AI safety you often have to turn ill-defined, messy intuitions into formal problem statements before you can start working on them. In other areas of AI, people are more likely to have already formalized the problem for you.
It’s important to be your own harshest critic. This is cultivated in some other fields, such as computer security and (in a different way) in mathematics. But ML tends to encourage a sloppy attitude here.
Both of these I think are fairly easily measurable from looking at someone’s past work and talking to them, though.
Identifying highly capable individuals is indeed hard, but I don’t think this is any more of a problem in AI safety research than in other fields. I’ve been involved in screening in two different industries (financial trading and, more recently, AI research). In both cases there’s always been a lot of guesswork involved, and I don’t get the impression it’s any better in other sectors. If anything I’ve found screening in AI easier: at least you can actually read the person’s work, rather than everything behind behind an NDA (common in many industries).
Quite. I think that my model of Eli was setting the highest standard possible—not merely a good researcher, but a great one, the sort of person who can bring whole new paradigms/subfields into existence (Kahneman & Tversky, Von Neumann, Shannon, Einstein, etc), and then noting that because the tails come apart (aka regressional goodharting), optimising for the normal metrics used in standard hiring practices won’t get you these researchers (I realise that probably wasn’t true for Von Neumann, but I think it was true for all the others).
I like the breakdown of those two bullet points, a lot, and I want to think more about them.
I bet that you could do that, yes. But that seems like a different question than making a scalable system that can do it.
In any case, Ben articulates the view that generated the comment above, above.