I don’t think it’s quite right that scaling labs universally need more in the way of “Iterators” than other archetypes. For instance, from Anthropic’s Interpretability April update (https://transformer-circuits.pub/2024/april-update/index.html), we currently believe management is the most important role our team is hiring for right now:
Managers—We see this as the most important role that we’re hiring for right now.
Our growth is likely to be bottlenecked on management capacity, and finding the right fit for the team could make a huge difference to our long-term success.
Filling this role has been challenging because we’re looking for someone with experience in a research or engineering environment, who is excited about and experienced with people and project management, and who is enthusiastic about our research agenda and mission.
I’d also add that “Research Engineer” is an extremely important profile to doing a lot of scaling lab work and feels more important to me than this post suggests (at least when I think of how much Anthropic Interpretability is interested in these roles). E.g. our recent paper Scaling Monosemanticitywould have been completely, 100% impossible without research engineers tackling seriously hard engineering challenges. [See the Author Contributions/”Infrastructure, Tooling, and Core Algorithmic Work” section for a flavor of the sorts of work involved.]
Without commenting too much on a specific org (anonymity commitments, sorry!), I think we’re in agreement here and that the information you provided doesn’t conflict with the findings of the report (although, since your comment is focused on a single org in a way that the report is simply not licensed to be, your comment is somewhat higher resolution).
One manager creates bandwidth for 5-10 additional Iterator hires, so the two just aren’t weighted the same wrt something like ‘how many of each should we have in a MATS cohort?’ In a sense, a manger is responsible for ~half the output of their team, or is “worth” 2.5-5 employees (if, counterfactually, you wouldn’t have been able to hire those folks at all). This is, of course, conditional on being able to get those employees once you hire the manager. Many orgs also hire managers from within, especially if they have a large number of folks in associate positions who’ve been with the org > 1 year and have the requisite soft skills to manage effectively.
If you told me “We need x new safety teams from scratch at an existing org”, probably I would want to produce (1-2)x Amplifiers (to be managers), and (5-10)x Iterators. Keeping in mind the above note about internal hires, this pushes the need (in terms of ‘absolute number of heads that can do the role’) for Amplifiers relative to Iterators down somewhat.
Fwiw, I think that research engineer is a pretty Iterator-specced role, although with different technical requirements from, i.e. “Member of Technical Staff” and “Research Scientist”, and that pursuing an experimental agenda that requires building a lot of your own tools (with an existing software development background) is probably great prep for that position. My guess is that MATS scholars focused on evals, demos, scalable oversight, or control could make strong research engineers down the line, and that things like CodeSignal tests would help catch strong Research Engineers in the wild.
...we’re looking for someone with experience in a research or engineering environment, who is excited about and experienced with people and project management, and who is enthusiastic about our research agenda and mission.
I’d also predict that, if management becomes a massive bottleneck to Anthropic scaling, they would restructure somewhat to make the prerequisites for these roles a little less demanding (as has DeepMind, with their People Managers, as opposed to Research Leads, and as have several growing technical orgs, as mentioned in the post).
I don’t think it’s quite right that scaling labs universally need more in the way of “Iterators” than other archetypes. For instance, from Anthropic’s Interpretability April update (https://transformer-circuits.pub/2024/april-update/index.html), we currently believe management is the most important role our team is hiring for right now:
I’d also add that “Research Engineer” is an extremely important profile to doing a lot of scaling lab work and feels more important to me than this post suggests (at least when I think of how much Anthropic Interpretability is interested in these roles). E.g. our recent paper Scaling Monosemanticity would have been completely, 100% impossible without research engineers tackling seriously hard engineering challenges. [See the Author Contributions/”Infrastructure, Tooling, and Core Algorithmic Work” section for a flavor of the sorts of work involved.]
Without commenting too much on a specific org (anonymity commitments, sorry!), I think we’re in agreement here and that the information you provided doesn’t conflict with the findings of the report (although, since your comment is focused on a single org in a way that the report is simply not licensed to be, your comment is somewhat higher resolution).
One manager creates bandwidth for 5-10 additional Iterator hires, so the two just aren’t weighted the same wrt something like ‘how many of each should we have in a MATS cohort?’ In a sense, a manger is responsible for ~half the output of their team, or is “worth” 2.5-5 employees (if, counterfactually, you wouldn’t have been able to hire those folks at all). This is, of course, conditional on being able to get those employees once you hire the manager. Many orgs also hire managers from within, especially if they have a large number of folks in associate positions who’ve been with the org > 1 year and have the requisite soft skills to manage effectively.
If you told me “We need x new safety teams from scratch at an existing org”, probably I would want to produce (1-2)x Amplifiers (to be managers), and (5-10)x Iterators. Keeping in mind the above note about internal hires, this pushes the need (in terms of ‘absolute number of heads that can do the role’) for Amplifiers relative to Iterators down somewhat.
Fwiw, I think that research engineer is a pretty Iterator-specced role, although with different technical requirements from, i.e. “Member of Technical Staff” and “Research Scientist”, and that pursuing an experimental agenda that requires building a lot of your own tools (with an existing software development background) is probably great prep for that position. My guess is that MATS scholars focused on evals, demos, scalable oversight, or control could make strong research engineers down the line, and that things like CodeSignal tests would help catch strong Research Engineers in the wild.
I’d also predict that, if management becomes a massive bottleneck to Anthropic scaling, they would restructure somewhat to make the prerequisites for these roles a little less demanding (as has DeepMind, with their People Managers, as opposed to Research Leads, and as have several growing technical orgs, as mentioned in the post).