Thanks to the authors for taking the time to think about how to improve our organization and the field of AI takeover prevention as a whole. I share a lot of the concerns mentioned in this post, and I’ve been spending a lot of my attention trying to improve some of them (though I also have important disagreements with parts of the post).
Here’s some information that perhaps supports some of the points made in the post and adds texture, since it seems hard to properly critique a small organization without a lot of context and inside information. (This is adapted from my notes over the past few months.)
Most importantly, I am eager to increase our rate of research output – and critically to have that increase be sustainable because it’s done by a more stable and well-functioning team. I don’t think we should be satisfied with the current output rate, and I think this rate being too low is in substantial part due to not having had the right organizational shape or sufficiently solid management practices (which, in empathy with the past selves of the Redwood leadership team, is often a tricky thing for young organizations to figure out, and is perhaps especially tricky in this field).
I think the most important error that we’ve made so far is trying to scale up too quickly. I feel bad about the ways in which this has contributed to people who’ve worked here having an unexpectedly bad experience. I believe this was upstream of other organizational mistakes and that it put stress on our relative inexperience in management. While having fewer staff gives fewer people a chance to have roles working on our type of AI alignment research, I expect it will help increase the management quality per person. For example, I think there will be more and better opportunities for researchers at Redwood to grow, which is something I’ve been excited to focus on. I think scaling too quickly was somewhat downstream of not having an extremely clear articulation of what specific flavor of research output we are aiming to produce and, in turn, having a tested organization that we believe reliably produces those outputs.
I think this was an unforced error on our part – for example, Holden and Ajeya expressed concerns to me about this multiple times. My thinking at the time was something like “this sure seems like a pretty confusing field in a lot of ways, and (something something act-omission bias) I’m worried that if we chose an unrealistically high standard for clarity to gate on for organizational growth, then we might learn more slowly than we might otherwise, and fail to give people opportunities to contribute to the field.” I now think that I was wrong about this.
With that said, I’ll also briefly note some of the ways I disagree with the content and framing of this post:
We think our “causal scrubbing” work is our most significant output so far – substantially more important than, for example, our “Interpretability in the Wild” work.
At the beginning of our adversarial training project, we reviewed the literature (including the papers in the list that the above post links to) and discussed the project proposal with relevant experts. I think we made important mistakes in that project, but I don’t think that we failed to understand the state of the field.
I am moderately optimistic about Redwood’s current trajectory and our potential to contribute to making the future go well. I feel substantially better about the place that we’re in now relative to where we were, say, 6 months ago. We remain a relatively young organization making an unusual bet.
I really appreciate feedback, and if anyone reading this wants to send feedback to us about Redwood, you can email info at rdwrs.com or, if you prefer anonymity, visit www.admonymous.co/redwood.
Ditto pseudonym, I recognize from another comment that there is an upcoming Constellation post from the original poster and a more effortful response forthcoming there, but I still think that despite receiving this piece in advance I am kind of surprised the following were not responded to?
Lack of Senior ML Research Staff
Lack of Comm… w/ ML Community
Conflicts of interest with funders
I guess people are busy and this is not a priority—seems like people are mostly thinking about Underwhelming Research Output (and Nate himself seems to say as much here)
Hi Nate, can you comment a bit more about this section?
We’ve heard multiple cases of people being fired after something negative happens in their life (personal, conflict at work, etc) that causes them to be temporarily less productive at work. While Redwood management have made some efforts to offer support to staff (e.g. offering unpaid leave on some occasions), we believe it may not have been done consistently, and are aware of cases where termination happened with little warning.
I feel like this would be among the more negative updates I would make about Redwood if true, but think it would be possible that there are differences in how a specific event is seen by different parties. Specifically, these seem to reflect weaker organizational or management practices that aren’t to do with Redwood making an “unusual bet” (though relevant to it being a young organization).
Specifically:
Has Redwood ever terminated someone for losing productivity that they otherwise wouldn’t have, due to a personal life event?
Does Redwood have a policy around leave that includes support for personal life events?
Does Redwood have a clear termination process including warnings before a termination where reasonable, and opportunities for an employee to course-correct with the support of the organization?
Does Redwood have a clear termination process including warnings before a termination where reasonable
I think I’m an unusual case, but I found out a short term contract had been ended early through an automated email, and I received no response when contacting several Redwood staff to check if I had been terminated.
I think this is very uncharacteristic though: they’re all good people and I’m net optimistic about Redwood’s future. I think they can improve their communication around hiring/trialling/firing processes though.
Edit: I’ve chatted with Buck and it seems like this was a communication problem.
I know nothing about this organisation, and very little about this field, but this is an impressively humble and open response from a leader of an org in the face of a very critical article. No comment on content, but I appreciate the approach @Nate Thomas
Thanks to the authors for taking the time to think about how to improve our organization and the field of AI takeover prevention as a whole. I share a lot of the concerns mentioned in this post, and I’ve been spending a lot of my attention trying to improve some of them (though I also have important disagreements with parts of the post).
Here’s some information that perhaps supports some of the points made in the post and adds texture, since it seems hard to properly critique a small organization without a lot of context and inside information. (This is adapted from my notes over the past few months.)
Most importantly, I am eager to increase our rate of research output – and critically to have that increase be sustainable because it’s done by a more stable and well-functioning team. I don’t think we should be satisfied with the current output rate, and I think this rate being too low is in substantial part due to not having had the right organizational shape or sufficiently solid management practices (which, in empathy with the past selves of the Redwood leadership team, is often a tricky thing for young organizations to figure out, and is perhaps especially tricky in this field).
I think the most important error that we’ve made so far is trying to scale up too quickly. I feel bad about the ways in which this has contributed to people who’ve worked here having an unexpectedly bad experience. I believe this was upstream of other organizational mistakes and that it put stress on our relative inexperience in management. While having fewer staff gives fewer people a chance to have roles working on our type of AI alignment research, I expect it will help increase the management quality per person. For example, I think there will be more and better opportunities for researchers at Redwood to grow, which is something I’ve been excited to focus on. I think scaling too quickly was somewhat downstream of not having an extremely clear articulation of what specific flavor of research output we are aiming to produce and, in turn, having a tested organization that we believe reliably produces those outputs.
I think this was an unforced error on our part – for example, Holden and Ajeya expressed concerns to me about this multiple times. My thinking at the time was something like “this sure seems like a pretty confusing field in a lot of ways, and (something something act-omission bias) I’m worried that if we chose an unrealistically high standard for clarity to gate on for organizational growth, then we might learn more slowly than we might otherwise, and fail to give people opportunities to contribute to the field.” I now think that I was wrong about this.
With that said, I’ll also briefly note some of the ways I disagree with the content and framing of this post:
We think our “causal scrubbing” work is our most significant output so far – substantially more important than, for example, our “Interpretability in the Wild” work.
At the beginning of our adversarial training project, we reviewed the literature (including the papers in the list that the above post links to) and discussed the project proposal with relevant experts. I think we made important mistakes in that project, but I don’t think that we failed to understand the state of the field.
I am moderately optimistic about Redwood’s current trajectory and our potential to contribute to making the future go well. I feel substantially better about the place that we’re in now relative to where we were, say, 6 months ago. We remain a relatively young organization making an unusual bet.
I really appreciate feedback, and if anyone reading this wants to send feedback to us about Redwood, you can email info at rdwrs.com or, if you prefer anonymity, visit www.admonymous.co/redwood.
Ditto pseudonym, I recognize from another comment that there is an upcoming Constellation post from the original poster and a more effortful response forthcoming there, but I still think that despite receiving this piece in advance I am kind of surprised the following were not responded to?
Lack of Senior ML Research Staff
Lack of Comm… w/ ML Community
Conflicts of interest with funders
I guess people are busy and this is not a priority—seems like people are mostly thinking about Underwhelming Research Output (and Nate himself seems to say as much here)
Hi Nate, can you comment a bit more about this section?
I feel like this would be among the more negative updates I would make about Redwood if true, but think it would be possible that there are differences in how a specific event is seen by different parties. Specifically, these seem to reflect weaker organizational or management practices that aren’t to do with Redwood making an “unusual bet” (though relevant to it being a young organization).
Specifically:
Has Redwood ever terminated someone for losing productivity that they otherwise wouldn’t have, due to a personal life event?
Does Redwood have a policy around leave that includes support for personal life events?
Does Redwood have a clear termination process including warnings before a termination where reasonable, and opportunities for an employee to course-correct with the support of the organization?
I think I’m an unusual case, but I found out a short term contract had been ended early through an automated email, and I received no response when contacting several Redwood staff to check if I had been terminated.I think this is very uncharacteristic though: they’re all good people and I’m net optimistic about Redwood’s future. I think they can improve their communication around hiring/trialling/firing processes though.
Edit: I’ve chatted with Buck and it seems like this was a communication problem.
I know nothing about this organisation, and very little about this field, but this is an impressively humble and open response from a leader of an org in the face of a very critical article. No comment on content, but I appreciate the approach @Nate Thomas