I want to second what TheAthenians is asking here. I think thereâs plenty that can be valuable from evaluations (or even critiques) like this, and thereâs a lot that youâve (Omega) posted here and in the Redwood Critique which is useful to know. In particular:
Startups in AI alignment are likely to be over-indexed in driven, technically capable people, but not those with extensive experience in how to run organisations[1]
Navigating the pressure between gaining funding by promising exciting capabilities to funders and stopping contributing to ârace dynamicsâ[2] is a hard problem that these orgs need to be aware of, and have structures against them to prevent this
Other labs outside of the EA/âLW âingroupâ may have done comparable/âbetter research, or provided better outputs, and might be a better use of future AI Safety funding[3]
I think itâs a very good sign that youâve shared both drafts so far with Redwood and Conjecture. No notes here, good job :)
However, after reading this post and reflecting, I canât help but agree with the somewhat sceptical mood of most(?) commenters here? In particular:
You donât say why youâre doing this series, either in this post, or the Redwood one, or in your shortform. That feels really strange to meânot to have a âPost 0â in the sequence saying, for example âAs people who work full time on AI Safety, we have noticed problems in some prestigious EA/âAIS organisations. This series is an attempt to make these concerns public, and hopefully improve the field etc. etc.â. Just out of the gate: âIn this series, we evaluate AI safety organizations that have received more than $10 million per year in funding.â[4]
The anonymisation kind of irks me. I get sometimes itâs necessary, and there may be repercussions in the industy. But to the extent those repercussions are because you posted high-profile, highly-negative assements of organisations that turned out to not be accurate...[5] thatâs not wholly unjustified? At the very least, your confidence seems far above the claims made in this piece (as marius mentions), and the confidential sources basically turn this into a âhe said /â she saidâ between this post and Conjectureâs response
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I canât help wonder if these should be separated outâthere seems to be some âcorrectâ paradigm that the authors of âOmegaâ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
Thank you for raising this point â youâre right that we donât explain why we are writing this series, and we will update the sequence description to be more transparent on that point. The reasons you suggest are basically correct.
With increased attention to TAIS there are many people trying to get into TAIS roles. Without significant context on organizations, new entrants to the field will tend to go to TAIS organizations based on their prominence caused by factors such as total funding, media coverage, volume of output, etc. Much of the discussion we have observed around TAIS organizations, especially criticisms of them, happens behind closed doors in conversations that junior people are usually not privy to. We wish to help disseminate this information more broadly to enable individuals to make a better informed decision.
We are concerned âthat the attractiveness of working at an organization that is connected to the EA or TAIS communities makes it more likely for community members to take jobs at such organizations even if this will result in a lower lifetime impact than alternatives. Conjectureâs sponsorship of TAIS field building efforts may also lead new talent, who are unfamiliar with Conjectureâs history, to have an overly rosy impression of them.â
Regarding anonymization, we are also frustrated that we are not able to share more details. The sources we cite are credible to us (we believe the people who brought them to us to have high integrity). We try to provide relevant context where we can but donât always have control over this. We donât think an issue being based on (who you) trust, means that we shouldnât bring these issues to light. We would encourage people who are making active decisions about their potential employment or collaboration with Conjecture to speak to people they trust and draw their own conclusions. We plan to edit all our recommendations to say this more explicitly.
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I canât help wonder if these should be separated outâthere seems to be some âcorrectâ paradigm that the authors of âOmegaâ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
We believe that an organization should be graded on multiple metrics. Their outputs are where we would put the most weight. However, their strategy and governance are also key. The last year has brought into sharp relief the importance of strong organizational governance.
We donât believe that there is a specific âparadigmâ we advocate for. We would support the TAIS community pursuing a diversified research agenda.
Thanks for your thoughtful response here (and elsewhere). I definitely think youâre acting in good faith (again, I think sharing your evaluations with the labs beforehand and seeking information/âclarification from them is really big evidence of this), and I have appreciated both posts even if I was more on the critical side for this one. Iâm sorry that youâve found the response to this post difficult and I apologise if I contributed to that unfairly. I look forward to you continuing the series (I think with Anthropic?).
On the object level I donât think we actually disagree that muchâIâm very much in agreement with your sentiment on organisational governance, and how much this has been shown to be crucial in both the EA and AI spaces over the past year. I think allowing critical evaluations of AI Safety from inside the field is important to making sure the field stays healthy. I agree that many can achieve good impact working outside of explicitly EA-aligned organisations, and to not give those organisations a âpassâ because of that affiliationâespecially if they are in the early stages of their career. And I agree that rate at which Conjecture has scaled at will likely lead to organisational problems that may impact the quality of their output.
So on reflection, perhaps why my reaction to this post was more mixed than my reaction to the Redwood Post was is because you made some very strong and critical claims about Conjecture but the evidence you presented was often vague or a statement of your own beliefs.[1] So, for example, numerous concerns about Connorâs behaviour are stated in the article, but I donât have much to update on apart from âThe authors of Omega interpret these events as a sign of poor/âuntrustworthy characterâ, and if I donât share the same interpretation (or to the same degree),[2] our beliefs canât converge any more unless further evidence/âcontext of those claims is provided.
The same goes for technical assessments about the quality of Conjectureâs workâwhere the evidence is simply: âWe believe most of Conjectureâs publicly available research to date is low-quality.â Perhaps Iâm asking for more technical details about your evaluation of research lab output here which might not be what the post is designed for, but itâs probably the kind of evidence that would convince me most here.
A final example is on whether Conjecture has damaged the AI Safety cause among UK policymakers. Given the writers, sources, and policymakers in question are all anonymous I simply have very little ability to adjudicate the extent to which this claim is true. This is, I think, the downside of your decision to remain anonymousâit means that any trust that the authors of Omega have built up with their work in the AI Safety community canât be used to vouch for these claims where the evidence is more ambiguous.
I do accept, and take it as a point in your favour, that this may in large part be due to Conjectureâs reluctance to co-ordinate with the rest of the ML Community and make their work more available for public scrutiny
Hi JWS, Just wanted to let you know that weâve posted our introduction to the series. We hope it adds some clarity to the points youâve raised here for others.
I want to second what TheAthenians is asking here. I think thereâs plenty that can be valuable from evaluations (or even critiques) like this, and thereâs a lot that youâve (Omega) posted here and in the Redwood Critique which is useful to know. In particular:
Startups in AI alignment are likely to be over-indexed in driven, technically capable people, but not those with extensive experience in how to run organisations[1]
Navigating the pressure between gaining funding by promising exciting capabilities to funders and stopping contributing to ârace dynamicsâ[2] is a hard problem that these orgs need to be aware of, and have structures against them to prevent this
Other labs outside of the EA/âLW âingroupâ may have done comparable/âbetter research, or provided better outputs, and might be a better use of future AI Safety funding[3]
I think itâs a very good sign that youâve shared both drafts so far with Redwood and Conjecture. No notes here, good job :)
However, after reading this post and reflecting, I canât help but agree with the somewhat sceptical mood of most(?) commenters here? In particular:
You donât say why youâre doing this series, either in this post, or the Redwood one, or in your shortform. That feels really strange to meânot to have a âPost 0â in the sequence saying, for example âAs people who work full time on AI Safety, we have noticed problems in some prestigious EA/âAIS organisations. This series is an attempt to make these concerns public, and hopefully improve the field etc. etc.â. Just out of the gate: âIn this series, we evaluate AI safety organizations that have received more than $10 million per year in funding.â[4]
The anonymisation kind of irks me. I get sometimes itâs necessary, and there may be repercussions in the industy. But to the extent those repercussions are because you posted high-profile, highly-negative assements of organisations that turned out to not be accurate...[5] thatâs not wholly unjustified? At the very least, your confidence seems far above the claims made in this piece (as marius mentions), and the confidential sources basically turn this into a âhe said /â she saidâ between this post and Conjectureâs response
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I canât help wonder if these should be separated outâthere seems to be some âcorrectâ paradigm that the authors of âOmegaâ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
Given our demographics, I think this is just a perennial EA problem
The OpenAI failure mode
Iâm not taking an object-level position on this, but itâs something that needs to be monitored occasionally
https://ââforum.effectivealtruism.org/ââposts/ââDaRvpDHHdaoad9Tfu/ââcritiques-of-prominent-ai-safety-labs-redwood-research
For the record, I donât think Iâm in a position to give an answer. This is a hypothetical/âconditional point
Thank you for raising this point â youâre right that we donât explain why we are writing this series, and we will update the sequence description to be more transparent on that point. The reasons you suggest are basically correct.
With increased attention to TAIS there are many people trying to get into TAIS roles. Without significant context on organizations, new entrants to the field will tend to go to TAIS organizations based on their prominence caused by factors such as total funding, media coverage, volume of output, etc. Much of the discussion we have observed around TAIS organizations, especially criticisms of them, happens behind closed doors in conversations that junior people are usually not privy to. We wish to help disseminate this information more broadly to enable individuals to make a better informed decision.
We are concerned âthat the attractiveness of working at an organization that is connected to the EA or TAIS communities makes it more likely for community members to take jobs at such organizations even if this will result in a lower lifetime impact than alternatives. Conjectureâs sponsorship of TAIS field building efforts may also lead new talent, who are unfamiliar with Conjectureâs history, to have an overly rosy impression of them.â
Regarding anonymization, we are also frustrated that we are not able to share more details. The sources we cite are credible to us (we believe the people who brought them to us to have high integrity). We try to provide relevant context where we can but donât always have control over this. We donât think an issue being based on (who you) trust, means that we shouldnât bring these issues to light. We would encourage people who are making active decisions about their potential employment or collaboration with Conjecture to speak to people they trust and draw their own conclusions. We plan to edit all our recommendations to say this more explicitly.
We believe that an organization should be graded on multiple metrics. Their outputs are where we would put the most weight. However, their strategy and governance are also key. The last year has brought into sharp relief the importance of strong organizational governance.
We donât believe that there is a specific âparadigmâ we advocate for. We would support the TAIS community pursuing a diversified research agenda.
Thanks for your thoughtful response here (and elsewhere). I definitely think youâre acting in good faith (again, I think sharing your evaluations with the labs beforehand and seeking information/âclarification from them is really big evidence of this), and I have appreciated both posts even if I was more on the critical side for this one. Iâm sorry that youâve found the response to this post difficult and I apologise if I contributed to that unfairly. I look forward to you continuing the series (I think with Anthropic?).
On the object level I donât think we actually disagree that muchâIâm very much in agreement with your sentiment on organisational governance, and how much this has been shown to be crucial in both the EA and AI spaces over the past year. I think allowing critical evaluations of AI Safety from inside the field is important to making sure the field stays healthy. I agree that many can achieve good impact working outside of explicitly EA-aligned organisations, and to not give those organisations a âpassâ because of that affiliationâespecially if they are in the early stages of their career. And I agree that rate at which Conjecture has scaled at will likely lead to organisational problems that may impact the quality of their output.
So on reflection, perhaps why my reaction to this post was more mixed than my reaction to the Redwood Post was is because you made some very strong and critical claims about Conjecture but the evidence you presented was often vague or a statement of your own beliefs.[1] So, for example, numerous concerns about Connorâs behaviour are stated in the article, but I donât have much to update on apart from âThe authors of Omega interpret these events as a sign of poor/âuntrustworthy characterâ, and if I donât share the same interpretation (or to the same degree),[2] our beliefs canât converge any more unless further evidence/âcontext of those claims is provided.
The same goes for technical assessments about the quality of Conjectureâs workâwhere the evidence is simply: âWe believe most of Conjectureâs publicly available research to date is low-quality.â Perhaps Iâm asking for more technical details about your evaluation of research lab output here which might not be what the post is designed for, but itâs probably the kind of evidence that would convince me most here.
A final example is on whether Conjecture has damaged the AI Safety cause among UK policymakers. Given the writers, sources, and policymakers in question are all anonymous I simply have very little ability to adjudicate the extent to which this claim is true. This is, I think, the downside of your decision to remain anonymousâit means that any trust that the authors of Omega have built up with their work in the AI Safety community canât be used to vouch for these claims where the evidence is more ambiguous.
I do accept, and take it as a point in your favour, that this may in large part be due to Conjectureâs reluctance to co-ordinate with the rest of the ML Community and make their work more available for public scrutiny
For the record, my only contact with Connor personally has been to chat with him over a beer at the EAG afterparty
Hi JWS, Just wanted to let you know that weâve posted our introduction to the series. We hope it adds some clarity to the points youâve raised here for others.