Why are you doing critiques instead of evaluations? This seems like you’re deliberately only looking for bad things instead of trying to do a balanced investigation into the impact of an organization.
This seems like bad epistemics and will likely lead to a ton of not necessarily warranted damage to orgs that are trying to do extremely important work. Not commenting on the content of your criticisms of Redwood or Conjecture, but your process.
Knowing there’s a group of anonymous people who are explicitly looking to find fault with orgs feels like an instance of EA culture rewarding criticism to the detriment of the community as a whole. Generally, I can see that you’re trying to do good, but your approach makes me feel like the EA community is hostile and makes me not want to engage with it.
I want to second what TheAthenians is asking here. I think there’s plenty that can be valuable from evaluations (or even critiques) like this, and there’s a lot that you’ve (Omega) posted here and in the Redwood Critique which is useful to know. In particular:
Startups in AI alignment are likely to be over-indexed in driven, technically capable people, but not those with extensive experience in how to run organisations[1]
Navigating the pressure between gaining funding by promising exciting capabilities to funders and stopping contributing to ‘race dynamics’[2] is a hard problem that these orgs need to be aware of, and have structures against them to prevent this
Other labs outside of the EA/LW ‘ingroup’ may have done comparable/better research, or provided better outputs, and might be a better use of future AI Safety funding[3]
I think it’s a very good sign that you’ve shared both drafts so far with Redwood and Conjecture. No notes here, good job :)
However, after reading this post and reflecting, I can’t help but agree with the somewhat sceptical mood of most(?) commenters here? In particular:
You don’t say why you’re doing this series, either in this post, or the Redwood one, or in your shortform. That feels really strange to me—not to have a “Post 0” in the sequence saying, for example “As people who work full time on AI Safety, we have noticed problems in some prestigious EA/AIS organisations. This series is an attempt to make these concerns public, and hopefully improve the field etc. etc.”. Just out of the gate: “In this series, we evaluate AI safety organizations that have received more than $10 million per year in funding.”[4]
The anonymisation kind of irks me. I get sometimes it’s necessary, and there may be repercussions in the industy. But to the extent those repercussions are because you posted high-profile, highly-negative assements of organisations that turned out to not be accurate...[5] that’s not wholly unjustified? At the very least, your confidence seems far above the claims made in this piece (as marius mentions), and the confidential sources basically turn this into a “he said / she said” between this post and Conjecture’s response
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I can’t help wonder if these should be separated out—there seems to be some ‘correct’ paradigm that the authors of ‘Omega’ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
Thank you for raising this point – you’re right that we don’t explain why we are writing this series, and we will update the sequence description to be more transparent on that point. The reasons you suggest are basically correct.
With increased attention to TAIS there are many people trying to get into TAIS roles. Without significant context on organizations, new entrants to the field will tend to go to TAIS organizations based on their prominence caused by factors such as total funding, media coverage, volume of output, etc. Much of the discussion we have observed around TAIS organizations, especially criticisms of them, happens behind closed doors in conversations that junior people are usually not privy to. We wish to help disseminate this information more broadly to enable individuals to make a better informed decision.
We are concerned “that the attractiveness of working at an organization that is connected to the EA or TAIS communities makes it more likely for community members to take jobs at such organizations even if this will result in a lower lifetime impact than alternatives. Conjecture’s sponsorship of TAIS field building efforts may also lead new talent, who are unfamiliar with Conjecture’s history, to have an overly rosy impression of them.”
Regarding anonymization, we are also frustrated that we are not able to share more details. The sources we cite are credible to us (we believe the people who brought them to us to have high integrity). We try to provide relevant context where we can but don’t always have control over this. We don’t think an issue being based on (who you) trust, means that we shouldn’t bring these issues to light. We would encourage people who are making active decisions about their potential employment or collaboration with Conjecture to speak to people they trust and draw their own conclusions. We plan to edit all our recommendations to say this more explicitly.
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I can’t help wonder if these should be separated out—there seems to be some ‘correct’ paradigm that the authors of ‘Omega’ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
We believe that an organization should be graded on multiple metrics. Their outputs are where we would put the most weight. However, their strategy and governance are also key. The last year has brought into sharp relief the importance of strong organizational governance.
We don’t believe that there is a specific “paradigm” we advocate for. We would support the TAIS community pursuing a diversified research agenda.
Thanks for your thoughtful response here (and elsewhere). I definitely think you’re acting in good faith (again, I think sharing your evaluations with the labs beforehand and seeking information/clarification from them is really big evidence of this), and I have appreciated both posts even if I was more on the critical side for this one. I’m sorry that you’ve found the response to this post difficult and I apologise if I contributed to that unfairly. I look forward to you continuing the series (I think with Anthropic?).
On the object level I don’t think we actually disagree that much—I’m very much in agreement with your sentiment on organisational governance, and how much this has been shown to be crucial in both the EA and AI spaces over the past year. I think allowing critical evaluations of AI Safety from inside the field is important to making sure the field stays healthy. I agree that many can achieve good impact working outside of explicitly EA-aligned organisations, and to not give those organisations a ‘pass’ because of that affiliation—especially if they are in the early stages of their career. And I agree that rate at which Conjecture has scaled at will likely lead to organisational problems that may impact the quality of their output.
So on reflection, perhaps why my reaction to this post was more mixed than my reaction to the Redwood Post was is because you made some very strong and critical claims about Conjecture but the evidence you presented was often vague or a statement of your own beliefs.[1] So, for example, numerous concerns about Connor’s behaviour are stated in the article, but I don’t have much to update on apart from “The authors of Omega interpret these events as a sign of poor/untrustworthy character”, and if I don’t share the same interpretation (or to the same degree),[2] our beliefs can’t converge any more unless further evidence/context of those claims is provided.
The same goes for technical assessments about the quality of Conjecture’s work—where the evidence is simply: “We believe most of Conjecture’s publicly available research to date is low-quality.” Perhaps I’m asking for more technical details about your evaluation of research lab output here which might not be what the post is designed for, but it’s probably the kind of evidence that would convince me most here.
A final example is on whether Conjecture has damaged the AI Safety cause among UK policymakers. Given the writers, sources, and policymakers in question are all anonymous I simply have very little ability to adjudicate the extent to which this claim is true. This is, I think, the downside of your decision to remain anonymous—it means that any trust that the authors of Omega have built up with their work in the AI Safety community can’t be used to vouch for these claims where the evidence is more ambiguous.
I do accept, and take it as a point in your favour, that this may in large part be due to Conjecture’s reluctance to co-ordinate with the rest of the ML Community and make their work more available for public scrutiny
Hi JWS, Just wanted to let you know that we’ve posted our introduction to the series. We hope it adds some clarity to the points you’ve raised here for others.
Thanks for highlighting this potential issue. We’d like to clarify that our intention is to evaluate both the positives and negatives. In retrospect, calling our posts “critiques” may have given the wrong impression: although it’s consistent with historical usage of the word[1] it does tend to carry a negative connotation. Ultimately our evaluation of Conjecture ended up fairly negative, exacerbating this impression: we expect future posts in the series on organizations where we have a more mixed evaluation to have a greater mix of positives and negatives.
You are right that overall we focus more on negatives than positives. We believe this is justified since organizations are already incentivized to make the positive case for themselves, and routinely do so in public announcements as well as private recruitment and funding pitches. By contrast, there is little reward from highlighting negatives. Indeed, we’re publishing this anonymously (foregoing any credit we could get from bringing these issues to attention) in order to protect against retaliation.
Our goal is not to make the EA community more hostile, and we’re certainly sorry that this post made you want to engage less with the community. We would not subject an individual or a small organization to this level of scrutiny. Our series is targeted only at large organizations above a certain bar of funding. With a $10 mn budget and a team of 20+ people, we do not think that Conjecture has will be threatened by a small volunteer group of anonymous individuals. If our arguments are specious, Conjecture would only need to dedicate a small fraction of their resources to rebutting them.
We appreciate your bringing attention to these points, and will be updating out sequence description and the introduction to each post to clarify these points in the next few days.
You are right that overall we focus more on negatives than positives. We believe this is justified since organizations are already incentivized to make the positive case for themselves, and routinely do so in public announcements as well as private recruitment and funding pitches.
As a potential consumer of your critiques/evaluations, I would prefer that you distribute your focus exactly to the degree that you believe it to be warranted in light of your independent impression of the org in question, rather than try to rectify a possible imbalance by deliberately erring in the opposite direction.
Hi Pablo, thanks for you comment. We want to clarify that we aren’t trying to balance the critiques in a certain way, just that it so happens that the organizations that are next on our list will have a greater mix of positives and negatives.
Why are you doing critiques instead of evaluations? This seems like you’re deliberately only looking for bad things instead of trying to do a balanced investigation into the impact of an organization.
This seems like bad epistemics and will likely lead to a ton of not necessarily warranted damage to orgs that are trying to do extremely important work. Not commenting on the content of your criticisms of Redwood or Conjecture, but your process.
Knowing there’s a group of anonymous people who are explicitly looking to find fault with orgs feels like an instance of EA culture rewarding criticism to the detriment of the community as a whole. Generally, I can see that you’re trying to do good, but your approach makes me feel like the EA community is hostile and makes me not want to engage with it.
I want to second what TheAthenians is asking here. I think there’s plenty that can be valuable from evaluations (or even critiques) like this, and there’s a lot that you’ve (Omega) posted here and in the Redwood Critique which is useful to know. In particular:
Startups in AI alignment are likely to be over-indexed in driven, technically capable people, but not those with extensive experience in how to run organisations[1]
Navigating the pressure between gaining funding by promising exciting capabilities to funders and stopping contributing to ‘race dynamics’[2] is a hard problem that these orgs need to be aware of, and have structures against them to prevent this
Other labs outside of the EA/LW ‘ingroup’ may have done comparable/better research, or provided better outputs, and might be a better use of future AI Safety funding[3]
I think it’s a very good sign that you’ve shared both drafts so far with Redwood and Conjecture. No notes here, good job :)
However, after reading this post and reflecting, I can’t help but agree with the somewhat sceptical mood of most(?) commenters here? In particular:
You don’t say why you’re doing this series, either in this post, or the Redwood one, or in your shortform. That feels really strange to me—not to have a “Post 0” in the sequence saying, for example “As people who work full time on AI Safety, we have noticed problems in some prestigious EA/AIS organisations. This series is an attempt to make these concerns public, and hopefully improve the field etc. etc.”. Just out of the gate: “In this series, we evaluate AI safety organizations that have received more than $10 million per year in funding.”[4]
The anonymisation kind of irks me. I get sometimes it’s necessary, and there may be repercussions in the industy. But to the extent those repercussions are because you posted high-profile, highly-negative assements of organisations that turned out to not be accurate...[5] that’s not wholly unjustified? At the very least, your confidence seems far above the claims made in this piece (as marius mentions), and the confidential sources basically turn this into a “he said / she said” between this post and Conjecture’s response
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I can’t help wonder if these should be separated out—there seems to be some ‘correct’ paradigm that the authors of ‘Omega’ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
Given our demographics, I think this is just a perennial EA problem
The OpenAI failure mode
I’m not taking an object-level position on this, but it’s something that needs to be monitored occasionally
https://forum.effectivealtruism.org/posts/DaRvpDHHdaoad9Tfu/critiques-of-prominent-ai-safety-labs-redwood-research
For the record, I don’t think I’m in a position to give an answer. This is a hypothetical/conditional point
Thank you for raising this point – you’re right that we don’t explain why we are writing this series, and we will update the sequence description to be more transparent on that point. The reasons you suggest are basically correct.
With increased attention to TAIS there are many people trying to get into TAIS roles. Without significant context on organizations, new entrants to the field will tend to go to TAIS organizations based on their prominence caused by factors such as total funding, media coverage, volume of output, etc. Much of the discussion we have observed around TAIS organizations, especially criticisms of them, happens behind closed doors in conversations that junior people are usually not privy to. We wish to help disseminate this information more broadly to enable individuals to make a better informed decision.
We are concerned “that the attractiveness of working at an organization that is connected to the EA or TAIS communities makes it more likely for community members to take jobs at such organizations even if this will result in a lower lifetime impact than alternatives. Conjecture’s sponsorship of TAIS field building efforts may also lead new talent, who are unfamiliar with Conjecture’s history, to have an overly rosy impression of them.”
Regarding anonymization, we are also frustrated that we are not able to share more details. The sources we cite are credible to us (we believe the people who brought them to us to have high integrity). We try to provide relevant context where we can but don’t always have control over this. We don’t think an issue being based on (who you) trust, means that we shouldn’t bring these issues to light. We would encourage people who are making active decisions about their potential employment or collaboration with Conjecture to speak to people they trust and draw their own conclusions. We plan to edit all our recommendations to say this more explicitly.
We believe that an organization should be graded on multiple metrics. Their outputs are where we would put the most weight. However, their strategy and governance are also key. The last year has brought into sharp relief the importance of strong organizational governance.
We don’t believe that there is a specific “paradigm” we advocate for. We would support the TAIS community pursuing a diversified research agenda.
Thanks for your thoughtful response here (and elsewhere). I definitely think you’re acting in good faith (again, I think sharing your evaluations with the labs beforehand and seeking information/clarification from them is really big evidence of this), and I have appreciated both posts even if I was more on the critical side for this one. I’m sorry that you’ve found the response to this post difficult and I apologise if I contributed to that unfairly. I look forward to you continuing the series (I think with Anthropic?).
On the object level I don’t think we actually disagree that much—I’m very much in agreement with your sentiment on organisational governance, and how much this has been shown to be crucial in both the EA and AI spaces over the past year. I think allowing critical evaluations of AI Safety from inside the field is important to making sure the field stays healthy. I agree that many can achieve good impact working outside of explicitly EA-aligned organisations, and to not give those organisations a ‘pass’ because of that affiliation—especially if they are in the early stages of their career. And I agree that rate at which Conjecture has scaled at will likely lead to organisational problems that may impact the quality of their output.
So on reflection, perhaps why my reaction to this post was more mixed than my reaction to the Redwood Post was is because you made some very strong and critical claims about Conjecture but the evidence you presented was often vague or a statement of your own beliefs.[1] So, for example, numerous concerns about Connor’s behaviour are stated in the article, but I don’t have much to update on apart from “The authors of Omega interpret these events as a sign of poor/untrustworthy character”, and if I don’t share the same interpretation (or to the same degree),[2] our beliefs can’t converge any more unless further evidence/context of those claims is provided.
The same goes for technical assessments about the quality of Conjecture’s work—where the evidence is simply: “We believe most of Conjecture’s publicly available research to date is low-quality.” Perhaps I’m asking for more technical details about your evaluation of research lab output here which might not be what the post is designed for, but it’s probably the kind of evidence that would convince me most here.
A final example is on whether Conjecture has damaged the AI Safety cause among UK policymakers. Given the writers, sources, and policymakers in question are all anonymous I simply have very little ability to adjudicate the extent to which this claim is true. This is, I think, the downside of your decision to remain anonymous—it means that any trust that the authors of Omega have built up with their work in the AI Safety community can’t be used to vouch for these claims where the evidence is more ambiguous.
I do accept, and take it as a point in your favour, that this may in large part be due to Conjecture’s reluctance to co-ordinate with the rest of the ML Community and make their work more available for public scrutiny
For the record, my only contact with Connor personally has been to chat with him over a beer at the EAG afterparty
Hi JWS, Just wanted to let you know that we’ve posted our introduction to the series. We hope it adds some clarity to the points you’ve raised here for others.
Thanks for highlighting this potential issue. We’d like to clarify that our intention is to evaluate both the positives and negatives. In retrospect, calling our posts “critiques” may have given the wrong impression: although it’s consistent with historical usage of the word[1] it does tend to carry a negative connotation. Ultimately our evaluation of Conjecture ended up fairly negative, exacerbating this impression: we expect future posts in the series on organizations where we have a more mixed evaluation to have a greater mix of positives and negatives.
You are right that overall we focus more on negatives than positives. We believe this is justified since organizations are already incentivized to make the positive case for themselves, and routinely do so in public announcements as well as private recruitment and funding pitches. By contrast, there is little reward from highlighting negatives. Indeed, we’re publishing this anonymously (foregoing any credit we could get from bringing these issues to attention) in order to protect against retaliation.
Our goal is not to make the EA community more hostile, and we’re certainly sorry that this post made you want to engage less with the community. We would not subject an individual or a small organization to this level of scrutiny. Our series is targeted only at large organizations above a certain bar of funding. With a $10 mn budget and a team of 20+ people, we do not think that Conjecture has will be threatened by a small volunteer group of anonymous individuals. If our arguments are specious, Conjecture would only need to dedicate a small fraction of their resources to rebutting them.
We appreciate your bringing attention to these points, and will be updating out sequence description and the introduction to each post to clarify these points in the next few days.
“a report that discusses a situation or the writings or ideas of someone and offers a judgment about them”; Cambridge English Dictionary
As a potential consumer of your critiques/evaluations, I would prefer that you distribute your focus exactly to the degree that you believe it to be warranted in light of your independent impression of the org in question, rather than try to rectify a possible imbalance by deliberately erring in the opposite direction.
Hi Pablo, thanks for you comment. We want to clarify that we aren’t trying to balance the critiques in a certain way, just that it so happens that the organizations that are next on our list will have a greater mix of positives and negatives.