This is an anonymous account.
Omega
Karma: 1,103
Hi JWS, Just wanted to let you know that we’ve posted our introduction to the series. We hope it adds some clarity to the points you’ve raised here for others.
An Introduction to Critiques of prominent AI safety organizations
We’ve updated the recommendation about working at Conjecture.
We posted the Redwood post several weeks late on LW, which might explain the low karma on LW.
Hi Bruce, thanks for this thoughtful comment. We think Conjecture needs to address key concerns before we would recommend working there, although we could imagine Conjecture being the best option for a small fraction of people who are (a) excited by their current CoEm approach, (b) can operate independently in an environment with limited mentorship, (c) are confident they can withstand internal pressure (if there is a push to work on capabilities). As a result of these (and other) comments in this comment thread, we will be updating our recommendation to work at Conjecture.
That being said, we expect it to be rare that an individual would have an offer from Conjecture but not have access to other opportunities that are better than independent research. In practice many organizations end up competing for the same, relatively small pool of the very top candidates. Our guess is that most individuals who could receive an offer from Conjecture could pursue one of the paths outlined above in our replies to Marius such as being a research assistant or PhD student in academia, or working in an ML engineering position in an applied team at a major tech company (if not from more promising places like the ones we discuss in the original post). We think these positions can absorb a fairly large amount of talent, although we note that most AI/ML fields are fairly competitive.
Thanks for your offer to help Oli, we really appreciate it. We’ll reach out via DM.
Thanks for sharing your experience, this kind of information is really helpful for us to know.
(personal, emotional reflection)
On a personal note, the past few days have been pretty tough for me. I noticed I took the negative feedback pretty hard.
I hope we have demonstrated that we are acting in good faith, willing to update and engage rigorously with feedback and criticism, but some of the comments made me feel like people thought we were trying to be deceptive or mislead people. It’s pretty difficult to take that in when it’s so far from our intentions.
We try not to let the fact that our posts are anonymous mean we can say things that aren’t as rigorous, but sometimes it feels like people don’t realize that we are people too. I think comments might be phrased differently if we weren’t anonymous.
I think it’s especially hard when this post has taken many weekends to complete, and we’ve invested several hours this week in engaging with comments, which is a tough trade off against other projects.
Brief reflections on the Conjecture post and it’s reception
(Written from the non-technical primary author)
Reception was a lot more critical than I expected. As last time, many good points were raised that pointed out areas where we weren’t clear
We shared it with reviewers (especially ones who we would expect to disagree with us) hoping to pre-empt these criticisms. The gave useful feedback.
However, what we didn’t realize was that the people engaging with our post in the comments were quite different from our reviewers and didnt share the background knowledge that our reviewers did
We included our end line views (based on feedback previously that we didn’t do this enough) and I think it’s those views that felt very strong to people.
It’s really, really hard to share the right level of detail and provide adequate context. I think this post managed to be both too short and too long.
Short: because we didn’t make as many explicit comparisons benchmarking research
Long: we felt we needed to add context on several points that weren’t obvious to low context people.
When editing a post it’s pretty challenging to figure out what assumptions you can assume and what your reader won’t know, because there’s a broad range of knowledge. I think nested thoughts could be helpful for making posts reasonable length
We initially didn’t give as much detail in some areas because the other (technical) author is time-limited and didn’t think it was critical. The post editing process is extremely long for a post of this size and gravity, so we had to make decisions on when to stop iterating.
Overall, I think the post still generated some interesting and valuable discussion, and I hope it at the very least causes people to think more critically about where they end up working.
I am sad that Conjecture didn’t engage with the post as much as we would have liked.
I think it’s difficult to strike a balance of ‘say what you believe to be true’ and ‘write something people aren’t put off by’
I think some people expected their views to be reflected in our critique. I think I’m sympathetic to that to some extent, but I think you can err to far in that direction (and I’ve seen pushback the other way as well). It feels like with this post, people felt very strongly (many comments were pretty strongly stated) such that it wasn’t just a disagreement but people felt it was a hitpiece. Il
I think I want to get better at communicating that, ultimately, these are the views of a very small group of people, these topics are very high uncertainty, and there will be disagreements, but that doesn’t mean we have a hidden agenda or something we are trying to push. (I’ll probably add this to our intro).
I’d be thrilled to see others write their own evaluations of these or other orgs.
We didn’t do some super basic things which feel obvious in retrospect e.g. explain why we are writing this series. But context is important when people are primed to respond negatively to a post.
Changes we plan to make:
Recruiting “typical” readers for our next review round
Hiring a copyeditor so we can spend more time on substance
Figuring out other ways to save time. Ideally, getting other technical contributors on board would be great (it would improve the quality and hopefully provide a slightly different perspective). Unfortunately, it’s hard to get people to do unpaid, anonymous work that might get a lot of pushback.
Posting an intro post with basic context we can point people to
The next post will be on anthropic and have (substantively) different critiques. I’d ideally like to spend some time figuring out how to murphyjitsu it so that we can meet people where they are at
I want to ensure that we get more engagement from anthropic (although I can imagine they might not engage much for different reasons to Conjecture—e.g. NDAs and what they are allowed to say publicly)
We appreciate you sharing your impression of the post. It’s definitely valuable for us to understand how the post was received, and we’ll be reflecting on it for future write-ups.
1) We agree it’s worth taking into account aspects of an organization other than their output. Part of our skepticism towards Conjecture – and we should have made this more explicit in our original post (and will be updating it) – is the limited research track record of their staff, including their leadership. By contrast, even if we accept for the sake of argument that ARC has produced limited output, Paul Christiano has a clear track record of producing useful conceptual insights (e.g. Iterated Distillation and Amplification) as well as practical advances (e.g. Deep RL From Human Preferences) prior to starting work at ARC. We’re not aware of any equally significant advances from Connor or other key staff members at Conjecture; we’d be interested to hear if you have examples of their pre-Conjecture output you find impressive.
We’re not particularly impressed by Conjecture’s process, although it’s possible we’d change our mind if we knew more about it. Maintaining high velocity in research is certainly a useful component, but hardly sufficient. The Builder/Breaker method proposed by ARC feels closer to a complete methodology. But this doesn’t feel like the crux for us: if Conjecture copied ARC’s process entirely, we’d still be much more excited about ARC (per-capita). Research productivity is a product of a large number of factors, and explicit process is an important but far from decisive one.
In terms of the explicit comparison with ARC, we would like to note that ARC Theory’s team size is an order of magnitude smaller than Conjecture. Based on ARC’s recent hiring post, our understanding is the theory team consists of just three individuals: Paul Christiano, Mark Xu and Jacob Hilton. If ARC had a team ten times larger and had spent close to $10 mn, then we would indeed be disappointed if there were not more concrete wins.
2) Thanks for the concrete examples, this really helps tease apart our disagreement.
We are overall glad that the Simulators post was written. Our view is that it could have been much stronger had it been clearer which claims were empirically supported versus hypotheses. Continuing the comparison with ARC, we found ELK to be substantially clearer and a deeper insight. Admittedly ELK is one of the outputs people in the TAIS community are most excited by so this is a high bar.
The stuff on SVDs and sparse coding [...] was a valuable contribution. I’d still say it was less influential than e.g. toy models of superposition or causal scrubbing but neither of these were done by like 3 people in two weeks.
This sounds similar to our internal evaluation. We’re a bit confused by why “3 people in two weeks” is the relevant reference class. We’d argue the costs of Conjecture’s “misses” need to be accounted for, not just their “hits”. Redwood’s team size and budget are comparable to that of Conjecture, so if you think that causal scrubbing is more impressive than Conjecture’s other outputs, then it sounds like you agree with us that Redwood was more impressive than Conjecture (unless you think the Simulator’s post is head and shoulders above Redwood’s other output)?
Thanks for sharing the data point this influenced independent researchers. That’s useful to know, and updates us positively. Are you excited by those independent researchers’ new directions? Is there any output from those researchers you’d suggest we review?
3) We remain confident in our sources regarding Conecture’s discussion with VCs, although it’s certainly conceivable that Conjecture was more open with some VCs than others. To clarify, we are not claiming that Connor or others at Conjecture did not mention anything about their alignment plans or interest in x-risk to VCs (indeed, this would be a barely tenable position for them given their public discussion of these plans), simply that their pitch gave the impression that Conjecture was primarily focused on developing products. It’s reasonable for you to be skeptical of this if your sources at Conjecture disagree; we would be interested to know how close to the negotiations those staff were, although understand this may not be something you can share.
4) We think your point is reasonable. We plan to reflect this recommendation and will reply here when we have an update.
5) This certainly depends on what “general industry” refers to: a research engineer at Conjecture might well be better for ML skill-building than, say, being a software engineer at Walmart. But we would expect ML teams at top tech companies, or working with relevant professors, to be significantly better for skill-building. Generally we expect quality of mentorship to be one of the most important components of individuals developing as researchers and engineers. The Conjecture team is stretched thin as a result of rapid scaling, and had few experienced researchers or engineers on staff in the first place. By contrast, ML teams at top tech companies will typically have a much higher fraction of senior researchers and engineers, and professors at leading universities comprise some of the best researchers in the field. We’d be curious to hear your case for Conjecture as skill building; without that it’s hard to identify where our main disagreement lies.
Here’s a more detailed response to these comments:
… You’re zooming in on every possible critique, and determining it’s an organization that shouldn’t have talent directed toward it.
We chose to write up critiques we felt were directly relevant to our end-line views (our goal was not to write up every possible critique). As we explain in another comment: “We believe that an organization should be graded on multiple metrics. Their outputs are where we would put the most weight. However, their strategy and governance are also key. The last year has brought into sharp relief the importance of strong organizational governance.”
We are supportive of the EA community pursuing a diversified research agenda, and individual organizations pursuing hits based agendas (we talk about that more in the first couple paragraphs of this comment). However, we do think that choosing the right organizations can make a difference, since top candidates often have the option for working at many organizations.
This is because we don’t agree that every alignment organization right now has relatively poor results. Here are some examples of results we find impressive, and organizations we think would be better places to work at than Conjecture:
Conceptual advance: ELK (ARC);
State-of-the-art in practically deployable method: constitutional AI (Anthropic);
Benchmarking of safety relevant problems: Trojan Detection Competition (CAIS)
Mechanistic interpretabilty: causal scrubbing (Redwood); toy models of superposition (Anthropic)
Thank you for raising this point – you’re right that we don’t explain why we are writing this series, and we will update the sequence description to be more transparent on that point. The reasons you suggest are basically correct.
With increased attention to TAIS there are many people trying to get into TAIS roles. Without significant context on organizations, new entrants to the field will tend to go to TAIS organizations based on their prominence caused by factors such as total funding, media coverage, volume of output, etc. Much of the discussion we have observed around TAIS organizations, especially criticisms of them, happens behind closed doors in conversations that junior people are usually not privy to. We wish to help disseminate this information more broadly to enable individuals to make a better informed decision.
We are concerned “that the attractiveness of working at an organization that is connected to the EA or TAIS communities makes it more likely for community members to take jobs at such organizations even if this will result in a lower lifetime impact than alternatives. Conjecture’s sponsorship of TAIS field building efforts may also lead new talent, who are unfamiliar with Conjecture’s history, to have an overly rosy impression of them.”
Regarding anonymization, we are also frustrated that we are not able to share more details. The sources we cite are credible to us (we believe the people who brought them to us to have high integrity). We try to provide relevant context where we can but don’t always have control over this. We don’t think an issue being based on (who you) trust, means that we shouldn’t bring these issues to light. We would encourage people who are making active decisions about their potential employment or collaboration with Conjecture to speak to people they trust and draw their own conclusions. We plan to edit all our recommendations to say this more explicitly.
I think the blurring between organisational design, strategy, and governance are somewhat separate to the research paradigm question. I can’t help wonder if these should be separated out—there seems to be some ‘correct’ paradigm that the authors of ‘Omega’ would like more funding and research in AI Safety, beyond correcting the organisational practices critiqued in this post and the Redwood one.
We believe that an organization should be graded on multiple metrics. Their outputs are where we would put the most weight. However, their strategy and governance are also key. The last year has brought into sharp relief the importance of strong organizational governance.
We don’t believe that there is a specific “paradigm” we advocate for. We would support the TAIS community pursuing a diversified research agenda.
Hi Pablo, thanks for you comment. We want to clarify that we aren’t trying to balance the critiques in a certain way, just that it so happens that the organizations that are next on our list will have a greater mix of positives and negatives.
Regarding your specific concerns about our recommendations:
1) We address this point in our response to Marius (5th paragraph)
2) As we note in the relevant section: “We think there is a reasonable risk that Connor and Conjecture’s outreach to policymakers and media is alarmist and may decrease the credibility of x-risk.” This kind of relationship-building is unilateralist when it can decrease goodwill amongst policymakers.
3) To be clear, we do not expect Conjecture to have the same level of “organizational responsibility” or “organizational competence” (we aren’t sure what you mean by those phrases and don’t use them ourselves) as OpenAI or Anthropic. Our recommendation was for Conjecture to have a robust corporate governance structure. For example, they could change their corporate charter to implement a “springing governance” structure such that voting equity (but not political equity) shift to an independent board once they cross a certain valuation threshold. As we note in another reply, Conjecture’s infohazard policy has no legal force, and therefore is not as strong as either OpenAI or Anthropic’s corporate governance models. As we’ve noted already, we have concerns about both OpenAI and Anthropic despite having these models in place: Conjecture doesn’t even have those, which makes us more concerned.
[Note: we edited point 3) for clarity on June 13 2023]
Responding to both comments in this thread, we have written a reply to TheAthenians’s comment which addresses the points raised regarding our focus on criticism.
We appreciate your detailed reply outlining your concerns with the post.
Our understanding is that your key concern is that we are judging Conjecture based on their current output, whereas since they are pursuing a hits-based strategy we should expect in the median case for them to not have impressive output. In general, we are excited by hits-based approaches, but we echo Rohin’s point: how are we meant to evaluate organizations if not by their output? It seems healthy to give promising researchers sufficient runway to explore, but $10 million dollars and a team of twenty seems on the higher end of what we would want to see supported purely on the basis of speculation. What would you suggest as the threshold where we should start to expect to see results from organizations?
We are unsure where else you disagree with our evaluation of their output. If we understand correctly, you agree that their existing output has not been that impressive, but think that it is positive they were willing to share preliminary findings and that we have too high a bar for evaluating such output. We’ve generally not found their preliminary findings to significantly update our views, whereas we would for example be excited by rigorous negative results that save future researchers from going down dead-ends. However, if you’ve found engaging with their output to be useful to your research then we’d certainly take that as a positive update.
Your second key concern is that we provide limited evidence for our claims regarding the VCs investing in Conjecture. Unfortunately for confidentiality reasons we are limited in what information we can disclose: it’s reasonable if you wish to consequently discount this view. As Rohin said, it is normal for VCs to be profit-seeking. We do not mean to imply these VCs are unusually bad for VCs, just that their primary focus will be the profitability of Conjecture, not safety impact. For example, Nat Friedman has expressed skepticism of safety (e.g. this Tweet) and is a strong open-source advocate, which seems at odds with Conjecture’s info-hazard policy.
We have heard from multiple sources that Conjecture has pitched VCs on a significantly more product-focused vision than they are pitching EAs. These sources have either spoken directly to VCs, or have spoken to Conjecture leadership who were part of negotiation with VCs. Given this, we are fairly confident on the point that Conjecture is representing themselves differently to separate groups.
We believe your third key concern is our recommendations are over-confident. We agree there is some uncertainty, but think it is important to make actionable recommendations, and based on the information we have our sincerely held belief is that most individuals should not work at Conjecture. We would certainly encourage individuals to consider alternative perspectives (including expressed in this comment) and to ultimately make up their own mind rather than deferring, especially to an anonymous group of individuals!
Separately, I think we might consider the opportunity cost of working at Conjecture higher than you. In particular, we’d generally evaluate skill-building routes fairly highly: for example, being a research assistant or PhD student in academia, or working in an ML engineering position in an applied team at a major tech company. These are generally close to capabilities-neutral, and can make individuals vastly more productive. Given the limited information on CogEm it’s hard to assess whether it will or won’t work, but we think there’s ample evidence that there are better places to develop skills than Conjecture.
We wholeheartedly agree that it is important to maintain high epistemic standards during the critique. We have tried hard to differentiate between well-established facts, our observations from sources, and our opinion formed from those. For example, the About Conjecture section focuses on facts; the Criticisms and Suggestions section includes our observations and opinions; and Our Views on Conjecture are more strongly focused on our opinions. We’d welcome feedback on any areas where you feel we over-claimed.
Changing your mind in the face of new evidence is certainly commendable. In this case, we were highlighting that Connor has switched from confidently holding one extreme position (founding an organization dedicated to open-source output with all research conducted in public) to the opposite extreme (founding an organization with one of the most restrictive non-disclosure policies) without any substantial new evidence, and with little in the way of a public explanation.
In particular, we wanted to highlight that Conjecture may in the future radically depart from their current info-hazard policy. To the best of our knowledge, the info-hazard policy has no legal force: it is a policy maintained at the discretion of Conjecture leadership. Given Connor has previously radically changed his mind without corresponding extreme changes in the world, we should not be surprised if a major change of strategy occurs again. As such, we would suggest viewing their info-hazard policy as a short-term stance not a long-term commitment. This isn’t necessarily a bad thing – we’d like Conjecture to share more details on their CogEm approach, for example. However, since the info-hazard policy has been repeatedly highlighted by Conjecture as a reason to trust them over other labs, we felt it was important to observe the flimsy base it is built on.
Separately, we think the degree of Connor’s stated confidence in these and other beliefs (e.g. 99% on AGI by 2100, hat-tip to Ryan Carey) is far from demonstrating good epistemics. A charitable interpretation is that he tends to express himself more strongly than he truly feels. A cynical interpretation would be that his claims are strongly influenced by the incentives he is currently experiencing, a natural human tendency: for example, championing open-source when it was financially and socially rewarded at EleutherAI, and championing secrecy when working with a more cautious group of individuals.
(Thanks to your comment, we reviewed our post and noticed this point (which was made more clearly in earlier drafts) had been de-emphasized. We will be editing the post to make this point more clear in the key points, the top-level of the section and in our views section as well.)
Thanks for highlighting this potential issue. We’d like to clarify that our intention is to evaluate both the positives and negatives. In retrospect, calling our posts “critiques” may have given the wrong impression: although it’s consistent with historical usage of the word[1] it does tend to carry a negative connotation. Ultimately our evaluation of Conjecture ended up fairly negative, exacerbating this impression: we expect future posts in the series on organizations where we have a more mixed evaluation to have a greater mix of positives and negatives.
You are right that overall we focus more on negatives than positives. We believe this is justified since organizations are already incentivized to make the positive case for themselves, and routinely do so in public announcements as well as private recruitment and funding pitches. By contrast, there is little reward from highlighting negatives. Indeed, we’re publishing this anonymously (foregoing any credit we could get from bringing these issues to attention) in order to protect against retaliation.
Our goal is not to make the EA community more hostile, and we’re certainly sorry that this post made you want to engage less with the community. We would not subject an individual or a small organization to this level of scrutiny. Our series is targeted only at large organizations above a certain bar of funding. With a $10 mn budget and a team of 20+ people, we do not think that Conjecture has will be threatened by a small volunteer group of anonymous individuals. If our arguments are specious, Conjecture would only need to dedicate a small fraction of their resources to rebutting them.
We appreciate your bringing attention to these points, and will be updating out sequence description and the introduction to each post to clarify these points in the next few days.
- ^
“a report that discusses a situation or the writings or ideas of someone and offers a judgment about them”; Cambridge English Dictionary
- ^
The post is now live!
While we’re taking a short break from writing criticisms, I (the non-technical author) was wondering if people would be find it valuable for us to share (brief) thoughts what we’ve learnt so far from writing these first two critiques—such as how to get feedback, balance considerations, anonymity concerns, things we wish would be different in the ecosystem to make it easier for people to provide criticisms etc.
Especially keen to write for the audience of those who want to write critiques
Keen to hear what specific things (if any) people would be curious to hear
We’re always open to providing thoughts / feedback / inputs if you are trying to write a critique. I’d like to try and encourage more good-faith critiques that enable productive discourse.