I work at CEA on the Community Health team doing deep dives into concerns within the EA or adjacent communities. Feel free to reach out to me with any reports or concerns about conduct related to the EA community, whether involving people, spaces, or organizations (explicitly EA or not). I’m happy to talk about vague or small concerns.
(Opinions here my own by default though will sometimes speak in a professional capacity).
Personal website: www.chanamessinger.com
Post: OAI NDA drama, do we know that Anthropic does not do similar things? I heard something that reassured me but I no longer know what it was. Interested in what people have heard or know (feel free to DM me) or general takes on the sitaution.
Huh—this both feels like something I’m sympathetic to worrying about and matches what I’ve seen people say about similar issues around the internet. Why does it seem uncharitable to you?
I appreciate the tone and my read of the aim of this—feels like it’s trying to clarify what’s at stake and cause more understanding and better discussion.
I’m not sure what you mean by “real way”. One of the central ways it’s culturally understood that that word and certain uses of “gay” are bad to use is to be contemptuous about things one doesn’t like or are insufficiently masculine. That seems like an important and real way it can be used for harm, not only literally meaning to call a gay person a slur.
I appreciate you writing this up. I think it might be worth people who know him putting some effort into setting up a chat. Very plausibly people are and I don’t know anything about it, but people might also underweight the value of a face to face conversation coming from someone who understands you and shares a lot of your worldview expressing concerns.
This is also a argument for the forum’s existence generally, if many of the arguments would otherwise be had on Twitter.
Leopold Aschenbrenner is starting a cross between a hedge fund and a think tank for AGI. I have read only the sections of Situational Awareness most relevant to this project, and I don’t feel nearly like I understand all the implications, so I could end up being quite wrong. Indeed, I’ve already updated towards a better and more nuanced understanding of Aschenbrenner’s points, in ways that have made me less concerned than I was to begin with. But I want to say publicly that the hedge fund idea makes me nervous.
Before I give my reasons, I want to say that it seems likely most of the relevant impact comes not from the hedge fund but from the influence the ideas from Situational Awareness have on policymakers and various governments, as well as the influence and power Aschenbrenner and any cohort he builds wield. This influence may come from this hedge fund or be entirely incidental to it. I mostly do not address this here, but it does make all of the below less important.
I also believe that some (though not all) of my concerns about the hedge fund are based on specific disagreements with Aschenbrenner’s views. I discuss some of those below, but a full rebuttal this is not (and many of the points of disagreement I don’t yet feel confident in my view on). There is still plenty to do to hash out the actual empirical questions at hand.
Why I am nervous
A hedge fund investing in AI related investments means Aschenbrenner and his investors will gain financially from more and accelerated AGI progress. This seems to me to be one of the most important dynamics (excluding the points about influence above). That creates an incentive to create more AGI progress, even at the cost of safety, which seems quite concerning. I will say that Leopold has a good track record here around turning down money in not signing an NDA at Open AI despite loss of equity.
Aschenbrenner expresses strong support for the liberal democratic world to maintain a lead on AI advancement, and ensure that China does not reach an AI-based decisive military advantage over the United States[1]. The hedge fund, then, presumably aims to both support the goal of maintaining an AI lead over China and profit off of it. In my current view, this approach increases race dynamics and increases the risks of the worst outcomes (though my view on this has softened somewhat since my first draft, for reasons similar to what Zvi clarifies here[2]).
I especially think that it risks unnecessary competition when cooperation—the best outcome—could still be possible. It seems notable, for example, that no Chinese version of the Situational Awareness piece has come to my attention; going first in such a game both ensures you are first and that the game is played at all.
It’s also important that the investors (e.g. Patrick Collison) appear to be more focused on economic and technological development, and less concerned about risks from AI. The incentives of this hedge fund are therefore likely to point towards progress and away from slowing down for safety reasons.
There are other potential lines of thought here I have not yet fleshed out including:
The value of aiming to orient the US government and military attention to AGI (seems like a huge move with unclear sign)
The degree to which this move is unilateralist on Aschenbrenner’s part
How much money could be made and how much power the relevant people (e.g. Aschenbrenner and his investors) will have through investment and being connected to important decisions.
If a lot of money and/or power could be acquired, especially over AGI development, then there’s a healthy default skepticism I think should be applied to their actions and decision-making.
Specifics about Aschenbrenner himself. Different people in the same role would take very different actions, so specifics about his views, ways of thinking, and profile of strengths and weaknesses may be relevant.
Ways that the hedge fund could in fact be a good idea:
EA and AI causes could really use funder diversification. If Aschenbrenner intends to use the money he makes to support these issues, that could be very valuable (though I’ve certainly become somewhat more concerned with moonshot “become a billionaire to save the world” plans than I used to be).
The hedge fund could position Aschenbrenner to have a deep understanding of and connections within the AI landscape, making the think tank outputs very good, and causing important future decisions to be made better.
Aschenbrenner of course could be right about the value of the US government’s involvement, maintaining a US lead, and the importance of avoiding Chinese military supremacy over the US. In that case, him achieving his goals would of course be good. Cruxes include the likelihood of international cooperation, the possibility of international bans, probability of catastrophic outcomes from AI and the likelihood of “muddling through” on alignment.
I’m interested in hearing takes, ways I could be wrong, fleshing out of my arguments, or any other thoughts people have relevant to this. Happy to have private chats in DMs to discuss as well.
To be clear, Aschenbrenner wants that lead to exist to avoid a tight race in which safety and caution are thrown to the winds. If we can achieve that lead primarily through infosecurity (something he emphasizes), then added risks are low; but I think the views expressed in Situational Awareness also imply the importance of staying technologically ahead of China as their AI research improves. This comes with precisely the risks of creating and accelerating a race of this nature.
Additionally, when I read his description of the importance of even a two month lead, it implied to me that if the longer, more comfortable lead is lost, there will be strong reasons for the US to advance quickly so as to avoid China reaching superintelligence and subsequent military dominance first (which doesn’t mean he thinks we should actually do this if the time came). This seems to fairly explicitly describe the tight race scenario. I don’t think Aschenbrenner believes this would be a good situation to be in, but nonetheless thinks that’s what the true picture is.
From Zvi’s post: “He confirms he very much is NOT saying this:
The race to ASI is all that matters.
The race is inevitable.
We might lose.
We have to win.
Trying to win won’t mean all of humanity loses.
Therefore, we should do everything in our power to win.
I strongly disagree with this first argument. But so does Leopold.
Instead, he is saying something more like this:
ASI, how it is built and what we do with it, will be all that matters.
ASI is inevitable.
A close race to ASI between nations or labs almost certainly ends badly.
Our rivals getting to ASI first would also be very bad.
Along the way we by default face proliferation and WMDs, potential descent into chaos.
The only way to avoid a race is (at least soft) nationalization of the ASI effort.
With proper USG-level cybersecurity we can then maintain our lead.
We can then use that lead to ensure a margin of safety during the super risky and scary transition to superintelligence, and to negotiate from a position of strength.”
I was thinking about to what extent NDAs (either non-disclosure or non-disparagement agreements) played a role in the 2018 blowup at Alameda Research (since if there were a lot, that could be a throughline between messiness at Alameda and messiness at Open AI recently).
Here’s what I’ve collected from public records:
Not mentioned as far as I can tell in Going Infinite
Ben West: “I don’t want to speak for this person, but my own experience was pretty different. For example: Sam was fine with me telling prospective AR employees why I thought they shouldn’t join (and in fact I did do this),[4] and my severance agreement didn’t have any sort of non-disparagement clause. This comment says that none of the people who left had a non-disparagement clause, which seems like an obvious thing a person would do if they wanted to use force to prevent disparagement.[5]” From here
Kerry Vaughn: “Information about pre-2018 Alameda is difficult to obtain because the majority of those directly involved signed NDAs before their departure in exchange for severance payments. I am aware of only one employee who did not. The other people who can spreak freely on the topic are early investors in Alameda and members of the EA community who heard about Alameda from those directly involved before they signed their NDAs”. From here.
ftxthrowaway: “Lastly, my severance agreement didn’t have a non-disparagement clause, and I’m pretty sure no one’s did. I assume that you are not hearing from staff because they are worried about the looming shitstorm over FTX now, not some agreement from four years ago.” From here (it’s a response to the previous)
nbouscal: “I’m the person that Kerry was quoting here, and am at least one of the reasons he believed the others had signed agreements with non-disparagement clauses. I didn’t sign a severance agreement for a few reasons: I wanted to retain the ability to sue, I believed there was a non-disparagement clause, and I didn’t want to sign away rights to the ownership stake that I had been verbally told I would receive. Given that I didn’t actually sign it, I could believe that the non-disparagement clauses were removed and I didn’t know about it, and people have just been quiet for other reasons (of which there are certainly plenty).” From here (it’s a response to the previous)
Later says “I do think I was probably just remembering incorrectly about this to be honest, I looked back through things from then and it looks like there was a lot of back-and-forth about the inclusion of an NDA (among other clauses), so it seems very plausible that it was just removed entirely during that negotiation (aside from the one in the IP agreement).” Link here.
arthrowaway: “Also no non-disparagement clause in my agreement. FWIW I was one of the people who negotiated the severance stuff after the 2018 blowup, and I feel fairly confident that that holds for everyone. (But my memory is crappy, so that’s mostly because I trust the FB post about what was negotiated more than you do.)” From here (it’s in the same thread as the above)
Overall this tells a story where NDAs weren’t a big part of the Alameda story (since I think Ben West and nbouscal at least left during the 2018 blowup, but folks should correct me if I’m wrong). This is a bit interesting to me.
Interested in if others have different takeaways.
Buck, do you have any takes on how good this seems to you / how good the arguments in the manifesto for doing this work seem to you? (No worries if not or you don’t want to discuss publicly)
Also relevant: AI companies aren’t really using external evaluators
Say more about Conjecture’s structure?
Sam’s comment: https://twitter.com/sama/status/1790518031640347056
Jan Leike has also left: https://www.nytimes.com/2024/05/14/technology/ilya-sutskever-leaving-openai.html
Jan Leike, who ran the Super Alignment team alongside Dr. Sutskever, has also resigned from OpenAI. His role will be taken by John Schulman, another company co-founder.
This was a delight to read! I found the fact that an essay competition in 1837 was a successful activist move really striking!
This is mentioned here, but I want to double down on the value of “asking around about the organization and what the experiences of others were”.
I talked to someone recently in tech about whether there were good ways to find out if working at any given tech organization was right for you, and he said basically no, that it was hard to get an accurate picture, that the resources that had tried to do this in the field (like Blind) added some information but gave warped impressions from who posted there. (That said, from a quick skim, it seems a lot better than nothing to me! I’m glad it exists)
So basically my current view is it’s hard, and you might need to get information from a bunch of different sources.
I think people do not get karma from the baseline +1 or +2 that comes with making a new comment.
Thank you so much for this!
As the post says above, I’d like to share updates the team has made on its policies based on the internal review we did following the Time article and Owen’s statement as a manager on the team and the person who oversaw the internal review. (My initial description of the internal review is here). In general, these changes have been progressing prior to knowing the boards’ determinations, though thinking from Zach and the EV legal team has been an important input throughout.
Changes
Overall we spent dozens of hours over multiple calendar months in discussions and doing writeups, both internally to our team and getting feedback from Interim CEA CEO Ben West and others. Several team members did retrospectives or analyses on the case, and we consulted with external people (two EAs with some experience thinking about these topics as well as seven professionals in HR, law, consulting and ombuds) for advice on our processes generally.
From this we created a list of practices to change and additional steps to add. The casework team also reflected on many past cases to check that these changes were robust and applicable across a wide variety of casework.
Our changes, in rough order of importance:
Defaulting more often to getting input from more people on the team (especially if a case involves someone with power in the EA community) by having multiple caseworkers on a case and/or involving managers for oversight
Specifically, most cases now get seen by more than one person, particularly cases that involve someone with sufficient power in the community (confidentiality constraints may limit how many caseworkers work on a case.)
For the cases we take on, we have a lower bar for proactively reaching out to members to the community in order to get more information. For example, if this same case came up again, we would weigh more highly the importance of getting additional information from the woman Zach mentions above. Our previous approach pointed in this direction as well, but this change will make it less likely we miss an important opportunity.
We are creating a system that automatically flags to us when complaints about the same person come to different caseworkers, making us more likely to catch patterns of poor behavior (this previously was caught in a less systematic way)
We have lowered the bar for what’s considered a personal conflict of interest and improved our systems so it is easier to pass a case on to another caseworker. For instance, if the same case came up again, we would have it owned by someone else.
We’ve added and improved written internal guidelines for considerations to take into account during cases, particularly for cases involving someone with power in the EA community. Previously these were distributed across different documents and tacit understanding; they are now more formalized.
Automatic reminders for follow-ups on old cases to see if any more action is needed
An update towards overcommunication and explicitness when giving feedback to people whose behavior we’d like to change; an update against assuming the feedback has been understood and incorporated
An update towards escalations to boards, legal teams and HR departments where relevant.
Better systems (reminders, templates in meetings, etc) for making sure balls aren’t dropped based on when or how information comes in
The casework side of our team meets monthly to check that we are using our systems effectively, that we have implemented these changes with new and existing cases, and to update our internal guidelines as needed.
Some notes on the changes
I’m glad we have these changes, though I think it’s worth noting my guess that a big part of what will end up driving change here is also in large part due to CH team members’ individual reflections on the original events (Time article + statement from Owen + original case).
The list of changes is not exhaustive, since many of the changes stemming from this internal review (depending on your bar for “policy change”) are quite small e.g. “adding a consideration to our list of factors that may make a case more serious.” I think those “nudge” type changes are important and useful for incorporating data without over-updating or over-bureacratizing, though they are harder to assess externally. In general I’m trying to convey our high level changes without creating a laundry list. Nonetheless I hope the descriptions of our process changes help people make accurate updates and have accurate expectations about our team.
There is some broader context, which is that the casework team’s approach in January of 2023 (before the Time article came out) was already in significant ways different than in 2021. For instance, since Catherine joined in late 2021 (and even more so since Charlotte joined in 2023), the casework team has been able to double-check cases more often and have a lower bar for passing on cases where there was a personal connection. The changes as a result of the internal review add on to these previous shifts.
We’ll be checking in as a team at regular intervals to reflect on the value of these changes at getting better outcomes. This refers both to whether the increased time and effort per case looks like the right call (relative to e.g. handling more cases) and whether the decisions made are overall better (rather than the changes overfitting to this specific instance).
What’s still underway
In addition to the changes in how we handle personal COIs (with some person one team member has a COI with), we’re still working on a general approach to institutional COIs and what constitutes one, e.g. if a case involves someone in EV leadership. So far our planned approach is to ask for a person external to EV to give an independent view when such an institutional COI comes up in a case we take on, but it’s not obvious that there will always be an appropriate person willing. This will need more work to figure out.
As Zach indicates above, we’ll be working on a more formalized version of our current processes for escalation, COIs, and anything else the boards ask for.
I really don’t know how the norms of professional investigative journalism work, but I imagine a lot hinges on whether the source of concern / subject of the piece is the repository of a large amount of relevant information about the central claims.
e.g. the point is “how much work do you need to put in to make sure your claims are accurate” and then sometimes that implies “no need to get replies from the subject of the piece because you can get the information elsewhere” and sometimes that implies “you have to engage a bunch with the subject of the piece because they have relevant information.”
Love this, and love that there was also audio