Researching Causality and Safe AI at Oxford.
Previously, founder (with help from Trike Apps) of the EA Forum.
Discussing research etc at https://twitter.com/ryancareyai.
RyanCarey
Karma: 9,335
It may not be worth becoming a research lead under many worldviews.
I’m with you on almost all of your essay, regarding the advantages of a PhD, and the need for more research leads in AIS, but I would raise another kind of issue—there are not very many career options for a research lead in AIS at present. After a PhD, you could pursue:
Big RFPs. But most RFPs from large funders have a narrow focus area—currently it tends to be prosaic ML, safety, and mechanistic interpretability. And having to submit to grantmakers’ research direction somewhat defeats the purpose of being a research lead.
Joining an org working on an adjacent research direction. But they may not exist, depending on what you’re excited to work on.
Academia. But you have to be willing to travel, teach a lot, and live on well below the salary for a research contributor.
Little funders (like LTFF). But they may take 3+ months to apply for, and only last a year at a time, and they won’t respond to your emails for an explanation of this.
Get hired by as a researcher at OpenPhil? But very few will be hired and given research autonomy here.
For a many research leads, these options won’t be very attractive, and I find it hard to feel positive about convincing people to become research leads until better opportunities are in place. What would make me excited? I think we should have:
A. Research agenda agnostic RFPs. There needs to be some way for experienced AI safety researchers to figure out whether AI safety is actually a viable long-term career for them. Currently, there’s no way to get OpenPhil’s opinion on this—you simply have to wait years until they notice you. But there aren’t very many AI safety researchers, and there should be a way for them to run this test so that they can decide which way to direct their lives.
Concrete proposal: OpenPhil should say “we want applications from AIS researchers who we might be excited about as individuals, even if we don’t find their research exciting” and should start an RFP along these lines.
B. MLGA (Make LTFF great again). I’m not asking much here, but they should be faster, be calibrated on their timelines, respond to email in case of delays, offer multi-year grants.
Concrete proposal: LTFF should say “we want to fund people for multiple years at a time, and we will resign if we can’t get our grantmaking process work properly
C. At least one truly research agenda-agnostic research organisation, that will hire research leads to pursue their own research interests.
Concrete proposal: Folks should found an academic department-style research organisation that hires research leads, gets them office space and visas, and gives them a little support to apply for grants to support their teams. Of course this requires a level of interest from OpenPhil and other grantmakers in supporting this organisation.
Finally, I conclude on a personal note. As Adam knows, and other readers may deduce, I myself am a research lead underwhelmed with options (1-5). I would like to fix C (or A-B) and am excited to talk about ways of achieving this, but a big part of me just wants to leave AIS for a while, as these options are so much stronger, from a selfish perspective. Given that AIS has been this way for years, I suspect many others might leave before these issues are fixed.
Thanks for engaging with my criticism in a positive way.
Regarding how timely the data ought to be, I don’t think live data is necessary at all—it would be sufficient in my view to post updated information every year or two.
I don’t think “applied in the last 30 days” is quite the right reference class, however, because by-definition, the averages will ignore all applications that have been waiting for over one month. I think the most useful kind of statistics would:
Restrict to applications from n to n+m months ago, where n>=3
Make a note of what percentage of these applicants haven’t received a response
Give a few different percentiles for decision-timelines, e.g. 20th, 50th, 80th, 95th percentiles.
Include a clear explanation of which applications are being included, or excluded, for example, are you including applications that were not at all realistic, and so were rejected as soon as they landed on your desk?
With such statistics on the website, applications would have a much better sense of what they can expect from the process.
I had a similar experience with 4 months of wait (uncalibrated grant decision timelines on the website) and unresponsiveness to email with LTFF, and I know a couple of people who had similar problems. I also found it pretty “disrespectful”.
Its hard to understand why a) they wouldn’t list the empirical grant timelines on their website, and b) why they would have to be so long.
There is an “EA Hotel”, which is decently-sized, very intensely EA, and very cheap.
Occasionally it makes sense for people to accept very low cost-of-living situations. But a person’s impact is usually a lot higher than their salary. Suppose that a person’s salary is x, their impact 10x, and their impact is 1.1 times higher when they live in SF, due to proximity to funders and AI companies. Then you would have to cut costs by 90% to make it worthwhile to live elsewhere. Otherwise, you would essentially be stepping over dollars to pick up dimes.
Of course there are some theoretical reasons for growing fast. But theory only gets you so far, on this issue. Rather, this question depends on whether growing EA is promising currently (I lean against) compared to other projects one could grow. Even if EA looks like the right thing to build, you need to talk to people who have seen EA grow and contract at various rates over the last 15 years, to understand which modes of growth have been healthier, and have contributed to gained capacity, rather than just an increase in raw numbers. In my experience, one of the least healthy phases of EA was when there was the heaviest emphasis on growth, perhaps around 1.5-4 years ago, whereas it seemed to do better pretty-much all of the other times.
Yes, they were involved in the first, small, iteration of EAG, but their contributions were small compared to the human capital that they consumed. More importantly, they were a high-demand group that caused a lot of people serious psychological damage. For many, it has taken years to recover a sense of normality. They staged a partial takeover of some major EA institutions. They also gaslit the EA community about what they were doing, which confused and distracted decent-sized subsections of the EA communtiy for years.
I watched The Master a couple of months ago, and found to be a simultaneously compelling and moving description of the experience of cult membership, that I would recommend.
Interesting point, but why do these people think that climate change is going to cause likely extinction? Again, it’s because their thinking is politics-first. Their side of politics is warning of a likely “climate catastrophe”, so they have to make that catastrophe as bad as possible—existential.
I think that disagreement about the size of the risks is part of the equation. But it’s missing what is, for at least a few of the prominent critics, the main element—people like Timnit, Kate Crawford, Meredith Whittaker are bought in leftie ideologies focuses on things like “bias”, “prejudice”, and “disproportionate disadvantage”. So they see AI as primarily an instrument of oppression. The idea of existential risk cuts against the oppression/justice narrative, in that it could kill everyone equally. So they have to opposite it.
Obviously this is not what is happening with all people in the FATE AI, or AI Ethics community, but I do think it’s what’s driving some of the loudest voices, and that we should be clear-eyed about it.
I guess you’re right, but even so I’d ask:
Is it 11 new orgs, or will some of them stick together (perhaps with CEA) when they leave?
What about other orgs not on the website, like GovAI and Owain’s team?
Separately, are any teams going to leave CEA?
Related to (1) is the question: which sponsored projects are definitely being spun out?
Hmm, OK. Back when I met Ilya, about 2018, he was radiating excitement that his next idea would create AGI, and didn’t seem sensitive to safety worries. I also thought it was “common knowledge” that his interest in safety increased substantially between 2018-22, and that’s why I was unsurprised to see him in charge of superalignment.
Re Elon-Zillis, all I’m saying is that it looked to Sam like the seat would belong to someone loyal to him at the time the seat was created.
You may well be right about D’Angelo and the others.
The main thing that I doubt is that Sam knew at the time that he was gifting the board to doomers. Ilya was a loyalist and non-doomer when appointed. Elon was I guess some mix of doomer and loyalist at the start. Given how AIS worries generally increased in SV circles over time, more likely than not some of D’Angelo, Hoffman, and Hurd moved toward the “doomer” pole over time.
Nitpicks:
I think Dario and others would’ve also been involved in setting up the corporate structure
Sam never gave the “doomer” faction a near majority. That only happened because 2-3 “non-doomers” left and Ilya flipped.
Causal Foundations is probably 4-8 full-timers, depending on how you count the small-to-medium slices of time from various PhD students. Several of our 2023 outputs seem comparably important to the deception paper:
Towards Causal Foundations of Safe AGI, The Alignment Forum—the summary of everything we’re doing.
Characterising Decision Theories with Mechanised Causal Graphs, arXiv—the most formal treatment yet of TDT and UDT, together with CDT and EDT in a shared framework.
Human Control: Definitions and Algorithms, UAI—a paper arguing that corrigibility is not exactly the right thing to be aiming for, to assure good shut down behaviour.
Discovering Agents, Artificial Intelligence Journal—an investigation of the “retargetability” notion of agency.
What if you just pushed it back one month—to late June?
2 - I’m thinking more of the “community of people concerned about AI safety” than EA.
1,3,4- I agree there’s uncertainty, disagreement and nuance, but I think if NYT’s (summarised) or Nathan’s version of events is correct (and they do seem to me to make more sense to me than other existing accounts) then the board look somewhat like “good guys”, albeit ones that overplayed their hand, whereas Sam looks somewhat “bad”, and I’d bet that over time, more reasonable people will come around to such a view.
It’s a disappointing outcome—it currently seems that OpenAI is no more tied to its nonprofit goals than before. A wedge has been driven between the AI safety community and OpenAI staff, and to an extent, Silicon Valley generally.
But in this fiasco, we at least were the good guys! The OpenAI CEO shouldn’t control its nonprofit board, or compromise the independence of its members, who were doing broadly the right thing by trying to do research and perform oversight. We have much to learn.
Yeah I think EA just neglects the downside of career whiplash a bit. Another instance is how EA orgs sometimes offer internships where only a tiny fraction of interns will get a job, or hire and then quickly fire staff. In a more ideal world, EA orgs would value rejected & fired applicants much more highly than non-EA orgs, and so low-hit-rate internships, and rapid firing would be much less common in EA than outside.
It looks like, on net, people disagree with my take in the original post.
I just disagreed with the OP because it’s a false dichotomy; we could just agree with the true things that activists believe, and not the false ones, and not go based on vibes. We desire to believe that mech-interp is mere safety-washing iff it is, and so on.
It actually was not just neutrally listed as a “possible” project, because it was the fourth bullet point under “Projects and programs we’d like to see” here.