AI safety researcher
Thomas Kwa
Karma: 3,583
Crossposted from LessWrong.
Maybe I’m being cynical, but I’d give >30% that funders have declined to fund AI Safety Camp in its current form for some good reason. Has anyone written the case against? I know that AISC used to be good by talking to various colleagues, but I have no particular reason to believe in its current quality.
MATS has steadily increased in quality over the past two years, and is now more prestigious than AISC. We also have Astra, and people who go directly to residencies at OpenAI, Anthropic, etc. One should expect that AISC doesn’t attract the best talent.
If so, AISC might not make efficient use of mentor / PI time, which is a key goal of MATS and one of the reasons it’s been successful.
Why does the founder, Remmelt Ellen, keep linkposting writing by Forrest Landry which I’m 90% sure is obvious crankery? It’s not just my opinion; Paul Christiano said “the entire scientific community would probably consider this writing to be crankery”, one post was so obviously flawed it gets −46 karma, and generally the community response has been extremely negative. Some AISC work is directly about the content in question. This seems like a concern especially given the philosophical/conceptual focus of AISC projects, and the historical difficulty in choosing useful AI alignment directions without empirical grounding. [Edit: To clarify, this is not meant to be a character attack. I am concerned that Remmelt does not have the skill of distinguishing crankery from good research, even if he has substantially contributed to AISC’s success in the past.]
All but 2 of the papers listed on Manifund as coming from AISC projects are from 2021 or earlier. Because I’m interested in the current quality in the presence of competing programs, I looked at the two from 2022 or later: this in a second-tier journal and this in a NeurIPS workshop, with no top conference papers. I count 52 participants in the last AISC so this seems like a pretty poor rate, especially given that 2022 and 2023 cohorts (#7 and #8) could both have published by now. (though see this reply from Linda on why most of AISC’s impact is from upskilling)
The impact assessment was commissioned by AISC, not independent. They also use the number of AI alignment researchers created as an important metric. But impact is heavy-tailed, so the better metric is value of total research produced. Because there seems to be little direct research, to estimate the impact we should count the research that AISC alums from the last two years go on to produce. Unfortunately I don’t have time to do this.
Surely as their gold standard “career change” pin-up story, they could find a higher EV career change.
You’re assuming that the EV of switches from global health to biosecurity is lower than the EV of switching from something else to biosecurity. Even though global health is better than most cause areas, this could be false in practice for at least two reasons
If the impact of biosecurity careers is many times higher than the impact of global health, and people currently in global health are slightly more talented, altruistic, or hardworking.
If people currently in global health are not doing the most effective global health interventions.
This article just made HN. It’s a report saying that 39 of 50 top offsetting programs are likely junk, 8 “look problematic”, and 3 lack sufficient information, with none being found good.
I think most climate people are very suspicious of charities like this, rather than or in addition to not believing in ethical offsetting. See this Wendover Productions video on problematic, non-counterfactual, and outright fraudulent climate offsets. I myself am not confident that CATF offsets are good and would need to do a bunch of investigation, and most people are not willing to do this starting from, say, an 80% prior that CATF offsets are bad.
Upvoted. I don’t agree with all of these takes but they seem valuable and underappreciated.
But with no evidence, just your guesses. IMO we should wait until things shake out and even then the evidence will require lots of careful interpretation. Also EA is 2⁄3 male, which means that even minor contributions of women to scandals could mean they cause proportionate harms.
I’m looking for AI safety projects with people with some amount of experience. I have 3⁄4 of a CS degree from Caltech, one year at MIRI, and have finished the WMLB and ARENA bootcamps. I’m most excited about activation engineering, but willing to do anything that builds research and engineering skill.
If you’ve published 2 papers in top ML conferences or have a PhD in something CS related, and are interested in working with me, send me a DM.
Upvoted for making an actual calculation with reasonable numbers.
Is there any evidence for this claim? One can speculate about how average personality gender differences would affect p(scandal), but you’ve just cited two cases where women caused huge harms, which seems to argue neutrally or against you.
Who tends to be clean?
With all the scandals in the last year or two, has anyone looked at which recruitment sources are least likely to produce someone extremely net negative in direct impact or to the community (i.e. a justified scandal)? Maybe this should inform outreach efforts.
In addition to everything mentioned so far, there’s the information and retributive justice effect of the public exposé, which can be positive. As long as it doesn’t devolve into a witch hunt, we want to discourage people from using EA resources and trust in the ways Nonlinear did, and this only works if it’s public. If this isn’t big enough, think about the possibility of preventing FTX. (I don’t know if the actual fraud was preventable, but negative aspects of SBF’s character and the lack of separation between FTX and Alameda could have been well substantiated and made public. Just the reputation of EAs doing due diligence here could have prevented a lot of harm.)
When I wrote the comment, it wasn’t clear to me what the aim of the post was, and I thought Rockwell’s reply clarified this. I just misinterpreted “accomplishments” at the top as being about impact rather than community. So I’m now glad this post exists, though citing metrics still bothers me a bit.
Fair point about the independent research funding bar. I think the impact of CAIS and FAR are hard to deny, simply because they both have several impressive papers.
Crop yields are extremely low in much of Africa so my guess is there’s potential for farmed animals to be fed while keeping constant or even decreasing land use.
Some questions I would be interested in:
Where will Sub-Saharan Africa stand in terms of meat consumption and especially chicken consumption as the standard of living increases? When/if Nigeria hits China’s current GDP per capita of $12.5k do we expect more or less meat consumption than China has?
Then there’s the welfare side. As countries get richer we have seen animal welfare get worse due to factory farming, then perhaps slightly better due to ability to afford animal welfare measures. Will we see the same trajectory in Africa, or should we expect something different like the ability to “leapfrog” to some of the worst animal agriculture practices, or even past them?
Visualization is pretty important in exploratory mechanistic interp work, but this is more about fast research code: see any of Neel’s exploratory notebooks.
When Redwood had a big interpretability team, they were also developing their own data viz tooling. This never got open-sourced, and this could have been due to lack of experience by the people who wrote such tooling. Anthropic has their own libraries too, Transformerlens could use more visualization, and I hear David Bau’s lab is developing a better open-source interpretability library. My guess is there is more impact if you’re willing to participate in interp research yourself, but still probably some opportunities to mostly do data viz at some interp shop.
With regard to bottlenecks being on knowing where/how to look, the important thing is to work with the right team. From a quick glance the Learning Interpretability Tool is not focused on mechinterp, and the field of interp is so much larger than the subset targeted at alignment that you’d likely have more impact at something more targeted. In your position I’d likely talk to a bunch of empirical alignment researchers about their frontend / data viz needs, see if a top tier team like Superalignment is hiring, and have an 80k call while developing a good inside view on the problem
What object-level achievements has EA NYC had, in terms of money moved, talent found and placed in organizations, policy and technical achievements by those organizations, etc.? To be a bit blunt, the EA community and community members are not the point of EA, impact is the point.
I think funding is a bottleneck. Everything I’ve heard suggests the funding environment is really tight: CAIS is not hiring due to lack of funding. FAR is only hiring one RE in the next few months due to lack of funding. Less than half of this round of MATS scholars were funded for independent research. I think this is because there are not really 5-10 EA funders able to fund at large scale, just OP and SFF; OP is spending less than they were pre-FTX. At LTFF the bar is high, LTFF’s future is uncertain, and they tend not to make huge grants anyway. So securing funding should be a priority for anyone trying to start an org.
Edit: I now think the impact of these orgs is uncertain enough that one should not conclude with certainty there is a funding bottleneck.
This currently has +154 karma on EA Forum and only +24 on LW, with similar exposure on each site, so I think it’s fair to say that the reception is positive here and negative on LW. Maybe it’s worth thinking about why.
My comment is not an ad hominem. An ad hominem attack would be if someone is arguing point X and you distract from X by attacking their character. I was questioning only Remmelt’s ability to distinguish good research from crankery, which is directly relevant to the job of an AISC organizer, especially because some AISC streams are about the work in question by Forrest Landry. I apologize if I was unintentionally making some broader character attack. Whether it’s obnoxious is up to you to judge.