AI safety researcher
Thomas Kwa
Karma: 3,548
This article just made HN. It’s a report saying that 39 of 50 top offsetting programs are likely junk, 8 “look problematic”, and 3 lack sufficient information, with none being found good.
I think most climate people are very suspicious of charities like this, rather than or in addition to not believing in ethical offsetting. See this Wendover Productions video on problematic, non-counterfactual, and outright fraudulent climate offsets. I myself am not confident that CATF offsets are good and would need to do a bunch of investigation, and most people are not willing to do this starting from, say, an 80% prior that CATF offsets are bad.
Upvoted. I don’t agree with all of these takes but they seem valuable and underappreciated.
But with no evidence, just your guesses. IMO we should wait until things shake out and even then the evidence will require lots of careful interpretation. Also EA is 2⁄3 male, which means that even minor contributions of women to scandals could mean they cause proportionate harms.
I’m looking for AI safety projects with people with some amount of experience. I have 3⁄4 of a CS degree from Caltech, one year at MIRI, and have finished the WMLB and ARENA bootcamps. I’m most excited about activation engineering, but willing to do anything that builds research and engineering skill.
If you’ve published 2 papers in top ML conferences or have a PhD in something CS related, and are interested in working with me, send me a DM.
Upvoted for making an actual calculation with reasonable numbers.
Is there any evidence for this claim? One can speculate about how average personality gender differences would affect p(scandal), but you’ve just cited two cases where women caused huge harms, which seems to argue neutrally or against you.
Who tends to be clean?
With all the scandals in the last year or two, has anyone looked at which recruitment sources are least likely to produce someone extremely net negative in direct impact or to the community (i.e. a justified scandal)? Maybe this should inform outreach efforts.
In addition to everything mentioned so far, there’s the information and retributive justice effect of the public exposé, which can be positive. As long as it doesn’t devolve into a witch hunt, we want to discourage people from using EA resources and trust in the ways Nonlinear did, and this only works if it’s public. If this isn’t big enough, think about the possibility of preventing FTX. (I don’t know if the actual fraud was preventable, but negative aspects of SBF’s character and the lack of separation between FTX and Alameda could have been well substantiated and made public. Just the reputation of EAs doing due diligence here could have prevented a lot of harm.)
When I wrote the comment, it wasn’t clear to me what the aim of the post was, and I thought Rockwell’s reply clarified this. I just misinterpreted “accomplishments” at the top as being about impact rather than community. So I’m now glad this post exists, though citing metrics still bothers me a bit.
Fair point about the independent research funding bar. I think the impact of CAIS and FAR are hard to deny, simply because they both have several impressive papers.
Crop yields are extremely low in much of Africa so my guess is there’s potential for farmed animals to be fed while keeping constant or even decreasing land use.
Some questions I would be interested in:
Where will Sub-Saharan Africa stand in terms of meat consumption and especially chicken consumption as the standard of living increases? When/if Nigeria hits China’s current GDP per capita of $12.5k do we expect more or less meat consumption than China has?
Then there’s the welfare side. As countries get richer we have seen animal welfare get worse due to factory farming, then perhaps slightly better due to ability to afford animal welfare measures. Will we see the same trajectory in Africa, or should we expect something different like the ability to “leapfrog” to some of the worst animal agriculture practices, or even past them?
Visualization is pretty important in exploratory mechanistic interp work, but this is more about fast research code: see any of Neel’s exploratory notebooks.
When Redwood had a big interpretability team, they were also developing their own data viz tooling. This never got open-sourced, and this could have been due to lack of experience by the people who wrote such tooling. Anthropic has their own libraries too, Transformerlens could use more visualization, and I hear David Bau’s lab is developing a better open-source interpretability library. My guess is there is more impact if you’re willing to participate in interp research yourself, but still probably some opportunities to mostly do data viz at some interp shop.
With regard to bottlenecks being on knowing where/how to look, the important thing is to work with the right team. From a quick glance the Learning Interpretability Tool is not focused on mechinterp, and the field of interp is so much larger than the subset targeted at alignment that you’d likely have more impact at something more targeted. In your position I’d likely talk to a bunch of empirical alignment researchers about their frontend / data viz needs, see if a top tier team like Superalignment is hiring, and have an 80k call while developing a good inside view on the problem
What object-level achievements has EA NYC had, in terms of money moved, talent found and placed in organizations, policy and technical achievements by those organizations, etc.? To be a bit blunt, the EA community and community members are not the point of EA, impact is the point.
I think funding is a bottleneck. Everything I’ve heard suggests the funding environment is really tight: CAIS is not hiring due to lack of funding. FAR is only hiring one RE in the next few months due to lack of funding. Less than half of this round of MATS scholars were funded for independent research. I think this is because there are not really 5-10 EA funders able to fund at large scale, just OP and SFF; OP is spending less than they were pre-FTX. At LTFF the bar is high, LTFF’s future is uncertain, and they tend not to make huge grants anyway. So securing funding should be a priority for anyone trying to start an org.
Edit: I now think the impact of these orgs is uncertain enough that one should not conclude with certainty there is a funding bottleneck.
This currently has +154 karma on EA Forum and only +24 on LW, with similar exposure on each site, so I think it’s fair to say that the reception is positive here and negative on LW. Maybe it’s worth thinking about why.
Manifold markets related to this:
Fixed
You’re assuming that the EV of switches from global health to biosecurity is lower than the EV of switching from something else to biosecurity. Even though global health is better than most cause areas, this could be false in practice for at least two reasons
If the impact of biosecurity careers is many times higher than the impact of global health, and people currently in global health are slightly more talented, altruistic, or hardworking.
If people currently in global health are not doing the most effective global health interventions.