I think you’re maybe overstating how much more promising grad students are than undergrads for short-term technical impact. Historically, people without much experience in AI safety have often produced some of the best work. And it sounds like you’re mostly optimizing for people who can be in a position to make big contributions within two years; I think that undergrads will often look more promising than grad students given that time window.
Buck
Karma: 6,588
I agree with you that people seem to somewhat overrate getting jobs in AI companies.
However, I do think there’s good work to do inside AI companies. Currently, a lot of the quality-adjusted safety research happens inside AI companies. And see here for my rough argument that it’s valuable to have safety-minded people inside AI companies at the point where they develop catastrophically dangerous AI.
Tentative implications:
People outside of labs are less likely to have access to the very best models and will have less awareness of where the state of the art is.
Warning shots are somewhat less likely as highly-advanced models may never be deployed externally.
We should expect to know less about where we’re at in terms of AI progress.
Working at labs is perhaps more important than ever to improve safety and researchers outside of labs may have little ability to contribute meaningfully.
Whistleblowing and reporting requirements could become more important as without them government would have little ability to regulate frontier AI.
Any regulation based solely on deployment (which has been quite common) should be adjusted to take into account that the most dangerous models may be used internally long before they’re deployed.
For what it’s worth, I think that the last year was an update against many of these claims. Open source models currently seem to be closer to state of the art than they did a year ago or two years ago. Currently, researchers at labs seem mostly in worse positions to do research than researchers outside labs.
I very much agree that regulations should cover internal deployment, though, and I’ve been discussing risks from internal deployment for years.
Well known EA sympathizer Richard Hanania writes about his donation to the Shrimp Welfare Project.
When we were talking about this in 2012 we called it the “poor meat-eater problem”, which I think is clearer.
seems like the marginal value is much higher
I’ve done this.
I think this is a very good use of time and encourage people to do it.
yeah I totally agree
Alex Wellerstein notes the age distribution of Manhattan Project employees:
Sometimes people criticize EA for having too many young people; I think that this age distribution is interesting context for that.
[Thanks to Nate Thomas for sending me this graph.]
Note: When an earlier private version of these notes was circulated, a senior figure in technical AI safety strongly contested my description. They believe that the Anthropic SAE work is much more valuable than the independent SAE work, as both were published around the same time, but the Anthropic work provides sufficient evidence to be worth extending by other researchers, whereas the independent research was not dispositive.
For the record, if the researcher here was COI’d, eg working at Anthropic, I think you should say so, and you should also substantially discount what they said.
I’d bet against that but not confident
I don’t think he says anything in the manifesto about why AI is going to go better if he starts a “hedge fund/think tank”.
I haven’t heard a strong case for him doing this project but it seems plausibly reasonable. My guess is I’d think it was a suboptimal choice if I heard his arguments and thought about it, but idk.
I’d bet that he didn’t mean black people here.
For what it’s worth, I’m 75% confident that Hanania didn’t mean black people with the “animals” comment.
I think it’s generally bad form to not take people at their word about the meaning of their statements, though I’m also very sympathetic to the possibility of provocateurs exploiting charity to get away with dogwhistles (and I think Hanania deserves more suspicion of this than most), so I feel mixed about you using it as an example here.
I don’t think Hanania is exactly well-positioned to build support on the right; he constantly talks about how much contempt he has for conservatives.
He lays out the relevant part of his perspective in “The Free World Must Prevail” and “Superalignment” in his recent manifesto.
I think it’s pretty unreasonable to call him a Nazi—he’d hate Nazis, because he loves Jews and generally dislikes dumb conservatives.
I agree that he seems pretty racist.
Most importantly, it seems to me that the people in EA leadership that I felt were often the most thoughtful about these issues took a step back from EA, often because EA didn’t live up to their ethical standards, or because they burned out trying to affect change and this recent period has been very stressful
Who on your list matches this description? Maybe Becca if you think she’s thoughtful on these issues? But isn’t that one at most?
I think increasing the value of good futures is probably higher importance, but much less tractable