AI safety, governance, and alignment research and field buiilding.
Gabriel Mukobi
Karma: 353
Thanks for this! One response I’ve read about Claim 1 in particular is something along the lines of “promising students tend to emphasize CB while in school because they are positioned to have a greater impact that way, but when they graduate many (most?) of them slow down CB and move into up-skilling/direct work.”
That may still not be an ideal scenario under Claim 1 because it means those promising people are delayed in realizing their direct impact, but I wonder if anyone has anecdotal evidence of this kind of effect persistent or decaying after university?
Thanks for writing this, just filled out the form! I’m excited for a more coordinated community around AI safety field-building in universities!
The Tree of Life: Stanford AI Alignment Theory of Change
Thanks Chris! Yeah, I think that’s definitely a potential downside—I imagine we’ll definitely meet at least a couple of people smack in the middle of the quarter who go “Wow, that sounds cool, and also I never heard about this before” (this has happened with other clubs at Stanford).
I think the ways to mitigate failure due to this effect are:
Outreach early (when students still read their emails) and widely (to hopefully get the invitation message to everyone who would consider joining).
Don’t turn down students we find later, instead offer a lot of encouragement and a follow-up plan to join the reading group next term (10-week quarters for us).
This has the downside of latency in that we’re potentially delaying an impactful career, though maybe this isn’t significant compared to the latency of taking enough courses/doing enough research to graduate.
Semester schools could also offer the reading group twice per ~15-week semester, though that sounds a bit intense.
For promising students waiting for the next term, still offer them 1-on-1s, suggest things for them to read/work on independently, invite them to special reading group events like visits from guest alignment professionals, and maybe invite them to some core-only events.
Alternatively, if we find a student who has already read a bit, we could consider having them join a reading group cohort part-way through the year.
Good points! This reminds me of the recent Community Builders Spend Too Much Time Community Building post. Here are some thoughts about this issue:
Field-building and up-skilling don’t have to be orthogonal. I’m hopeful that a lot of an organizer’s time in such a group would involve doing the same things general members going through the system would be doing, like facilitating interesting reading group discussions or working on interesting AI alignment research projects. As the too much time post suggests, maybe just doing the cool learning stuff is a great way to show that we’re serious, get new people interested, and keep our group engaged.
Like Trevor Levin says in that reply, I think field-building is more valuable now than it will be as we get closer to AGI, and I think direct work will be more valuable later than it is now. Moreso, I think field-building while you’re a university student is significantly more valuable than field-building after you’ve graduated.
I don’t necessarily think the most advanced students will always need to be organizers under this model. I think there’s a growing body of EAs who want to help with AI alignment field-building but don’t necessarily think they’re the best fit for direct work (maybe they’re underconfident though), and this could be a great opportunity for them to help with little opportunity costs.
I’m really hopeful about several new orgs people are starting for field-wide infrastructure that could help offset a lot of the operational costs of this, including orgs that might be able to hire professional ops people to support a local group.
That’s not to say I recommend every student who’s really into AI safety delay their personal growth to work on starting a university group. Just that if you have help and think you could have a big impact, it might be worth considering letting off the solo up-skilling pedal to add in some more field-building.
I can’t unread this comment:
“Humanity could, theoretically, last for millions of centuries on Earth alone.” I find this claim utterly absurd. I’d be surprised if humanity outlasts this century.
Ughh they’re so close to getting it! Maybe this should give me hope?
Wow quite a strong claim, and as a longtermist mostly working on AI, not sure how to feel about it yet.🤔 But I’ll take a stab at a different thing:
However, If you care about causes such as poverty, health or animals, and you think your community could update based on a post saying “Cause Y will be affected by AI”, leave a comment and I will think about writing about it.
A few initial concrete examples:
Poverty: Transformative AI could radically transform the global economy, which could lead to the end of poverty as we know it and immense wealth for all or perhaps tremendous inequality.
(Mental) health: We’re likely to see really good AI chatbots pop up in the shorter term (i.e. before TAI, kinda starting now) which could potentially serve as ultra-low-cost but very useful therapists/friends.
Animals: AI could make alternative proteins much more efficient to produce, bringing a much sooner switch away from animal agriculture (and might even be necessary for cultivated meat to be competitive). Some are already working on this.
Based on AI timelines, I would be surprised if all these things didn’t happen in the next ~50 years, which feels quite “neartermist” to me.
I agree. I think it’s also worth pointing out that is a different metric from (the total number of people matching some criteria). One takeaway I got from the post is that while the probabilities might still be different between schools (“75th percentile SAT score falls very gradually”), the number of “smart” students might be comparable at different schools because many non-elite schools tend to be larger than private elite schools (further, since there are also many more non-elite schools, I might expect but that’s besides the point).
Practically, for EA outreach, this maybe implies that university outreach might be harder at big non-elite-but-still-good schools: as Joseph points out below, the median student might be less qualified, so you’d have to sample more to “sift” through all the students and find the top students. But EA outreach isn’t random sampling—it appeals to certain aptitudes, can use nerd-sniping/self-selection effects, and can probably be further improved to select for the most smart/agentic/thoughtful/truth-seeking/whatever you care about students, and there might be a comparable number of those at different universities regardless of elite-ness. If the heavy tail of impact is what matters, this makes me update towards believing EA outreach (and students starting EA groups) at non-elite-but-still-good universities could be as good as at elite universities.
Thanks for writing this! A couple days I was also wondering “🤔 could I use janky-statistics to estimate how the number of top N% most ‘promising’ students at X small elite private university compares to Y large non-elite public university?” But I’m glad you used non-janky statistics to evaluate this much better than I would have.
I’ll add a few more ways this could be wrong:
“Intelligence” could come in different qualities in different communities, e.g. maybe MIT tends to be more promising for maths (25% SAT Math = 780 😅), Harvard tends to be more promising for business and law, and yet other universities could be more promising in other things.
As others have pointed out, maybe “intelligence” (as measures by standardized test scores) isn’t the only metric to care about, and maybe we should be selecting for EQ/agency/truth-seekingness/etc. which could be differently distributed among schools than “intelligence.”
We don’t actually need the most “intelligent” students for a lot of things, maybe “medium intelligent” students among decent universities can still robustly understand EA ideas but be more agentic/impact-minded and be good for the community (this updates me more in favor of your point).
Conversely, maybe the bar for certain things (e.g. AI alignment technical researcher) should be much higher. Maybe the small-tail distribution of test scores doesn’t map well to the long-tailed distribution of impact in highly technical things, and we actually care about the 0.01% of students across some metric which could differ more significantly between schools.
I’m not sure how much to believe some of these, and some of them being right could mean the point presented in the post is not strong enough and there could be more “promising” students at non-elite universities than elite ones. Either way, I think it’s reasonable to update away from assuming EA outreach and community building should be focused on elite schools (if you’re a student at a non-elite school reading this, consider learning more about EA, getting mentorship, and starting a university group!).
I had no idea there were this many nematodes—is wild animal welfare just nematode welfare?! Do Rethink, WAI, or others have any research on them?
Thanks for building this, def seems like a way to save a lot of organizer time (and I appreciate how it differentiates things from a Bible group or a cult)!
To me, it seems like the main downside will be the lack of direct engagement between new people and established EAs. In a normal reading group, a participant meets and talks with a facilitator on day 1, and then every week between every 1-3 hours of EA-related reading. In this system, it seems like they don’t really get to meet and talk with someone until they go through a significant amount of independent exploration and write a reflection, and I wonder if the combination of that high required activation energy with little human-to-human guidance might cause you to lose some potentially good students as you go from the predicted 40 to 20.
You could try offering “cheaper” 1:1s to these people early, but that seems less efficient than having several of them in a weekly reading group discussion which would defeat the point. That’s not to say I don’t think this is the right move for your situation. Just that I’m extra curious about how this factor might play out, and I’m excited for you to test this system and share the results with other groups!
Is Civilization on the Brink of Collapse? - Kurzgesagt
They just added to it so it’s now “Is Civilization on the Brink of Collapse? And Could We Recover?” but it still seems to not answer the first question.
I’ll also plug Microsoft Edge as a great tool for this: There’s both a desktop browser and a mobile app, and it has a fantastic built-in Read Aloud feature that works in both. You just click the Read Aloud icon or press Ctrl/Cmd+Shift+U on a keyboard and it will start reading your current web page or document out loud!
It has hundreds of neural voices (Microsoft calls them “Natural” voices) in dozens of languages and dialects, and you can change the reading speed too. I find the voices to be among the best I’ve heard, and the super low activation energy of not having to copy-paste anything or switch to another window means I use it much more often than when I tried apps like Neural Reader.
Sidenote, but as a browser, since it’s Chromium-based it’s basically the same as Google Chrome (you can even install extensions from the Chrome Web Store) but with slightly less bloat and better performance.
Levelling Up in AI Safety Research Engineering
Thanks, that’s a good point! I was very uncertain about that, it was mostly a made-up number. I do think the time to implement an ML paper depends wildly on how complex the paper is (e.g. a new training algorithm paper necessitates a lot more time to test it than a post-hoc interpretability paper that uses pre-trained models) and how much you implement (e.g. rewrite the code but don’t do any training vs evaluate the key result to get the most important graph vs try to replicate almost all of the results).
I now think my original 10-20 hours per paper number was probably an underestimate, but it feels really hard to come up with a robust estimate here and I’m not sure how valuable it would be, so I’ve removed that parenthetical from the text.
Thanks for sharing your experiences, too! As for transformers, yeah it seems pretty plausible that you could specialize in a bunch of traditional Deep RL methods and qualify as a good research engineer (e.g. very employable). That’s what several professionals seem to have done, e.g. Daniel Ziegler.
But maybe that’s changing, and it’s worth it to start learning things. It seems like most of the new RL papers incorporate some kind of transformer encoder in the loop, if not basically being a straight-up Decision Transformer.
Good find, added!
Sure!
Thanks for making this!
In the future, you may want to ask just one full name question for people who don’t fit neatly into the first name/last name split.