AI safety, governance, and alignment research and field buiilding.
GabeM
Karma: 409
Thanks for this post! I appreciate the transparency, and I’m sorry for all this suckiness.
Could one additional easyish structural change be making applications due even earlier for EAGx? I feel like the EA community has a bad tendency of having apps for things open until very soon before the actual thing, and maybe an earlier due date gives people more time to figure out if they’re going and creates more buffer before catering number deadlines. Ofc, this costs some extra organizer effort as you have to plan more ahead, but I expect that’s more of a shifting thing rather than an whole lot of extra work.
Ha thanks Vael! Yeah, that seems hard to standardize but potentially quite useful to use levels like these for hiring, promotions, and such. Let me know how it goes if you try it!
Thanks! Forgot about cloud computing, added a couple of courses to the Additional Resources of Level 4: Deep Learning.
Oh lol I didn’t realize that was a famous philosopher until now, someone commented from a Google account with that name! Removed Ludwig.
Sure!
Good find, added!
Thanks for sharing your experiences, too! As for transformers, yeah it seems pretty plausible that you could specialize in a bunch of traditional Deep RL methods and qualify as a good research engineer (e.g. very employable). That’s what several professionals seem to have done, e.g. Daniel Ziegler.
But maybe that’s changing, and it’s worth it to start learning things. It seems like most of the new RL papers incorporate some kind of transformer encoder in the loop, if not basically being a straight-up Decision Transformer.
Thanks, that’s a good point! I was very uncertain about that, it was mostly a made-up number. I do think the time to implement an ML paper depends wildly on how complex the paper is (e.g. a new training algorithm paper necessitates a lot more time to test it than a post-hoc interpretability paper that uses pre-trained models) and how much you implement (e.g. rewrite the code but don’t do any training vs evaluate the key result to get the most important graph vs try to replicate almost all of the results).
I now think my original 10-20 hours per paper number was probably an underestimate, but it feels really hard to come up with a robust estimate here and I’m not sure how valuable it would be, so I’ve removed that parenthetical from the text.
Levelling Up in AI Safety Research Engineering
I’ll also plug Microsoft Edge as a great tool for this: There’s both a desktop browser and a mobile app, and it has a fantastic built-in Read Aloud feature that works in both. You just click the Read Aloud icon or press Ctrl/Cmd+Shift+U on a keyboard and it will start reading your current web page or document out loud!
It has hundreds of neural voices (Microsoft calls them “Natural” voices) in dozens of languages and dialects, and you can change the reading speed too. I find the voices to be among the best I’ve heard, and the super low activation energy of not having to copy-paste anything or switch to another window means I use it much more often than when I tried apps like Neural Reader.
Sidenote, but as a browser, since it’s Chromium-based it’s basically the same as Google Chrome (you can even install extensions from the Chrome Web Store) but with slightly less bloat and better performance.
They just added to it so it’s now “Is Civilization on the Brink of Collapse? And Could We Recover?” but it still seems to not answer the first question.
Is Civilization on the Brink of Collapse? - Kurzgesagt
Thanks for building this, def seems like a way to save a lot of organizer time (and I appreciate how it differentiates things from a Bible group or a cult)!
To me, it seems like the main downside will be the lack of direct engagement between new people and established EAs. In a normal reading group, a participant meets and talks with a facilitator on day 1, and then every week between every 1-3 hours of EA-related reading. In this system, it seems like they don’t really get to meet and talk with someone until they go through a significant amount of independent exploration and write a reflection, and I wonder if the combination of that high required activation energy with little human-to-human guidance might cause you to lose some potentially good students as you go from the predicted 40 to 20.
You could try offering “cheaper” 1:1s to these people early, but that seems less efficient than having several of them in a weekly reading group discussion which would defeat the point. That’s not to say I don’t think this is the right move for your situation. Just that I’m extra curious about how this factor might play out, and I’m excited for you to test this system and share the results with other groups!
I had no idea there were this many nematodes—is wild animal welfare just nematode welfare?! Do Rethink, WAI, or others have any research on them?
Thanks for writing this! A couple days I was also wondering “🤔 could I use janky-statistics to estimate how the number of top N% most ‘promising’ students at X small elite private university compares to Y large non-elite public university?” But I’m glad you used non-janky statistics to evaluate this much better than I would have.
I’ll add a few more ways this could be wrong:
“Intelligence” could come in different qualities in different communities, e.g. maybe MIT tends to be more promising for maths (25% SAT Math = 780 😅), Harvard tends to be more promising for business and law, and yet other universities could be more promising in other things.
As others have pointed out, maybe “intelligence” (as measures by standardized test scores) isn’t the only metric to care about, and maybe we should be selecting for EQ/agency/truth-seekingness/etc. which could be differently distributed among schools than “intelligence.”
We don’t actually need the most “intelligent” students for a lot of things, maybe “medium intelligent” students among decent universities can still robustly understand EA ideas but be more agentic/impact-minded and be good for the community (this updates me more in favor of your point).
Conversely, maybe the bar for certain things (e.g. AI alignment technical researcher) should be much higher. Maybe the small-tail distribution of test scores doesn’t map well to the long-tailed distribution of impact in highly technical things, and we actually care about the 0.01% of students across some metric which could differ more significantly between schools.
I’m not sure how much to believe some of these, and some of them being right could mean the point presented in the post is not strong enough and there could be more “promising” students at non-elite universities than elite ones. Either way, I think it’s reasonable to update away from assuming EA outreach and community building should be focused on elite schools (if you’re a student at a non-elite school reading this, consider learning more about EA, getting mentorship, and starting a university group!).
I agree. I think it’s also worth pointing out that is a different metric from (the total number of people matching some criteria). One takeaway I got from the post is that while the probabilities might still be different between schools (“75th percentile SAT score falls very gradually”), the number of “smart” students might be comparable at different schools because many non-elite schools tend to be larger than private elite schools (further, since there are also many more non-elite schools, I might expect but that’s besides the point).
Practically, for EA outreach, this maybe implies that university outreach might be harder at big non-elite-but-still-good schools: as Joseph points out below, the median student might be less qualified, so you’d have to sample more to “sift” through all the students and find the top students. But EA outreach isn’t random sampling—it appeals to certain aptitudes, can use nerd-sniping/self-selection effects, and can probably be further improved to select for the most smart/agentic/thoughtful/truth-seeking/whatever you care about students, and there might be a comparable number of those at different universities regardless of elite-ness. If the heavy tail of impact is what matters, this makes me update towards believing EA outreach (and students starting EA groups) at non-elite-but-still-good universities could be as good as at elite universities.
Wow quite a strong claim, and as a longtermist mostly working on AI, not sure how to feel about it yet.🤔 But I’ll take a stab at a different thing:
However, If you care about causes such as poverty, health or animals, and you think your community could update based on a post saying “Cause Y will be affected by AI”, leave a comment and I will think about writing about it.
A few initial concrete examples:
Poverty: Transformative AI could radically transform the global economy, which could lead to the end of poverty as we know it and immense wealth for all or perhaps tremendous inequality.
(Mental) health: We’re likely to see really good AI chatbots pop up in the shorter term (i.e. before TAI, kinda starting now) which could potentially serve as ultra-low-cost but very useful therapists/friends.
Animals: AI could make alternative proteins much more efficient to produce, bringing a much sooner switch away from animal agriculture (and might even be necessary for cultivated meat to be competitive). Some are already working on this.
Based on AI timelines, I would be surprised if all these things didn’t happen in the next ~50 years, which feels quite “neartermist” to me.
I can’t unread this comment:
“Humanity could, theoretically, last for millions of centuries on Earth alone.” I find this claim utterly absurd. I’d be surprised if humanity outlasts this century.
Ughh they’re so close to getting it! Maybe this should give me hope?
Good points! This reminds me of the recent Community Builders Spend Too Much Time Community Building post. Here are some thoughts about this issue:
Field-building and up-skilling don’t have to be orthogonal. I’m hopeful that a lot of an organizer’s time in such a group would involve doing the same things general members going through the system would be doing, like facilitating interesting reading group discussions or working on interesting AI alignment research projects. As the too much time post suggests, maybe just doing the cool learning stuff is a great way to show that we’re serious, get new people interested, and keep our group engaged.
Like Trevor Levin says in that reply, I think field-building is more valuable now than it will be as we get closer to AGI, and I think direct work will be more valuable later than it is now. Moreso, I think field-building while you’re a university student is significantly more valuable than field-building after you’ve graduated.
I don’t necessarily think the most advanced students will always need to be organizers under this model. I think there’s a growing body of EAs who want to help with AI alignment field-building but don’t necessarily think they’re the best fit for direct work (maybe they’re underconfident though), and this could be a great opportunity for them to help with little opportunity costs.
I’m really hopeful about several new orgs people are starting for field-wide infrastructure that could help offset a lot of the operational costs of this, including orgs that might be able to hire professional ops people to support a local group.
That’s not to say I recommend every student who’s really into AI safety delay their personal growth to work on starting a university group. Just that if you have help and think you could have a big impact, it might be worth considering letting off the solo up-skilling pedal to add in some more field-building.
Curious what you mean by this. One version of chance is “uniform prediction of AGI over future years” which obviously seems worse than Metaculus, but perhaps you meant a more specific baseline?
Personally, I think forecasts like these are rough averages of what informed individuals would think about these questions. Yes, you shouldn’t defer to them, but it’s also useful to recognize how that community’s predictions have changed over time.