Co-founding BlueDot Impact, focusing on AI safety talent pipeline strategy.
Have a background consisting of a brief research stint on pessimistic agents (reinforcement learning), ML engineering & product ownership, and Physics
Jamie B
Karma: 884
Thanks for engaging!
Two sessions. One to discuss the readings and another for people to bounce their takes off others in the cohort.
Sounds like a fun experiment! I found that just open discussion sometimes leads to less valuable discussion, so in both cases I’d focus on a few specific discussion prompts / trying to help people come to a conclusion on some question. I linked to something about learning activities in the main post, which I think helps with session design. As with anything though, I think trying it out is the only way to know for sure, so feel free to ignore me.
without assuming knowledge of AGISF
I’d be keen to hear specifically what the pre-requisite knowledge is—just in order to inform people if they ‘know enough’ to take your course. Maybe it’s weeks 1-3 of the alignment course? Agree with your assessment that further courses can be more specific, though.
I agree that one of the best ways to do this would be to just create a curriculum, send it around to people and then additionally collect feedback from people who have gone through the course
Sounds right! I would encourage you trying to front-load some of the work before creating a curriculum though. Without knowing how expert you are in agent foundations yourself—I’d suggest trying to take steps that mean your first stab is close enough for giving feedback to seem valuable to the people you ask, and so it’s not a huge lift to get from 1st draft to final product and there are no nasty surprises from people who would have done it completely differently.
I.e. what if you ask 3-5 experts what they think the most important part of agent foundations is, and maybe try to conduct 30 min interviews with them to solicit the story they would tell in a curriculum? You can also ask them their top recommended resources, and why they recommend it. That would be a strong start, I think.
Do I need a technical background to work on AI Governance? I think no, not really. Quick take because I don’t justify many of my claims.
Context: I haver been a technical ML engineer and (briefly) a researcher, and I’m now trying to work on AI governance (and spending a lot of time speaking to people who do work on AI governance).
Examples of things that are useful to understand to do AI governance:
1. Knowing about the train, test, deploy cycle at industrial AI companies.
2. 1. Knowing the psyche of ML engineers at those orgs.
3. Knowing which media channels machine learning engineers & researchers use to stay on top of news, including twitter & ML companies.
You don’t get any of those insights by doing an ML coursera course. It might be fun / gratifying to do that course for other reasons, but I think it won’t make you better at governance. It’s better to have a few friends who are ML engineers and to get them to sketch out what it’s like at a lab, some day (or—more costly but more thorough—to take a role at a lab, technical or nontechnical).
What I do think you need to engage with technically is not to be afraid to read below the surface of techincal memes—but I think not much below the surface.
Concrete example: watermarking.It’s enough for policymakers to be able to read a few watermarking papers and understand:
a) watermarking is a way of tagging your model’s outputs to prove it was produced by AI
b) There are no tried & tested, reliable watermarking methods at the moment.
Where I see nontechnincal folk fall down (less so in this community) is when they throw out the term ‘watermarking’ but couldn’t tell you about what methods can be used or what the reliability of those methods is. I think that can be read about, and you don’t need to have direct experience having tried to watermarking something (I certainly haven’t).
I revisit this post from time to time, and had a new thought!
Did you consider at the time talent needs in the civil service & US congress? If so, would you consider these differently now?
This might just be the same as “doing policy implementation”, and would therefore be quite similar to Angelina’s comment. My question is inspired by the rapid growth in interest in AI regulation in the UK & US governments since this post, which led me to consider potential talent needs on those teams.
Yes—the best thing to do is to sign up and work through the curriculum in your own time!
Thanks for the post!
There was consensus that it would be good if CEA replaced one of its (currently) three annual conferences with a conference that’s explicitly framed as being about x-risk or AI-risk focused conference.
In response to a corresponding prompt (“ … at least one of the EAGs should get replaced by an x-risk or AI-risk focused conference …”)I’m curious if you felt the thrust was that the group thought it’s good if CEA in particular replace the activity of running its 3rd EAG with running an AI safety conference, or that there should be an AI safety conference?
In general when we talk about ‘cause area specific field building’, the purpose that makes most sense to me is to build a community around those cause areas, which people who don’t buy the whole EA philosophy can join if they spot a legible cause they think is worthwhile working on.
I’m a little hesitant to default to repurpose existing EA institutions, communities and events to house the proposed cause area specific field building. It seems to me that the main benefit of cause area specific field building is to potentially build something new, fresh and separate from the other cultural norms and beliefs that the EA community brings with it.
Perhaps the crux for me is “is this a conference for EAs interested in AI safety, or is it a conference for anyone interested in AI safety?” If the latter, this points away from an EA-affiliated conference (though I appreciate there are pragmatic questions around “who else would do it”). A fresh feel and new audience might still be achievable in the case that CEA runs the conference ops, but I imagine it would be important to bear in mind during CEA’s branding, outreach and choices made during the execution of such a conference.
We’ll aim to release a short post about this by the end of the week!
I also sometimes use naturalreaders. Unfortunately I find it a bit… unnatural at times.
I’ve been really enjoying Type III Audio’s reader on this forum, though!
That sounds great to me, thanks!
I totally agree there’s a gap here. At BlueDot Impact (/ AGI safety fundamentals), we’re currently working on understanding the pipeline for ourselves.
We’ll be launching another governance course in the next week, and in the longer term we will publish more info on governance careers on our website, as and when we establish the information for ourselves.
In the meantime, there’s great advice on this account, mostly targeted at people in the US, but there might be some transferrable lessons:
https://forum.effectivealtruism.org/users/us-policy-careers
Thanks for highlighting that there were other 2 announcements that I didn’t focus on in this post.
Whilst the funding announcement may be positive, I didn’t expect that it would have strong implications for alignment research—so I chose to ignore it in this post. I didn’t spend more than a minute checking my assumption there, though.
RE the announcement of further OMB policies- I totally agree that it sounds like it could be important for alignment / risk reduction. I omitted that announcement mostly because I didn’t have very much context to know what those policies would entail, given the announcement was quite light on details at this point. Thanks for shedding some light on what it could mean!
FWIW, I think this post makes progress and could work in the contexts of some groups. As a concrete example, it would probably work for me as an organiser of one-off courses, and probably for organisers of one-off retreats or internships.
I appreciate the thrust of comments pointing out imperfections in e.g. local group settings, but I just want to be careful that we don’t throw out the proposal just because it doesn’t work for everyone in all contexts; I think it’s better to start with an an imperfect starting point and to iterate on that where it doesn’t work in specific contexts, rather than to try to come up with the perfect policy in-theory and get paralysed when we can’t achieve that.
Thanks for highlighting this!
Great, thanks for writing this up! I don’t work in policy, but it seems to be an extremely pragmatic and helpful guide from an outside-perspective.
A question—is being a US citizen a hard requirement for all of this advice?
If not a hard requirement, what hidden (or explicit) barriers would you expect a non-citizen to face?
I also think that power dynamics are the source of the biggest problems in the work/social overlap, so a flatter power structure might be a good way of avoiding some of the pitfalls and abuses of the work/social overlap.
Do you think that in abstract that professional/social overlap is less of a problem when the power structure is flatter, or that having a flatter power structure is something that EA could actually achieve?
I’m curious because, to deal with potential abuse of power, I would prefer a more explicit power structure (which sounds like an opposite conclusion to your suggestion).
My first assumption is that power structures are an unavoidable fact in any group of people. I assume that trying to enact a flatter power structure might actually cash out as pretending the power structure doesn’t exist [this might be where we disagree!].
Pretending that power structures are flat leads to plausibly permissable abuse of the actual underlying power structure. However strictly acknowledging a power structure means one is forced to acknowledge the power dynamic.
So to encourage healthy relationships, I would have called for making power structures explicit, in EA or any group.
Thanks for exploring this issue! I agree that there could be more understanding between AI safety & the wider AI community, and I’m curious to do more thinking about this.
I think each of the 3 claims you make in the body of the text are broadly true. However I don’t think they directly back up the claim in the title that “AI safety is not separate from near-term applications”.
I think there are some important ways that AI safety is distinct; it goes 1 step further by imagining the capabilities of future systems, and trying to anticipate ways they could go wrong ahead of time. I think there are some research questions it’d be hard to work on if the AI safety field wasn’t separate from current-day application research. E.g. agent foundations, inner misalignment and detecting deception.
I think I agree with much of your sentiment still. To illustrate what I mean, I would like it to be true that:
Important AI current-day-application safety issues are worked on by many people, and there is mutual respect between our communities
Work done by near-term application researchers is known about and leverageable by the AGI safety community
Ultimately, there is still a distinct, accessible AGI safety community that works on issues distinct to advanced, general AI systems
I wrote this guide for Cambridge, UK, when Cambridge EA CIC was running a hiring round.
I think a guide for Cambridge based on your template would still be valuable (but I won’t do it any time soon). In my guide, I was focused on 1) a broader audience (including ‘non-EAs’) and 2) moving to Cambridge rather than visiting temporarily.
Someone brought a game called “Confident?” into the Cambridge office. It’s basically a competitive gamification of callibration training.
You are rewarded for having the smallest confidence window of all the players, and penalised if your answer is outside of your confidence window.
Super fun!
EA knowledge is not required. Thanks for asking!
Thanks for trying this and writing it up :-) I think there might well be some benefits to getting through the programme intensively, like:
+ It takes less time so you can do whatever you want to do next (you mention reading more researchers’ agenda)
+ A better bonding experience if you do it in-person, which might only be possible in a 1-week intensive session if you don’t all live in the same cityMy perceived drawbacks:
- Less digestion & retention of the content (you highlighted this one)
- Less opportunity to mix with other people doing the programme (we hope to spark more of this next time we run the global programme)
- Might be harder to have access to facilitators / more knowledgeable people (you also stated this is important)Overall, I think continuing the global programme suits people who couldn’t take the time to do an intensive version, and intensive version suits people who prefer not to do virtual reading groups.
I am currently pursuing a couple of projects that are intended to appeal to the sensibilities of AI researchers who aren’t in the alignment community already. This has already been very useful for informing the communications and messaging I would use for those. I can see myself referring back to this often, when pursuing other field building activities too. Thanks a lot for publishing this!
What I had in mind was “shows up to all 8 discussion groups for the taught part of the course”. I also didn’t check this figure, so that was from memory.
True, there are lots of ways to define it (e.g. finishing the readings, completing the project, etc)