I’m a generalist and open sourcerer that does a bit of everything, but perhaps nothing particularly well. I’m currently the AI Safety Group Support Lead at CEA.
I was previously a Software Engineer in the Worldview Investigations Team at Rethink Priorities.
Hope to see you next year! 🤝
At last, a biblically accurate Qualy The Lightbulb
[Opinion exclusively my own]
I think this framing has a lot of value. Funnily enough, I’ve heard tales of groups like this from the early days of EA groups, when people were just figuring things out, and this pattern would sometimes pop up.
I do want to push back a little bit on this:
But before deferring, I think it’s important to realize that you’re deferring, and to make sure that you understand and trust who or what you’re deferring to (and perhaps to first have an independent impression). Many intro fellowship curricula (eg the EA handbook) come across more as manifestos than research agendas—and are often presented as an ideology, not as evidence that we can use to make progress on our research question.
The EA handbook (which nowadays is what the vast majority of groups use for their intro fellowships) includes three “principles-first” weeks (weeks 1, 2, and 7), which are meant to help participants develop their own opinions with the help of only the basic EA tools or concepts.
Furthermore, week 7 (“What do you think”) includes a reading of independent impressions, and learning that concept (and discussing where EA might be wrong) is one of the key objectives of the week:
A key concept for this session is the importance of forming independent impressions. In the long run, you’re likely to gain a deeper understanding of important issues if you think through the arguments for yourself. But (since you can’t reason through everything) it can still sometimes make sense to defer to others when you’re making decisions.
In the past, a lot of work has been put in trying to calibrate how “principles-based” or “cause-oriented” intro fellowships should be, and I think the tradeoff can be steep for university groups since people can get rapidly disenchanted by abstract philosophical discussion about cause prioritization (as you mention). This can also lead to people treating EA as a purely intellectual exercise, instead of thinking of concrete ways in which EA ideas should (for example) change their career plans.
That said, I think there are other ways in which we could push groups further along in this direction, for example:
We could create programming (like fellowships or workshops) around cause prioritization, exploring different frameworks and tools in the field. Not just giving a taste of these frameworks (like the handbook does), but also teaching hands-on skills that participants can use for actual cause prioritization research.
We could push for more discussion centered around exploratory cause research, for example, by creating socials or events in which participants try to come up with cause candidates and do some preliminary research on how promising they are (i.e. a “cause exploration hackathon”).
I know there has been some previous work in this direction. For example, there’s this workshop template, or this fellowship on global priorities research. But I think we don’t yet have a killer demo, and I would be excited about groups experimenting with something like this.
X-Risk sentiment in the audience: at one point in the debate, one participant asked the audience who thought AI was an existential risk. From memory, around 2/3s of students put up their hands.
Do you have a rough sense of how many of these had interacted with AI Safety programming/content from your group? Like, was a substantial part of the audience just members from your group who had heard EA arguments about AIS?
Thanks for the awesome work!
I love this post because over EAG last weekend I talked with a couple other people about songs with EA themes, and we thought about making a forum post with a list.
I like many of the songs by Vienna Teng, particularly Landsailor, which is “An ode to shipping logistics, city lights, globalized agriculture, and our interconnected world.”
As a bonus, there’s also the The Precipice EDM remix (thanks @michel for flagging this one the other day lol).
Even beyond Head On, I think the most obviously EA song in the album is Visions:
(...)
Visions
Imagining the worlds that could be
Shaping a mosaic of fates
For all sentient beings
Visions
Cycles of growth and decay
Cascading chains of events
With no one to praise or blame
Visions
Avoidable suffering and pain
We are patiently inching our way
Toward unreachable utopias
Visions
Enslaved by the forces of nature
Elevated by mindless replicators
Challenged to steer our collective destiny
Ironically, I think I may have listened to this song dozens or hundreds of time before someone pointed out that José González was EA-adjacent, had sung at an EAG and had written this song to explicitly include EA themes.
The above makes me think that you should therefore be even more skeptical of OAA’s chances of success than you are about Gaia’s chances.
I am, but OAA also seems less specific, and it’s harder to evaluate its feasibility compared to something more concrete (like this proposal).
In fact, we think that if there are sufficiently many AI agents and decision intelligence systems that are model-based, i.e., use some kinds of executable state-space (“world”) models to do simulations, hypothesise counterfactually about different courses of actions and external conditions (sometimes in collaboration with other agents, i.e., planning together), and deploy regularisation techniques (from Monte Carlo aggregation of simulation results to amortized adversarial methods suggested by Bengio on slide 47 here) to permit compositional reasoning about risk and uncertaintly that scales beyond the boundary of a single agent, the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much “by default” because a lot of scientists and industry players will work in parallel to build some versions and local patches of it.
My problem with this is that it sounds good, but this argument relies on many hidden premises, that make me inherently skeptical of any strong claims like “(…) the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much ‘by default’”.
I think this could be addressed by a convincing MVP, and I think that you’re working on that, so I won’t push further on this point.
It’s fine with me and most other people except for e/accs, for now, but what about the time when the cost of training powerful/dangerous models will drop so much that anyone can buy a chip to train the next rogue AI for 1000$? How does compute governance look in this world?
The current best proposals for compute governance rely on very specific types of math. I don’t think throwing blockchain or DAOs at the problem makes a lot of sense, unless you find an instance of the very specific set of problems they’re good at solving.
My priors against the crypto world comes mostly from noticing a lot of people throwing tools to problems without a clear story of how these tools actually solve the problem. This has happened so many times that I have come to generally distrust crypto/blockchain proposals unless they give me a clear explanation of why using these technologies makes sense.
But I think the point I made here was kinda weak anyway (it was, at best, discrediting by association), so I don’t think it makes sense to litigate this particular point.
Compare with Collective Intelligence Project. It has started with the mission to “fix governance” (and pretty much “help to counteract Moloch” in the domain of political economy, too, they barely didn’t use this concept, or maybe they even did, I don’t want to check it now), and now they “pivoted” to AI safety and achieved great legibility on this path: e.g., they partner with OpenAI, apparently, on more than one project now. Does this mean that CIP is a “solution looking for a problem”? No, it’s just the kind of project that naturally lends to helps both with Moloch and AI safety. I’d say the same could be said of Gaia Network (if it is realised in some forms) and this lies pretty much in plain sight.
I find this decently convincing, actually. Like, maybe, I’m pattern matching too much on other projects which have in the past done something similar (just lightly rebranding themselves while tacking a completely different problem).
Overall, I still don’t feel very good about the overall feasibility of this project, but I think you were right to push back on some of my counterarguments here.
I think this would be more the result of new orgs rather than bigger orgs? Like I would argue that we currently don’t have anything near the optimal amount of orgs dedicated to training programs, and as funding increases, we will probably get a lot of them.
I think I’m in some ways confused about this. I think it’s true that the hiring situation is hard, but my priors say that this is likely to change fast[1] and that the downside risk for many people is probably low: especially in the technical side, time upskilling for AI Safety is probably not time completely wasted for the industry at large[2].
Are there any particular things you think we could do better? I think one could be just in general being less quick to suggest AIS as a career path for people who might be in risk for financial hardship as a result. Career guides in general do seem very oriented to people who can often take the risk of spending months unemployed, and doing upskilling or job hunting.[3]
Both as a result of higher funding and people funding a lot of orgs whenever there is both excess talent and funding overhang.
Especially ML-wise. But this is probably less true (if at all) for people upskilling in AI Safety policy and governance, strategy and fieldbuilding, etc.
Something something rich western countries
(even though they don’t seem to directly convince people to become EAs)
I want to flag that in general, convincing people to become EAs, or more precisely, creating cool spaces for more people to get into EA, is a thing that people in the community do actually do a lot. I did this myself a few years ago, by starting an EA group at my university. I’m guessing there might be several hundred EA community builders around the world, it’s just that they don’t generally focus on wealthy individuals specifically.[1]
Do you know why it tends to be difficult to convince people?
A general answer to this is that the core ideas of EA have some significant inferential distance for most people, that is, there’s usually a lot of context that you need to explain to someone before they get why EAs care so much about anti-malaria bednets, animal suffering or AI Safety. It’s also the case that some EA conclusions tend to be very counterintuitive and run against people’s previously held beliefs, adding to the difficulty. [2]
It can be much easier to pitch them in any of these individual cause areas, but this means you trade-off generality: maybe you get someone to care about animal suffering, but they end up donating to organizations that are much less impactful than the EA standard.[3]
And going into more speculative territory[4], I think people with a lot of money might be more skeptical of people wanting their money, which kinda makes sense. Philanthropists tend to be one-issue donors: they think about something that is meaningful to them (like education, homelessness, or dogs) and then tend to focus their donations heavily in that issue. Persuading them otherwise means not only explaining EA ideas, but also making them realize that they should stop doing what they’re doing, which is hard.
And would I be able to contact any of these 3 people you know? No worries if not!
Let me see what I can do, I’ve sent you a message through the forum!
PD: I want to push back on something I’ve said earlier: “My impression is that if you find a tractable way of doing this consistently, then you probably should”.
I should probably add some nuance: you shouldn’t pursue a job just because you think it’s a priori impactful. Whether you like this job, whether you feel like you could do it sustainably, is very important. And EA shouldn’t necessarily determine your entire life, there’s balance to be had. It’s also obviously very important to check this possibility against others, you shouldn’t just dig into the first impactful job you find.
I do think a lot of people have done this in ad-hoc basis though.
See The explanatory obstacle of EA for some concrete examples.
This is not to say there isn’t value to this. I think you can often convince people that they should donate to big funds (say, the Animal Welfare Fund), but this tends to be a tougher sell. In some cases, like say, the Lead Exposure Elimination Project, the EA context might be completely unnecessary for potential funders.
I’m not an expert on this, please don’t take my guesses too seriously!
This is great! I’ve publicly spoken about AI Safety a couple of times, and I’ve found some analogies to be tremendously useful. There’s one (which I’ve just submitted), that I particularly like:
I find myself thinking back to the early days of Covid. There were weeks when it was clear that lockdowns were coming, that the world was tilting into crisis, and yet normalcy reigned, and you sounded like a loon telling your family to stock up on toilet paper. There was the difficulty of living in exponential time, the impossible task of speeding policy and social change to match the rate of viral replication. I suspect that some of the political and social damage we still carry from the pandemic reflects that impossible acceleration. There is a natural pace to human deliberation. A lot breaks when we are denied the luxury of time.
But that is the kind of moment I believe we are in now. We do not have the luxury of moving this slowly in response, at least not if the technology is going to move this fast.
From this op-ed by Ezra Klein.
There are some relevant awesome lists (AIS, Alignment, ML Interpretability), but none of them are both up to date and on topic. There’s also alignment.dev, but not all the projects are open source, and it’s very infrastructure-oriented.
I wouldn’t be that surprised if I’m missing such a list, but AFAIK it doesn’t exist, and plausibly someone should work on this! (Maybe coordinate through AED?)
Changed it to a note. As for the latter, my intuition is that we should probably hedge for the full spectrum, from no experience to some wet bio background (but the case where we get an expert seems much more unlikely).
This is just a top-notch post. I love to see detailed analyses like this. Props.