Personally, I feel fairly strongly convinced to favor interventions that could help the future past 20 years from now. (A much lighter version of “Longtermism”).
If I had a budget of $10B, I’d probably donate a fair bit to some existing AI safety groups. But it’s tricky to know what to do with, say, $10k. And the fact that the SFF, OP, and others have funded some of the clearest wins makes it harder to know what’s exciting on-the-margin.
I feel incredibly unsatisfied with the public EA dialogue around AI safety strategy now. From what I can tell, there’s some intelligent conversation happening by a handful of people at the Constellation coworking space, but a lot of this is barely clear publicly. I think many people outside of Constellation are working on simplified models, like “AI is generally dangerous, so we should slow it all down,” as opposed to something like, “Really, there are three scary narrow scenarios we need to worry about.”
I recently spent a week in DC and found it interesting. But my impression is that a lot of people there are focused on fairly low-level details, without a great sense of the big-picture strategy. For example, there’s a lot of work into shovel-ready government legislation, but little thinking on what the TAI transition should really look like.
This sort of myopic mindset is also common in the technical space, where I meet a bunch of people focused on narrow aspects of LLMs, without much understanding of how their work exactly fits into the big picture of AI alignment. As an example, a lot of work seems like it would help with misuse risk, even when the big-picture EAs seem much more focused on accident risk.
Some (very) positive news is that we do have far more talent in this area than we did 5 years ago, and there’s correspondingly more discussion. But it still feels very chaotic.
A bit more evidence—it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting, and AI policy. But all of this still represents perhaps ~30% or so of their total budget, and I don’t sense that that’s about to change. Overall this comes off as measured and cautious. Meanwhile, it’s been difficult to convince other large donors to get into this area. (Other than Jaan Tallinn, he might well have been the strongest dedicated donor here).
Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it’s just that the AI safety people have migrated to other blogs and organizations.
But again, I’m very hopeful that we can find interventions that will help in the long-term, so few of these excite me. I’d expect and hope that interventions that help the long-term future would ultimately improve animal welfare and more.
So on one hand, AI risk seems like the main intervention area for the long-term, but on the other, the field is a bit of a mess right now.
I feel quite frustrated that EA doesn’t have many other strong recommendations for other potential donors interested in the long-term. For example, I’d really hope that there could be good interventions to make the US government or just US epistemics more robust, but I barely see any work in that area.
“Forecasting” is one interesting area—it currently does have some dedicated support from OP. But it honestly seems to be in a pretty mediocre state to me right now. There might be 15-30 full-time people in the space at this point, and there’s surprisingly little in terms of any long-term research agendas.
Hi Ozzie – Peter Favaloro here; I do grantmaking on technical AI safety at Open Philanthropy. Thanks for this post, I enjoyed it.
I want to react to this quote: …it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?)
I agree that over the past year or two our grantmaking in technical AI safety (TAIS) has been too bottlenecked by our grantmaking capacity, which in turn has been bottlenecked in part by our ability to hire technical grantmakers. (Though also, when we’ve tried to collect information on what opportunities we’re missing out on, we’ve been somewhat surprised at how few excellent, shovel-ready TAIS grants we’ve found.)
Over the past few months I’ve been setting up a new TAIS grantmaking team, to supplement Ajeya’s grantmaking. We’ve hired some great junior grantmakers and expect to publish an open call for applications in the next few months. After that we’ll likely try to hire more grantmakers. So stay tuned!
OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting
Yeah, I find myself very confused by this state of affairs. Hundreds of people are being funneled through the AI safety community-building pipeline, but there’s little funding for them to work on things once they come out the other side.[1]
As well as being suboptimal from the viewpoint of preventing existential catastrophe, this also just seems kind of common-sense unethical. Like, all these people (most of whom are bright-eyed youngsters) are being told that they can contribute, if only they skill up, and then they later findout that that’s not the case.
These community-building graduates can, of course, try going the non-philanthropic route—i.e., apply to AGI companies or government institutes. But there are major gaps in what those organizations are working on, in my view, and they also can’t absorb so many people.
Yea, I think this setup has been incredibly frustrating downstream. I’d hope that people from OP with knowledge could publicly reflect on this, but my quick impression is that some of the following factors happened: 1. OP has had major difficulties/limitations around hiring in the last 5+ years. Some of this is lack of attention, some is that there aren’t great candidates, some is a lack of ability. This effected some cause areas more than others. For whatever reason, they seemed to have more success hiring (and retaining talent) for community than for technical AI safety. 2. I think there’s been some uncertainties / disagreements into how important / valuable current technical AI safety organizations are to fund. For example, I imagine if this were a major priority from those in charge of OP, more could have been done. 3. OP management seems to be a bit in flux now. Lost Holden recently, hiring a new head of GCR, etc. 4. I think OP isn’t very transparent and public with explaining their limitations/challenges publicly. 5. I would flag that there are spots at Anthropic and Deepmind that we don’t need to fund, that are still good fits for talent. 6. I think some of the Paul Christiano—connected orgs were considered a conflict-of-interest, given that Ajeya Cotra was the main grantmaker. 7. Given all of this, I think it would be really nice if people could at least provide warnings about this. Like, people entering the field are strongly warned that the job market is very limited. But I’m not sure who feels responsible / well placed to do this.
Thanks for the comment, I think this is very astute.
~
Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it’s just that the AI safety people have migrated to other blogs and organizations.
I think there’s a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.
I don’t think that all AI safety orgs are actually fully funded since there are orgs that OP cannot fund for reasons (see Trevor’s post and also OP’s individual recommendations in AI) other than cost-effectiveness and also OP cannot and should not fund 100% of every org (it’s not sustainable for orgs to have just one mega-funder; see also what Abraham mentioned here). Also there is room for contrarian donation takes like Michael Dickens’s.
I think there’s a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.
That makes sense, but I’m feeling skeptical. There are just so many AI safety orgs now, and the technical ones generally aren’t even funded by OP.
On AI safety, I think it’s fairly likely (40%?) that the risk of x-risk (upon a lot of reflection) in the next 20 years is less than 20%, and that the entirety of the EA scene might be reducing it to say 15%.
This means that the entirety of the EA AI safety scene would help the EV of the world by ~5%.
On one hand, this is a whole lot. But on the other, I’m nervous that it’s not ambitious enough, for what could be one of the most [combination of well-resourced, well-meaning, and analytical/empirical] groups of our generation.
One thing I like about epistemic interventions is that the upper-bounds could be higher.
(There are some AI interventions that are more ambitious, but many do seem to be mainly about reducing x-risk by less than an order of magnitude, not increasing the steady-state potential outcome)
I’d also note here that an EV gain of 5% might not be particularly ambitious. It could well be the case that many different groups can do this—so it’s easier than it might seem if you think goodness is additive instead of multiplicative.
Around EA Priorities:
Personally, I feel fairly strongly convinced to favor interventions that could help the future past 20 years from now. (A much lighter version of “Longtermism”).
If I had a budget of $10B, I’d probably donate a fair bit to some existing AI safety groups. But it’s tricky to know what to do with, say, $10k. And the fact that the SFF, OP, and others have funded some of the clearest wins makes it harder to know what’s exciting on-the-margin.
I feel incredibly unsatisfied with the public EA dialogue around AI safety strategy now. From what I can tell, there’s some intelligent conversation happening by a handful of people at the Constellation coworking space, but a lot of this is barely clear publicly. I think many people outside of Constellation are working on simplified models, like “AI is generally dangerous, so we should slow it all down,” as opposed to something like, “Really, there are three scary narrow scenarios we need to worry about.”
I recently spent a week in DC and found it interesting. But my impression is that a lot of people there are focused on fairly low-level details, without a great sense of the big-picture strategy. For example, there’s a lot of work into shovel-ready government legislation, but little thinking on what the TAI transition should really look like.
This sort of myopic mindset is also common in the technical space, where I meet a bunch of people focused on narrow aspects of LLMs, without much understanding of how their work exactly fits into the big picture of AI alignment. As an example, a lot of work seems like it would help with misuse risk, even when the big-picture EAs seem much more focused on accident risk.
Some (very) positive news is that we do have far more talent in this area than we did 5 years ago, and there’s correspondingly more discussion. But it still feels very chaotic.
A bit more evidence—it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting, and AI policy. But all of this still represents perhaps ~30% or so of their total budget, and I don’t sense that that’s about to change. Overall this comes off as measured and cautious. Meanwhile, it’s been difficult to convince other large donors to get into this area. (Other than Jaan Tallinn, he might well have been the strongest dedicated donor here).
Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it’s just that the AI safety people have migrated to other blogs and organizations.
But again, I’m very hopeful that we can find interventions that will help in the long-term, so few of these excite me. I’d expect and hope that interventions that help the long-term future would ultimately improve animal welfare and more.
So on one hand, AI risk seems like the main intervention area for the long-term, but on the other, the field is a bit of a mess right now.
I feel quite frustrated that EA doesn’t have many other strong recommendations for other potential donors interested in the long-term. For example, I’d really hope that there could be good interventions to make the US government or just US epistemics more robust, but I barely see any work in that area.
“Forecasting” is one interesting area—it currently does have some dedicated support from OP. But it honestly seems to be in a pretty mediocre state to me right now. There might be 15-30 full-time people in the space at this point, and there’s surprisingly little in terms of any long-term research agendas.
Hi Ozzie – Peter Favaloro here; I do grantmaking on technical AI safety at Open Philanthropy. Thanks for this post, I enjoyed it.
I want to react to this quote:
…it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?)
I agree that over the past year or two our grantmaking in technical AI safety (TAIS) has been too bottlenecked by our grantmaking capacity, which in turn has been bottlenecked in part by our ability to hire technical grantmakers. (Though also, when we’ve tried to collect information on what opportunities we’re missing out on, we’ve been somewhat surprised at how few excellent, shovel-ready TAIS grants we’ve found.)
Over the past few months I’ve been setting up a new TAIS grantmaking team, to supplement Ajeya’s grantmaking. We’ve hired some great junior grantmakers and expect to publish an open call for applications in the next few months. After that we’ll likely try to hire more grantmakers. So stay tuned!
That sounds exciting, thanks for the update. Good luck with team building and grantmaking!
Yeah, I find myself very confused by this state of affairs. Hundreds of people are being funneled through the AI safety community-building pipeline, but there’s little funding for them to work on things once they come out the other side.[1]
As well as being suboptimal from the viewpoint of preventing existential catastrophe, this also just seems kind of common-sense unethical. Like, all these people (most of whom are bright-eyed youngsters) are being told that they can contribute, if only they skill up, and then they later find out that that’s not the case.
These community-building graduates can, of course, try going the non-philanthropic route—i.e., apply to AGI companies or government institutes. But there are major gaps in what those organizations are working on, in my view, and they also can’t absorb so many people.
Yea, I think this setup has been incredibly frustrating downstream. I’d hope that people from OP with knowledge could publicly reflect on this, but my quick impression is that some of the following factors happened:
1. OP has had major difficulties/limitations around hiring in the last 5+ years. Some of this is lack of attention, some is that there aren’t great candidates, some is a lack of ability. This effected some cause areas more than others. For whatever reason, they seemed to have more success hiring (and retaining talent) for community than for technical AI safety.
2. I think there’s been some uncertainties / disagreements into how important / valuable current technical AI safety organizations are to fund. For example, I imagine if this were a major priority from those in charge of OP, more could have been done.
3. OP management seems to be a bit in flux now. Lost Holden recently, hiring a new head of GCR, etc.
4. I think OP isn’t very transparent and public with explaining their limitations/challenges publicly.
5. I would flag that there are spots at Anthropic and Deepmind that we don’t need to fund, that are still good fits for talent.
6. I think some of the Paul Christiano—connected orgs were considered a conflict-of-interest, given that Ajeya Cotra was the main grantmaker.
7. Given all of this, I think it would be really nice if people could at least provide warnings about this. Like, people entering the field are strongly warned that the job market is very limited. But I’m not sure who feels responsible / well placed to do this.
Thanks for the comment, I think this is very astute.
~
I think there’s a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.
I don’t think that all AI safety orgs are actually fully funded since there are orgs that OP cannot fund for reasons (see Trevor’s post and also OP’s individual recommendations in AI) other than cost-effectiveness and also OP cannot and should not fund 100% of every org (it’s not sustainable for orgs to have just one mega-funder; see also what Abraham mentioned here). Also there is room for contrarian donation takes like Michael Dickens’s.
That makes sense, but I’m feeling skeptical. There are just so many AI safety orgs now, and the technical ones generally aren’t even funded by OP.
For example: https://www.lesswrong.com/posts/9n87is5QsCozxr9fp/the-big-nonprofits-post
While a bunch of these salaries are on the high side, not all of them are.
On AI safety, I think it’s fairly likely (40%?) that the risk of x-risk (upon a lot of reflection) in the next 20 years is less than 20%, and that the entirety of the EA scene might be reducing it to say 15%.
This means that the entirety of the EA AI safety scene would help the EV of the world by ~5%.
On one hand, this is a whole lot. But on the other, I’m nervous that it’s not ambitious enough, for what could be one of the most [combination of well-resourced, well-meaning, and analytical/empirical] groups of our generation.
One thing I like about epistemic interventions is that the upper-bounds could be higher.
(There are some AI interventions that are more ambitious, but many do seem to be mainly about reducing x-risk by less than an order of magnitude, not increasing the steady-state potential outcome)
I’d also note here that an EV gain of 5% might not be particularly ambitious. It could well be the case that many different groups can do this—so it’s easier than it might seem if you think goodness is additive instead of multiplicative.