I definitely sympathize, though I’d phrase things differently.
As I’ve notedbefore, I think much of the cause is just that the community incentives very much come from the funding. And right now, we only have a few funders, and those funders are much more focused on AI Safety specifics then they are things like rationality/epistemics/morality. I think these people are generally convinced on specific AI Safety topics and unconvinced by a lot of more exploratory / foundational work.
For example, this is fairly clear at OP. Their team focused on “EA” is formally called “GCR Capacity Building.” The obvious goal is to “get people into GCR jobs.”
You mention a frustration about 80k. But 80k is getting a huge amount of their funding from OP, so it makes sense to me that they’re doing the sorts of things that OP would like.
Personally, I’d like to see more donations come from community members, to be aimed at community things. I feel that the EA scene has really failed here, but I’m hopeful there could be changes.
I don’t mean to bash OP / SFF / others. I think they’re doing reasonable things given their worldviews, and overall I think they’re both very positive. I’m just pointing out that they represent about all the main funding we have, and that they just aren’t focused on the EA things some community members care about.
Right now, I think that EA is in a very weak position. There just aren’t that many people willing to put in time or money to push forward the key EA programs and mission, other than using it as a way to get somewhat narrow GCR goals.
Or, in your terms, I think that almost no one is actually funding the “Soul” of EA, including the proverbial EA community.
I guess this might be the “last, properly funded EA analysis” unless something came out after that which I missed (“last” in that going forward it seems funders are doubling down on AI and might not rethink this decision in the near future)? I think the takeaway from this work by Rethink Priorities for me is that it is not at all unreasonable to focus on other things than AI, as going all in on AI seemed to require a set of quite extreme beliefs/assumptions. Would be happy to be corrected if my simple take-away might be overly naive.
I think to me cubic or faster value increases and that we will mostly have a future with very low risk and that it is only now, or during a few periods that risk will be extremely high. In a sense, I see these assumptions in tension as high value often is accompanied by high risk. I was just made aware that even sending digital being to far-away galaxies looks extremely expensive energy-wise, even if one keep only the minimum power requirement during a multi-year travel between solar systems. I guess in essence, I feel like to justify these assumptions one would have to really look into what these assumptions materially mean, and use historical precedent and reasonable analysis across a wide range of scenarios to see if they make sense. For me this is more intuition and a scepticism that enough work have been done to get certainty of these assumptions. To some degree, I also feel like AI safety was a direction where funders might get more of a feeling “of doing something”—something I have been at fault at myself. Something like just chipping away at the stubborn problems of poverty/global health, or animal welfare is likely to remain “unsolved” problems even with billions more invested. Moreover, they do not have novelty, and these “industries” are less prone to be affected while AI is new and one can see more systemic effects. Maybe this last point actually drives at something supporting AI safety—it might be more tractable in a sense. Sorry this was long and not underpinned by much analysis so would welcome any analysis on these points, especially analysis that might change my mind.
I do think one issue people may be underrating is that we might just not bother with space colonization, if the distances and costs mean that no one on Earth will ever see significant material gain from it.
I think that given a few generations of expansion to different stars in all directions, it is not implausible (i.e. at least 25% chance) that X-risk becomes extremely low (i.e. under 1 in 100,000 per century, once there are say, 60 colonies with expansion plans, and a lot less once there are 1000 colonies.) After all, we’ve already survived a million years, and most X-risks not from AI seem mostly to apply to single planet civilizations, plus the lightspeed barrier makes it hard for a risk to reach everywhere at once. But I think I agree that thinking through this stuff is very, very hard, and I’m sympathetic to David Thorstad’s claim that if we keep finding ways current estimates of the value of X-risk reduction could be wildly wrong, at some point we should just lose trust in current estimates (see here for Thorstad making the claim: https://reflectivealtruism.com/2023/11/03/mistakes-in-the-moral-mathematics-of-existential-risk-part-5-implications/), even though I am a lot less confident than Thorstad is that very low future per year risk is an “extreme” assumption.
It is disturbing to me how much Thorstad’s work on this stuff seems to have been ignored by leading orgs; it is very serious work criticizing key assumptions that they base their decisions on, even if I personally think he tends to push points in his favour a bit far. I assume the same is true for the Rethink report you cite, although it is long and complicated enough, unlike Thorstad’s short blog posts, that I haven’t read any of it.
Actually reading this again, I think maybe you have a point about complexity of arguments/assumptions. Not sure if it is Occam’s Razor, but if one has to contort an argument into this weird, windy argument with unusual assumptions—maybe this hard attempt at something like “rationalization” should be a warning flag. That said, the world is complex and unpredictable, so perhaps reasoning about it is complex too—I guess this is an age-old debate with no clear answer!
Animal welfare on the other hands seems so extremely easy to argue is important. Global poverty a little less so but still easier than x-risk (more about whether handing out mosquito nets is better than economic growth, democracy, human rights, etc.).
My wish here is that specific people running orgs and projects were made of tougher stuff re following funding incentives. For example, it doesn’t seem like your project is at serious risk of defunding if you’re 20-30% more explicit about the risks you care about or what personally motivates you to do this work.
There are probably only about 200 people on Earth with the context x competence for OP to enthusiastically fund for leading on this work – they have bargaining power to frame their projects differently. Yet on this telling, they bow to incentives to be the very-most-shining star by OP’s standard, so they can scale up and get more funding. I would just make the trade off the other way: be smaller and more focused on things that matter.
I think social feedback loops might bend back around to OP as well if they had fewer options. Indeed, this might have been the case before FTX. The point of the piece is that I see the inverse happening, I just might be more agnostic about whether the source is OP or specific project leaders. Either or both can correct if they buy my story.
> For example, it doesn’t seem like your project is at serious risk of defunding if you’re 20-30% more explicit about the risks you care about or what personally motivates you to do this work.
I suspect that most nonprofit leaders feel a great deal of funding insecurity. There’s always neat new initiatives that a group would love to expand to, and also, managers hate the risk of potentially needing to fire employees. They’re often thinking about funding on the margins—either they are nervous about firing a few employees, or they are hoping to expand to new areas.
> There are probably only about 200 people on Earth with the context x competence for OP to enthusiastically fund for leading on this work
I think there’s more competition. OP covers a lot of ground. I could easily see them just allocating a bit more money to human welfare later on, for example.
> My wish here is that specific people running orgs and projects were made of tougher stuff re following funding incentives.
I think that the issue of incentives runs deeper than this. It’s not just a matter of leaders straightforwardly understanding the incentives and taking according actions. It’s also that people will start believing things that are convenient to said incentives, that leaders will be chosen who seem to be good fits for the funding situation, and so on. The people who really believe in other goals often get frustrated and leave.
I’d guess that the leaders of these orgs feel more aligned with the OP agenda then they do the agenda you outline, for instance.
Agree on most of this too. I wrote too categorically about the risk of “defunding.” You will be on a shorter leash if you take your 20-30% independent-view discount. I was mostly saying that funding wouldn’t go to zero and crash your org.
I further agree on cognitive dissonance + selection effects.
Maybe the main disagreement is that OP is ~a fixed monolith. I know people there. They’re quite EA in my accounting; much like I think of many leaders at grantees. There’s room in these joints. I think current trends are driven by “deference to the vibe” on both sides of the grant-making arrangement. Everyone perceives plain speaking about values and motivations as cringe and counterproductive and it thereby becomes the reality.
I’m sure org leaders and I have disagreements along these lines, but I think they’d also concede they’re doing some substantial amount of deliberate deemphasis of what they regard as their terminal goals in service of something more instrumental. They do probably disagree with me that it is best all-things-considered to undo this, but I wrote the post to convince them!
For what it’s worth, I find some of what’s said in this thread quite surprising.
Reading your post, I saw you describing two dynamics:
Principles-first EA initiatives are being replaced by AI safety initiatives
AI safety initiatives founded by EAs, which one would naively expect to remain x-risk focused, are becoming safety-washed (e.g., your BlueDot example)
I understood @Ozzie’s first comment on funding to be about 1. But then your subsequent discussion with Ozzie seems to also point to funding as explaining 2.[1]
While Open Phil has opinions within AI safety that have alienated some EAs—e.g., heavy emphasis on pure ML work[2]—my impression was that they are very much motivated by ‘real,’ x-risk-focused AI safety concerns, rather than things like discrimination and copyright infringement. But it sounds like you might actually think that OP-funded AI safety orgs are feeling pressure from OP to be less about x-risk? If so, this is a major update for me, and one that fills me with pessimism.
For example, you say, “[OP-funded orgs] bow to incentives to be the very-most-shining star by OP’s standard, so they can scale up and get more funding. I would just make the trade off the other way: be smaller and more focused on things that matter.”
I think OP and grantees are synced up on xrisk (or at least GCRs) being the terminal goal. My issue is that their instrumental goals seem to involve a lot of deemphasizing that focus to expand reach/influence/status/number of allies in ways that I worry lend themselves to mission/value drift.
The BlueDot example seems different to what I was pointing at.
I would flag that lack of EA funding power sometimes makes xrisk less of an issue.
Like, some groups might not trust that OP/SFF will continue to support them, and then do whatever they think they need to in order to attract other money—and this often is at odds with xrisk prioritization.
(I clearly see this as a issue with the broader world, not with OP/SFF)
I definitely sympathize, though I’d phrase things differently.
As I’ve noted before, I think much of the cause is just that the community incentives very much come from the funding. And right now, we only have a few funders, and those funders are much more focused on AI Safety specifics then they are things like rationality/epistemics/morality. I think these people are generally convinced on specific AI Safety topics and unconvinced by a lot of more exploratory / foundational work.
For example, this is fairly clear at OP. Their team focused on “EA” is formally called “GCR Capacity Building.” The obvious goal is to “get people into GCR jobs.”
You mention a frustration about 80k. But 80k is getting a huge amount of their funding from OP, so it makes sense to me that they’re doing the sorts of things that OP would like.
Personally, I’d like to see more donations come from community members, to be aimed at community things. I feel that the EA scene has really failed here, but I’m hopeful there could be changes.
I don’t mean to bash OP / SFF / others. I think they’re doing reasonable things given their worldviews, and overall I think they’re both very positive. I’m just pointing out that they represent about all the main funding we have, and that they just aren’t focused on the EA things some community members care about.
Right now, I think that EA is in a very weak position. There just aren’t that many people willing to put in time or money to push forward the key EA programs and mission, other than using it as a way to get somewhat narrow GCR goals.
Or, in your terms, I think that almost no one is actually funding the “Soul” of EA, including the proverbial EA community.
I just have to call out the amazing work by Rethink Priorities and those that funded this sequence of analyses (not sure who that is, would welcome info!): https://forum.effectivealtruism.org/s/WdL3LE5LHvTwWmyqj
I guess this might be the “last, properly funded EA analysis” unless something came out after that which I missed (“last” in that going forward it seems funders are doubling down on AI and might not rethink this decision in the near future)? I think the takeaway from this work by Rethink Priorities for me is that it is not at all unreasonable to focus on other things than AI, as going all in on AI seemed to require a set of quite extreme beliefs/assumptions. Would be happy to be corrected if my simple take-away might be overly naive.
What are the “extreme beliefs” you have in mind?
I think to me cubic or faster value increases and that we will mostly have a future with very low risk and that it is only now, or during a few periods that risk will be extremely high. In a sense, I see these assumptions in tension as high value often is accompanied by high risk. I was just made aware that even sending digital being to far-away galaxies looks extremely expensive energy-wise, even if one keep only the minimum power requirement during a multi-year travel between solar systems. I guess in essence, I feel like to justify these assumptions one would have to really look into what these assumptions materially mean, and use historical precedent and reasonable analysis across a wide range of scenarios to see if they make sense. For me this is more intuition and a scepticism that enough work have been done to get certainty of these assumptions. To some degree, I also feel like AI safety was a direction where funders might get more of a feeling “of doing something”—something I have been at fault at myself. Something like just chipping away at the stubborn problems of poverty/global health, or animal welfare is likely to remain “unsolved” problems even with billions more invested. Moreover, they do not have novelty, and these “industries” are less prone to be affected while AI is new and one can see more systemic effects. Maybe this last point actually drives at something supporting AI safety—it might be more tractable in a sense. Sorry this was long and not underpinned by much analysis so would welcome any analysis on these points, especially analysis that might change my mind.
I do think one issue people may be underrating is that we might just not bother with space colonization, if the distances and costs mean that no one on Earth will ever see significant material gain from it.
I think that given a few generations of expansion to different stars in all directions, it is not implausible (i.e. at least 25% chance) that X-risk becomes extremely low (i.e. under 1 in 100,000 per century, once there are say, 60 colonies with expansion plans, and a lot less once there are 1000 colonies.) After all, we’ve already survived a million years, and most X-risks not from AI seem mostly to apply to single planet civilizations, plus the lightspeed barrier makes it hard for a risk to reach everywhere at once. But I think I agree that thinking through this stuff is very, very hard, and I’m sympathetic to David Thorstad’s claim that if we keep finding ways current estimates of the value of X-risk reduction could be wildly wrong, at some point we should just lose trust in current estimates (see here for Thorstad making the claim: https://reflectivealtruism.com/2023/11/03/mistakes-in-the-moral-mathematics-of-existential-risk-part-5-implications/), even though I am a lot less confident than Thorstad is that very low future per year risk is an “extreme” assumption.
It is disturbing to me how much Thorstad’s work on this stuff seems to have been ignored by leading orgs; it is very serious work criticizing key assumptions that they base their decisions on, even if I personally think he tends to push points in his favour a bit far. I assume the same is true for the Rethink report you cite, although it is long and complicated enough, unlike Thorstad’s short blog posts, that I haven’t read any of it.
Actually reading this again, I think maybe you have a point about complexity of arguments/assumptions. Not sure if it is Occam’s Razor, but if one has to contort an argument into this weird, windy argument with unusual assumptions—maybe this hard attempt at something like “rationalization” should be a warning flag. That said, the world is complex and unpredictable, so perhaps reasoning about it is complex too—I guess this is an age-old debate with no clear answer!
Animal welfare on the other hands seems so extremely easy to argue is important. Global poverty a little less so but still easier than x-risk (more about whether handing out mosquito nets is better than economic growth, democracy, human rights, etc.).
I agree with all of this.
My wish here is that specific people running orgs and projects were made of tougher stuff re following funding incentives. For example, it doesn’t seem like your project is at serious risk of defunding if you’re 20-30% more explicit about the risks you care about or what personally motivates you to do this work.
There are probably only about 200 people on Earth with the context x competence for OP to enthusiastically fund for leading on this work – they have bargaining power to frame their projects differently. Yet on this telling, they bow to incentives to be the very-most-shining star by OP’s standard, so they can scale up and get more funding. I would just make the trade off the other way: be smaller and more focused on things that matter.
I think social feedback loops might bend back around to OP as well if they had fewer options. Indeed, this might have been the case before FTX. The point of the piece is that I see the inverse happening, I just might be more agnostic about whether the source is OP or specific project leaders. Either or both can correct if they buy my story.
I see it a bit differently.
> For example, it doesn’t seem like your project is at serious risk of defunding if you’re 20-30% more explicit about the risks you care about or what personally motivates you to do this work.
I suspect that most nonprofit leaders feel a great deal of funding insecurity. There’s always neat new initiatives that a group would love to expand to, and also, managers hate the risk of potentially needing to fire employees. They’re often thinking about funding on the margins—either they are nervous about firing a few employees, or they are hoping to expand to new areas.
> There are probably only about 200 people on Earth with the context x competence for OP to enthusiastically fund for leading on this work
I think there’s more competition. OP covers a lot of ground. I could easily see them just allocating a bit more money to human welfare later on, for example.
> My wish here is that specific people running orgs and projects were made of tougher stuff re following funding incentives.
I think that the issue of incentives runs deeper than this. It’s not just a matter of leaders straightforwardly understanding the incentives and taking according actions. It’s also that people will start believing things that are convenient to said incentives, that leaders will be chosen who seem to be good fits for the funding situation, and so on. The people who really believe in other goals often get frustrated and leave.
I’d guess that the leaders of these orgs feel more aligned with the OP agenda then they do the agenda you outline, for instance.
Agree on most of this too. I wrote too categorically about the risk of “defunding.” You will be on a shorter leash if you take your 20-30% independent-view discount. I was mostly saying that funding wouldn’t go to zero and crash your org.
I further agree on cognitive dissonance + selection effects.
Maybe the main disagreement is that OP is ~a fixed monolith. I know people there. They’re quite EA in my accounting; much like I think of many leaders at grantees. There’s room in these joints. I think current trends are driven by “deference to the vibe” on both sides of the grant-making arrangement. Everyone perceives plain speaking about values and motivations as cringe and counterproductive and it thereby becomes the reality.
I’m sure org leaders and I have disagreements along these lines, but I think they’d also concede they’re doing some substantial amount of deliberate deemphasis of what they regard as their terminal goals in service of something more instrumental. They do probably disagree with me that it is best all-things-considered to undo this, but I wrote the post to convince them!
For what it’s worth, I find some of what’s said in this thread quite surprising.
Reading your post, I saw you describing two dynamics:
Principles-first EA initiatives are being replaced by AI safety initiatives
AI safety initiatives founded by EAs, which one would naively expect to remain x-risk focused, are becoming safety-washed (e.g., your BlueDot example)
I understood @Ozzie’s first comment on funding to be about 1. But then your subsequent discussion with Ozzie seems to also point to funding as explaining 2.[1]
While Open Phil has opinions within AI safety that have alienated some EAs—e.g., heavy emphasis on pure ML work[2]—my impression was that they are very much motivated by ‘real,’ x-risk-focused AI safety concerns, rather than things like discrimination and copyright infringement. But it sounds like you might actually think that OP-funded AI safety orgs are feeling pressure from OP to be less about x-risk? If so, this is a major update for me, and one that fills me with pessimism.
For example, you say, “[OP-funded orgs] bow to incentives to be the very-most-shining star by OP’s standard, so they can scale up and get more funding. I would just make the trade off the other way: be smaller and more focused on things that matter.”
At the expense of, e.g., more philosophical approaches
I think OP and grantees are synced up on xrisk (or at least GCRs) being the terminal goal. My issue is that their instrumental goals seem to involve a lot of deemphasizing that focus to expand reach/influence/status/number of allies in ways that I worry lend themselves to mission/value drift.
Yea, I broadly agree with Mjreard here.
The BlueDot example seems different to what I was pointing at.
I would flag that lack of EA funding power sometimes makes xrisk less of an issue.
Like, some groups might not trust that OP/SFF will continue to support them, and then do whatever they think they need to in order to attract other money—and this often is at odds with xrisk prioritization.
(I clearly see this as a issue with the broader world, not with OP/SFF)