I am a Program Officer at Longview Philanthropy, where I conduct investigations in artificial intelligence (AI), with a focus on digital minds and EU AI governance.
I previously completed a Ph.D. in economics at Stanford University, with focuses on the impacts of policies on future generations and the economics of animal welfare. I was a National Science Foundation Graduate Research Fellow and a Global Priorities Fellow with the Forethought Foundation. I was previously a Senior Research Analyst at Northwestern University’s Global Poverty Research Lab and Innovations for Poverty Action. I am also a board member of Animal Charity Evaluators.
zdgroff
Karma: 2,339
Thanks for writing this Will. I feel a bit torn on this so will lay out some places where I agree and some where I disagree:
I agree that some of these AI-related cause areas beyond takeover risk deserve to be seen as their own cause areas as such and that lumping them all under “AI” risks being a bit inaccurate.
That said, I think the same could be said of some areas of animal work—wild animal welfare, invertebrate welfare, and farmed vertebrate welfare should perhaps get their own billing. And then this can keep expanding—see, e.g., OP’s focus areas, which list several things under the global poverty bracket.
Perhaps on balance I’d vote for poverty, animals, AI takeover, and post-AGI governance or something like that.
I also very much agree that “making the transition to a post-AGI society go well” beyond AI takeover is highly neglected given its importance.
I’m not convinced EA is intellectually adrift and tend to agree with Nick Laing’s comment. My quick take is that it feels boring to people who’ve been in it a while but still is pretty incisive for people who are new to it, which describes most of the world.
I think principles-first EA goes better with a breadth of focuses and cause areas, because it shows the flexibility of the principles and the room for disagreement within them. I tend to think that too much focus on AI can take away with this, so it would concern me if >50-60% of the discussion were around AI.
I very much agree with the PR mentality comments—in particular, I find many uses of the “EA adjacent” term to be farcical. I added effective altruism back into my Twitter bio inspired by this post and @Alix Pham’s.
I agree it would be good for the EA Forum to be a place where more of the AI discussion happens, and I think it’s particularly suited for post-AGI society—it’s been a good place for digital minds conversations, for example.
So I guess I come down on the side of thinking (a) members of the EA community should recognize that there’s a lot more to discuss around AI than takeover, and it merits a rich and varied conversation, but (b) I would be wary of centering the transition to a post-AGI society go well at the expense of other cause areas.
I’d add to this that you do also have the possibility that 1-3 happen, but they happen much later than many people currently think. My personal take is that the probability that ‘either AGI’s impact comes in more than ten years or it’s not that radical’ is >50%, certainly far more than 0%.
I’ve been interested to see this book since I came across the idea. I think the argument for this being a problem from a variety of perspectives is pretty compelling.
For me, probably the key chapter is “Dodging the asteroid. And other benefits of other people.” I’m also interested in how population issues could interact with AI-driven changes.
The Peter Singer quote is interesting—I’m a bit surprised given his past views on population ethics. I’m wondering if he’s updated his views.
Agree that it’s not certain or obvious that AI risk is the most pressing issue (though it is 80k’s best guess & my personal best guess
Yeah, FWIW, it’s mine too. Time will tell how I feel about the change in the end. That EA Forum post on the 80K-EA community relationship feels very appropriate to me, so I think my disagreement is about the application.
I think that existential risks from various issues with AGI (especially if one includes trajectory changes) are high enough that one needn’t accept fanatical views to prioritise them
I think the argument you linked to is reasonable. I disagree, but not strongly. But I think it’s plausible enough that AGI concerns (from an impartial cause prioritization perspective) require fanaticism that there should still be significant worry about it. My take would be that this worry means an initially general EA org should not overwhelmingly prioritize AGI.
By my read, that post and the excerpt from it are about the rhetorical motivation for existential risk rather than the impartial ethical motivation. I basically agree that longtermism is not the right framing in most conversations, and it’s also not necessary for thinking existential risk work would be more valuable than the marginal public dollar.
I included the qualifier “From an altruistic cause prioritization perspective” because I think that from an impartial cause prioritization perspective, the case is different. If you’re comparing existential risk to animal welfare and global health, the links in my comment I think make the case pretty persuasively that you need longtermism.
I’m not sure exactly what this change will look like, but my current impression from this post leaves me disappointed. I say this as someone who now works on AI full-time and is mostly persuaded of strong longtermism. I think there’s enough reason for uncertainty about the top cause and value in a broad community that central EA organizations should not go all-in on a single cause. This seems especially the case for 80,000 Hours, which brings people in by appealing to a general interest in doing good.
Some reasons for thinking cause diversification by the community/central orgs is good:
From an altruistic cause prioritization perspective, existential risk seems to require longtermism, including potentially fanatical views (see Christian Tarsney, Rethink Priorities). It seems like we should give some weight to causes that are non-fanatical.
Existential risk is not most self-identified EAs’ top cause, and about 30% of self-identified EAs say they would not have gotten involved if it did not focus on their top cause (EA survey). So it does seem like you miss an audience here.
Organizations like 80,000 Hours set the tone for the community, and I think there’s good rule-of-thumb reasons to think focusing on one issue is a mistake. As 80K’s problem profile on factory farming says, factory farming may be the greatest moral mistake humanity is currently making, and it’s good to put some weight on rules of thumb in addition to expectations.
Timelines have shortened, but it doesn’t seem obvious whether the case for AGI being an existential risk has gotten stronger or weaker. There are signs of both progress and setbacks, and evidence of shorter timelines but potentially slower takeoff.
I’m also a bit confused because 80K seemed to recently re-elevate some non-existential risk causes on its problem profiles (great power war and factory farming; many more under emerging challenges). This seemed like the right call and part of a broader shift away from going all-in on longtermism in the FTX era. I think that was a good move and that keeping an EA community that is not only AGI is valuable.
I think I agree with the Moral Power Laws hypothesis, but it might be irrelevant to the question of whether to try to improve the value of the future or work on extinction risk.
My thought is this: the best future is probably a convergence of many things going well, such as people being happy on average, there being many people, the future lasting a long time, and maybe some empirical/moral uncertainty stuff. Each of these things plausibly has a variety of components, creating a long tail. Yet you’d need expansive, simultaneous efforts on many fronts to get there. In practice, even a moderately sized group of people is only going to make a moderate to small push on a single front, or very small pushes on many fronts. This means the value we could plausibly affect, obviously quite loosely speaking, does not follow a power law.
The value of the future conditional on civilization surviving seems positive to me, but not robustly so. I think the main argument for its being positive is theoretical (e.g., Spreading happiness to the stars seems little harder than just spreading), but the historical/contemporary record is ambiguous.
The value of improving the future seems more robustly positive if it is tractable. I suspect it is not that much less tractable than extinction risk work. I think a lot of AI risk satisfies this goal as well as the x-risk goal for reasons Will MacAskill gives in What We Owe the Future. Understanding, developing direct interventions for, and designing political processes for digital minds seem like plausible candidates. Some work on how to design democratic institutions in the age of AI also seems plausibly tractable enough to compete with extinction risk.
This is a stimulating and impressively broad post.
I want to press a bit on whether these trends are necessarily bad—I think they are, but there are a few reasons why I wonder about it.
1) Secrecy: While secrecy makes it difficult or impossible to know if a system is a moral patient, it also prevents rogue actors from quickly making copies of a sentient system or obtaining a blueprint for suffering. (It also prevents rogue actors from obtaining a blueprint for flourishing, which supports your point.) How do you think about this?
2 and 3) If I understand correctly, the worry here is that AI multiplies at a speed that outpaces our understanding, making it less likely that humanity handles digital minds wisely. Some people are bullish on digital minds (i.e., think they would be good in and of themselves). Some also think other architectures would be more likely to be sentient than transformers. Wider exploration and AI-driven innovation plausibly have the effect of just increasing the population of digital minds. How do you weigh this against the other considerations?
I think I’d be more worried about pulling out entirely than a delayed release, but either one seems possible (but IMO unlikely).
What seems less likely to work?
Work with the EU and the UK
Trump is far less likely to take regulatory inspiration from European countries and generally less likely to regulate. On the other-hand perhaps under a 2028 Dem administration we would see significant attention on EU/UK regulations.
The EU/UK are already scaling back the ambitions of their AI regulations out of fear that Trump would retaliate if they put limits on US companies.
Interesting—I’ve had the opposite take for the EU. The low likelihood of regulation in the US seems like it would make EU regulation more important since that might be all there is. (The second point still stands, but it’s still unclear how much that retaliation will happen and what impact it will have.)
It depends on aspects of the Brussels’ effect, and I guess it could be that a complete absence of US regulation means companies just pull out of the EU in response to regulation there. Maybe recent technical developments make that more likely. On net, I’m still inclined to think these updates increase the importance of EU stuff.For the UK, I think I’d agree—UK work seems to get a lot of its leverage from the relationship with the US.
I think I might add this to my DIY, atheist, animal-rights Haggadah.
TLDR: Graduating Stanford economics Ph.D. primarily interested in research or grantmaking work to improve the long-term future or animal welfare.
Skills & background: My job-market details, primarily aimed at economics academia, are on my website. I am an applied microeconomist (meaning empirical work and applied theory), with my research largely falling in political economy (econ + poli sci), public economics (econ of policy impacts), and behavioral/experimental economics.I have been involved in effective altruism for 10+ years, including having been a Senior Programme Associate at Longview Philanthropy in 2022, Vice Board Chair at Animal Charity Evaluators (board member 2020-present), Global Priorities Fellow with the Global Priorities Institute and the Forethought Foundation (2019-2020), and a Senior Research Analyst at Innovations for Poverty Action and the Northwestern Global Poverty Research Lab (RA 2014-2018).
Some examples of my work:
Persistence in Policy: Evidence from Close Votes
This is my economics “job market paper.”
AGI Catastrophe and Takeover: Some Reference Class-Based Priors (Forum post)
This won a Second Prize in the Open Philanthropy AI Worldviews Contest
Does suffering dominate enjoyment in the animal kingdom? An update to welfare biology
I also organized an Economics of Animal Welfare session at the Stanford Institute for Theoretical Economics.
Location/remote: Flexible. I am currently in the Bay Area but would be happy with other U.S. metropolitan areas and open to the UK as well. I’m happy to be remote but prefer to have somewhere to work around other people.
Availability & type of work: I am looking for a full-time job after I graduate in June 2023. I can potentially start part-time as early as March. I plan to take some time off sometime in the next year.I will probably decide as to my next step within a month or two.
Resume/CV/LinkedIn: LinkedIn; CV
Email/contact: zdgroff@gmail.com
Other notes: The problem area I am most interested in working on is what can be done to improve the long-term future, if anything, beyond extinction risk. My ideal job would probably involve a mix of research and making things happen.
[Edited to add the second sentence of the paragraph beginning, “Putting these together.”]
The primary result doesn’t speak to this, but secondary results can shed some light on it. Overall, I’d guess persistence is a touch less for policies with much more support, but note that the effect of proposing a policy on later policy is likely much larger than the effect of passing a policy conditional on its having been proposed.
The first thing to note is that there are really two questions here we might want to ask:
What is the effect of passing a policy change, conditional on its having been proposed, when its support is not marginal?
What is the effect of proposing a policy change when its support is not marginal?
I’ll speak to (1) first.
The main thing we can do is look at states that require a supermajority to pass a referendum in Appendix Figure E15. This does not directly answer (1) because, while it allows us to look at referendums whose support is well above 50%, it is looking at cases where you need more than 50% to revisit the referendum. Nevertheless, it gives us some information. First, things look similar for the most part. Second, it looks like maybe there’s a higher chance that supermajority referendums pass later on, especially in the first decade, though it’s very noisy statistically. Third, repeal is slightly less common, though this is again noisy and also confounded with the higher difficulty of repealing one of these.
In the latest version of the paper, I include a simulation (Section 5.1) that allows me to simulate some relevant experiments, though these are currently not in the paper. In my simulation, I can simulate the effect of passing an initiative (a referendum by petition) with varying levels of support. It is generally the case that, for policies that get proposed and have support above 55%, persistence is about 25% smaller at 100 years for the reason you give: these policies are more likely to pass eventually. (I do this by simulating a world where, holding voter support constant, I randomly assign policies to be passed or not.)
Putting these together, I think it would be reasonable to think either that the effect of passage is similar for policies with widespread support, or that it is somewhat smaller. You can also look at the discussion of state legislation in section 6, which does not rely on close votes (though plausibly selects for things being marginal by focusing on adoptions of policies by states where similar states lack those policies).
Turning to (2), we should expect the effect of proposing a policy on whether that policy is in effect later on to be much larger than the effect of passing a policy conditional on its being proposed. Appendix Figure E20 (formerly D20) is one attempt to get at this and suggests the effect of successfully proposing a policy is ~50% larger than the effect of passing a proposed one. One could also imagine simulating this—but that exercise requires some unclear assumptions, so I’m inclined to go with Appendix Figure E20 here.*
One underlying theme in all of this is that the people who propose policies are very much in the driver’s seat. Persistence largely appears to be a result of the fact that small numbers of people can set policies based on whether they decide to pursue policy changes or not.
*One could also imagine simulating this, but the problem there is that the vast majority of policies one could conceive of probably have approximately nobody who cares about them (e.g., minor tweaks to the language of some law, declaring that pistachio is the best ice cream flavor and offering an infinitesimal subsidy for it). My calibration has the policies that get proposed as being in the tail of the distribution in terms of how much people care about them. As a result, if we look at policies that don’t get proposed, basically nobody would ever bother trying to repeal or revisit them.
Yep, at the risk of omitting others, Lukas Freund as well.
Yes, it’s a good point that benefits and length of the period are not independent, and I agree with the footnote too.
I would note that the factors I mentioned there don’t seem like they should change things that much for most issues. I could see using 50-100 years rather than, e.g., 150 years as my results would seem to suggest, but I do think 5-10 years is an order of magnitude off.
Easy Q to answer so doesn’t take much time! In economics, the norm is not to publish your job market paper until after the market for various reasons. (That way, you don’t take time away from improving the paper, and the department that hires you gets credit.)
We will see before long how it publishes!
I look at some things you might find relevant here. I try to measure the scale of the impact of a referendum. I do this two ways. I have just a subjective judgment on a five-point scale, and then I also look at predictions of the referendum’s fiscal impact from the secretary of state. Neither one is predictive. I also look at how many people would be directly affected by a referendum and how much news coverage there was before the election cycle. These predict less persistence.
This is something I plan to do more, but they can’t vary that much because when I look at variables that vary across states (e.g., requirements to get on the ballot), I don’t see much of a difference.
I’m not totally sure what your question is, but I think you might be interpreting my results as saying that close referendums are especially persistent. I’m only focusing on close referendums because it’s a natural experiment—I’m not saying there’s something special about them otherwise. I’m just estimating the effect of passing a marginal referendum on whether the policy is in place later on. I can try to think about whether this holds for things that are not close by looking at states with supermajority requirements or by looking at legislation, and it looks like things are similar when they’re not as close.
I’m a grantmaker at Longview and manage the Digital Sentience Fund—thought I’d share my thinking here: “backchaining from… making the long-term future go well conditional on no Al takeover” is my goal with the fund (with the restriction of being related to the wellbeing of AIs in a somewhat direct way), though we might disagree on how that’s best achieved through funding. Specifically, the things you’re excited about would probably be toward the top of the list of things I’m excited about, but I also think broader empirical and philosophical work and field-building are some of the best ways to get there.
Relative to Lukas’s post, I’d say my goals are, in order, 5 and 2, then 4, then 3 and 1. An additional goal is improving the design of models that might be sticky over the long term.
All of the things on those lists require technical and policy researchers, engineers, lawyers, etc. that basically don’t currently exist in large numbers, so I do think fairly broad field building is important. There are pretty tight limits to how targeted field building can be: you can target, e.g., ML versus law, and you can suggest topics, but you’re basically just creating new specialists in the fields you pick who then pursue the topics you want.
Our recent funding opportunities targeted ML and neuroscience, so more around understanding minds than things like the role in society and trade. I’d guess that we repeat this but also run one or add in suggested topics that focus more on law, trade, etc.
Realistically addressing many of the things on those lists likely also requires a mature field of understanding AI minds, so I think empirical and philosophical work on sentience feeds into it.
To get concrete, recent distribution of funds and donations we’ve advised on (which is a decent approximation of where things go) looks like ~50% field building, of which maybe 10% is on things like the role of AI minds in society (including, e.g., trade) vs understanding AI minds; 40% research, of which maybe 25% is on the role of AI minds in society; a bit more than 10% lab-facing work, and 10% other miscellaneous things like communications and preparatory policy work. Generally the things I’m most excited to grow are lab-facing work and work on the role of AI minds in society.
(I also care about “averting Al takeover” and factor that in, though it’s not the main goal and gets less weight.)