(Posting in a personal capacity unless stated otherwise.) I help allocate Open Phil’s resources to improve the governance of AI with a focus on avoiding catastrophic outcomes. Formerly co-founder of the Cambridge Boston Alignment Initiative, which supports AI alignment/safety research and outreach programs at Harvard, MIT, and beyond, co-president of Harvard EA, Director of Governance Programs at the Harvard AI Safety Team and MIT AI Alignment, and occasional AI governance researcher. I’m also a proud GWWC pledger and vegan.
tlevin
Karma: 2,287
One Year in DC
I basically agree with this (and might put the threshold higher than $100, probably much higher for people actively pursuing policy careers), with the following common exceptions:
It seems pretty low-cost to donate to a candidate from Party X if...
You’ve already made donations to Party X. Larger and more recent ones are slightly worse, but as Daniel notes, even small ones from several elections ago can come back to bite.
You don’t see a realistic world where you go into the federal government during a Party Y administration even if you didn’t donate to Party X, because...
You don’t think you could go into the federal government at all (though as Daniel notes, you may not realize at the time of making the donation that you might want to later; what I have in mind is like, you have significantly below average people skills, and/or you’ve somehow disqualified yourself).
You have a permanently discoverable digital paper trail of criticizing Party Y, e.g. social media posts, op-eds, etc.
You just don’t think you’d be able to stomach working in a Party Y administration. (Though consider asking, would you really not be able to stomach it for a few years if it seemed like an amazing career and impact opportunity?)
I don’t know the weeds of the moral parliament view, but my suspicion is that this argument relies on too low of a level of ethical views (that is, “not meta enough”). That’s still just a utilitarian frame with empirical uncertainty. The kind of “credences on different moral views” I have in mind is more like:
I want my moral actions to be guided by some mix of like, 25% bullet-biting utilitarianism (in which case, insects are super important in expectation), 25% virtue ethics (in which case they’re a small consideration—you don’t want to go out of your way to hurt them, but you’re not obligated to do much in particular, and you should be way more focused on people or other animals who you have relationships with and obligations towards), 15% some kind of “stewardship of humanity” (where you maybe just want to avoid actively being a monster but should be focused elsewhere), 10% libertarianism (where it’s quite unclear how you’d treat insects), and 25% spread across other views, which mostly just points towards not being super-fanatical about any of the others. So something like 30% of me thinks insect suffering is a big deal, which is enough for me to take it seriously but not enough for me to drop the stuff that more like 75% of me thinks is a big deal; in other words I think it’s moderately important.
I don’t know what my actual numbers are, and I’m not sure each of these views is really what the respective philosophy would say about insect welfare; I’m just saying, it’s easy in this kind of framework to wind up having lots of moderate priorities that each seem extremely important on certain ethical views.
I think it’s reasonable to say “I put some credence on moral views that imply insect suffering is very important and some credence on moral views that imply it’s not important; all things considered, I think it’s moderately important.”
A couple other comments are gesturing at this, but this logic could be applied to all kinds of things: existential risk is probably “either” extremely important or not at all important if you plug different empirical and ethical views into a formula and trust the answer; likewise present-day global health, or political polarization, or developed-world mental health, etc. Eventually, you can either (1) go all in on a particular ethical and meta-ethical theory, (2) be inconsistent, or (3) combine all these considerations into a balanced whole, in which probably a lot of things that pencil as “extremely important” in some views wind up being a moderately high priority. I don’t think it’s obvious that (3) is right, but this post does not make an argument that (1) is right, and I think the burden of proof is on the side arguing explicitly against moderation and intuitive conclusions.
One reason to think (3) is right is to look at the track records. You say you “cannot be a moderate Christian.” I don’t think religious fundamentalists have morally outperformed religious moderates. There are lots of people who take religious values seriously but not fanatically; some of the leaders of the world’s greatest social movements used a lot of religious thinking and rhetoric without trying to follow every letter of the Bible.
I definitely agree there are plenty of ways we should reach elites and non-elites alike that aren’t statistical models of timelines, and insofar as the resources going towards timeline models (in terms of talent, funding, bandwidth) are fungible with the resources going towards other things, maybe I agree that more effort should be going towards the other things (but I’m not sure—I really think the timeline models have been useful for our community’s strategy and for informing other audiences).
But also, they only sometimes create a sense of panic; I could see specificity being helpful for people getting out of the mode of “it’s vaguely inevitable, nothing to be done, just gotta hope it all works out.” (Notably the timeline models sometimes imply longer timelines than the vibes coming out of the AI companies and Bay Area house parties.)
There’s a grain that I agree with here, which is that people excessively plan around a median year for AGI rather than a distribution for various events, and that planning around that kind of distribution leads to more robust and high-expected-value actions (and perhaps less angst).
However, I strongly disagree with the idea that we already know “what we need.” Off the top of my head, several ways narrowing the error bars on timelines—which I’ll operationalize as “the distribution of the most important decisions with respect to building transformative AI”—would be incredibly useful:To what extent will these decisions be made by the current US administration, or by people governed by the current administration? This affects the political strategy everyone—including, I propose, PauseAI—should adopt.
To what extent will the people making the most important AI decisions remember stuff people said in 2025? This is very important for the relative usefulness of public communications versus research, capacity-building, etc.
Are these decisions soon enough that the costs of being “out of the action” outweigh the longer-term benefits of e.g. going to grad school, developing technical expertise, etc? Clearly relevant for lots of individuals who want to make a big impact.
When should philanthropists spend their resources? As I and others have written, there are several considerations that point towards spending later; these are weakened a lot if the key decisions are in the next few years.
To what extent will the most transformative models be technically similar to the ones we have today? That answer determines the value of technical safety research.
I also strongly disagree with the framing that the important thing is us knowing what we know. Yes, people who have been immersed in AI content for years often believe that very scary and/or awesome AI capabilities are coming within the decade. But most people, including most of the people who might take the most important actions, are not in this category and do not share this view (or at least don’t seem to have internalized it). Work that provides an empirical grounding for AI forecasts has already been very useful in bringing attention to AGI and its risks from a broader set of people, including in governments, who would otherwise be focused on any one of the million other problems in the world.
Giving now vs giving later, in practice, is a thorny tradeoff. I think these add up to roughly equal considerations, so my currently preferred policy is to split my donations 50-50, i.e. give 5% of my income away this year and save/invest 5% for a bigger donation later. (None of this is financial/tax advice! Please do your own thinking too.)
In favor of giving now (including giving a constant share of your income every year/quarter/etc, or giving a bunch of your savings away soon):
Simplicity.
The effects of your donation might have compounding returns, e.g. field-building gets more people doing great stuff, this can in turn build the field, etc., or be path-dependent, e.g. someone does some writing that establishes better concepts for the field.
Value drift: maybe you don’t trust your future self to give as much, or to be as good at picking good stuff. (Some commitment mechanisms exist for this, like DAFs, but that really only fixes the “give as much” problem, and there are lots of opportunities that DAFs can’t fund, such as 501c4 advocacy organizations, individuals, political campaigns, etc.)
Expropriation risk: you might lose the money, including via global catastrophe.
In favor of giving later:
Value of information: especially in a fast-changing field like AI, we’ll continue learning more about what kinds of interventions work as time goes on.
Philanthropic learning: basically the opposite of value drift: you specifically might become a wiser donor, especially if you’re currently young and/or new to the field.
Returns to scale: it’s probably better to make e.g. a single $150k donation than ten donations averaging $15k, because orgs can act pretty decisively with an amount like that, like hire somebody or run a program. (Eventually you hit diminishing returns, but not for most individual donors.)
Tax bunching (only applies to donations that you can write off): in my understanding, at least in the US, there’s a threshold below which you effectively can’t write off donations (the standard deduction), so there’s effectively a fixed cost in any year that you make donations. This makes donating a fixed amount every year a pretty suboptimal strategy, other things equal; if you’re donating an amount below or not that far above the standard deduction to c3 orgs every year, you might be able to save or donate significantly more if you instead donate once every few years.
Are you a US resident who spends a lot of money on rideshares + food delivery/pickup? If so, consider the following:
Costco members can buy up to four Uber gift cards of $50 value every two weeks (that is, 2 packs of 2 $50 gift cards). Now, and I think typically, these sell at 20% off face value.
Costco membership costs $65/year.
It takes ~2 minutes per gift card all-in.
You can use them on rides, scooters, and Uber Eats.
According to o3-mini-high, this means it’s worth it if you spend $1625 / (5 - how much you value your marginal minute) per year on these services, if you get no other use out of the Costco membership. (If you do, this number goes down, of course.)
Hooray, you now have more money for donations, consumption, savings, or investment for a small time cost!
I was not paid by Costco or Uber to say this, I swear.
I think the opposite might be true: when you apply it to broad areas, you’re likely to mistake low neglectedness for a signal of low tractability, and you should just look at “are there good opportunities at current margins.” When you start looking at individual solutions, it starts being quite relevant whether they have already been tried. (This point already made here.)
Would it be good to solve problem P?
Can I solve P?
What is gained by adding the third thing? If the answer to #2 is “yes,” then why does it matter if the answer to #3 is “a lot,” and likewise in the opposite case, where the answers are “no” and “very few”?Edit: actually yeah the “will someone else” point seems quite relevant.
Fair enough on the “scientific research is super broad” point, but I think this also applies to other fields that I hear described as “not neglected” including US politics.
Not talking about AI safety polling, agree that was highly neglected. My understanding, reinforced by some people who have looked into the actually-practiced political strategies of modern campaigns, is that it’s just a stunningly under-optimized field with a lot of low-hanging fruit, possibly because it’s hard to decouple political strategy from other political beliefs (and selection effects where especially soldier-mindset people go into politics).
I sometimes say, in a provocative/hyperbolic sense, that the concept of “neglectedness” has been a disaster for EA. I do think the concept is significantly over-used (ironically, it’s not neglected!), and people should just look directly at the importance and tractability of a cause at current margins.
Maybe neglectedness useful as a heuristic for scanning thousands of potential cause areas. But ultimately, it’s just a heuristic for tractability: how many resources are going towards something is evidence about whether additional resources are likely to be impactful at the margin, because more resources mean its more likely that the most cost-effective solutions have already been tried or implemented. But these resources are often deployed ineffectively, such that it’s often easier to just directly assess the impact of resources at the margin than to do what the formal ITN framework suggests, which is to break this hard question into two hard ones: you have to assess something like the abstract overall solvability of a cause (namely, “percent of the problem solved for each percent increase in resources,” as if this is likely to be a constant!) and the neglectedness of the cause.
That brings me to another problem: assessing neglectedness might sound easier than abstract tractability, but how do you weigh up the resources in question, especially if many of them are going to inefficient solutions? I think EAs have indeed found lots of surprisingly neglected (and important, and tractable) sub-areas within extremely crowded overall fields when they’ve gone looking. Open Phil has an entire program area for scientific research, on which the world spends >$2 trillion, and that program has supported Nobel Prize-winning work on computational design of proteins. US politics is a frequently cited example of a non-neglected cause area, and yet EAs have been able to start or fund work in polling and message-testing that has outcompeted incumbent orgs by looking for the highest-value work that wasn’t already being done within that cause. And so on.
What I mean by “disaster for EA” (despite the wins/exceptions in the previous paragraph) is that I often encounter “but that’s not neglected” as a reason not to do something, whether at a personal or organizational or movement-strategy level, and it seems again like a decent initial heuristic but easily overridden by taking a closer look. Sure, maybe other people are doing that thing, and fewer or zero people are doing your alternative. But can’t you just look at the existing projects and ask whether you might be able to improve on their work, or whether there still seems to be low-hanging fruit that they’re not taking, or whether you could be a force multiplier rather than just an input with diminishing returns? (Plus, the fact that a bunch of other people/orgs/etc are working on that thing is also some evidence, albeit noisy evidence, that the thing is tractable/important.) It seems like the neglectedness heuristic often leads to more confusion than clarity on decisions like these, and people should basically just use importance * tractability (call it “the IT framework”) instead.
It’s also just jargon-y. I call them “AI companies” because people outside the AGI memeplex don’t know what an “AI lab” is, and (as you note) if they infer from someone’s use of that term that the frontier developers are something besides “AI companies,” they’d be wrong!
Biggest disagreement between the average worldview of people I met with at EAG and my own is something like “cluster thinking vs sequence thinking,” where people at EAG are like “but even if we get this specific policy/technical win, doesn’t it not matter unless you also have this other, harder thing?” and I’m more like, “Well, very possibly we won’t get that other, harder thing, but still seems really useful to get that specific policy/technical win, here’s a story where we totally fail on that first thing and the second thing turns out to matter a ton!”
Skepticism towards claims about the views of powerful institutions
Thanks, glad to hear it’s helpful!
Re: more examples, I co-sign all of my teammates’ AI examples here—they’re basically what I would’ve said. I’d probably add Tarbell as well.
Re: my personal donations, I’m saving for a bigger donation later; I encounter enough examples of very good stuff that Open Phil and other funders can’t fund, or can’t fund quickly enough, that I think there are good odds that I’ll be able to make a really impactful five-figure donation over the next few years. If I were giving this year, I probably would’ve gone the route of political campaigns/PACs.
Re: sub-areas, there are some forms of policy advocacy and moral patienthood research for which small-to-medium-size donors could be very helpful. I don’t have specific opportunities in mind that I feel like I can make a convincing public pitch for, but people can reach out if they’re interested.
A case for donating to AI risk reduction (including if you work in AI)
I hope to eventually/maybe soon write a longer post about this, but I feel pretty strongly that people underrate specialization at the personal level, even as there are lots of benefits to pluralization at the movement level and large-funder level. There are just really high returns to being at the frontier of a field. You can be epistemically modest about what cause or particular opportunity is the best, not burn bridges, etc, while still “making your bet” and specializing; in the limit, it seems really unlikely that e.g. having two 20 hr/wk jobs in different causes is a better path to impact than a single 40 hr/wk job.
I think this applies to individual donations as well; if you work in a field, you are a much better judge of giving opportunities in that field than if you don’t, and you’re more likely to come across such opportunities in the first place. I think this is a chronically underrated argument when it comes to allocating personal donations.
Thanks for running this survey. I find these results extremely implausibly bearish on public policy—I do not think we should be even close to indifferent between improving the AI policy of the country that can make binding rules on all of the leading labs plus many key hardware inputs and has a $6 trillion budget and the most powerful military on earth by 5% and having $8.1 million more dollars for a good grantmaker, or having 32.5 “good video explainers,” or having 13 technical AI academics. I’m biased, of course, but IMO the surveyed population is massively overrating the importance of the alignment community relative to the US government.
Having a savings target seems important. (Not financial advice.)
I sometimes hear people in/around EA rule out taking jobs due to low salaries (sometimes implicitly, sometimes a little embarrassedly). Of course, it’s perfectly understandable not to want to take a significant drop in your consumption. But in theory, people with high salaries could be saving up so they can take high-impact, low-paying jobs in the future; it just seems like, by default, this doesn’t happen. I think it’s worth thinking about how to set yourself up to be able to do it if you do find yourself in such a situation; you might find it harder than you expect.
(Personal digression: I also notice my own brain paying a lot more attention to my personal finances than I think is justified. Maybe some of this traces back to some kind of trauma response to being unemployed for a very stressful ~6 months after graduating: I just always could be a little more financially secure. A couple weeks ago, while meditating, it occurred to me that my brain is probably reacting to not knowing how I’m doing relative to my goal, because 1) I didn’t actually know what my goal is, and 2) I didn’t really have a sense of what I was spending each month. In IFS terms, I think the “social and physical security” part of my brain wasn’t trusting that the rest of my brain was competently handling the situation.)
So, I think people in general would benefit from having an explicit target: once I have X in savings, I can feel financially secure. This probably means explicitly tracking your expenses, both now and in a “making some reasonable, not-that-painful cuts” budget, and gaming out the most likely scenarios where you’d need to use a large amount of your savings, beyond the classic 3 or 6 months of expenses in an emergency fund. For people motivated by EA principles, the most likely scenarios might be for impact reasons: maybe you take a public-sector job that pays half your current salary for three years, or maybe you’d need to self-fund a new project for a year; how much would it cost to maintain your current level of spending, or a not-that-painful budget-cut version? Then you could target that amount (in addition to the emergency fund, so you’d still have that at the end of the period); once you have that, you could feel more secure/spend less brain space on money, donate more of your income, and be ready to jump on a high-impact, low-paying opportunity.
Of course, you can more easily hit that target if you can bring down your expenses—you both lower the required amount in savings and you save more each month. So, maybe some readers would also benefit from cutting back a bit, though I think most EAs are pretty thrifty already.
(This is hardly novel—Ben Todd was publishing related stuff on 80k in 2015. But I guess I had to rediscover it, so posting here in case anyone else could use the refresher.)