(Posting in a personal capacity unless stated otherwise.) I work at Coefficient Giving with a focus on helping humanity better navigate transformative AI, especially through US public policy. Formerly co-founder of the Cambridge Boston Alignment Initiative, which supports AI alignment/safety research and outreach programs at Harvard, MIT, and beyond, co-president of Harvard EA, Director of Governance Programs at the Harvard AI Safety Team and MIT AI Alignment, and occasional AI governance researcher. I’m also a proud GWWC pledger and vegan.
tlevin
Karma: 2,510
What SB 53, California’s new AI law, does
Weak-downvoted; I think it’s fair game to say an org acted in an untrustworthy way, but I think it’s pretty essential to actually sketch the argument rather than screenshotting their claims and not specifying what they’ve done that contradicts the claims. It seems bad to leave the reader in a position of being like, “I don’t know what the author means, but I guess Epoch must have done something flagrantly contradictory to these goals and I shouldn’t trust them,” rather than elucidating the evidence so the reader can actually “form their own judgment.” Ben_West then asked in two comments for these specifics, and I still don’t know what you mean (and I think I’m pretty high-percentile among forum readers on the dimension of “familiar with drama/alleged bad behavior of AI safety orgs”).
Would remove the downvote if you fill in the implicit part of the argument here: what information/explanation would a reader need to know what you mean by “it certainly seems to me that the AI Safety community was too ready to trust Epoch” in the context of these screenshots?
(Speaking for myself as someone who has also recommended donating to Horizon, not Julian or OP)
I basically think the public outputs of the fellows is not a good proxy for the effectiveness of the program (or basically any talent program). The main impact of talent programs, including Horizon, seems better measured by where participants wind up shortly after the program (on which Horizon seems objectively strong), plus a subjective assessment of how good the participants are. There just isn’t a lot of shareable data/info on the latter, so I can’t do much better than just saying “I’ve spent some time on this (rather than taking for granted that they’re good) and I think they’re good on average.” (I acknowledge that this is not an especially epistemically satisfying answer.)
I appreciate these analyses, but given the very high sensitivity of the bottom lines to parameters like how welfare ranges correspond to neuron counts or other facts about the animals in question, I find it implausible that the best donation option is to fund the intervention with the highest mean estimate rather than either 1) fund more research into those parameters or 2) save/invest until such research has happened. Maybe future posts could examine the tradeoff between funding/waiting for such research versus funding the direct interventions now?
I think this is comparing apples and oranges: biological capabilities on benchmarks (AFAIK not that helpful in real-world lab settings yet) versus actual economic impact. The question is whether real world bio capabilities will outstrip real world broad economic capabilities.
It’s certainly possible that an AI will trigger a biorisk if-then commitment before it has general capabilities capable of 10% cumulative GDP growth. But I would be pretty surprised if we get a system so helpful that it could counterfactually enable laypeople to dramatically surpass the current state of the art in the specific domain of bio-offense without having previously gotten systems that are pretty helpful at counterfactually enabling professionals to do their jobs somewhat better and automate some routine tasks. I think your claim implies something like, as AI automates things, it will hit “making a bioweapon that ends the world, which no one can currently do” before it hits “the easiest ~15% of the stuff we already do, weighted by market value” (assuming labor is ~2/3 of GDP). This seems unlikely, especially since bioweapons involves a bunch of physical processes where AIs seem likely to struggle mightily for a while, though again I concede not impossible.
In terms of whether “most AI safety people” believe this, consider that the great takeoff speeds debate was operationalized in terms of whether AI would produce a cumulative 100% growth in four years before it produced 100% growth in one year. To the extent that this debate loosely tracked a debate within the community more broadly, it seems to imply a large constituency for the view that we will see much more than 10% cumulative growth before AI becomes existentially scary.
Why does the high generality of AI capabilities imply that a similar level of capabilities produces 10% cumulative GDP growth and extinction?
I think this picture of EA ignoring stable totalitarianism is missing the longtime focus on China.
Also, see this thread on Open Phil’s ability to support right-of-center policy work.
It feels like there’s an obvious trade between the EA worldview on AI and Thiel’s, where the strategy is “laissez faire for the kinds of AI that cause late-90s-internet-scale effects (~10% cumulative GDP growth), aggressive regulation for the kinds of AI that inspire the ‘apocalyptic fears’ that he agrees should be taken seriously, and require evaluations of whether a given frontier AI poses those risks at the pre-deployment stage so you know which of these you’re dealing with.”
Indeed, this is pretty much the “if-then” policy structure Holden proposes here, seemingly with the combination of skepticism of capabilities and distrust of regulation very much in mind.
Obviously the devil (as it were) is in the details. But it feels like there are a bunch of design features that would move in this direction: very little regulation of AI systems that don’t trigger very high capability thresholds (i.e. nothing currently available), aiming for low-cost and accurate risk evaluations for specific threat models like very powerful scheming, self-improvement, and bioterrorism uplift. Idk, maybe I’m failing the ideological turing test here and Thiel would say this is already a nanny state proposal or would lapse into totalitarianism, but like, there’s a huge gulf between capabilities that can get you ~10% cumulative GDP growth and capabilities that can kill billions of people—really feels like there’s some governance structure that allows/promotes the former and regulates the latter.
I notice a pattern in my conversations where someone is making a career decision: the most helpful parts are often prompted by “what are your strengths and weaknesses?” and “what kinds of work have you historically enjoyed or not enjoyed?”
I can think of a couple cases (one where I was the recipient of career decision advice, another where I was the advice-giver) where we were kinda spinning our wheels, going over the same considerations, and then we brought up those topics >20 minutes into the conversation and immediately made more progress than the rest of the call to that point.
Maybe this is because in EA circles people have already put a ton of thought into considerations like “which of these jobs would be more impactful conditional on me doing a 8⁄10 job or better in them” and “which of these is generally better for career capital (including skill development, networks, and prestige),” so it’s the conversational direction with the most low-hanging fruit. Another frame is that this is another case of people underrating personal fit relative to the more abstract/generally applicable characteristics of a job.
Yeah interesting. To be clear, I’m not saying e.g. Manifund/Manival are net negative because of adverse selection. I do think additional grant evaluation capacity seems useful, and the AI tooling here seems at least more useful than feeding grants into ChatGPT. I suppose I agree that adverse selection is a smaller problem in general than those issues, though once you consider tractability, it seems deserving of some attention.
Cases where I’d be more worried about adverse selection, and would therefore more strongly encourage potential donors:
The amount you’re planning to give is big. Downside risks from funding one person to do a project are usually pretty low; empowering them to run an org is a different story. (Also, smaller grants are more likely to have totally flown under the radar of the big funders.)
The org/person has been around for a while.
The project is risky.
In those cases, especially for six-figure-and-up donations, people should feel free to supplement their own evaluation (via Manival or otherwise!) by checking in with professional grantmakers; Open Phil now has a donor advisory function that you can contact at donoradvisory@openphilanthropy.org.
(For some random feedback: I picked an applicant I was familiar with, was surprised by its low score, ran it through the “Austin config,” and it turns out it was losing a bunch of points for not having any information about the team’s background; only problem is, it had plenty of information about the team’s background! Not sure what’s goin on there. Also, weakly held, but I think when you run a config it should probably open a new tab rather than taking you away from the main page?)
Can you say more about how this / your future plans solve the adverse selection problems? (I imagine you’re already familiar with this post, but in case other readers aren’t, I recommend it!)
Having a savings target seems important. (Not financial advice.)
I sometimes hear people in/around EA rule out taking jobs due to low salaries (sometimes implicitly, sometimes a little embarrassedly). Of course, it’s perfectly understandable not to want to take a significant drop in your consumption. But in theory, people with high salaries could be saving up so they can take high-impact, low-paying jobs in the future; it just seems like, by default, this doesn’t happen. I think it’s worth thinking about how to set yourself up to be able to do it if you do find yourself in such a situation; you might find it harder than you expect.
(Personal digression: I also notice my own brain paying a lot more attention to my personal finances than I think is justified. Maybe some of this traces back to some kind of trauma response to being unemployed for a very stressful ~6 months after graduating: I just always could be a little more financially secure. A couple weeks ago, while meditating, it occurred to me that my brain is probably reacting to not knowing how I’m doing relative to my goal, because 1) I didn’t actually know what my goal is, and 2) I didn’t really have a sense of what I was spending each month. In IFS terms, I think the “social and physical security” part of my brain wasn’t trusting that the rest of my brain was competently handling the situation.)
So, I think people in general would benefit from having an explicit target: once I have X in savings, I can feel financially secure. This probably means explicitly tracking your expenses, both now and in a “making some reasonable, not-that-painful cuts” budget, and gaming out the most likely scenarios where you’d need to use a large amount of your savings, beyond the classic 3 or 6 months of expenses in an emergency fund. For people motivated by EA principles, the most likely scenarios might be for impact reasons: maybe you take a public-sector job that pays half your current salary for three years, or maybe you’d need to self-fund a new project for a year; how much would it cost to maintain your current level of spending, or a not-that-painful budget-cut version? Then you could target that amount (in addition to the emergency fund, so you’d still have that at the end of the period); once you have that, you could feel more secure/spend less brain space on money, donate more of your income, and be ready to jump on a high-impact, low-paying opportunity.
Of course, you can more easily hit that target if you can bring down your expenses—you both lower the required amount in savings and you save more each month. So, maybe some readers would also benefit from cutting back a bit, though I think most EAs are pretty thrifty already.
(This is hardly novel—Ben Todd was publishing related stuff on 80k in 2015. But I guess I had to rediscover it, so posting here in case anyone else could use the refresher.)
[linkpost] One Year in DC
I basically agree with this (and might put the threshold higher than $100, probably much higher for people actively pursuing policy careers), with the following common exceptions:
It seems pretty low-cost to donate to a candidate from Party X if...
You’ve already made donations to Party X. Larger and more recent ones are slightly worse, but as Daniel notes, even small ones from several elections ago can come back to bite.
You don’t see a realistic world where you go into the federal government during a Party Y administration even if you didn’t donate to Party X, because...
You don’t think you could go into the federal government at all (though as Daniel notes, you may not realize at the time of making the donation that you might want to later; what I have in mind is like, you have significantly below average people skills, and/or you’ve somehow disqualified yourself).
You have a permanently discoverable digital paper trail of criticizing Party Y, e.g. social media posts, op-eds, etc.
You just don’t think you’d be able to stomach working in a Party Y administration. (Though consider asking, would you really not be able to stomach it for a few years if it seemed like an amazing career and impact opportunity?)
I don’t know the weeds of the moral parliament view, but my suspicion is that this argument relies on too low of a level of ethical views (that is, “not meta enough”). That’s still just a utilitarian frame with empirical uncertainty. The kind of “credences on different moral views” I have in mind is more like:
I want my moral actions to be guided by some mix of like, 25% bullet-biting utilitarianism (in which case, insects are super important in expectation), 25% virtue ethics (in which case they’re a small consideration—you don’t want to go out of your way to hurt them, but you’re not obligated to do much in particular, and you should be way more focused on people or other animals who you have relationships with and obligations towards), 15% some kind of “stewardship of humanity” (where you maybe just want to avoid actively being a monster but should be focused elsewhere), 10% libertarianism (where it’s quite unclear how you’d treat insects), and 25% spread across other views, which mostly just points towards not being super-fanatical about any of the others. So something like 30% of me thinks insect suffering is a big deal, which is enough for me to take it seriously but not enough for me to drop the stuff that more like 75% of me thinks is a big deal; in other words I think it’s moderately important.
I don’t know what my actual numbers are, and I’m not sure each of these views is really what the respective philosophy would say about insect welfare; I’m just saying, it’s easy in this kind of framework to wind up having lots of moderate priorities that each seem extremely important on certain ethical views.
I think it’s reasonable to say “I put some credence on moral views that imply insect suffering is very important and some credence on moral views that imply it’s not important; all things considered, I think it’s moderately important.”
A couple other comments are gesturing at this, but this logic could be applied to all kinds of things: existential risk is probably “either” extremely important or not at all important if you plug different empirical and ethical views into a formula and trust the answer; likewise present-day global health, or political polarization, or developed-world mental health, etc. Eventually, you can either (1) go all in on a particular ethical and meta-ethical theory, (2) be inconsistent, or (3) combine all these considerations into a balanced whole, in which probably a lot of things that pencil as “extremely important” in some views wind up being a moderately high priority. I don’t think it’s obvious that (3) is right, but this post does not make an argument that (1) is right, and I think the burden of proof is on the side arguing explicitly against moderation and intuitive conclusions.
One reason to think (3) is right is to look at the track records. You say you “cannot be a moderate Christian.” I don’t think religious fundamentalists have morally outperformed religious moderates. There are lots of people who take religious values seriously but not fanatically; some of the leaders of the world’s greatest social movements used a lot of religious thinking and rhetoric without trying to follow every letter of the Bible.
I definitely agree there are plenty of ways we should reach elites and non-elites alike that aren’t statistical models of timelines, and insofar as the resources going towards timeline models (in terms of talent, funding, bandwidth) are fungible with the resources going towards other things, maybe I agree that more effort should be going towards the other things (but I’m not sure—I really think the timeline models have been useful for our community’s strategy and for informing other audiences).
But also, they only sometimes create a sense of panic; I could see specificity being helpful for people getting out of the mode of “it’s vaguely inevitable, nothing to be done, just gotta hope it all works out.” (Notably the timeline models sometimes imply longer timelines than the vibes coming out of the AI companies and Bay Area house parties.)
There’s a grain that I agree with here, which is that people excessively plan around a median year for AGI rather than a distribution for various events, and that planning around that kind of distribution leads to more robust and high-expected-value actions (and perhaps less angst).
However, I strongly disagree with the idea that we already know “what we need.” Off the top of my head, several ways narrowing the error bars on timelines—which I’ll operationalize as “the distribution of the most important decisions with respect to building transformative AI”—would be incredibly useful:To what extent will these decisions be made by the current US administration, or by people governed by the current administration? This affects the political strategy everyone—including, I propose, PauseAI—should adopt.
To what extent will the people making the most important AI decisions remember stuff people said in 2025? This is very important for the relative usefulness of public communications versus research, capacity-building, etc.
Are these decisions soon enough that the costs of being “out of the action” outweigh the longer-term benefits of e.g. going to grad school, developing technical expertise, etc? Clearly relevant for lots of individuals who want to make a big impact.
When should philanthropists spend their resources? As I and others have written, there are several considerations that point towards spending later; these are weakened a lot if the key decisions are in the next few years.
To what extent will the most transformative models be technically similar to the ones we have today? That answer determines the value of technical safety research.
I also strongly disagree with the framing that the important thing is us knowing what we know. Yes, people who have been immersed in AI content for years often believe that very scary and/or awesome AI capabilities are coming within the decade. But most people, including most of the people who might take the most important actions, are not in this category and do not share this view (or at least don’t seem to have internalized it). Work that provides an empirical grounding for AI forecasts has already been very useful in bringing attention to AGI and its risks from a broader set of people, including in governments, who would otherwise be focused on any one of the million other problems in the world.
Giving now vs giving later, in practice, is a thorny tradeoff. I think these add up to roughly equal considerations, so my currently preferred policy is to split my donations 50-50, i.e. give 5% of my income away this year and save/invest 5% for a bigger donation later. (None of this is financial/tax advice! Please do your own thinking too.)
In favor of giving now (including giving a constant share of your income every year/quarter/etc, or giving a bunch of your savings away soon):
Simplicity.
The effects of your donation might have compounding returns, e.g. field-building gets more people doing great stuff, this can in turn build the field, etc., or be path-dependent, e.g. someone does some writing that establishes better concepts for the field.
Value drift: maybe you don’t trust your future self to give as much, or to be as good at picking good stuff. (Some commitment mechanisms exist for this, like DAFs, but that really only fixes the “give as much” problem, and there are lots of opportunities that DAFs can’t fund, such as 501c4 advocacy organizations, individuals, political campaigns, etc.)
Expropriation risk: you might lose the money, including via global catastrophe.
In favor of giving later:
Value of information: especially in a fast-changing field like AI, we’ll continue learning more about what kinds of interventions work as time goes on.
Philanthropic learning: basically the opposite of value drift: you specifically might become a wiser donor, especially if you’re currently young and/or new to the field.
Returns to scale: it’s probably better to make e.g. a single $150k donation than ten donations averaging $15k, because orgs can act pretty decisively with an amount like that, like hire somebody or run a program. (Eventually you hit diminishing returns, but not for most individual donors.)
Tax bunching (only applies to donations that you can write off): in my understanding, at least in the US, there’s a threshold below which you effectively can’t write off donations (the standard deduction), so there’s effectively a fixed cost in any year that you make donations. This makes donating a fixed amount every year a pretty suboptimal strategy, other things equal; if you’re donating an amount below or not that far above the standard deduction to c3 orgs every year, you might be able to save or donate significantly more if you instead donate once every few years.
Consider whether you’re comparatively advantaged to give to non-tax-deductible things.
(Not financial advice.) I think people—especially donors who are giving >$100k/year—often default to thinking that they should stick to tax-deductible giving, because they have an unusually high “501c3 multiplier” due to high marginal income tax rates or low cost basis for capital gains taxes. I claim this is a mistake for some donors, because what matters is whether your 501c3 multiplier is unusually high relative to the average dollar in the donor mix, which is usually coming from other people in very high tax brackets.
People who do have unusually high “501c3 multipliers” include those with employer matches to 501c3 donations. For a 1:1 match for cash donations, I think the multiplier is something like 3.5x, and even higher if you’re donating appreciated assets like equity.[1] I would guess that you need to have a multiplier at least that good to actually be comparatively advantaged [ETA: because I think lots of the dollars from individual donors in the EA giving space come from people with 1:1 or better employer matches, like Google or Anthropic].[2]
The reason this matters is that if too many people think they’re comparatively advantaged for tax-deductible giving, then non-tax-deductible opportunities (e.g. 501c4 advocacy, political giving, even future 501c3s awaiting their 501c3 determination) will unduly struggle to fundraise, so the best marginal opportunities are often going to be in that category.
If your donation budget is $10,000 (of post-tax income) and you’re, say, a single San Franciscan making $500k (and therefore paying a 42.53% marginal tax rate, per SmartAsset), I think this means you could donate ~$17,400 in cash (a 1.74x multiplier) and deduct that from your income, reducing your tax burden by $7,400 = $10,000 from your post-tax income. Then your 1:1 employer match means the charity gets double that, or $34,800 (a 3.48x multiplier). If you’re donating assets that have appreciated, rather than cash, you also avoid paying taxes on those assets, which drives the multiplier up further.
Some people don’t have an employer match but are giving equities instead of cash, so you might think they’d have an unusually high 501c3 multiplier because their donations both mean they don’t have to pay taxes from selling the assets and they can write off the deduction. By my math/in my understanding of how taxes work, it’s pretty hard for this to get to 3.5x, because:
I believe you can also give (some kinds of?) appreciated assets to 501c4s without the 501c4 paying tax on that, unless the 501c4 also engages in certain political activities. So the first half of the logic—“they don’t have to pay taxes from selling the assets”—also applies to (some) non-tax-deductible donations, and you’re left with the writeoff, which is just a ~1.74x multiplier.
I believe that when you donate short-term-appreciated assets, you can only write off the cost basis, not the fair market value, and the long-term-appreciated tax rate is low enough (20% for the highest tax brackets) that I don’t think it can get you to 3.5x. But I haven’t totally crunched the numbers here.