Researching donation opportunities. Previously: ailabwatch.org.
Zach Stein-Perlman
Karma: 5,724
Crossposting Ryan’s comment on LW:
It seems pretty likely to me that you instead want AIs to be risk seeking for reasons discussed here. Takeover attempts that are very unlikely to succeed might speculatively actually be a great trade from the perspective of humanity and risk seeking/risk neutral AI in that they reduce overall takeover risk while being good for this AI (while deals are way less useful to humanity due to being less clear evidence). Risk avoidant AIs might do nothing or just take deals until AIs take over later (and accepting deals might not be a good strategy for them depending on their views about the chance of AI takeover and some other factors).
I also think the implicit story about how we steer these traits doesn’t really hold together and assumes a type of generalization I find somewhat implausible if we condition on AIs being egregiously misaligned.
Over on LessWrong I wrote some posts about prioritization research and donations.
Yes. Most people will directionally-agree (which is maybe a problem you were trying to solve by adding “probably”) but maybe that’s OK.
If AGI goes well for humans, it’ll probably [i.e. ≥70%] go well for animals
I think it doesn’t really make sense to do a sliding-scale vote on ≥70%. If my credence on (AGI goes well for animals | AGI goes well for humans) is 60%, then I’m just a no; if my credence is 80%, then I’m just a yes.
One way people could interpret the sliding-scale is expressing their confidence/stability in their judgment about whether the probability is greater or less than 70%. But that’s somewhat deranged and it’s not clear how to make it precise and I think everyone will just be confused.
It would be totally reasonable to vote on a scale from 0% to 100% for P(AGI goes well for animals | AGI goes well for humans), rather than voting from fully-disagree to fully-agree for ≥70%. Obviously that requires a little recoding. But making the voting scale more flexible, rather than just from fully-disagree to fully-agree, will benefit other debates too.
I haven’t engaged with your posts and so don’t know the arguments.
I respect that you and a few others legitimately feel deeply clueless. Alice and Bob are just whining about how not everything is clear-cut.
My claim is that if you’re worried, the correct response is to actually try to make the astronomical problem/cause go better, not to give up on it. I think if you’re savvy you will probably find a way to make the astronomical thing go better—such as doing strategy/prioritization/deconfusion work, or working on robustly good intermediate desiderata, or building skills/money in case there’s more clarity in the future—rather than ultimately thinking there’s nothing I can do to make the thing go better.
I.
Once upon a time, there was an EA named Alice. EA made a lot of sense to Alice, and she believed that some niche problems/causes were astronomically bigger than others. But she eventually decided that (1) the theories of change were confusing/suspicious and (2) there’s substantial evidence that a bunch of EA work is net-negative. So she decided to become a teacher or doctor or something.
II.
Alice made a mistake! If she thinks that some problems/causes are astronomically bigger than others, and she’s skeptical of certain approaches, she should look for better approaches, not give up on those problems/causes! For example, she could:
Find an intervention (in the great problems/causes) that she believes in, and do that
Defer to people who she really respects on the topic
Try to understand the problem and possible interventions; do strategy/prioritization/deconfusion work (for herself or maybe benefitting the whole community)
Develop relevant skills and/or save up money, and set herself up to notice if there’s more clarity or great opportunities in the future
Accept sign-uncertainty and do positive-EV stuff
III.
This is actually about my friend Bob who’s sometimes like I work on AI safety but I feel clueless about whether we’re actually helping, and I see that farmed animal suffering is a huge problem, and I want to go work on farmed animal welfare. If Bob still believes that the AI stuff is astronomically more important than the animal stuff, Bob is making the same mistake as Alice!
Two hours before you posted this, MacAskill posted a brief explanation of viatopianism.
This essay is the first in a series that discusses what a good north star [for post-superintelligence society] might be. I begin by describing a concept that I find helpful in this regard:
Viatopia: an intermediate state of society that is on track for a near-best future, whatever that might look like.
Viatopia is a waystation rather than a final destination; etymologically, it means “by way of this place”. We can often describe good waystations even if we have little idea what the ultimate destination should be. A teenager might have little idea what they want to do with their life, but know that a good education will keep their options open. Adventurers lost in the wilderness might not know where they should ultimately be going, but still know they should move to higher ground where they can survey the terrain. Similarly, we can identify what puts humanity in a good position to navigate towards excellent futures, even if we don’t yet know exactly what those futures look like.
In the past, Toby Ord and I have promoted the related idea of the “long reflection”: a stable state of the world where we are safe from calamity, and where we reflect on and debate the nature of the good life, working out what the most flourishing society would be. Viatopia is a more general concept: the long reflection is one proposal for what viatopia would look like, but it need not be the only one.
I think that some sufficiently-specified conception of viatopia should act as our north star during the transition to superintelligence. In later essays I’ll discuss what viatopia, concretely, might look like; this note will just focus on explaining the concept.
. . .
Unlike utopianism, it cautions against the idea of having some ultimate end-state in mind. Unlike protopianism, it attempts to offer a vision for where society should be going. It focuses on achieving whatever society needs to be able to steer itself towards a truly wonderful outcome.
I think I’m largely on board. I think I’d favor doing some amount of utopian planning (aiming for something like hedonium and acausal trade). Viatopia sounds less weird than utopias like that. I wouldn’t be shocked if Forethought talked relatively more about viatopia because it sounds less weird. I would be shocked if they push us in the direction of anodyne final outcomes. I agree with Peter that stuff is “convex” but I don’t worry that Forethought will have us tile the universe with compromisium. But I don’t have much private info.
Quick take on longtermist donations for giving tuesday.
My favorite donation opportunity is Alex Bores’s congressional campaign. I also like Scott Wiener’s congressional campaign.
If you have to donate to a normal longtermist 501c3, I think Forethought, METR, and The Midas Project—and LTFF/ARM and Longview’s Frontier AI Fund—are good and can use more money (and can’t take Good Ventures money). But I focus on evaluating stuff other than normal longtermist c3s, because other stuff seems better and has been investigated much less; I don’t feel very strongly about my normal longtermist c3 recommendations.
Some friends and I have nonpublic recommendations less good than Bores but ~4x as good as the normal longtermist c3s above, according to me.
+1
Random take: people underrate optionality / information value. Even within EA, few opportunities are within 5x of the best opportunities (even on the margin), due to inefficiencies in the process by which people get informed about donation opportunities. Waiting to donate is great if it increases your chances of donating very well. Almost all of my friends regret their past donations; they wish they’d saved money until they were better-informed.
Random take: there are still some great c3 opportunities, but hopefully after the Anthropic people eventually get liquidity they’ll fill all of the great c3 opportunities.
Some public c3 donation opportunities I like are The Midas Project (small funding gap + no industry money), Forethought, and LTFF/ARM.
Random take: you should really invest your money to get a high return rate.
I’m not sure what we should be doing now! But I expect that people can make progress if they backchain from the von Neumann probes, whereas my impression is that most people entering the “digital sentience” space never think about the von Neumann probes.
Oh, clarification: it’s very possible that there aren’t great grant opportunities by my lights. It’s not like I’m aware of great opportunities that the other Zach isn’t funding. I should have focused more on expected grants than Zach’s process.
Thanks. I’m somewhat glad to hear this.
One crux is that I’m worried that broad field-building mostly recruits people to work on stuff like “are AIs conscious” and “how can we improve short-term AI welfare” rather than “how can we do digital-minds stuff to improve what the von Neumann probes tile the universe with.” So the field-building feels approximately zero-value to me — I doubt you’ll be able to steer people toward the important stuff in the future.
A smaller crux is that I’m worried about lab-facing work similarly being poorly aimed.
I endorse Longview’s Frontier AI Fund; I think it’ll give to high-marginal-EV AI safety c3s.
I do not endorse Longview’s Digital Sentience Fund. (This view is weakly held. I haven’t really engaged.) I expect it’ll fund misc empirical and philosophical “digital sentience” work plus unfocused field-building — not backchaining from averting AI takeover or making the long-term future go well conditional on no AI takeover. I feel only barely positive about that. (I feel excited about theoretical work like this.)
$500M+/year in GCR spending
Wait, how much is it?https://www.openphilanthropy.org/grants/page/4/?q&focus-area%5B0%5D=global-catastrophic-risks&yr%5B0%5D=2025&sort=high-to-low&view-list=true lists $240M in 2025 so far.
I have a decent understanding of some of the space. I feel good about marginal c4 money for AIPN and SAIP. (I believe AIPN now has funding for most of 2026, but I still feel good about marginal funding.)
There are opportunities to donate to politicians and PACs which seem 5x as impactful as the best c4s. These are (1) more complicated and (2) public. If you’re interested in donating ≥$20K to these, DM me. This is only for US permanent residents.
I’m confident the timing was a coincidence. I agree that (novel, thoughtful, careful) posting can make things happen.
In most cases, especially high-stakes cases, cost of living is small relative to prioritization/productivity/information benefits. (View not justified here.)