We do have an on record prediction from Yudkowsky, actually: in 1999 he predicted that drexler-style nanotech would arrive in 2010 (using some quite embarrassing reasoning, if I may comment as a physicist). He was also predicting that nanotech would “powerful enough to destroy the planet”, which is why he wanted to build the singularity himself, something he thought the singularity institute could accomplish by 2008.

This seems to be an instance of crying wolf by literally the exact same person that is crying wolf today.

Think the main reason it doesn’t get talked about much is that impoverishing other countries was baked into the whole “America First” idea in the first place, including the [obviously incorrect] beliefs that trade is essentially zero sum so making these countries poorer is necessary to make Americans richer. But Trump also got votes from a lot of Americans whose main concern was rising prices, so it’s particularly salient that the first major effect of blanket tariff increase on consumer goods will be their cost of living going up...

(I think also the effects of US tariff levels on the typical <$2 a day person are relatively indirect: most of them aren’t involved in direct exports to the US from countries likely to be major tariff losers, especially if he turns out to be far more interested in restricting imports of Chinese manufactured alternatives to US luxury goods than cheap foodstuffs. Lower global economic output will slow their local economies down too, but that impact feels less tangible, and to an extent is balanced out by other factors like China’s increased interest in trading with the global South and whatever happens to energy prices.)

Thank you for writing this! I wish I had internalized some of these points more while I was at university, and guess others will feel the same.

One thing in particular that I recognized is viewing 80,000 Hours (and the EA community more broadly) as offering definitive answers, rather than tools and questions. Looking back, I realize I maybe fell into that mindset. I almost expected that if I just followed the “right” path they laid out and worked as hard as I could, I’d maximize my impact. That was, of course, a very soothing thought, drastically simplifying the complexity of my career choice. But it was also very wrong, and I’m grateful that this quick take is now there to point this out :)

Now the post is updated with 2024 numbers :)

I didn’t include Longview Philanthropy because they’re a smaller funder and a lot of their funding seems to come from Open Philanthropy. There is a column called “Other” that serves as a catch-all for any funders I left out.

I took a look at Founder’s Pledge but their donations don’t seem that relevant to AI safety to me.

Thanks for posting this. My comment is on the viability of donor screening; I’m looking for places to reduce the level of burden to make this more viable.

I’m not going to focus on the specific details of the CC policy, but will mention one big issue that I think makes it a weak model in general. It comes across too close to one-size-fits-most, rather than being heavily risk stratified. That stratification should depend on the amount, the presence or absence of yellow/​orange/​red flags in a more cursory screen, and so on.

For at least some organizations, I like the $10K threshold for performing some sort of donor screening. That’s in part based on my gut that third parties may see a five-figure donation as significant. I could be persuaded to go higher depending on the size of the organization and the nature of its work. Maybe I’m showing my own biases, but I think it would be hard for the general public to get or at least stay mad at a bednet-distribution charity that took a problematic $50K donation as long as it promptly disgorged it upon detection, and those orgs’ effectiveness is less dependent on public esteem anyway. Tolerances for policy or meta work would probably be less.

But for orgs on the relatively higher screening side of things: in the $10K to perhaps $25K or $50K range, that should probably look more like 15-30 minutes of time from a junior staffer /​ contractor /​ intelligent college student intern looking for yellow, orange, or red flags, not a full-fledged review. If no flags are found, clear the donor for three years (unless their donations go up a level or the organization happens to learn of concerns). If there are flags, send the file to the development director (“DD”) or other mid-level staff. The DD could clear yellow flags or decline red flags, or do a medium-to-full fat writeup if necessary.

In my view, the primary point of vetting one’s midsize donors for most orgs is to stop donations from red-klaxon donors—e.g., SBF’s parents, most people who have been convicted of crimes involving moral turpitude or fraud, people who are readily known to have engaged in pretty shady behavior with crypto initial coin offerings, etc. It is not realistic to scope midsize-donor investigation to catch everyone from whom one wouldn’t take a donations after a bespoke custom investigation. At this level, I don’t think it is necessary to consider an individual donor’s employer unless it presents a conflict of interest, the donor is a senior executive, or the employer is really problematic/​controversial.

Moving to the $50K-$250K range, I predict that a lot of donors will still be low risk and should be able to screen OK even at stricter orgs without involvement of senior staff or a lot of paperwork. Anyone whose source of income is being a lawyer at a typical law firm, a physician doing typical physician things, a trader at somewhere like Jane Street, etc. shouldn’t be too hard to clear efficiently. One potential crux is that I’m relatively more concerned that the donations are cleanly obtained and relatively less concerned about the donor’s non-financial activities (as it were). So I don’t necessarily think most orgs would have many donors who need the full-dress vetting treatment.

When does a donor become more than “midsize” for screening purposes? One non-exclusive indicator: when the org cannot afford to disgorge a donor’s money going back several years immediately and without grave impacts on the viability of the org’s continuing work, then that donor is not midsize. In other words, perhaps post-acceptance screening (cf. post-claims underwriting in insurance) can be a viable part of the overall vetting strategy . . . but only if the org can immediately spit back the money that it wouldn’t have taken had it conducted a full investigation upfront.

I’ll check feasibility with GiveWell’s donor numbers because I know where to find those. There were 138 100K+ donors through GiveWell in 2022, plus an unclear number of anonymous donors (p. 12). A few hundred screened donors seems appropriate for an org moving as much money as GiveWell, especially since most of its donors are repeat donors and I do not think full reinvestigations would be necessary each year.

What to do with the anonymous money is likely to be a major issue here, though. If an org is going to do that, I guess the only mitigation may be healthy reserves for post-acceptance screening if the donor’s identity becomes known and is problematic?

Mo, thank you for chiming in. Yes, you understood the key point, and you summarised it very well! In my reply to Jan, I expanded on your point about why I think calculating the expected value is not possible for AI safety. Feel free to check it out.

I am curious, though: do you disagree with the idea that a worldview diversification approach at an individual level is the preferred strategy? You understood my point, but how true do you think it is?

Hi Jan, I appreciate the kind words and the engagement!

You correctly summarized the two main arguments. I will start my answer by making sure we are on the same page regarding what expected value is.

Here is the formula I am using:

• (EV) = (p) × (+V) + (1 − p) × (−W)

• p = probability

• V = magnitude of the good outcome

• −W = magnitude of the bad outcome

As EAs, we are trying to maximize EV.

Given that I believe we are extremely morally uncertain about most causes, here is the problem I have encountered: Even if we could reliably estimate the probability of how good or bad a certain outcome is, and even how large +V and −W are, we still don’t know how to evaluate the overall intervention due to moral uncertainty.

For example, while the Risk-Averse Welfarist Consequentialism worldview would “aim to increase the welfare of individuals, human or otherwise, without taking long-shot bets or risking causing bad outcomes,” the Total Welfarist Consequentialism worldview would aim to maximize the welfare of all individuals, present or future, human or otherwise.

In other words, the two worldviews would interpret the formula (even if we perfectly knew the value of each variable) quite differently.

Which one is true? I don’t know. And this does not even take into account other worldviews, such as Egalitarianism, Nietzscheanism, or Kantianism, and others that exist.

To make matters worse, we are not only morally uncertain, but I am also saying that we can’t reliably estimate the probability of how likely a certain +V or −W is to come into existence through the objectives of AI safety. This is linked to my discussion with Jim about determinate credences (since I didn’t initially understand this concept well, ChatGPT gave me a useful explanation).

I think complex cluelessness leads to the expected value formula breaking down (for AI safety and most other longtermist causes) because we simply don’t know what p is, and I, at least, don’t have a determinate credence higher than 50%.

Even if we were to place a bet on a certain worldview (like Total Welfarist Consequentialism), this wouldn’t solve the problem of complex cluelessness or our inability to determine p in the context of AI safety (in my opinion).

This suggests that this cause shouldn’t be given any weight in my portfolio and implies that even specializing in AI safety on a community level doesn’t make sense.

Having said that, there are quite likely some causes where we can have a determinate credence way above 50% for p in the EV formula. And in these cases, a worldview diversification strategy seems to be the best option. This requires making a bet on a set or one worldview, though. Otherwise, we can’t interpret the EV formula, and doing good might not be possible.

Here is an (imperfect) example of how this might look and why a WDS could be the best to pursue. The short answer to why WDS is preferred appears to be that, given moral uncertainty, we don’t need to choose one out of 10 different worldviews and hope that it is correct; instead, we can diversify across different ones. Hence, this seems to be a much more robust approach to doing good.

What do you make of this?

1. Become conversational in Spanish so I can talk to my fianceé′s family easily.

2. Work out ten times per month (3x/​week with leeway)

3. Submit 12 short stories about transformative AI to publishers this year.
   
   More details here. Ongoing mission: get a literary agent for my novel!

I’ve been meaning to write about what it’s like trying to figure out your career direction between 18 and 21 while being part of the EA community—a time that feels, at least for me, like the most uncertain and exploratory part of life. You’re not just asking yourself what you want to do, but also grappling with questions about impact and doing the most good, which adds another hard layer to an already complex period.

For anyone at university who’s getting introduced to EA and feeling overwhelmed about career decisions, I feel like I want to share some thoughts. I’ve been there—feeling unsure about whether I was making the “right” choices or if I was doing enough to have impact. I’ve seen others around that age wrestle with the same questions. I still wrestle with my impact. You have my empathy.

If that’s where you are right now, maybe a few scattered pieces of advice from someone a little further down the road could help. I won’t turn this into a long essay, but if any of this resonates—or if you want more specific guidance—I’d be happy to expand on these thoughts:

1. 80,000 Hours is not gospel: Of course, they don’t claim to be gospel, and explicitly want you to explore different options in your career and provide tools for you to do the thinking. But it’s very easy to just default to their listed career options and cause areas. 80,000 Hours won’t help you make the right choices if you aren’t willing to accept that it’s just one piece of your career puzzle. Most people don’t just end up working on what they want by default, unfortunately.

2. Think beyond conventional impact paths: On that note, if something doesn’t fit neatly into how a typical EA career pans out, that might feel uncomfortable. But that’s okay—outside this community, people do all sorts of things in the world. Going to university doesn’t automatically prepare you for AI policy work or give you operations skills. You’ll probably need to get experience in the outside world that isn’t an EA career path, and that can be hard if all you’ve consumed at university is EA philosophy and its traditional career paths. This is why not pigeon-holing yourself is a good idea (see point 4).

3. Get ready for ego hits: Yes, there’s plenty written about how EA jobs are hard to get, but lots of jobs that seem shiny and potentially useful for career capital will reject you—because you’re not the only one who wants that shiny job. You might get lucky and end up right where you want to be, but you’re probably just like everybody else: inexperienced and trying to make it in the world. Each job application can take weeks of effort. You can make it to interview rounds, all excited about the possibility of doing something you want, only to be rejected because someone else has 10 years more experience than you. This will happen, and it will be hard. You just have to get back up and try again. I found it useful to remind myself every time I got rejected that ‘they can reject me, but they can’t kill my spirit’—and that helped muster up the motivation to push forward.

4. Don’t let EA become your whole life: I switched my degree to be very AI Governance focused (which maybe is paying off), made EA friends, went to EA retreats. It’s so enticing because the university EA community tends to be interesting, thoughtful, and ambitious—that’s pleasant to be around and can mean your life gets wrapped up in it. Getting invited to EA Global conferences in the Bay Area when you’re a twenty-year-old at university hits that status-seeking part of your brain hard. People think it’s really cool, and it feels good when they do. I wish I could say I was above caring what others think, but my brain (like most people’s) is wired to chase social validation at times. While there’s plenty of advice out there about letting go of status-seeking—and you should definitely work on that—I think it’s important to acknowledge how these dynamics can pull you deeper into making EA your whole identity. I strongly suggest building yourself in other communities and finding interests outside the movement. This advice might seem obvious to any adult, but when you’re at university finding your people, and those people happen to offer both intellectual stimulation and status boosts, it’s really easy to stick to the comfortable option.

5. Don’t dismiss grades—they’re part of the bigger picture: I absorbed some wrong advice about grades not mattering through the rationality community. But they do matter: not just for masters applications, but as a signal to employers about your ability to work hard and follow through. Even if EA jobs don’t always list grade requirements, having good grades demonstrates competence and work ethic. More importantly, engaging deeply with your subject teaches you how to tackle difficult problems and work systematically—skills that matter regardless of where you end up. And actually trying with your degree and doing well can make university a much more pleasant experience.

Hope that’s useful to somebody.

It might also be the safest way to “incubate” an AI Safety organisation (for all the Entrepreneurs out there).

Can you expand on this a bit? Do you mean field-building orgs, or any AIS org in general? :)

Thanks for mentioning all of these! Bluedot has put out so much awesome stuff for the broader fieldbuilding community!

We tried this before with a few groups, and found that adding a third party to organizing a course (especially for smaller courses) can actually add more work than it saves.

Could you expand on this a bit more? This is basically the main crux for us to be running this kind of program, or at least for including groups who would otherwise be able to put up their own courses. Can you think of some specific failure modes that we should be mindful of?

Love this analysis and I’ve been wondering why no one talks about it. There are two motivations that makes sense to me for why analysts don’t talk about this:

1. Political framing—putting American interests first is the way to persuade policymakers to listen to you.

2. Genuine nationalism—these analysts actually care more about the harms to Americans than to foreigners.

It bothers me to not be able to distinguish between these.

If you haven’t read it, this article is a convincing argument for why containing harmful policies by the West should be a main focus for development policy.

I think there is the possibility that he focused on the seemingly “right” goal, in the wrong way.

Appreciate the post! Maybe explore different charity topics, get hands on with volunteering, talk to/​see the people who suffer, and decide how you could help with your skill sets

″ Our Boring Advice For Teens ”

Larks, I’ve noticed that your comments on this post focus on criticizing the specific rules in the CC donor policy document that I shared as an example for other EA organizations to consider. You rightly point out that some of these rules might be impractical for EA orgs to follow. However, I feel frustrated because I think your responses have missed the point of my original post, which was to highlight the value of having a written donor screening policy, and to offer the CC document as a resource that other orgs can legally copy and adapt to their own needs (under the CC-BY license), rather than to recommend that others adopt the policy verbatim.

I would appreciate it if we could keep the conversation focused on discussing the broader idea of implementing donor screening policies or processes for EA orgs, rather than getting bogged down in the specifics of this document. By focusing on the bigger picture, we can ensure that EA organizations develop systems that promote transparency and trust, especially in light of past controversies such as FTX. I think this approach would help build more confidence in the community, while still allowing for flexibility in implementation.

If you have thoughts on how these policies or processes could be better adapted to fit the needs of EA organizations, I’d be glad to discuss them here. I hope we can continue this conversation in a collaborative and constructive way.

If you do have a determinate credence above 50% for AI safety work, how do you arrive at this conclusion?

It happens that I do not. But I would if I believed there was evidence robust to unknown unknowns in favor of assuming “AI Safety work” is good, factoring in all the possible consequences from now until the end of time. This would require robust reasons to believe that current AI safety work actually increases rather than decreases safety overall AND that increased safety is actually good all things considered (e.g., that human disempowerment is actually bad overall). (See Guillaume’s comment on the distinction). I won’t elaborate on what would count as “evidence robust to unknown unknowns” in such a context but this is a topic for a future post/​paper, hopefully.

Next, I want to push back on your claim that if ii) is correct, everything collapses. I agree that this would lead to the conclusion that we are probably entirely clueless about longtermist causes, probably the vast majority of causes in the world. However, it would make me lean toward near-term areas with much shorter causal chains, where there is a smaller margin of error—for example, caring for your family or local animals, which carry a low risk of backfiring.

Sorry, I didn’t mean to argue against that. I just meant that work you are clueless about (e.g. maybe AI safety work in your case?) shouldn’t be given any weight in your diversified portfolio. I didn’t mean to make any claim about what I personnally think we should or shouldn’t be clueless about. The “everything falls apart” was unclear and probably unwarranted.

Ah yeah sorry, the doc just has a summary of our OKRs, which is meant to broadly communicate what our team is working on. It’s not the actual text from our OKRs. I’ve added a sentence to clarify that.

I’m seeing lots of O’s and no KR’s. Is this intentional?

(My understanding is that one of the main benefits of having KPIs or OKRs is the accountability that comes from having specific and measurable (i.e. quantifiable) outcomes that you can track and evaluate, and I can’t see any of those in the document.)

Interested to hear other thoughts

Same here! :-)