I just learned about Zipline, the world’s largest autonomous drone delivery system, from YouTube tech reviewer Marques Brownlee’s recent video, so I was surprised to see Zipline pop up in a GiveWell grant writeup of all places. I admittedly had the intuition that if you’re optimising for cost-effectiveness as hard as GW do, and that your prior is as skeptical as theirs is, then the “coolness factor” would’ve been stripped clean off whatever interventions pass the bar, and Brownlee’s demo both blew my mind with its coolness (he placed an order on mobile for a power bank and it arrived by air in thirty seconds flat, yeesh) and also seemed the complete opposite of cost-effective (caveating that I know nothing about drone delivery economics). Quoting their “in a nutshell” section:
In December 2024, GiveWell recommended a $54,620 grant to Zipline for a six-month scoping project. Zipline will use this time to review ways that they could use drones to increase vaccination uptake, especially in hard-to-reach areas with low vaccination coverage and high rates of vaccine-preventable diseases. …
We recommended this grant because:
Drones are an intuitive way to bring vaccines closer to communities with low coverage rates, especially when demand for vaccination exists, but conditions like difficult terrain, poor infrastructure, weak cold chain, or insecurity make it difficult for families to access immunizations.
This grant aligns with our strategy of making several scoping grants to high-potential organizations to source promising ideas for solving bottlenecks in the routine immunization system, and then testing these concepts.
Okay, but what about cost-effectiveness? Their “main reservations” section says
Evidence on cost-effectiveness of drones for vaccine delivery is limited, and we have not modeled the cost-effectiveness of the types of programs that Zipline plans to consider, nor the value of information for this scoping grant.
An internal review conducted in 2023 focused on drones for health was generally skeptical about there being many opportunities in this area that would meet GiveWell’s bar, although this scoping grant will focus on the most promising scenario (remote areas with high rates of vaccine-preventable diseases and low vaccination coverage rates).
Is there any evidence of cost-effectiveness at all then? According to Zipline, yes — e.g. quoting the abstract from their own 2025 modelling study:
Objectives: In mid-2020, the Ghana Health Service introduced Zipline’s aerial logistics (centralized storage and delivery by drones) in the Western North Region to enhance health supply chain resilience. This intervention led to improved vaccination coverage in high-utilization districts. This study assessed the cost-effectiveness of aerial logistics as an intervention to improve immunization coverage.
Methods: An attack rate model, adjusted for vaccination coverage and vaccine efficacy, was used to estimate disease incidence among vaccinated and unvaccinated populations, focusing on 17 022 infants. Incremental cost-effectiveness ratios of US dollar per averted disability-adjusted life-year (DALY) were evaluated from societal and government perspectives, using real-world operations data. …
Results: In 2021, aerial logistics averted 688 disease cases. Incremental cost-effectiveness ratios were $41 and $58 per averted DALY from the societal and government perspectives, respectively. The intervention was cost-saving when at least 20% of vaccines delivered by aerial logistics replaced those that would have been delivered by ground transportation, with potential government savings of up to $250 per averted DALY. Probabilistic sensitivity analysis confirmed the robustness of these findings.
That’s super cost-effective. For context, the standard willingness-to-pay to avert a DALY is 1x per capita GDP or $2,100 in Ghana, so 35-50x higher. Also:
… we calculated that aerial logistics facilitated the completion of an additional 14 979 full immunization courses… We estimated that 4 children’s lives (95% CI 2–7) were saved in these districts during 2021. … the intervention averted a total of $20 324 in treatment costs and $2819 for caregivers between lost wages and transport.
At a cost of $0.66 per incremental FIC (fully immunized child),this approach outperforms other delivery methods analyzed in the review, including the most cost-effective category of interventions identified, namely “Delivery Approach” interventions, such as monthly immunization by mobile teams in villages and the enhancement of satellite clinic immunization practices.
(GW notes that they’d given Zipline’s study a look and “were unable to quickly assess how key parameters like program costs and the impact of the program on vaccination uptake and disease were being estimated”. Neither can I. Still pretty exciting)
Can confirm; Zipline is ridiculously cool. I saw their P1 Drones in action in Ghana and met some of their staff at EA conferences. Imo, Zipline is one of the most important organizations around for last-mile delivery infrastructure. They’re a key partner for GAVI, and they save lives unbelievably cost-effectively by transporting commodities like snakebite antivenom within minutes to people who need it.
Their staff and operations are among the most high-performing of any organization I’ve ever seen. Here are some pics from a visit I took to their bay area office in October 2024. I’d highly recommend this Mark Rober video, and checking out Zipline’s website. If any software engineers are interested in high-impact work, I would encourage you to apply to Zipline!
Yep Snakebite is one of the few slamdunk usecases for me here. Until we design a cheap, heat stable antivenom I think drones that can get there in under an hour might be the best option in quite a wide range of places.
Zipline have been around for about 10 years I think—boy do they have the cool factor. One big issue is that they can only carry as really tiny amount of stuff. Also the places where they can potentially save money have to be super hard to access, because a dirt cheap motorcycle which can go 50km for a dollar of fuel can carry 50x as much weight.
My lukewarm take is that hey have done well, but as with most things haven’t quite lived up to their initial hype.
Nice! I’ve been enjoying your quick takes / analyses, and find your writing style clear/easy to follow. Thanks Mo! (I think this could have been a great top level post FWIW, but to each their own :) )
Is there a good list of the highest leverage things a random US citizen (probably in a blue state) can do to cause Trump to either be removed from office or seriously constrained in some way? Anyone care to brainstorm?
Like the safe state/swing state vote swapping thing during the election was brilliant—what analogues are there for the current moment, if any?
In case this is useful to anyone in the future: LTFF does not provide funding for-profit organizations. I wasn’t able to find mentions of this online, so I figured I should share.
I was made aware of this after being rejected today for applying to LTFF as a for-profit. We updated them 2 weeks ago on our transition into a non-profit, but it was unfortunately too late, and we’ll need to send a new non-profit application in the next funding round.
Yall, I have been off and on distracted from my work by intense and unpleasant outrage/disgust at immigration enforcement, ever since Trump’s first campaign speech close to ten years ago. I have few visceral moral convictions, and this is the strongest. I wish my strongest conviction was a positive one, where I’m brimming with hope and warmth. But instead, anger is more salient.
I don’t know what to do about it. I dont think spending kajillions of dollars of lawyers so that the victims’ lives can be somewhat less inconvenienced passes the ITN test, and i don’t have kajillions of dollars. So it’s basically like, I know I’m not gonna do anything about it, I just have to sit and try to let it distract me as little as possible. Total bummer, feels very disempowering.
It’d be great to be able to transmute these feelings into a positive vibe.
For you or others reading this, I can really recommend protesting if you’re not already. I also doubt it passes the ITN test (although, I wouldn’t discount it!), but it does provide (a) a good outlet for your feelings and (b) a real sense that you’re not alone, that there are people out there who are gonna fight alongside you. I come back from protests feeling a mix of emotions, but depressed and disempowered is rarely one of them.
In light of recent discourse on EA adjacency, this seems like a good time to publicly note that I still identify as an effective altruist, not EA adjacent.
I am extremely against embezzling people out of billions of dollars of money, and FTX was a good reminder of the importance of “don’t do evil things for galaxy brained altruistic reasons”. But this has nothing to do with whether or not I endorse the philosophy that “it is correct to try to think about the most effective and leveraged ways to do good and then actually act on them”. And there are many people in or influenced by the EA community who I respect and think do good and important work.
As do I brother, thanks for this declaration! I think now might not be the worst time ogir those who do identify directly as EAs to stay so to encourage the movement, especially some of the higher up thought and movement leaders. I don’t think a massive sign up form or anything drastic is necessary, just a few higher status people standing up and saying “hey, I still identify with this thing”.
That is if they think it isn’t an outdated term...
I’m curious what you both think of my impression that the focus on near-term AGI has completely taken over EA and sucked most of the oxygen out of the room.
I was probably one of the first 1,000 people to express an interest in organized effective altruism, back before it was called “effective altruism”. I remember being in the Giving What We Can group on Facebook when it was just a few hundred members, when they were still working on making a website. The focus then was exclusively on global poverty.
Later, when I was involved in a student EA group from around 2015 to 2017, global poverty was still front and centre, animal welfare and vegetarianism/veganism/reducetarianism was secondary, and the conversation about AI was nipping at the margins.
Fast forward to 2025 and it seems like EA is now primarily a millennialist intellectual movement focused on AGI either causing the apocalypse or creating utopia within the next 3-10 years (with many people believing it will happen within 5 years), or possibly as long as 35 years if you’re way far out on the conservative end of the spectrum.
This change has nothing to do with FTX and probably wouldn’t be a reason for anyone at Anthropic to distance themselves from EA, since Anthropic is quite boldly promoting a millennialist discourse around very near-term AGI.
But it is a reason for me not to feel an affinity with the EA movement anymore. It has fundamentally changed. It’s gone from tuberculosis to transhumanism. And that’s just not what I signed up for.
The gentle irony is that I’ve been interested in AGI, transhumanism, the Singularity, etc. for as long as I’ve been interested in effective altruism, if not a little longer. In principle, I endorse some version of many of these ideas.
But when I see the kinds of things that, for example, Dario Amodei and others at Anthropic are saying about AGI within 2 years, I feel unnerved. It feels like I’m at the boundary of the kind of ideas that it makes sense to try to argue against or rationally engage with. Because it doesn’t really feel like a real intellectual debate. It feels closer to someone experiencing some psychologically altered state, like mania or psychosis, where attempting to rationally persuade someone feels inappropriate and maybe even unkind. What do you even do in that situation?
I recently wrote here about why these super short AGI timelines make no sense to me. I read an article today that puts this into perspective. Apple is planning to eventually release a version of Siri that merges the functionality of the old, well-known version of Siri and the new soon-to-be-released version that is based on an LLM. The article says Apple originally wanted to release the merged version of Siri sooner, but now this has been delayed to 2027. Are we going to have AGI before Apple finishes upgrading Siri? These ideas don’t live in the same reality.
To put a fine point on it, I would estimate the probability of AGI being created by January 1, 2030 to be significantly less than the odds of Jill Stein winning the U.S. presidential election in 2028 as the Green Party candidate (not as the leader of either the Democratic or Republican primary), which, to be clear, I think will be roughly as likely as her winning in 2024, 2020, or 2016 was. I couldn’t find any estimates of Stein’s odds of winning either the 2028 election or past elections from prediction markets or election forecast models. At one point, electionbettingodds.com gave her 0.1%, but I don’t know if they massively rounded up or if those odds were distorted by a few long-shot bets on Stein. Regardless, I think it’s safe to say the odds of AGI being developed by January 1, 2030 are significantly less than 0.1%.
If I am correct (and I regret to inform you that I am correct), then I have to imagine the credibility of EA will diminish significantly over the next 5 years. Because, unlike FTX scamming people, belief in very near-term AGI is something that many people in EA have consciously, knowingly, deliberately signed up for. Whereas many of the warning signs about FTX were initially only known to insiders, the evidence against very near-term AGI is out in the open, meaning that deciding to base the whole movement on it now is a mistake that is foreseeable and… I’m sorry to say… obvious.
I feel conflicted saying things like this because I can see how it might come across as mean and arrogant. But I don’t think it’s necessarily unkind to try to give someone a reality check under unusual, exceptional circumstances like these.
I think EA has become dangerously insular and — despite the propaganda to the contrary — does not listen to criticism. The idea that EA has abnormal or above-average openness to criticism (compared to what? the evangelical church?) seems only to serve the function of self-licensing. That is, people make token efforts at encouraging or engaging with criticism, and then, given this demonstration of their open-mindedness, become more confident in what they already believed, and feel licensed to ignore or shut down criticism in other instances.
It also bears considering what kind of criticism or differing perspectives actually get serious attention. Listening to someone who suggests that you slightly tweak your views is, from one perspective, listening to criticism, but, from another perspective, it’s two people who already agree talking to each other in an echo chamber and patting themselves on the back for being open-minded. (Is that too mean? I’m really trying not to be mean.)
On the topic of near-term AGI, I see hand-wavey dismissal of contrary views, whether they come from sources like Turing Prize winner and FAIR Chief AI Scientist Yann LeCun, surveys of AI experts, or superforecasters. Some people predict AGI will be created very soon and seemingly a much larger number think it will take much longer. Why believe the former and not the latter? I see people being selective in this way, but I don’t see them giving principled reasons for being selective.
Crucially, AGI forecasts are a topic where intuition plays a huge role, and where intuitions are contagious. A big part of the “evidence” for near-term AGI that people explicitly base their opinion on is what person X, Y, and Z said about when they think AGI will happen. Someone somewhere came up with the image of some people sitting in a circle just saying ever-smaller numbers to each other, back and forth. What exactly would prevent that from being the dynamic?
When it comes to listening to differing perspectives on AGI, what I have seen more often than engaging with open-mindedness and curiosity is a very unfortunate, machismo/hegemonic masculinity-style impulse to degrade or humiliate a person for disagreeing. This is the far opposite of “EA loves criticism”. This is trying to inflict pain on someone you see as an opponent. This is the least intellectually healthy way of engaging in discourse, besides, I guess, I don’t know, shooting someone with a gun if they disagree with you. You might as well just explicitly forbid and censor dissent.
I would like to believe that, in 5 years, the people in EA who have disagreed with me about near-term AGI will snap out of it and send me a fruit basket. But they could also do like Elon Musk, who, after predicting fully autonomous Teslas would be available in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, and 2024, and getting it wrong 9 years in a row, now predicts fully autonomous Teslas will be available in 2025.
In principle, you could predict AGI within 5 years and just have called it a few years too soon. If you can believe in very near-term AGI today, you will probably be able to believe in very near-term AGI when 2030 rolls around, since AI capabilities will only improve.
Or they could go the Ray Kurzweil route. In 2005, Kurzweil predicted that we would have “high-resolution, full-immersion, visual-auditory virtual reality” by 2010. In 2010, when he graded his own predictions, he called this prediction “essentially correct”. This was his explanation:
The computer game industry is rapidly moving in this direction. Technologies such as Microsoft’s Kinect allows players to control a videogame without requiring controllers by detecting the player’s body motions. Three-dimensional high-definition television is now available and will be used by a new generation of games that put the user in a full-immersion, high-definition, visual-auditory virtual reality environment.
Kurzweil’s gradings of his own predictions are largely like that. He finds a way to give himself a rating of “correct” or “essentially correct”. Even though he was fully incorrect. I wonder if Dario Amodei will do the same thing in 2030.
In 2030, there will be the option of doubling down on near-term AGI. Either the Elon Musk way — kick the can down the road — or the Ray Kurzweil way — revisionist history. And the best option will be some combination of both.
When people turn out to be wrong, it is not guaranteed to increase their humility or lead to soul searching. People can easily increase their defensiveness and their aggression toward people who disagree with them.
And, so, I don’t think merely being wrong will be enough on its own for EA to pull out of being a millennialist near-term AGI community. That can continue indefinitely even if AGI is over 100 years away. There is no guarantee that EA will self-correct in 5 years.
For these reasons, I don’t feel an affinity toward EA any more — it’s nothing like what it was 10 or 15 years ago — and I don’t feel much hope for it changing back, since I can imagine a scenario where it only gets worse 5 years from now.
Richard Ngo has a selection of open-questions in his recent post. One question that caught my eye:
How much censorship is the EA forum doing (e.g. of thought experiments) and why?
I originally created this account to share a thought experiment I suspected might be a little too ‘out there’ for the moderation team. Indeed, it was briefly redacted and didn’t appear in the comment section for a while (it does now). It was, admittedly, a slightly confrontational point and I don’t begrudge the moderation team for censoring it. They were patient and transparent in explaining why it was briefly redacted. You can read the comment and probably guess correctly why it was flagged.
Still, I am curious to hear of other cases like this. My guess is that in most of them, the average forum reader will side with the moderation team.
LessWrong publishes most of their rejected posts and comments on a separate webpage. I say ‘most’ as I suspect infohazards are censored from that list. I would be interested to hear the EA forum’s moderation team’s thoughts on this approach/whether it’s something they’ve considered, should they read this and have time to respond.[1]
Creating such a page would also allow them to collect on Ngo’s bounty, since they would be answer both how much censorship they do and (assuming they attach moderation notes) why
However, some content (like your first comment) requires additional back and forth internally (such as checking with moderators) and/or with the new user. This process involves various non-obvious judgement calls, which is what caused a long delay between your submitting the comment and us reaching out to you (plus the fact that many people were out over the winter holidays). In the case of your comment, we asked you to edit it and you didn’t respond to us or edit the comment for over a week, and then our facilitator felt bad for keeping you in the queue for so long so they approved your comment.
We currently do not use the rejected content feature that LW uses. Instead, almost all[1] of the content that may have been rejected under their system ends up appearing on the rest of our site, and we currently mostly rely on users voting to make content more or less visible (for example, karma affects where a post is displayed on the Frontpage). I plan to seriously consider whether we should start using the rejected content feature here soon; if so, then I expect that we’ll have the same page set up.
I think that, if we had been using the rejected content feature, the right move would have been for us to reject your comment instead of approving it.
My guess is that there are edge cases, but in practice we keep our queue clear, so my understanding is that users are typically not in limbo for more than a few days. Things like spam are not rejected — accounts that post spam are banned.
Thanks for taking the time to respond thoroughly! I sincerely appreciate that.
I can’t quite remember when I read the message sent from the facilitator, but my memory is that it was after the comment was restored (feel free to check on your end if that’s possible). I was slightly bummed out that a comment which took some effort to write was rejected and wasn’t super motivated to respond defending it.
At the time, I was aware that the metaphor was abrasive, but hoped I had sanded off the edges by adding a disclaimer at the start. It can be difficult to balance ‘writing the thing I honestly believe’ with ‘not upset anybody or make them uncomfortable when discussing moral issues 100% of the time.’ I did hum and haw over whether I should post it, but ultimately decided that most people wouldn’t be upset by the metaphor or would even agree with it’s accuracy (given that the meat/dairy industries are both rife with animal sexual abuse). Seeing as how it was interpreted as flame bait / trolling, I somewhat regret posting it.
On a final note; am I able to ask why you would reject it? I.e. do you believe I was trolling or flame baiting? I won’t be insulted either way, but would find it useful going forward to know how I should better write my comments.
Two final notes:
• I am pleased to hear you are considering a rejected content feature.
• I used the word ‘censorship’ in my original short form post and want to underscore that I don’t think it’s intrinsically bad to censor. I.e. the moderation team should be doing some level of censorship (and I suspect most forum users would agree).
Thanks for the feedback! I think moderation is tricky and I’m relatively new at it myself. I’m sad at how long users can get stuck in the queue, and I’d love to improve how fast we resolve moderation questions, but where exactly we draw these lines will probably be a learning process for me, and we’ll continue to iterate on that.
It looks like you submitted the comment on Dec 17, and our facilitator messaged you on Jan 6 (the delay partly being due to people being out for the holidays), and then they approved your comment a little over a week after messaging you. Yeah I agree that this was an edge case, and I don’t think you were being malicious, but I think you could have made your point more productively by, for example, just using “torture”.
I feel that using the rejected content feature would give our team more leeway to be opinionated about shaping the home page of our site (compared to now), and we’d feel somewhat free to reject things that don’t fit the type of discussions we want to see. For example, it looks like LW rejects posts from new users that don’t have a clear introduction. So I think if something is an edge case in the current system, then it would likely get rejected under the other system.
I recently came across Santi Ruiz from the Institute for Progress’s podcast and substack, Statecraft. I enjoyed going back through the archives and thought I’d share some of my favorites here.
The AI alignment community had a major victory in the regulatory landscape, and it went unnoticed by many.
The EU AI Act explicitly mentions “alignment with human intent” as a key focus area in relation to regulation of systemic risks.
As far as I know, this is the first time “alignment” has been mentioned by a law, or major regulatory text.
It’s buried in Recital 110, but it’s there. And it also makes research on AI Control relevant:
“International approaches have so far identified the need to pay attention to risks from potential intentional misuse or unintended issues of control relating to alignment with human intent”.
This means that alignment is now part of the EU’s regulatory vocabulary.
But here’s the issue: most AI governance professionals and policymakers still don’t know what it really means, or how your research connects to it.
I’m trying to build a space where AI Safety and AI Governance communities can actually talk to each other.
If you’re curious, I wrote an article about this, aimed at the corporate decision-makers that lack literacy on your area.
Would love any feedback, especially from folks thinking about how alignment ideas can scale into the policy domain.
Here is the Substack link (I also posted it on LinkedIn):
AIxBio looks pretty bad and it would be great to see more people work on it
We’re pretty close to having a country of virologists in a data center with AI models that can give detailed and accurate instructions for all steps of a biological attack — with recent reasoning models, we might have this already
These models have safeguards but they’re trivial to overcome — Pliny the Liberator manages to jailbreak every new model within 24 hours and open sources the jailbreaks
Open source will continue to be just a few months behind the frontier given distillation and amplification, and these can be fine-tuned to remove safeguards in minutes for less than $50
People say it’s hard to actually execute the biology work, but I don’t see any bottlenecks to bioweapon production that can’t be done by a bio undergrad with limitless scientific knowledge; on my current understanding, the bottlenecks are not manual dexterity bottlenecks like playing a violin which require years of practice, they are knowledge bottlenecks
Bio supply chain controls that make it harder to get ingredients aren’t working and aren’t on track to work
So it seems like we’re very close to democratizing (even bespoke) bioweapons. When I talk to bio experts about this they often reassure me that few people want to conduct a biological attack, but I haven’t seen much analysis on this and it seems hard to be highly confident.
While we gear up for a bioweapon democracy it seems that there are very few people working on worst-case bio, and most of the people working on it are working on access controls and evaluations. But I don’t expect access controls to succeed, and I expect evaluations to mostly be useful for scaring politicians, due in part to the open source issue meaning we just can’t give frontier models robust safeguards. The most likely thing to actually work is biodefense.
I suspect that too many people working on GCR have moved into working on AI alignment and reliability issues and too few are working on bio. I suspect there are bad incentives, given that AI is the new technology frontier and working with AI is good career capital, and given that AI work is higher status.
When I talk to people at the frontier of biosecurity, I learn that there’s a clear plan and funding available, but the work is bottlenecked by entrepreneurial people who can pick up a big project and execute on it autonomously — these people don’t even need a bio background. On my current guess, the next 3-5 such people who are ambivalent about what to do should go into bio rather than AI, in part because AI seems to be more bottlenecked by less generalist skills, like machine learning, communications, and diplomacy.
I think the main reasons that EAs are working on AI stuff over bio stuff is that there aren’t many good routes into worst case bio work afaict largely due to infohazard concerns from field building, and the x-risk case for biorisk not being very compelling (maybe due to infohazard concerns around threat models).
I think these are fair points, I agree the info hazard stuff has smothered a lot of talent development and field building, and I agree the case for x-risk from misaligned advanced AI is more compelling. At the same time, I don’t talk to a lot of EAs and people in the broader ecosystem these days who are laser focused on extinction over GCR, that seems like a small subset of the community. So I expect various social effects, making a bunch more money, and AI being really cool and interesting and fast-moving are probably a bigger deal than x-risk compellingness simpliciter. Or at least they have had a bigger effect on my choices!
But insufficiently successful talent development / salience / comms is probably the biggest thing, I agree.
Yup! The highest level plan is in Kevin Esvelt’s “Delay, Detect, Defend”: use access controls and regulation to delay worst-case pandemics, build a nucleic acid observatory and other tools to detect amino acid sequences for superpandemics, and defend by hardening the world against biological attacks.
The basic defense, as per DDD, is:
Develop and distribute adequate PPE to all essential workers
Make sure the supply chain is robust to ensure that essential workers can distribute food and essential supplies in the event of a worst-case pandemic
Environmental defenses like far-UVC that massively reduce the spread and replication rate of pandemic pathogens
IMO “delay” has so far basically failed but “detect” has been fairly successful (though incompletely). Most of the important work now needs to rapidly be done on the “defend” side of things.
There’s a lot more details on this and the biosecurity community has really good ideas now about how to develop and distribute effective PPE and rapidly scale environmental defenses. There’s also now interest in developing small molecule countermeasures that can stop pandemics early but are general enough to stop a lot of different kinds of biological attacks. A lot of this is bottlenecked by things like developing industrial-scale capacity for defense production or solving logistics around supply chain robustness and PPE distribution. Happy to chat more details or put you in touch with people better suited than me if it’s relevant to your planning.
Anthropic has been getting flak from some EAs for distancing itself from EA. I think some of the critique is fair, but overall, I think that the distancing is a pretty safe move.
Compare this to FTX. SBF wouldn’t shut up about EA. He made it a key part of his self-promotion. I think he broadly did this for reasons of self-interest for FTX, as it arguably helped the brand at that time.
I know that at that point several EAs were privately upset about this. They saw him as using EA for PR, and thus creating a key liability that could come back and bite EA.
And come back and bite EA it did, about as poorly as one could have imagined.
So back to Anthropic. They’re taking the opposite approach. Maintaining about as much distance from EA as they semi-honestly can. I expect that this is good for Anthropic, especially given EA’s reputation post-FTX.
And I think it’s probably also safe for EA.
I’d be a lot more nervous if Anthropic were trying to tie its reputation to EA. I could easily see Anthropic having a scandal in the future, and it’s also pretty awkward to tie EA’s reputation to an AI developer.
To be clear, I’m not saying that people from Anthropic should actively lie or deceive. So I have mixed feelings about their recent quotes for Wired. But big-picture, I feel decent about their general stance to keep distance. To me, this seems likely in the interest of both parties.
Do you think that distancing is ever not in the interest of both parties? If so, what is special about Anthropic/EA?
(I think it’s plausible that the answer is that distancing is always good; the negative risks of tying your reputation to someone always exceed the positive. But I’m not sure.)
Arguably, around FTX, it was better. EA and FTX both had strong brands for a while. And there were worlds in which the risk of failure was low.
I think it’s generally quite tough to get this aspect right though. I believe that traditionally, charities are reluctant to get their brands associated with large companies, due to the risks/downsides. We don’t often see partnerships between companies and charities (or say, highly-ideological groups) - I think that one reason why is that it’s rarely in the interests of both parties.
Typically companies want to tie their brands to very top charities, if anyone. But now EA has a reputational challenge, so I’d expect that few companies/orgs want to touch “EA” as a thing.
Arguably influencers are a often a safer option—note that EA groups like GiveWell and 80k are already doing partnerships with influencers. As in, there’s a decent variety of smart YouTube channels and podcasts that hold advertisements for 80k/GiveWell. I feel pretty good about much of this.
Arguably influencers are crafted in large part to be safe bets. As in, they’re very incentivized to not go crazy, and they have limited risks to worry about (given they represent very small operations).
Arguably influencers are a often a safer option—note that EA groups like GiveWell and 80k are already doing partnerships with influencers. As in, there’s a decent variety of smart YouTube channels and podcasts that hold advertisements for 80k/GiveWell. I feel pretty good about much of this.
This feels different to me. In most cases, there is a cultural understanding of the advertiser-ad seller relationship that limits the reputational risk. (I have not seen the “partnerships” in question, but assume there is money flowing in one direction and promotional consideration in the other.) To be sure, activists will demand for companies to pull their ads from a certain TV show when it does something offensive, to stop sponsoring a certain sports team, or so on. However, I don’t think consumers generally hold prior ad spend against a brand when it promptly cuts the relationship upon learning of the counterparty’s new and problematic conduct.
In contrast, people will perceive something like FTX/EA or Anthropic/EA as a deeper relationship rather than a mostly transactional relationship involving the exchange of money for eyeballs. Deeper relationships can have a sense of authenticity that increases the value of the partnership—the partners aren’t just in it for business reasons—but that depth probably increases the counterparty risks to each partner.
I hope my post was clear enough that distance itself is totally fine (and you give compelling reasons for that here). It’s ~implicitly denying present knowledge or past involvement in order to get distance that seems bad for all concerned. The speaker looks shifty and EA looks like something toxic you want to dodge.
Responding to a direct question by saying “We’ve had some overlap and it’s a nice philosophy for the most part, but it’s not a guiding light of what we’re doing here” seems like it strictly dominates.
A number of policy tools such as regulations, liability regimes or export controls—aimed at tackling AI risks - have already been explored, and mostly appear as promising and worth further iterations.
But AFAIK no one has so far come up with a concrete proposal to use tax policy tools to internalize AI risks. I wonder why, considering that policies, such as tobacco taxes, R&D tax credits, and 401(k), have been mostly effective. Tax policy also seems to be underutilized and neglected, given we already possess sophisticated institutions like tax agencies or tax policy research networks.
Safety measures spending of AI Companies seems to be relatively low, and we can expect that if competition intensifies, these expenses will be even lower.
So I’ve started to consider more seriously the idea of tax incentives—basically we can provide a tax credit or deduction for expenditures on AI safety measures like alignment research, cybersecurity or oversight mechanisms etc. which effectively could lower their cost. To illustrate: AI Company incurs safety researcher salary as a cost and then 50% of that cost can be additionally deducted from the tax base.
My guess was that such tool could influence the ratio of safety-to-capability spending. If implemented properly it could help mitigate competitive pressures affecting frontier AI labs by incentivising them to increase spending on AI safety measures.
Like any market intervention, we can justify such incentives if they correct market inefficiencies or generate positive externalities. In this case, lowering the cost of security measures helps internalize risk.
However there are many problems on path to design such tool effectively:
The crucial problem is that financial benefit from tax credit can’t match the expected value of increasing capabilities. Underlying incentives for capability breakthroughs are potentially orders of magnitude larger. So simply AI labs wouldn’t bother and keep the same level while getting extra money from incentives which is an obvious backlash.
However, if some AI Company plans to increase safety expenses due to their real concerns about risks or external pressures (boards, public etc.), perhaps they would be more willing to do it.
Also risk of keeping the same safety expenses level could be overcome by requiring a specific threshold of expenditures to benefit from the incentive.
The focus here is on inputs (spending) instead of outcomes (actual safety).
Implementing it would be pain in the ass, requiring creating specialised departments within IRS or delegating most of the work to NIST.
Defining the scope of qualified expenditures - it could be hard to separate safety from capabilities research cost. Keeping an eye on this later can be a considerable administrative cost.
Expected expenses could be incurred regardless of the public funding received if we just impose a strict requirement.
There could be a problem of safety washing—AI labs creating an impression and signalling that appropriate safety measures are implemented and benefiting from incentives while not decreasing the risk effectively.
I don’t know much about US tax system but I guess it could overlap with existing R&D tax incentives. However, existing incentives are unlikely to reduce the risk. if they are used for both safety and capabilities research then they
Currently most AI labs are in loss position so they can’t effectively benefit from such incentives unless some special feature is put in place, like refundable tax credits or the option to claim such relief/credit as soon as they make a taxable profit.
Perhaps direct government financing would be more effective. Or existing ideas (such as those mentioned earlier) would be more effective and we don’t have enough room for weaker solutions.
Maybe money isn’t a problem here as AI labs are more talent constrained. If the main bottleneck for effective safety work is a talented researcher, then making safety spending cheaper via tax credits might not significantly increase the amount of high-quality safety work done.
Is there something crucial that I am missing? Is it worth investigating further? So far it has more problems than the potential benefits so I don’t think it’s promising, but I’d love to hear your thoughts on it.
11. It would probably cost a good bit of political capital to get this through, which may have an opportunity cost. You may not even get public support from the AI companies because the proposal contains an implicit critique that they haven’t been doing enough on safety.
12. By the time the legislation got out of committee and through both houses, the scope of incentivized activity would probably be significantly broader than what x-risk people have in mind (e.g., reducing racial bias). Whether companies would prefer to invest more in x-risk safety vs. other incentivized topics is unclear to me.
I saw mentioned in one discussion about Dustin Moskowitz’s exit from the Effective Altruism Forum that him quitting isn’t by itself necessarily an omen, of the relationship between Good Ventures and the effective altruism ecosystem becoming much colder than it is now. While it wouldn’t be a binary for Dustin anyway, it may seem like his is a special case with implications for the relationship between EA and Good Ventures going forward. Nonetheless, while most of them may not be billionaires, as far as I’m aware, I know there high net worth donors among effective altruists have also mentioned before that they’ve almost always tended to avoid the EA Forum. That’s probably for the same reasons any major philanthropist might. In spite of that, those donors have decently known relationships with the effective altruism ecosystem, with them remaining continually and relatively stable.
How and to what extent Dustin quitting the forum will imply decreased funding EA causes receive from Good Ventures will perhaps only become apparent later. The main indicator of that could be the nature and amount of grants made throughout the rest of the year, based on Open Philanthropy’s recommendations. It may be likely there’s a positive correlation between the two outcomes of Dustin quitting the EA Forum, and Good Ventures support for a variety of projects affiliated/aligned with effective altruism potentially declining. Yet how exactly how strong that correlation may be is hardly certain, and many effective altruists may currently be overestimating it.
What’s the most cost-effective economic growth-boosting intervention? It’s cat mascots. I just learned about Tama the calico cat (via @thatgoodnewsgirl on Instagram), who “gained fame for being a railway station master and operating officer at Kishi Station on the Kishigawa Line in Kinokawa, Wakayama Prefecture, Japan”.
Tama was born in Kinokawa, Wakayama, and was raised with a group of stray cats that used to live close to Kishi Station. They were regularly fed by passengers and by Toshiko Koyama, the informal station manager at the time.
The station was near closure in 2004 because of financial problems on the rail line. Around this time, Koyama adopted Tama. Eventually the decision to close the station was withdrawn after the citizens demanded that it stay open.[3] In April 2006, the newly formed Wakayama Electric Railway destaffed all stations on the Kishigawa Line to cut costs, and at the same time evicted the stray cats from their shelter to make way for new roads leading to the stations. Koyama pleaded with Mitsunobu Kojima, president of Wakayama Electric Railway, to allow the cats to live inside Kishi Station; Kojima, seeing Tama as a maneki-neko (beckoning cat), agreed to the request.[4] …
In lieu of an annual salary, the railway provided Tama with a year’s worth of cat food and a gold name tag for her collar stating her name and position. A station master’s hat was specially designed and made to fit Tama, and took more than six months to complete.[6] In July 2008, a summer hat was also issued to Tama for hotter weather.[7] Tama’s original gold name tag was stolen by a visitor on October 10, 2007, but a replica was quickly made to replace it.[8]
The publicity from Tama’s appointment led to an increase in passengers by 17% for that month as compared to January 2006; ridership statistics for March 2007 showed a 10% increase over the previous financial year. A study estimated that the publicity surrounding Tama has contributed 1.1 billion yen to the local economy.[9]
That’s actually lowballing Tama. If you click on footnote 9, it’s ¥1.38 billion ($13.1 million at the exchange rate back then):
With 55,000 more people having used the Kishigawa Line than would normally be expected, Tama is being credited with a contribution to the local economy calculated to have reached as much as 1.1 billion yen (10.44 million dollars) in 2007 alone, according to a study announced last week.
Katsuhiro Miyamoto, a professor at Kansai University’s School of Accountancy, said picture books and other merchandise featuring the feline stationmaster also produced significant economic effects. A television appearance and other publicity surrounding Tama—who receives cat food in lieu of a salary—was worth 280 million yen, according to Miyamoto.
In contrast, Tama was really cheap. Gemini 2.5 Pro helpfully estimated the following cost breakdown, which adds up to a median of ¥110,000:
A year’s supply of food probably cost ¥30,000 - ¥60,000
Vet care could range from ¥30,000 - ¥70,000 annually to cover everything from routine check-ups to vaccinations, flea/tick prevention, and potential unexpected health issues
Her promotion to “super station master” (superintendent equivalent) came with an office (a converted ticket booth containing a litter box) estimated at ¥50,000 - ¥110,000, amortised over Tama’s nearly decade-long career
Tama’s two custom-made hats (summer and other seasons) weren’t cheap, probably ¥10,000 - ¥30,000 each, but worth including in the compensation package to retain top talent
In contrast her two gold-plated name tags (the first was stolen by a visitor) likely weren’t as expensive, ¥2,000 - ¥5,000
This works out to almost exactly 10,000:1 benefit-cost ratio just from increased ridership, and that’s excluding merchandise, TV appearance and other publicity etc — well beyond even Open Phil’s higher-than-ever 2,100x bar!
In fact, a few hundred thousand Tamas would already double global economic growth, and the world’s Tama-carrying capacity is probably 3-4 OOMs higher at least (the World Population Review’s estimate of over a billion cats worldwide today should be interpreted as a lower bound). Talk about feline-powered explosive economic growth potential…
Cats’ economic growth potential likely has a heavy-tailed distribution, because how else would cats knock things off shelves with their tail. As such, Open Philanthropy needs to be aware that some cats, like Tama, make much better mascots than other cats. One option would be to follow a hits-based strategy: give a bunch of areas cat mascots, and see which ones do the best. However, given the presence of animal welfare in the EA movement, hitting cats is likely to attract controversy. A better strategy would be to identify cats that already have proven economic growth potential and relocate them to areas most in need of economic growth. Tama makes up 0.00000255995% of Japan’s nominal GDP (or something thereabouts, I’m assuming all Tama-related benefits to GDP occurred in the year 2020). If these benefits had occurred in North Korea, they would be 0.00086320506% of nominal GDP or thereabouts. North Korea is also poorer, so adding more money to its economy goes further. Japan and North Korea are near each other, so transporting Tama to North Korea would be extremely cheap. Assuming Tama’s benefits are the same each year and are independent of location (which seems reasonable, I asked ChatGPT for an image of Tama in North Korea and it is still cute), catnapping Tama would be highly effective. One concern is that there might be downside risk, because people morally disapprove of kidnapping cats. On the other hand, people expressing moral disapproval of kidnapping cats are probably more likely to respect animal’s boundaries by not eating meat, thus making this an intervention that spans cause areas. In conclusion: EA is solved, all we have to do is kidnap some cats.
A few thoughts. First, it is a really cute story, and I’m glad you shared it. It feels very Japanese.
Second, marketing and tourism aren’t often considered as major areas for economic development and growth (at least not in the popular press books I’ve read or the EA circles I’ve been in), but this is a simple little case study to demonstrate that having a mascot (or anything else that people like, from fancy buildings to locations tied to things people like) can drive economic activity. But it is also hard to predict in advance what will be a hit. I bet that lots of places have beautiful murals, cute animals, historical importance, lovely scenery, and similar attractions without having much of a positive return on investment. For me, the notable think about Tama’s story is how little money was needed to add something special to the local station. A lot of investments related to tourism are far more expensive.
A final thought, one that maybe folks more versed in economics can help me with. Should we consider this an example of economic growth? Is this just shifting spending/consumption from one place to another? Would people who spent money to ride this train otherwise would have spent that money doing something else: riding a different train, visiting a park, etc.
If this were a story, there’d be some kind of academy taking in humanity’s top talent and skilling them up in alignment.
Most of the summer fellowships seem focused on finding talent that is immediately useful. And I can see how this is tempting given the vast numbers of experienced and talented folks seeking to enter the space. I’d even go so far as to suggest that the majority of our efforts should probably be focused on finding people who will be useful fairly quickly.
Nonetheless, it does seem as though there should be at least one program that aims to find the best talent (even if they aren’t immediately useful) and which provides them with the freedom to explore and the intellectual environment in which to do so.
I wish I could articulate my intuition behind this clearer, but the best I can say for now is that my intuition is that continuing to scale existing fellowships would likely provide decreasing marginal returns and such an academy wouldn’t be subject to this because it would be providing a different kind of talent.
Am I wrong that EAs working in AI (safety, policy, etc.) and who are now earning really well (easily top 1%) are less likely to donate to charity?
At least in my circles, I get the strong impression that this is the case, which I find kind of baffling (and a bit upsetting, honestly). I have some just-so stories for why this might be the case, but I’d rather hear others’ impressions, especially if they contradict mine (I might be falling prey to confirmation bias here since the prior should be that salary correlates positively with likelihood of donating among EAs regardless of sector).
There are at least three common justifications for not donating, each of which can be quite reasonable:
A high standard of living and saving up money are important selfish wants for EAs in AI, just as they are in broader society.
EAs in AI have needs (either career or personal) that require lots of money.
Donations are much lower impact than one’s career.
I don’t donate to charity other than animal product offsets; this is mainly due to 1 and 2. As for 1, I’m still early career enough that immediate financial stability is a concern. Also for me, forgoing luxuries like restaurant food and travel makes me demotivated enough that I have difficulty working. I have tried to solve this in the past but have basically given up and now treat these luxuries as partially needs rather than wants.
For people just above the top-1% threshold of $65,000, 3 and 2 are very likely. $65,000 is roughly the rate paid to marginal AI safety researchers, so donating 20% will bring only 20% of someone’s career impact even if the grantmakers find an opportunity as good as themself. If they also live in a HCOL area, 2 is very likely—in San Francisco the average rent for a 1bed is $2,962/month and an individual making less than $104,000 qualifies for public housing assistance!
But shouldn’t I have more dedication to the cause and donate anyway? I would prefer to instead spend more effort on getting better at my job (since I’m nowhere near the extremely high skillcap of AI safety research) and working more hours (possibly in ways that funge with donations eg by helping out grantmakers). I actually do care about saving for retirement, and finding a higher-paying job at a lab safety team just so I can donate is probably counterproductive, because trying to split one’s effort between two theories of change while compromising on both is generally bad (see the multipliers post). If I happened to get an equally impactful job that paid double, I would probably start donating after about a year, or sooner if donations were urgent and I expected high job security.
Most EAs want to be rich and close to power. Or at least they are way more into the “effective” optimization part than the altruism. They talk a big game but getting in early on a rising power (AI companies) is not altruistic. Especially not when you end up getting millions in compensation due to very rapid valuation increases.
I made a large amount of money in the 2021 crypto bom. I made a much smaller, though large for me, amount in the 2017 crash. I have never had a high paying job. Often I have had no job at all. My longterm partner has really bad health. So I’m perhaps unusually able to justify holding onto windfalls. I still gave away 50% pre-tax both times.
Its relatively common (I don’t know about rates) for such people to take pay-cuts rather than directly donate that percentage. I know some who could be making millions a year who are actually making hundreds. It makes sense they don’t feel the need to donate anything additional on top of that!
It’s not clear to me whether you’re talking about people who (a) do a voluntary salary sacrifice while working at an EA org, or (b) people who could have earned much more in industry but moved to a nonprofit so now earn much less than their hypothetical maximum earning potential.
In case (a), yes, their salary sacrifice should count towards their real donations.
But I think a practical moral philosophy wherein donation expectations are based on your actual material resources (and constraints), not your theoretical maximum earning potential, seems more justifiable. So I don’t think people who do (b) (which includes myself) should get to say that doing (b) liberates them from the same obligation to donate that would attend to a person in the same material circumstances with worse outside options.
I think the right stance here is a question of “should EA be praising such people or get annoyed they’re not giving up more if it wants to keep a sufficient filter for who it calls true believers”, and the answer there is obviously both groups are great & true believers and it seems dumb to get annoyed at either.
The 10% number was notably chosen for these practical reasons (there is nothing magic about that number), and to back-justify that decision with bad moral philosophy about “discharge of moral duty” is absurd.
I’m not going to defend my whole view here, but I want to give a though experiment as to why I don’t think that “shadow donations”—the delta between what you could earn if you were income-maximizing, and what you’re actually earning in your direct work job—are a great measure for the purposes of practical philosophy (though I agree they’re both a relevant consideration and a genuine sacrifice).
Imagine two twins, Anna and Belinda. Both have just graduated with identical grades, skills, degrees, etc. Anna goes directly from college work on AI safety at Safety Org, making $75,000 / year. Belinda goes to work for OpenMind doing safety-neutral work, making $1M per year total compensation. Belinda learns more marketable skills; she could make at least $1M / year indefinitely. Anna, on the other hand, has studiously plugged away at AI safety work, but since her work is niche, she can’t easily transfer these skills to do something that pays better.
Then imagine that, after three years, Belinda joins Anna at Safety Org. Belinda was not fired; she could have stayed at OpenMind and made $1M per year indefinitely. At this point, Anna has gotten a few raises and is making $100,000, and donating 3% of her salary. Belinda gets the same job on the same pay scale, and does equally good work, but donates nothing. Belinda reasons that, because she could still be $1M per year, she has “really” donated $900,000 of labor to Safety Org, and so has sacrificed roughly 90% of her income.
Is Belinda more altruistic than Anna? Which attitude should EAs aspire to?
To give some more color on my general view:
I don’t really think there’s a first-order fact of the matter as to who of these two (or anyone) is “more altruistic,” or what one’s “obligations” are. At bottom, there are just worlds with more or less value in them.
My view mostly comes from a practical view of how the EA community and project can be most impactful, credible, and healthy. I think the best attitude is closer to Anna’s than Belinda’s.
Donating also has other virtues over salary reductions, since it is concrete, measurable, and helps create a more diversified funding ecosystem.
To be clear, I think it’s great that people like Belinda exist, and they should be welcomed and celebrated in the community. But I don’t think the particular mindset of “well I have really sacrificed a lot because if I was purely selfish I could have made a lot more money” is one that we ought to recognize as particularly good or healthy.
I will note that my comment made no reference to who is “more altruistic”. I don’t know what that term means personally, and I’d rather not get into a semantics argument.
If you give the definition you have in mind, then we can argue over whether its smart to advocate that someone ought to be more altruistic in various situations, and whether it gets at intuitive notions of credit assignment.
I will also note that given the situation, its not clear to me Anna’s proper counterfactual here isn’t making $1M and getting nice marketable skills, since she and Belinda are twins, and so have the same work capacity & aptitudes.
To be clear, I think it’s great that people like Belinda exist, and they should be welcomed and celebrated in the community. But I don’t think the particular mindset of “well I have really sacrificed a lot because if I was purely selfish I could have made a lot more money” is one that we ought to recognize as particularly good or healthy.
I think this is the crux personally. This seems very healthy to me, in particular because it creates strong boundaries between the relevant person and EA. Note that burnout & overwork is not uncommon in EA circles! EAs are not healthy, and (imo) already give too much of themselves!
Why do you think its unhealthy? This seems to imply negative effects on the person reasoning in the relevant way, which seems pretty unlikely to me.
Suppose they’re triplets, and Charlotte, also initially identical, earns $1M/year just like Belinda, but can’t/doesn’t want to switch to safety. How much of Charlotte’s income should she donate in your worldview? What is the best attitude for the EA community?
I didn’t read Cullen’s comment as about 10%, and I think almost all of us would agree that this isn’t a magic number. Most would probably agree that it is too demanding for some and not demanding enough for others. I also don’t see anything in Cullen’s response about whether we should throw shade at people for not being generous enough or label them as not “true believers.”
Rather, Cullen commented on “donation expectations” grounded in “a practical moral philosophy.” They wrote about measuring an “obligation to donate.”
You may think that’s “bad moral philosophy,” but there’s no evidence of it being a post hoc rationalization of a 10% or other community giving norm here.
I disagree and think that b is actually totally sufficient justification. I’m taking as an assumption that we’re using an ethical theory that says people do not have an unbounded ethical obligation to give everything up to subsistence and that it is fine to set some kind of a boundary and fraction of your total budget of resources that you spend on altruistic purposes. Many people doing well paying altruistic careers (eg technical AI safety careers) could earn dramatically more money eg at least twice as much, if they were optimising for the highest paying career they could. I’m fairly sure I could be earning a lot more than I currently am if that was my main goal. But I consider the value of my labour from an altruistic perspective to exceed the additional money I could be donating and therefore do not see myself to have a significant additional ethical obligation to donate (though I do donate a fraction of my income anyway because I want to)
By foregoing a large amount of income for altruistic reasons, I think such people are spending a large amount of their resource budget on altruistic purposes, and that if they still have an obligation to donate more money that people in higher paying careers should be obliged to donate far more. Which is a consistent position, but not one I hold
I think I want to give (b) partial credit here in general. There may not be much practical difference between partial and full credit where the financial delta between a more altruistic job and a higher-salary job is high enough. But there are circumstances in which it might make a difference.
Without commenting on any specific person’s job or counterfactuals, I think it is often true that the person working a lower-paid but more meaningful job secures non-financial benefits not available from the maximum-salary job and/or avoids non-financial sacrifices associated with the maximum-salary job. Depending on the field, these could include lower stress, more free time, more pleasant colleagues, more warm fuzzies / psychological satisfaction, and so on. If Worker A earns 100 currency units doing psychologically meaningful, low to optimal stress work but similarly situated Worker B earns 200 units doing unpleasant work with little in the way of non-monetary benefits, treating the entire 100 units Worker A forewent as spent out of their resource budget on altruistic purposes does not strike a fair balance between Worker A and Worker B.
I don’t want to argue in anyone’s specific case, but I don’t think it’s universally true at all or even true the majority of the time that people that those working in AI could make more elsewhere. It sounds nice to say, but I think often people are earning more in AI jobs than they would elsewhere .
My reasoning was roughly that the machine learning skill set is also extremely employable in finance which tends to pay better. though openai salaries do get pretty high nowadays and if you value openai and anthropic equity at notably above their current market value, then plausibly, they’re higher paying. Definitely agreed it’s not universal.
My point is that, even though there’s a moral obligation, unless you think that high earning people in finance should be donating a very large fraction of their salary (so their post donation pay is less than the pay in AI safety), their de facto moral obligation has increased by the choice to do direct work, which is unreasonable to my eyes.
I would also guess that at least most people doing safety work at industry labs could get a very well paying role at a top tier finance firm? The talent bar is really high nowadays
I also want to point out that having better outside income-maximizing options makes you more financially secure than other people in your income bracket, all else equal, which pro tanto would give you more reason to donate than them.
My point is that “other people in the income bracket AFTER taking a lower paying job” is the wrong reference class.
Let’s say someone is earning $10mn/year in finance. I totally think they should donate some large fraction of their income. But I’m pretty reluctant to argue that they should donate more than 99% of it. So it seems completely fine to have a post donation income above $100K, likely far above.
If this person quits to take a job in AI Safety that pays $100K/year, because they think this is more impactful than their donations, I think it would be unreasonable to argue that they need to donate some of their reduced salary, because then their “maximum acceptable post donation salary” has gone down, even though they’re (hopefully) having more impact than if they donated everything above $100K
I’m picking fairly extreme numbers to illustrate the point, but the key point is that choosing to do direct work should not reduce your “maximum acceptable salary post donations”, and that at least according to my values, that max salary post donation is often above what they get paid in their new direct role.
I suppose what it comes down to is that I actually DO think it is morally better for the person earning $10m/year to donate $9.9m/year than $9m/year, about $900k/year better.
I want to achieve two things (which I expect you will agree with).
I want to “capture” the good done by anyone and everyone willing to contribute and I want them welcomed, accepted and appreciated by the EA community. This means that if a person who could earn $10m/year in finance and is “only” willing to contribute $1m/year (10%) to effective causes, I don’t want them turned away.
I want to encourage, inspire, motivate and push people to do better than they currently are (insofar as it’s possible). I think that includes an Anthropic employee earning $500k/year doing mech interp, a quant trader earning $10m/year, a new grad deciding what to do with their career and a 65-year old who just heard of EA.
I think it’s also reasonable for people to set limits for how much they are willing to do.
This is reasonable. I think the key point that I want to defend is that it seems wrong to say that choosing a more impactful job should mean you ought to have a lower post donation salary.
I personally think of it in terms of having some minimum obligation for doing your part (which I set at 10% by default), plus encouragement (but not obligation) to do significant amounts more good if you want to
My point is that “other people in the income bracket AFTER taking a lower paying job” is the wrong reference class.
Is there a single appropriate reference class here, as opposed to looking at multiple reference classes and weighting the results in some manner?
I agree that similarly situated person who decided to take a very high-paying job is a relevant reference class and should get some weight. However, it doesn’t follow that person with similar incomes working a non-impactful job is an irrelevant reference class or should get zero weight.
As Marcus notes, “[p]eople don’t choose to be smart enough to do ML work.” I would add that people don’t choose other factors that promote or inhibit their ability to choose a very high-paying job and/or a high-impact job (e.g., location and circumstances of birth, health, family obligations, etc.) In a pair of persons who are similarly situated economically, giving the more advantaged person a total pass on the moral obligation to donate money seems problematic to me. In this frame of reference, their advantages allowed them to land a more impactful job at the same salary as the less advantaged person—and in a sense we would be excusing them from a moral obligation because they are advantaged. (Giving the more privileged person a big break is also going to make it rather hard to establish substantial giving as a norm in the broader community, but that’s probably not in the scope of the question here.)
I don’t have a clear opinion on how to weight the two reference classes beyond an intuition that both classes should get perceptible weight. (It also seems plausible there are other reference classes to weigh as well, although I haven’t thought about what they might be.)
My argument is essentially that “similar income, non impactful job” is as relevant a reference class to the “similar income, impactful job person” as it is as a reference class to the “high income, non impactful job” person. I also personally think reference classes is the wrong way to think about it. If taking a more impactful job also makes someone obliged to take on a lower post donation salary (when they don’t have to), I feel like something has gone wrong, and the incentives are not aligned with doing the most good.
But I think a practical moral philosophy wherein donation expectations are based on your actual material resources (and constraints), not your theoretical maximum earning potential, seems more justifiable.
It’s complicated, I think. Based on your distinguishing (a) and (b), I am reading “salary sacrifice” as voluntarily taking less salary than was offered for the position you encumber (as discussed in, e.g., this post). While I agree that should count, I’m not sure (b) is not relevant.
The fundamental question to me is about the appropriate distribution of the fruits of one’s labors (“fruits”) between altruism and non-altruism. (Fruits is an imperfect metaphor, because I mean to include (e.g.) passive income from inherited wealth, but I’ll stick with it.)
We generally seem to accept that the more fruit one produces, the more (in absolute terms) it is okay to keep for oneself. Stated differently—at least for those who are not super-wealthy—we seem to accept that the marginal altruism expectation for additional fruits one produces is less than 100%. I’ll call this the “non-100 principle.” I’m not specifically defending that principle in this comment, but it seems to be assumed in EA discourse.
If we accept this principle, then consider someone who was working full-time in a “normal” job and earn a salary of 200 apples per year. They decide to go down to half-time (100-apple salary) and spend the half of their working hours producing 100 charitable pears for which they receive no financial benefit. [1]The non-100 principle suggests that it’s appropriate for this person to keep more of their apples than a person who works full-time to produce 100 apples (and zero pears). Their total production is twice as high, so they aren’t similarly situated to the full-time worker who produces the same number of apples. The decision to take a significantly less well-paid job seems analogous to splitting one’s time between remunerative and non-remunerative work. One gives up the opportunity to earn more salary in exchange for greater benefits that flow to others by non-donation means.
I am not putting too much weight on this thought experiment, but it does make me think that either the non-100 principle is wrong, or that the foregone salary counts for something in many circumstances even when it is not a salary sacrifice in the narrower sense.
I feel quite confused about the case where someone earns much less than their earning potential in another altruistically motivated but less impactful career doing work that uses a similar skillset (e.g. joining a think tank after working on policy at an AI company). This seems somewhere between A and B.
I’d consider this a question that doesn’t benefit from public speculation because every individual might have a different financial situation.
Truth be told “earning really well” is a very ambiguous category. Obviously, if someone were financially stable, eg. consistently earning high 5 figure or six figure dollars/euros/pounds/francs or more annually(and having non-trivial savings) and having a loan-free house, their spending would almost always reflect discretionary interests and personal opinions (like ’do I donate to charity or not”).
For everyone not financially stable, ‘donating to charity’ may not:
(a) be a discretionary decision and
(b) be a simple decision—that is, increasing charitable donations too soon comes at the expense of not investing in one’s personal ability to weather volatility, which has knock-on qualitative effects on career progression (especially to senior management roles), future earnings potential and lifetime charitable contributions. Additionally, not getting one’s personal finances in order early on contributes directly to great personal and family stress, which then has knock on effects on everything else.
tl;dr: when you’re broke, money allocation is a high-risk, high-stress headache. The long term solution is to prioritize becoming the opposite of broke, i.e; financially stable, first.
also see: Julia Wise’s post on the logistics of giving.
That analysis would be more compelling if the focus of the question were on a specific individual or small group. But, at least as I read it, the question is about the giving patterns of a moderately numerous subclass of EAs (working in AI + “earning really well”) relative to the larger group of EAs.
I’m not aware of any reason the dynamics you describe would be more present in this subclass than in the broader population. So a question asking about subgroup differences seems appropriate to me.
Edit: I see your point. Still, I’ll leave the below comment as-is, because from my (3rd world, generational financial instability, no health insurance, filial obligations etc.) point of view I think the perspective of a broke person ought to be represented.
But what counts as “numerous”, though? How many EAs are actually working in AI—fifty people? A hundred people? Who’s collecting data on this subgroup versus the larger EA group?
I agree that the question itself is appropriate and there’s nothing wrong with it. I was saying this question doesn’t benefit from public speculation, because, for one thing, there isn’t any reliable data for objective analysis, and for another, the logistics of an individual’s personal finance are a bigger factor in how or how much a person donates, at the non-millionaire level (in this subclass and the broader population).
My experience from the church is the salary doesn’t correlate will with likelihood of donating, although it does of course correlate with donating larger amounts of money.
If EAs working in AI policy and safety were serious about AI Doom being a near-term possibility, I would expect they would donate huge amounts towards that cause. A clear case of “revealed preferences” not just stated ones.
I think I was assuming people working in highly paid AI jobs were donating larger percentages of their income, but I haven’t seen data in either direction?
My experience from the church is the salary doesn’t correlate will with likelihood of donating, although it does of course correlate with donating larger amounts of money.
Yes, though I thought maybe among EAs there would be some correlation. 🤷
I think I was assuming people working in highly paid AI jobs were donating larger percentages of their income, but I haven’t seen data in either direction?
Yeah, me neither (which, again, is probably true; just not in my circles).
– A new section of concrete scenarios illustrating how AI can unintentionally suppress emergent thought
– A framing based on cold reading to explain how LLMs may anticipate user thoughts before they are fully formed
– Slight improvements in structure and flow for better accessibility
Examples included:
A student receives an AI answer that mirrors their in-progress insight and loses motivation
A researcher consults an LLM mid-theorizing, sees their intuition echoed, and feels their idea is no longer “theirs”
These additions aim to bridge the gap between abstract ethical structure and lived experience — making the argument more tangible and testable.
Feel free to revisit, comment, or share. And thank you again to those who engaged in the original thread — your input helped shape this improved version.
Japanese version also available (PDF, included in OSF link)
This post proposes a structural alternative to dark matter called the Central Tensional Return Hypothesis (CTRH). Instead of invoking unseen mass, CTRH attributes galactic rotation to directional bias from a radially symmetric tension field. The post outlines both a phenomenological model and a field-theoretic formulation, and invites epistemic scrutiny and theoretical engagement.
I’m interested in figuring out “what skill profiles are most leveraged for altruistic work after we get the first competent AI agents”?
I think this might be one of the most important questions for field-building orgs to work on (in any cause area but particularly AI safety). I think 80k and other AIS-motivated careers groups should try to think hard about this question and what it means for their strategy going forward.
I’m optimistic about productivity on the question of “what skill profiles are most leveraged after we get the first competent AI-agents” because:
few people have actually tried to think about what managing an AI workforce would look like
this doesn’t feel conceptually abstract (e.g. some people have already adopted LMs in their day-to-day workflows)
something like drop-in replacements for human remote workers are more likely than AIs less analogous to human workers if timelines are short (and if timelines are short, this work is more urgent than if timelines are long)
One important consideration is whether human managers are going to be good proxies for AI-agent managers. It’s plausible to me that the majority of object-level AI safety work that has ever be done will be done by AI agents. It could be that the current status quo persists, where people’s leverage routes through things like leadership and management skills, or it could be that ~everyone will become more leveraged proportional to their access to AI models (roughly their budget) due to AI agents having different affordances than humans, or something else.
For example, suppose that AI agents are much more tolerant, hardworking, and understanding of user intent for short tasks than human workers. I expect that having good strategic takes would become much more leveraged and having good internal stakeholder management skills would become much less leveraged (relative to today). If 80k (for example) thought that was likely, then maybe they should double down on finding people with great strategy takes and care less about typical management skills.
Maybe “junior” roles will be automated before “senior” roles almost uniformly across the economy. In that case, it’s even more valuable than it is right now to focus on getting “senior” people.
Or maybe coding will be automated before everything else, and for some reason people’s relative leverage inside the company stays pretty fixed, or it’s too hard to say which kinds of jobs will be most leveraged at software companies. Then “coding” still becomes much more leveraged per dollar than today, and it’s probably useful to find more value-aligned coders.
It would be surprising to me if the price of certain kinds of intellectual labour decreased by 100x, and this had little impact on people’s relative leverage.
(disclaimer: I’m not sure whether the above stories actually go through, they are meant to just be illustrative of the kind of thinking that seems undersupplied)
I just learned about Zipline, the world’s largest autonomous drone delivery system, from YouTube tech reviewer Marques Brownlee’s recent video, so I was surprised to see Zipline pop up in a GiveWell grant writeup of all places. I admittedly had the intuition that if you’re optimising for cost-effectiveness as hard as GW do, and that your prior is as skeptical as theirs is, then the “coolness factor” would’ve been stripped clean off whatever interventions pass the bar, and Brownlee’s demo both blew my mind with its coolness (he placed an order on mobile for a power bank and it arrived by air in thirty seconds flat, yeesh) and also seemed the complete opposite of cost-effective (caveating that I know nothing about drone delivery economics). Quoting their “in a nutshell” section:
Okay, but what about cost-effectiveness? Their “main reservations” section says
Is there any evidence of cost-effectiveness at all then? According to Zipline, yes — e.g. quoting the abstract from their own 2025 modelling study:
That’s super cost-effective. For context, the standard willingness-to-pay to avert a DALY is 1x per capita GDP or $2,100 in Ghana, so 35-50x higher. Also:
(GW notes that they’d given Zipline’s study a look and “were unable to quickly assess how key parameters like program costs and the impact of the program on vaccination uptake and disease were being estimated”. Neither can I. Still pretty exciting)
Can confirm; Zipline is ridiculously cool. I saw their P1 Drones in action in Ghana and met some of their staff at EA conferences. Imo, Zipline is one of the most important organizations around for last-mile delivery infrastructure. They’re a key partner for GAVI, and they save lives unbelievably cost-effectively by transporting commodities like snakebite antivenom within minutes to people who need it.
Their staff and operations are among the most high-performing of any organization I’ve ever seen. Here are some pics from a visit I took to their bay area office in October 2024. I’d highly recommend this Mark Rober video, and checking out Zipline’s website. If any software engineers are interested in high-impact work, I would encourage you to apply to Zipline!
Yep Snakebite is one of the few slamdunk usecases for me here. Until we design a cheap, heat stable antivenom I think drones that can get there in under an hour might be the best option in quite a wide range of places.
Zipline have been around for about 10 years I think—boy do they have the cool factor. One big issue is that they can only carry as really tiny amount of stuff. Also the places where they can potentially save money have to be super hard to access, because a dirt cheap motorcycle which can go 50km for a dollar of fuel can carry 50x as much weight.
My lukewarm take is that hey have done well, but as with most things haven’t quite lived up to their initial hype.
Nice! I’ve been enjoying your quick takes / analyses, and find your writing style clear/easy to follow. Thanks Mo! (I think this could have been a great top level post FWIW, but to each their own :) )
Is there a good list of the highest leverage things a random US citizen (probably in a blue state) can do to cause Trump to either be removed from office or seriously constrained in some way? Anyone care to brainstorm?
Like the safe state/swing state vote swapping thing during the election was brilliant—what analogues are there for the current moment, if any?
(Just quick random thoughts.)
The more that Trump is perceived as a liability for the party, the more likely they would go along with an impeachment after a scandal.
Reach out to Republicans in your state about your unhappiness about the recent behavior of the Trump administration.
Financially support investigative reporting on the Trump administration.
Go to protests?
Comment on Twitter? On Truth Social?
It’s possibly underrated to write concise and common sense pushback in the Republican Twitter sphere?
In case this is useful to anyone in the future: LTFF does not provide funding for-profit organizations. I wasn’t able to find mentions of this online, so I figured I should share.
I was made aware of this after being rejected today for applying to LTFF as a for-profit. We updated them 2 weeks ago on our transition into a non-profit, but it was unfortunately too late, and we’ll need to send a new non-profit application in the next funding round.
Yall, I have been off and on distracted from my work by intense and unpleasant outrage/disgust at immigration enforcement, ever since Trump’s first campaign speech close to ten years ago. I have few visceral moral convictions, and this is the strongest. I wish my strongest conviction was a positive one, where I’m brimming with hope and warmth. But instead, anger is more salient.
I don’t know what to do about it. I dont think spending kajillions of dollars of lawyers so that the victims’ lives can be somewhat less inconvenienced passes the ITN test, and i don’t have kajillions of dollars. So it’s basically like, I know I’m not gonna do anything about it, I just have to sit and try to let it distract me as little as possible. Total bummer, feels very disempowering.
It’d be great to be able to transmute these feelings into a positive vibe.
For you or others reading this, I can really recommend protesting if you’re not already. I also doubt it passes the ITN test (although, I wouldn’t discount it!), but it does provide (a) a good outlet for your feelings and (b) a real sense that you’re not alone, that there are people out there who are gonna fight alongside you. I come back from protests feeling a mix of emotions, but depressed and disempowered is rarely one of them.
In light of recent discourse on EA adjacency, this seems like a good time to publicly note that I still identify as an effective altruist, not EA adjacent.
I am extremely against embezzling people out of billions of dollars of money, and FTX was a good reminder of the importance of “don’t do evil things for galaxy brained altruistic reasons”. But this has nothing to do with whether or not I endorse the philosophy that “it is correct to try to think about the most effective and leveraged ways to do good and then actually act on them”. And there are many people in or influenced by the EA community who I respect and think do good and important work.
As do I brother, thanks for this declaration! I think now might not be the worst time ogir those who do identify directly as EAs to stay so to encourage the movement, especially some of the higher up thought and movement leaders. I don’t think a massive sign up form or anything drastic is necessary, just a few higher status people standing up and saying “hey, I still identify with this thing”.
That is if they think it isn’t an outdated term...
I’m curious what you both think of my impression that the focus on near-term AGI has completely taken over EA and sucked most of the oxygen out of the room.
I was probably one of the first 1,000 people to express an interest in organized effective altruism, back before it was called “effective altruism”. I remember being in the Giving What We Can group on Facebook when it was just a few hundred members, when they were still working on making a website. The focus then was exclusively on global poverty.
Later, when I was involved in a student EA group from around 2015 to 2017, global poverty was still front and centre, animal welfare and vegetarianism/veganism/reducetarianism was secondary, and the conversation about AI was nipping at the margins.
Fast forward to 2025 and it seems like EA is now primarily a millennialist intellectual movement focused on AGI either causing the apocalypse or creating utopia within the next 3-10 years (with many people believing it will happen within 5 years), or possibly as long as 35 years if you’re way far out on the conservative end of the spectrum.
This change has nothing to do with FTX and probably wouldn’t be a reason for anyone at Anthropic to distance themselves from EA, since Anthropic is quite boldly promoting a millennialist discourse around very near-term AGI.
But it is a reason for me not to feel an affinity with the EA movement anymore. It has fundamentally changed. It’s gone from tuberculosis to transhumanism. And that’s just not what I signed up for.
The gentle irony is that I’ve been interested in AGI, transhumanism, the Singularity, etc. for as long as I’ve been interested in effective altruism, if not a little longer. In principle, I endorse some version of many of these ideas.
But when I see the kinds of things that, for example, Dario Amodei and others at Anthropic are saying about AGI within 2 years, I feel unnerved. It feels like I’m at the boundary of the kind of ideas that it makes sense to try to argue against or rationally engage with. Because it doesn’t really feel like a real intellectual debate. It feels closer to someone experiencing some psychologically altered state, like mania or psychosis, where attempting to rationally persuade someone feels inappropriate and maybe even unkind. What do you even do in that situation?
I recently wrote here about why these super short AGI timelines make no sense to me. I read an article today that puts this into perspective. Apple is planning to eventually release a version of Siri that merges the functionality of the old, well-known version of Siri and the new soon-to-be-released version that is based on an LLM. The article says Apple originally wanted to release the merged version of Siri sooner, but now this has been delayed to 2027. Are we going to have AGI before Apple finishes upgrading Siri? These ideas don’t live in the same reality.
To put a fine point on it, I would estimate the probability of AGI being created by January 1, 2030 to be significantly less than the odds of Jill Stein winning the U.S. presidential election in 2028 as the Green Party candidate (not as the leader of either the Democratic or Republican primary), which, to be clear, I think will be roughly as likely as her winning in 2024, 2020, or 2016 was. I couldn’t find any estimates of Stein’s odds of winning either the 2028 election or past elections from prediction markets or election forecast models. At one point, electionbettingodds.com gave her 0.1%, but I don’t know if they massively rounded up or if those odds were distorted by a few long-shot bets on Stein. Regardless, I think it’s safe to say the odds of AGI being developed by January 1, 2030 are significantly less than 0.1%.
If I am correct (and I regret to inform you that I am correct), then I have to imagine the credibility of EA will diminish significantly over the next 5 years. Because, unlike FTX scamming people, belief in very near-term AGI is something that many people in EA have consciously, knowingly, deliberately signed up for. Whereas many of the warning signs about FTX were initially only known to insiders, the evidence against very near-term AGI is out in the open, meaning that deciding to base the whole movement on it now is a mistake that is foreseeable and… I’m sorry to say… obvious.
I feel conflicted saying things like this because I can see how it might come across as mean and arrogant. But I don’t think it’s necessarily unkind to try to give someone a reality check under unusual, exceptional circumstances like these.
I think EA has become dangerously insular and — despite the propaganda to the contrary — does not listen to criticism. The idea that EA has abnormal or above-average openness to criticism (compared to what? the evangelical church?) seems only to serve the function of self-licensing. That is, people make token efforts at encouraging or engaging with criticism, and then, given this demonstration of their open-mindedness, become more confident in what they already believed, and feel licensed to ignore or shut down criticism in other instances.
It also bears considering what kind of criticism or differing perspectives actually get serious attention. Listening to someone who suggests that you slightly tweak your views is, from one perspective, listening to criticism, but, from another perspective, it’s two people who already agree talking to each other in an echo chamber and patting themselves on the back for being open-minded. (Is that too mean? I’m really trying not to be mean.)
On the topic of near-term AGI, I see hand-wavey dismissal of contrary views, whether they come from sources like Turing Prize winner and FAIR Chief AI Scientist Yann LeCun, surveys of AI experts, or superforecasters. Some people predict AGI will be created very soon and seemingly a much larger number think it will take much longer. Why believe the former and not the latter? I see people being selective in this way, but I don’t see them giving principled reasons for being selective.
Crucially, AGI forecasts are a topic where intuition plays a huge role, and where intuitions are contagious. A big part of the “evidence” for near-term AGI that people explicitly base their opinion on is what person X, Y, and Z said about when they think AGI will happen. Someone somewhere came up with the image of some people sitting in a circle just saying ever-smaller numbers to each other, back and forth. What exactly would prevent that from being the dynamic?
When it comes to listening to differing perspectives on AGI, what I have seen more often than engaging with open-mindedness and curiosity is a very unfortunate, machismo/hegemonic masculinity-style impulse to degrade or humiliate a person for disagreeing. This is the far opposite of “EA loves criticism”. This is trying to inflict pain on someone you see as an opponent. This is the least intellectually healthy way of engaging in discourse, besides, I guess, I don’t know, shooting someone with a gun if they disagree with you. You might as well just explicitly forbid and censor dissent.
I would like to believe that, in 5 years, the people in EA who have disagreed with me about near-term AGI will snap out of it and send me a fruit basket. But they could also do like Elon Musk, who, after predicting fully autonomous Teslas would be available in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, and 2024, and getting it wrong 9 years in a row, now predicts fully autonomous Teslas will be available in 2025.
In principle, you could predict AGI within 5 years and just have called it a few years too soon. If you can believe in very near-term AGI today, you will probably be able to believe in very near-term AGI when 2030 rolls around, since AI capabilities will only improve.
Or they could go the Ray Kurzweil route. In 2005, Kurzweil predicted that we would have “high-resolution, full-immersion, visual-auditory virtual reality” by 2010. In 2010, when he graded his own predictions, he called this prediction “essentially correct”. This was his explanation:
Kurzweil’s gradings of his own predictions are largely like that. He finds a way to give himself a rating of “correct” or “essentially correct”. Even though he was fully incorrect. I wonder if Dario Amodei will do the same thing in 2030.
In 2030, there will be the option of doubling down on near-term AGI. Either the Elon Musk way — kick the can down the road — or the Ray Kurzweil way — revisionist history. And the best option will be some combination of both.
When people turn out to be wrong, it is not guaranteed to increase their humility or lead to soul searching. People can easily increase their defensiveness and their aggression toward people who disagree with them.
And, so, I don’t think merely being wrong will be enough on its own for EA to pull out of being a millennialist near-term AGI community. That can continue indefinitely even if AGI is over 100 years away. There is no guarantee that EA will self-correct in 5 years.
For these reasons, I don’t feel an affinity toward EA any more — it’s nothing like what it was 10 or 15 years ago — and I don’t feel much hope for it changing back, since I can imagine a scenario where it only gets worse 5 years from now.
Richard Ngo has a selection of open-questions in his recent post. One question that caught my eye:
I originally created this account to share a thought experiment I suspected might be a little too ‘out there’ for the moderation team. Indeed, it was briefly redacted and didn’t appear in the comment section for a while (it does now). It was, admittedly, a slightly confrontational point and I don’t begrudge the moderation team for censoring it. They were patient and transparent in explaining why it was briefly redacted. You can read the comment and probably guess correctly why it was flagged.
Still, I am curious to hear of other cases like this. My guess is that in most of them, the average forum reader will side with the moderation team.
LessWrong publishes most of their rejected posts and comments on a separate webpage. I say ‘most’ as I suspect infohazards are censored from that list. I would be interested to hear the EA forum’s moderation team’s thoughts on this approach/whether it’s something they’ve considered, should they read this and have time to respond.[1]
Creating such a page would also allow them to collect on Ngo’s bounty, since they would be answer both how much censorship they do and (assuming they attach moderation notes) why
Hi! I just want to start by clarifying that a user’s first post/comment doesn’t go up immediately while our facilitators/moderators check for spam or a clear norm violation (such as posting flame bait/clear trolling). Ideally this process takes no more than a day, though we currently don’t have anyone checking new users outside of approximately US Eastern Time business hours.
However, some content (like your first comment) requires additional back and forth internally (such as checking with moderators) and/or with the new user. This process involves various non-obvious judgement calls, which is what caused a long delay between your submitting the comment and us reaching out to you (plus the fact that many people were out over the winter holidays). In the case of your comment, we asked you to edit it and you didn’t respond to us or edit the comment for over a week, and then our facilitator felt bad for keeping you in the queue for so long so they approved your comment.
We currently do not use the rejected content feature that LW uses. Instead, almost all[1] of the content that may have been rejected under their system ends up appearing on the rest of our site, and we currently mostly rely on users voting to make content more or less visible (for example, karma affects where a post is displayed on the Frontpage). I plan to seriously consider whether we should start using the rejected content feature here soon; if so, then I expect that we’ll have the same page set up.
I think that, if we had been using the rejected content feature, the right move would have been for us to reject your comment instead of approving it.
My guess is that there are edge cases, but in practice we keep our queue clear, so my understanding is that users are typically not in limbo for more than a few days. Things like spam are not rejected — accounts that post spam are banned.
Hello!
Thanks for taking the time to respond thoroughly! I sincerely appreciate that.
I can’t quite remember when I read the message sent from the facilitator, but my memory is that it was after the comment was restored (feel free to check on your end if that’s possible). I was slightly bummed out that a comment which took some effort to write was rejected and wasn’t super motivated to respond defending it.
At the time, I was aware that the metaphor was abrasive, but hoped I had sanded off the edges by adding a disclaimer at the start. It can be difficult to balance ‘writing the thing I honestly believe’ with ‘not upset anybody or make them uncomfortable when discussing moral issues 100% of the time.’ I did hum and haw over whether I should post it, but ultimately decided that most people wouldn’t be upset by the metaphor or would even agree with it’s accuracy (given that the meat/dairy industries are both rife with animal sexual abuse). Seeing as how it was interpreted as flame bait / trolling, I somewhat regret posting it.
On a final note; am I able to ask why you would reject it? I.e. do you believe I was trolling or flame baiting? I won’t be insulted either way, but would find it useful going forward to know how I should better write my comments.
Two final notes:
• I am pleased to hear you are considering a rejected content feature.
• I used the word ‘censorship’ in my original short form post and want to underscore that I don’t think it’s intrinsically bad to censor. I.e. the moderation team should be doing some level of censorship (and I suspect most forum users would agree).
Thanks for the feedback! I think moderation is tricky and I’m relatively new at it myself. I’m sad at how long users can get stuck in the queue, and I’d love to improve how fast we resolve moderation questions, but where exactly we draw these lines will probably be a learning process for me, and we’ll continue to iterate on that.
It looks like you submitted the comment on Dec 17, and our facilitator messaged you on Jan 6 (the delay partly being due to people being out for the holidays), and then they approved your comment a little over a week after messaging you. Yeah I agree that this was an edge case, and I don’t think you were being malicious, but I think you could have made your point more productively by, for example, just using “torture”.
I feel that using the rejected content feature would give our team more leeway to be opinionated about shaping the home page of our site (compared to now), and we’d feel somewhat free to reject things that don’t fit the type of discussions we want to see. For example, it looks like LW rejects posts from new users that don’t have a clear introduction. So I think if something is an edge case in the current system, then it would likely get rejected under the other system.
Does anyone remember which EA org lets people specify in their wills that they are leaving money to effective charities?
https://www.givingwhatwecan.org/get-involved/bequests
I was thinking of a different organization, but thanks!
I recently came across Santi Ruiz from the Institute for Progress’s podcast and substack, Statecraft. I enjoyed going back through the archives and thought I’d share some of my favorites here.
Particularly Forum relevant
How to Save Twenty Million Lives
How to Secure Weapons-Grade Uranium
How to Hide the Manhattan Project
How to Salvage a Nuclear Waste Facility
How to Ban Biological Weapons
How to Catch a Lab Leak
Other standouts
How to Stage a Coup
How to Fix Crime in New York City
50 Thoughts on DOGE
How to Win an Election Against the Communists
The EU AI Act explicitly mentions “alignment with human intent” as a key focus area in relation to regulation of systemic risks.
As far as I know, this is the first time “alignment” has been mentioned by a law, or major regulatory text.
It’s buried in Recital 110, but it’s there. And it also makes research on AI Control relevant:
“International approaches have so far identified the need to pay attention to risks from potential intentional misuse or unintended issues of control relating to alignment with human intent”.
This means that alignment is now part of the EU’s regulatory vocabulary.
But here’s the issue: most AI governance professionals and policymakers still don’t know what it really means, or how your research connects to it.
I’m trying to build a space where AI Safety and AI Governance communities can actually talk to each other.
If you’re curious, I wrote an article about this, aimed at the corporate decision-makers that lack literacy on your area.
Would love any feedback, especially from folks thinking about how alignment ideas can scale into the policy domain.
Here is the Substack link (I also posted it on LinkedIn):
https://open.substack.com/pub/katalinahernandez/p/why-should-ai-governance-professionals?utm_source=share&utm_medium=android&r=1j2joa
My intuition says that this was a push from Future of Life Institute.
Thoughts? Did you know about this already?
AIxBio looks pretty bad and it would be great to see more people work on it
We’re pretty close to having a country of virologists in a data center with AI models that can give detailed and accurate instructions for all steps of a biological attack — with recent reasoning models, we might have this already
These models have safeguards but they’re trivial to overcome — Pliny the Liberator manages to jailbreak every new model within 24 hours and open sources the jailbreaks
Open source will continue to be just a few months behind the frontier given distillation and amplification, and these can be fine-tuned to remove safeguards in minutes for less than $50
People say it’s hard to actually execute the biology work, but I don’t see any bottlenecks to bioweapon production that can’t be done by a bio undergrad with limitless scientific knowledge; on my current understanding, the bottlenecks are not manual dexterity bottlenecks like playing a violin which require years of practice, they are knowledge bottlenecks
Bio supply chain controls that make it harder to get ingredients aren’t working and aren’t on track to work
So it seems like we’re very close to democratizing (even bespoke) bioweapons. When I talk to bio experts about this they often reassure me that few people want to conduct a biological attack, but I haven’t seen much analysis on this and it seems hard to be highly confident.
While we gear up for a bioweapon democracy it seems that there are very few people working on worst-case bio, and most of the people working on it are working on access controls and evaluations. But I don’t expect access controls to succeed, and I expect evaluations to mostly be useful for scaring politicians, due in part to the open source issue meaning we just can’t give frontier models robust safeguards. The most likely thing to actually work is biodefense.
I suspect that too many people working on GCR have moved into working on AI alignment and reliability issues and too few are working on bio. I suspect there are bad incentives, given that AI is the new technology frontier and working with AI is good career capital, and given that AI work is higher status.
When I talk to people at the frontier of biosecurity, I learn that there’s a clear plan and funding available, but the work is bottlenecked by entrepreneurial people who can pick up a big project and execute on it autonomously — these people don’t even need a bio background. On my current guess, the next 3-5 such people who are ambivalent about what to do should go into bio rather than AI, in part because AI seems to be more bottlenecked by less generalist skills, like machine learning, communications, and diplomacy.
I think the main reasons that EAs are working on AI stuff over bio stuff is that there aren’t many good routes into worst case bio work afaict largely due to infohazard concerns from field building, and the x-risk case for biorisk not being very compelling (maybe due to infohazard concerns around threat models).
I think these are fair points, I agree the info hazard stuff has smothered a lot of talent development and field building, and I agree the case for x-risk from misaligned advanced AI is more compelling. At the same time, I don’t talk to a lot of EAs and people in the broader ecosystem these days who are laser focused on extinction over GCR, that seems like a small subset of the community. So I expect various social effects, making a bunch more money, and AI being really cool and interesting and fast-moving are probably a bigger deal than x-risk compellingness simpliciter. Or at least they have had a bigger effect on my choices!
But insufficiently successful talent development / salience / comms is probably the biggest thing, I agree.
can you spell out the clear plan? feel free to DM me also
Yup! The highest level plan is in Kevin Esvelt’s “Delay, Detect, Defend”: use access controls and regulation to delay worst-case pandemics, build a nucleic acid observatory and other tools to detect amino acid sequences for superpandemics, and defend by hardening the world against biological attacks.
The basic defense, as per DDD, is:
Develop and distribute adequate PPE to all essential workers
Make sure the supply chain is robust to ensure that essential workers can distribute food and essential supplies in the event of a worst-case pandemic
Environmental defenses like far-UVC that massively reduce the spread and replication rate of pandemic pathogens
IMO “delay” has so far basically failed but “detect” has been fairly successful (though incompletely). Most of the important work now needs to rapidly be done on the “defend” side of things.
There’s a lot more details on this and the biosecurity community has really good ideas now about how to develop and distribute effective PPE and rapidly scale environmental defenses. There’s also now interest in developing small molecule countermeasures that can stop pandemics early but are general enough to stop a lot of different kinds of biological attacks. A lot of this is bottlenecked by things like developing industrial-scale capacity for defense production or solving logistics around supply chain robustness and PPE distribution. Happy to chat more details or put you in touch with people better suited than me if it’s relevant to your planning.
Anthropic has been getting flak from some EAs for distancing itself from EA. I think some of the critique is fair, but overall, I think that the distancing is a pretty safe move.
Compare this to FTX. SBF wouldn’t shut up about EA. He made it a key part of his self-promotion. I think he broadly did this for reasons of self-interest for FTX, as it arguably helped the brand at that time.
I know that at that point several EAs were privately upset about this. They saw him as using EA for PR, and thus creating a key liability that could come back and bite EA.
And come back and bite EA it did, about as poorly as one could have imagined.
So back to Anthropic. They’re taking the opposite approach. Maintaining about as much distance from EA as they semi-honestly can. I expect that this is good for Anthropic, especially given EA’s reputation post-FTX.
And I think it’s probably also safe for EA.
I’d be a lot more nervous if Anthropic were trying to tie its reputation to EA. I could easily see Anthropic having a scandal in the future, and it’s also pretty awkward to tie EA’s reputation to an AI developer.
To be clear, I’m not saying that people from Anthropic should actively lie or deceive. So I have mixed feelings about their recent quotes for Wired. But big-picture, I feel decent about their general stance to keep distance. To me, this seems likely in the interest of both parties.
Do you think that distancing is ever not in the interest of both parties? If so, what is special about Anthropic/EA?
(I think it’s plausible that the answer is that distancing is always good; the negative risks of tying your reputation to someone always exceed the positive. But I’m not sure.)
Arguably, around FTX, it was better. EA and FTX both had strong brands for a while. And there were worlds in which the risk of failure was low.
I think it’s generally quite tough to get this aspect right though. I believe that traditionally, charities are reluctant to get their brands associated with large companies, due to the risks/downsides. We don’t often see partnerships between companies and charities (or say, highly-ideological groups) - I think that one reason why is that it’s rarely in the interests of both parties.
Typically companies want to tie their brands to very top charities, if anyone. But now EA has a reputational challenge, so I’d expect that few companies/orgs want to touch “EA” as a thing.
Arguably influencers are a often a safer option—note that EA groups like GiveWell and 80k are already doing partnerships with influencers. As in, there’s a decent variety of smart YouTube channels and podcasts that hold advertisements for 80k/GiveWell. I feel pretty good about much of this.
Arguably influencers are crafted in large part to be safe bets. As in, they’re very incentivized to not go crazy, and they have limited risks to worry about (given they represent very small operations).
This feels different to me. In most cases, there is a cultural understanding of the advertiser-ad seller relationship that limits the reputational risk. (I have not seen the “partnerships” in question, but assume there is money flowing in one direction and promotional consideration in the other.) To be sure, activists will demand for companies to pull their ads from a certain TV show when it does something offensive, to stop sponsoring a certain sports team, or so on. However, I don’t think consumers generally hold prior ad spend against a brand when it promptly cuts the relationship upon learning of the counterparty’s new and problematic conduct.
In contrast, people will perceive something like FTX/EA or Anthropic/EA as a deeper relationship rather than a mostly transactional relationship involving the exchange of money for eyeballs. Deeper relationships can have a sense of authenticity that increases the value of the partnership—the partners aren’t just in it for business reasons—but that depth probably increases the counterparty risks to each partner.
I hope my post was clear enough that distance itself is totally fine (and you give compelling reasons for that here). It’s ~implicitly denying present knowledge or past involvement in order to get distance that seems bad for all concerned. The speaker looks shifty and EA looks like something toxic you want to dodge.
Responding to a direct question by saying “We’ve had some overlap and it’s a nice philosophy for the most part, but it’s not a guiding light of what we’re doing here” seems like it strictly dominates.
I agree.
I didn’t mean to suggest your post suggested otherwise—I was just focusing on another part of this topic.
Tax incentives for AI safety—rough thoughts
A number of policy tools such as regulations, liability regimes or export controls—aimed at tackling AI risks - have already been explored, and mostly appear as promising and worth further iterations.
But AFAIK no one has so far come up with a concrete proposal to use tax policy tools to internalize AI risks. I wonder why, considering that policies, such as tobacco taxes, R&D tax credits, and 401(k), have been mostly effective. Tax policy also seems to be underutilized and neglected, given we already possess sophisticated institutions like tax agencies or tax policy research networks.
Safety measures spending of AI Companies seems to be relatively low, and we can expect that if competition intensifies, these expenses will be even lower.
So I’ve started to consider more seriously the idea of tax incentives—basically we can provide a tax credit or deduction for expenditures on AI safety measures like alignment research, cybersecurity or oversight mechanisms etc. which effectively could lower their cost. To illustrate: AI Company incurs safety researcher salary as a cost and then 50% of that cost can be additionally deducted from the tax base.
My guess was that such tool could influence the ratio of safety-to-capability spending. If implemented properly it could help mitigate competitive pressures affecting frontier AI labs by incentivising them to increase spending on AI safety measures.
Like any market intervention, we can justify such incentives if they correct market inefficiencies or generate positive externalities. In this case, lowering the cost of security measures helps internalize risk.
However there are many problems on path to design such tool effectively:
The crucial problem is that financial benefit from tax credit can’t match the expected value of increasing capabilities. Underlying incentives for capability breakthroughs are potentially orders of magnitude larger. So simply AI labs wouldn’t bother and keep the same level while getting extra money from incentives which is an obvious backlash.
However, if some AI Company plans to increase safety expenses due to their real concerns about risks or external pressures (boards, public etc.), perhaps they would be more willing to do it.
Also risk of keeping the same safety expenses level could be overcome by requiring a specific threshold of expenditures to benefit from the incentive.
The focus here is on inputs (spending) instead of outcomes (actual safety).
Implementing it would be pain in the ass, requiring creating specialised departments within IRS or delegating most of the work to NIST.
Defining the scope of qualified expenditures - it could be hard to separate safety from capabilities research cost. Keeping an eye on this later can be a considerable administrative cost.
Expected expenses could be incurred regardless of the public funding received if we just impose a strict requirement.
There could be a problem of safety washing—AI labs creating an impression and signalling that appropriate safety measures are implemented and benefiting from incentives while not decreasing the risk effectively.
I don’t know much about US tax system but I guess it could overlap with existing R&D tax incentives. However, existing incentives are unlikely to reduce the risk. if they are used for both safety and capabilities research then they
Currently most AI labs are in loss position so they can’t effectively benefit from such incentives unless some special feature is put in place, like refundable tax credits or the option to claim such relief/credit as soon as they make a taxable profit.
Perhaps direct government financing would be more effective. Or existing ideas (such as those mentioned earlier) would be more effective and we don’t have enough room for weaker solutions.
Maybe money isn’t a problem here as AI labs are more talent constrained. If the main bottleneck for effective safety work is a talented researcher, then making safety spending cheaper via tax credits might not significantly increase the amount of high-quality safety work done.
Is there something crucial that I am missing? Is it worth investigating further? So far it has more problems than the potential benefits so I don’t think it’s promising, but I’d love to hear your thoughts on it.
11. It would probably cost a good bit of political capital to get this through, which may have an opportunity cost. You may not even get public support from the AI companies because the proposal contains an implicit critique that they haven’t been doing enough on safety.
12. By the time the legislation got out of committee and through both houses, the scope of incentivized activity would probably be significantly broader than what x-risk people have in mind (e.g., reducing racial bias). Whether companies would prefer to invest more in x-risk safety vs. other incentivized topics is unclear to me.
I saw mentioned in one discussion about Dustin Moskowitz’s exit from the Effective Altruism Forum that him quitting isn’t by itself necessarily an omen, of the relationship between Good Ventures and the effective altruism ecosystem becoming much colder than it is now. While it wouldn’t be a binary for Dustin anyway, it may seem like his is a special case with implications for the relationship between EA and Good Ventures going forward. Nonetheless, while most of them may not be billionaires, as far as I’m aware, I know there high net worth donors among effective altruists have also mentioned before that they’ve almost always tended to avoid the EA Forum. That’s probably for the same reasons any major philanthropist might. In spite of that, those donors have decently known relationships with the effective altruism ecosystem, with them remaining continually and relatively stable.
How and to what extent Dustin quitting the forum will imply decreased funding EA causes receive from Good Ventures will perhaps only become apparent later. The main indicator of that could be the nature and amount of grants made throughout the rest of the year, based on Open Philanthropy’s recommendations. It may be likely there’s a positive correlation between the two outcomes of Dustin quitting the EA Forum, and Good Ventures support for a variety of projects affiliated/aligned with effective altruism potentially declining. Yet how exactly how strong that correlation may be is hardly certain, and many effective altruists may currently be overestimating it.
We’ve just released the updated version of our structural alternative to dark matter: the Central Tensional Return Hypothesis (CTRH).
This version includes:
High-resolution, multi-galaxy CTR model fits
Comparative plots of CTR acceleration vs Newtonian gravity
Tension-dominance domains (zero-crossing maps)
Escape velocity validation using J1249+36
Structural scaling comparisons via the CTR “b” parameter
https://forum.effectivealtruism.org/posts/LA4Ma5NMALF3MQmvS/updated-structural-validation-of-the-central-tensional?utm_campaign=post_share&utm_source=link
We welcome engagement, critique, and comparative discussion with MOND or DM-based models.
What’s the most cost-effective economic growth-boosting intervention? It’s cat mascots. I just learned about Tama the calico cat (via @thatgoodnewsgirl on Instagram), who “gained fame for being a railway station master and operating officer at Kishi Station on the Kishigawa Line in Kinokawa, Wakayama Prefecture, Japan”.
The career section of her Wikipedia page astounded me:
That’s actually lowballing Tama. If you click on footnote 9, it’s ¥1.38 billion ($13.1 million at the exchange rate back then):
In contrast, Tama was really cheap. Gemini 2.5 Pro helpfully estimated the following cost breakdown, which adds up to a median of ¥110,000:
A year’s supply of food probably cost ¥30,000 - ¥60,000
Vet care could range from ¥30,000 - ¥70,000 annually to cover everything from routine check-ups to vaccinations, flea/tick prevention, and potential unexpected health issues
Litter, bedding, toys, and grooming supplies probably cost ¥10,000 - ¥20,000
Her promotion to “super station master” (superintendent equivalent) came with an office (a converted ticket booth containing a litter box) estimated at ¥50,000 - ¥110,000, amortised over Tama’s nearly decade-long career
Tama’s two custom-made hats (summer and other seasons) weren’t cheap, probably ¥10,000 - ¥30,000 each, but worth including in the compensation package to retain top talent
In contrast her two gold-plated name tags (the first was stolen by a visitor) likely weren’t as expensive, ¥2,000 - ¥5,000
This works out to almost exactly 10,000:1 benefit-cost ratio just from increased ridership, and that’s excluding merchandise, TV appearance and other publicity etc — well beyond even Open Phil’s higher-than-ever 2,100x bar!
In fact, a few hundred thousand Tamas would already double global economic growth, and the world’s Tama-carrying capacity is probably 3-4 OOMs higher at least (the World Population Review’s estimate of over a billion cats worldwide today should be interpreted as a lower bound). Talk about feline-powered explosive economic growth potential…
Tama’s office
I would imagine there are some replicability issues…
Love the post 🤩
Cats’ economic growth potential likely has a heavy-tailed distribution, because how else would cats knock things off shelves with their tail. As such, Open Philanthropy needs to be aware that some cats, like Tama, make much better mascots than other cats. One option would be to follow a hits-based strategy: give a bunch of areas cat mascots, and see which ones do the best. However, given the presence of animal welfare in the EA movement, hitting cats is likely to attract controversy. A better strategy would be to identify cats that already have proven economic growth potential and relocate them to areas most in need of economic growth. Tama makes up 0.00000255995% of Japan’s nominal GDP (or something thereabouts, I’m assuming all Tama-related benefits to GDP occurred in the year 2020). If these benefits had occurred in North Korea, they would be 0.00086320506% of nominal GDP or thereabouts. North Korea is also poorer, so adding more money to its economy goes further. Japan and North Korea are near each other, so transporting Tama to North Korea would be extremely cheap. Assuming Tama’s benefits are the same each year and are independent of location (which seems reasonable, I asked ChatGPT for an image of Tama in North Korea and it is still cute), catnapping Tama would be highly effective. One concern is that there might be downside risk, because people morally disapprove of kidnapping cats. On the other hand, people expressing moral disapproval of kidnapping cats are probably more likely to respect animal’s boundaries by not eating meat, thus making this an intervention that spans cause areas. In conclusion: EA is solved, all we have to do is kidnap some cats.
A few thoughts. First, it is a really cute story, and I’m glad you shared it. It feels very Japanese.
Second, marketing and tourism aren’t often considered as major areas for economic development and growth (at least not in the popular press books I’ve read or the EA circles I’ve been in), but this is a simple little case study to demonstrate that having a mascot (or anything else that people like, from fancy buildings to locations tied to things people like) can drive economic activity. But it is also hard to predict in advance what will be a hit. I bet that lots of places have beautiful murals, cute animals, historical importance, lovely scenery, and similar attractions without having much of a positive return on investment. For me, the notable think about Tama’s story is how little money was needed to add something special to the local station. A lot of investments related to tourism are far more expensive.
A final thought, one that maybe folks more versed in economics can help me with. Should we consider this an example of economic growth? Is this just shifting spending/consumption from one place to another? Would people who spent money to ride this train otherwise would have spent that money doing something else: riding a different train, visiting a park, etc.
I love that there is a disagree react: “hmm… no, seems like the most cost-effective economic growth boosting intervention is not in fact cat mascots”
If this were a story, there’d be some kind of academy taking in humanity’s top talent and skilling them up in alignment.
Most of the summer fellowships seem focused on finding talent that is immediately useful. And I can see how this is tempting given the vast numbers of experienced and talented folks seeking to enter the space. I’d even go so far as to suggest that the majority of our efforts should probably be focused on finding people who will be useful fairly quickly.
Nonetheless, it does seem as though there should be at least one program that aims to find the best talent (even if they aren’t immediately useful) and which provides them with the freedom to explore and the intellectual environment in which to do so.
I wish I could articulate my intuition behind this clearer, but the best I can say for now is that my intuition is that continuing to scale existing fellowships would likely provide decreasing marginal returns and such an academy wouldn’t be subject to this because it would be providing a different kind of talent.
Am I wrong that EAs working in AI (safety, policy, etc.) and who are now earning really well (easily top 1%) are less likely to donate to charity?
At least in my circles, I get the strong impression that this is the case, which I find kind of baffling (and a bit upsetting, honestly). I have some just-so stories for why this might be the case, but I’d rather hear others’ impressions, especially if they contradict mine (I might be falling prey to confirmation bias here since the prior should be that salary correlates positively with likelihood of donating among EAs regardless of sector).
There are at least three common justifications for not donating, each of which can be quite reasonable:
A high standard of living and saving up money are important selfish wants for EAs in AI, just as they are in broader society.
EAs in AI have needs (either career or personal) that require lots of money.
Donations are much lower impact than one’s career.
I don’t donate to charity other than animal product offsets; this is mainly due to 1 and 2. As for 1, I’m still early career enough that immediate financial stability is a concern. Also for me, forgoing luxuries like restaurant food and travel makes me demotivated enough that I have difficulty working. I have tried to solve this in the past but have basically given up and now treat these luxuries as partially needs rather than wants.
For people just above the top-1% threshold of $65,000, 3 and 2 are very likely. $65,000 is roughly the rate paid to marginal AI safety researchers, so donating 20% will bring only 20% of someone’s career impact even if the grantmakers find an opportunity as good as themself. If they also live in a HCOL area, 2 is very likely—in San Francisco the average rent for a 1bed is $2,962/month and an individual making less than $104,000 qualifies for public housing assistance!
But shouldn’t I have more dedication to the cause and donate anyway? I would prefer to instead spend more effort on getting better at my job (since I’m nowhere near the extremely high skillcap of AI safety research) and working more hours (possibly in ways that funge with donations eg by helping out grantmakers). I actually do care about saving for retirement, and finding a higher-paying job at a lab safety team just so I can donate is probably counterproductive, because trying to split one’s effort between two theories of change while compromising on both is generally bad (see the multipliers post). If I happened to get an equally impactful job that paid double, I would probably start donating after about a year, or sooner if donations were urgent and I expected high job security.
Most EAs want to be rich and close to power. Or at least they are way more into the “effective” optimization part than the altruism. They talk a big game but getting in early on a rising power (AI companies) is not altruistic. Especially not when you end up getting millions in compensation due to very rapid valuation increases.
I made a large amount of money in the 2021 crypto bom. I made a much smaller, though large for me, amount in the 2017 crash. I have never had a high paying job. Often I have had no job at all. My longterm partner has really bad health. So I’m perhaps unusually able to justify holding onto windfalls. I still gave away 50% pre-tax both times.
Most eas are simply not the real deal.
Its relatively common (I don’t know about rates) for such people to take pay-cuts rather than directly donate that percentage. I know some who could be making millions a year who are actually making hundreds. It makes sense they don’t feel the need to donate anything additional on top of that!
It’s not clear to me whether you’re talking about people who (a) do a voluntary salary sacrifice while working at an EA org, or (b) people who could have earned much more in industry but moved to a nonprofit so now earn much less than their hypothetical maximum earning potential.
In case (a), yes, their salary sacrifice should count towards their real donations.
But I think a practical moral philosophy wherein donation expectations are based on your actual material resources (and constraints), not your theoretical maximum earning potential, seems more justifiable. So I don’t think people who do (b) (which includes myself) should get to say that doing (b) liberates them from the same obligation to donate that would attend to a person in the same material circumstances with worse outside options.
I think the right stance here is a question of “should EA be praising such people or get annoyed they’re not giving up more if it wants to keep a sufficient filter for who it calls true believers”, and the answer there is obviously both groups are great & true believers and it seems dumb to get annoyed at either.
The 10% number was notably chosen for these practical reasons (there is nothing magic about that number), and to back-justify that decision with bad moral philosophy about “discharge of moral duty” is absurd.
I’m not going to defend my whole view here, but I want to give a though experiment as to why I don’t think that “shadow donations”—the delta between what you could earn if you were income-maximizing, and what you’re actually earning in your direct work job—are a great measure for the purposes of practical philosophy (though I agree they’re both a relevant consideration and a genuine sacrifice).
Imagine two twins, Anna and Belinda. Both have just graduated with identical grades, skills, degrees, etc. Anna goes directly from college work on AI safety at Safety Org, making $75,000 / year. Belinda goes to work for OpenMind doing safety-neutral work, making $1M per year total compensation. Belinda learns more marketable skills; she could make at least $1M / year indefinitely. Anna, on the other hand, has studiously plugged away at AI safety work, but since her work is niche, she can’t easily transfer these skills to do something that pays better.
Then imagine that, after three years, Belinda joins Anna at Safety Org. Belinda was not fired; she could have stayed at OpenMind and made $1M per year indefinitely. At this point, Anna has gotten a few raises and is making $100,000, and donating 3% of her salary. Belinda gets the same job on the same pay scale, and does equally good work, but donates nothing. Belinda reasons that, because she could still be $1M per year, she has “really” donated $900,000 of labor to Safety Org, and so has sacrificed roughly 90% of her income.
Anna, on the other hand, thinks that it is an immense privilege to be able to have a comfortable job where she can use her skills to do good, while still earning more than 99% of all people in the world. She knows that, if she had made different choices in life, she probably could have a higher earning potential. But that has never been her goal in life. Anna knows that the average person in her income bracket donate around 3% regardless of their outside job options, so it seems reasonable for her to at least match that.
Is Belinda more altruistic than Anna? Which attitude should EAs aspire to?
To give some more color on my general view:
I don’t really think there’s a first-order fact of the matter as to who of these two (or anyone) is “more altruistic,” or what one’s “obligations” are. At bottom, there are just worlds with more or less value in them.
My view mostly comes from a practical view of how the EA community and project can be most impactful, credible, and healthy. I think the best attitude is closer to Anna’s than Belinda’s.
Donating also has other virtues over salary reductions, since it is concrete, measurable, and helps create a more diversified funding ecosystem.
To be clear, I think it’s great that people like Belinda exist, and they should be welcomed and celebrated in the community. But I don’t think the particular mindset of “well I have really sacrificed a lot because if I was purely selfish I could have made a lot more money” is one that we ought to recognize as particularly good or healthy.
I will note that my comment made no reference to who is “more altruistic”. I don’t know what that term means personally, and I’d rather not get into a semantics argument.
If you give the definition you have in mind, then we can argue over whether its smart to advocate that someone ought to be more altruistic in various situations, and whether it gets at intuitive notions of credit assignment.
I will also note that given the situation, its not clear to me Anna’s proper counterfactual here isn’t making $1M and getting nice marketable skills, since she and Belinda are twins, and so have the same work capacity & aptitudes.
I think this is the crux personally. This seems very healthy to me, in particular because it creates strong boundaries between the relevant person and EA. Note that burnout & overwork is not uncommon in EA circles! EAs are not healthy, and (imo) already give too much of themselves!
Why do you think its unhealthy? This seems to imply negative effects on the person reasoning in the relevant way, which seems pretty unlikely to me.
Suppose they’re triplets, and Charlotte, also initially identical, earns $1M/year just like Belinda, but can’t/doesn’t want to switch to safety. How much of Charlotte’s income should she donate in your worldview? What is the best attitude for the EA community?
I didn’t read Cullen’s comment as about 10%, and I think almost all of us would agree that this isn’t a magic number. Most would probably agree that it is too demanding for some and not demanding enough for others. I also don’t see anything in Cullen’s response about whether we should throw shade at people for not being generous enough or label them as not “true believers.”
Rather, Cullen commented on “donation expectations” grounded in “a practical moral philosophy.” They wrote about measuring an “obligation to donate.”
You may think that’s “bad moral philosophy,” but there’s no evidence of it being a post hoc rationalization of a 10% or other community giving norm here.
I disagree and think that b is actually totally sufficient justification. I’m taking as an assumption that we’re using an ethical theory that says people do not have an unbounded ethical obligation to give everything up to subsistence and that it is fine to set some kind of a boundary and fraction of your total budget of resources that you spend on altruistic purposes. Many people doing well paying altruistic careers (eg technical AI safety careers) could earn dramatically more money eg at least twice as much, if they were optimising for the highest paying career they could. I’m fairly sure I could be earning a lot more than I currently am if that was my main goal. But I consider the value of my labour from an altruistic perspective to exceed the additional money I could be donating and therefore do not see myself to have a significant additional ethical obligation to donate (though I do donate a fraction of my income anyway because I want to)
By foregoing a large amount of income for altruistic reasons, I think such people are spending a large amount of their resource budget on altruistic purposes, and that if they still have an obligation to donate more money that people in higher paying careers should be obliged to donate far more. Which is a consistent position, but not one I hold
I think I want to give (b) partial credit here in general. There may not be much practical difference between partial and full credit where the financial delta between a more altruistic job and a higher-salary job is high enough. But there are circumstances in which it might make a difference.
Without commenting on any specific person’s job or counterfactuals, I think it is often true that the person working a lower-paid but more meaningful job secures non-financial benefits not available from the maximum-salary job and/or avoids non-financial sacrifices associated with the maximum-salary job. Depending on the field, these could include lower stress, more free time, more pleasant colleagues, more warm fuzzies / psychological satisfaction, and so on. If Worker A earns 100 currency units doing psychologically meaningful, low to optimal stress work but similarly situated Worker B earns 200 units doing unpleasant work with little in the way of non-monetary benefits, treating the entire 100 units Worker A forewent as spent out of their resource budget on altruistic purposes does not strike a fair balance between Worker A and Worker B.
Yeah this is fair
I don’t want to argue in anyone’s specific case, but I don’t think it’s universally true at all or even true the majority of the time that people that those working in AI could make more elsewhere. It sounds nice to say, but I think often people are earning more in AI jobs than they would elsewhere .
My reasoning was roughly that the machine learning skill set is also extremely employable in finance which tends to pay better. though openai salaries do get pretty high nowadays and if you value openai and anthropic equity at notably above their current market value, then plausibly, they’re higher paying. Definitely agreed it’s not universal.
Sure. But the average person working in AI is not at Jane St level like you and yes, OpenAI/Anthropic comp is extremely high.
I would also say that people still have a moral obligation. People don’t choose to be smart enough to do ML work.
My point is that, even though there’s a moral obligation, unless you think that high earning people in finance should be donating a very large fraction of their salary (so their post donation pay is less than the pay in AI safety), their de facto moral obligation has increased by the choice to do direct work, which is unreasonable to my eyes.
I would also guess that at least most people doing safety work at industry labs could get a very well paying role at a top tier finance firm? The talent bar is really high nowadays
I also want to point out that having better outside income-maximizing options makes you more financially secure than other people in your income bracket, all else equal, which pro tanto would give you more reason to donate than them.
My point is that “other people in the income bracket AFTER taking a lower paying job” is the wrong reference class.
Let’s say someone is earning $10mn/year in finance. I totally think they should donate some large fraction of their income. But I’m pretty reluctant to argue that they should donate more than 99% of it. So it seems completely fine to have a post donation income above $100K, likely far above.
If this person quits to take a job in AI Safety that pays $100K/year, because they think this is more impactful than their donations, I think it would be unreasonable to argue that they need to donate some of their reduced salary, because then their “maximum acceptable post donation salary” has gone down, even though they’re (hopefully) having more impact than if they donated everything above $100K
I’m picking fairly extreme numbers to illustrate the point, but the key point is that choosing to do direct work should not reduce your “maximum acceptable salary post donations”, and that at least according to my values, that max salary post donation is often above what they get paid in their new direct role.
I understand this. Good analogy.
I suppose what it comes down to is that I actually DO think it is morally better for the person earning $10m/year to donate $9.9m/year than $9m/year, about $900k/year better.
I want to achieve two things (which I expect you will agree with).
I want to “capture” the good done by anyone and everyone willing to contribute and I want them welcomed, accepted and appreciated by the EA community. This means that if a person who could earn $10m/year in finance and is “only” willing to contribute $1m/year (10%) to effective causes, I don’t want them turned away.
I want to encourage, inspire, motivate and push people to do better than they currently are (insofar as it’s possible). I think that includes an Anthropic employee earning $500k/year doing mech interp, a quant trader earning $10m/year, a new grad deciding what to do with their career and a 65-year old who just heard of EA.
I think it’s also reasonable for people to set limits for how much they are willing to do.
This is reasonable. I think the key point that I want to defend is that it seems wrong to say that choosing a more impactful job should mean you ought to have a lower post donation salary.
I personally think of it in terms of having some minimum obligation for doing your part (which I set at 10% by default), plus encouragement (but not obligation) to do significant amounts more good if you want to
Is there a single appropriate reference class here, as opposed to looking at multiple reference classes and weighting the results in some manner?
I agree that similarly situated person who decided to take a very high-paying job is a relevant reference class and should get some weight. However, it doesn’t follow that person with similar incomes working a non-impactful job is an irrelevant reference class or should get zero weight.
As Marcus notes, “[p]eople don’t choose to be smart enough to do ML work.” I would add that people don’t choose other factors that promote or inhibit their ability to choose a very high-paying job and/or a high-impact job (e.g., location and circumstances of birth, health, family obligations, etc.) In a pair of persons who are similarly situated economically, giving the more advantaged person a total pass on the moral obligation to donate money seems problematic to me. In this frame of reference, their advantages allowed them to land a more impactful job at the same salary as the less advantaged person—and in a sense we would be excusing them from a moral obligation because they are advantaged. (Giving the more privileged person a big break is also going to make it rather hard to establish substantial giving as a norm in the broader community, but that’s probably not in the scope of the question here.)
I don’t have a clear opinion on how to weight the two reference classes beyond an intuition that both classes should get perceptible weight. (It also seems plausible there are other reference classes to weigh as well, although I haven’t thought about what they might be.)
My argument is essentially that “similar income, non impactful job” is as relevant a reference class to the “similar income, impactful job person” as it is as a reference class to the “high income, non impactful job” person. I also personally think reference classes is the wrong way to think about it. If taking a more impactful job also makes someone obliged to take on a lower post donation salary (when they don’t have to), I feel like something has gone wrong, and the incentives are not aligned with doing the most good.
It’s complicated, I think. Based on your distinguishing (a) and (b), I am reading “salary sacrifice” as voluntarily taking less salary than was offered for the position you encumber (as discussed in, e.g., this post). While I agree that should count, I’m not sure (b) is not relevant.
The fundamental question to me is about the appropriate distribution of the fruits of one’s labors (“fruits”) between altruism and non-altruism. (Fruits is an imperfect metaphor, because I mean to include (e.g.) passive income from inherited wealth, but I’ll stick with it.)
We generally seem to accept that the more fruit one produces, the more (in absolute terms) it is okay to keep for oneself. Stated differently—at least for those who are not super-wealthy—we seem to accept that the marginal altruism expectation for additional fruits one produces is less than 100%. I’ll call this the “non-100 principle.” I’m not specifically defending that principle in this comment, but it seems to be assumed in EA discourse.
If we accept this principle, then consider someone who was working full-time in a “normal” job and earn a salary of 200 apples per year. They decide to go down to half-time (100-apple salary) and spend the half of their working hours producing 100 charitable pears for which they receive no financial benefit. [1]The non-100 principle suggests that it’s appropriate for this person to keep more of their apples than a person who works full-time to produce 100 apples (and zero pears). Their total production is twice as high, so they aren’t similarly situated to the full-time worker who produces the same number of apples. The decision to take a significantly less well-paid job seems analogous to splitting one’s time between remunerative and non-remunerative work. One gives up the opportunity to earn more salary in exchange for greater benefits that flow to others by non-donation means.
I am not putting too much weight on this thought experiment, but it does make me think that either the non-100 principle is wrong, or that the foregone salary counts for something in many circumstances even when it is not a salary sacrifice in the narrower sense.
How to measure pear output is tricky. The market rate for similar work in the for-profit sector may be the least bad estimate here.
I feel quite confused about the case where someone earns much less than their earning potential in another altruistically motivated but less impactful career doing work that uses a similar skillset (e.g. joining a think tank after working on policy at an AI company). This seems somewhere between A and B.
I’d consider this a question that doesn’t benefit from public speculation because every individual might have a different financial situation.
Truth be told “earning really well” is a very ambiguous category. Obviously, if someone were financially stable, eg. consistently earning high 5 figure or six figure dollars/euros/pounds/francs or more annually(and having non-trivial savings) and having a loan-free house, their spending would almost always reflect discretionary interests and personal opinions (like ’do I donate to charity or not”).
For everyone not financially stable, ‘donating to charity’ may not:
(a) be a discretionary decision and
(b) be a simple decision—that is, increasing charitable donations too soon comes at the expense of not investing in one’s personal ability to weather volatility, which has knock-on qualitative effects on career progression (especially to senior management roles), future earnings potential and lifetime charitable contributions. Additionally, not getting one’s personal finances in order early on contributes directly to great personal and family stress, which then has knock on effects on everything else.
tl;dr: when you’re broke, money allocation is a high-risk, high-stress headache. The long term solution is to prioritize becoming the opposite of broke, i.e; financially stable, first.
also see: Julia Wise’s post on the logistics of giving.
That analysis would be more compelling if the focus of the question were on a specific individual or small group. But, at least as I read it, the question is about the giving patterns of a moderately numerous subclass of EAs (working in AI + “earning really well”) relative to the larger group of EAs.
I’m not aware of any reason the dynamics you describe would be more present in this subclass than in the broader population. So a question asking about subgroup differences seems appropriate to me.
Edit: I see your point. Still, I’ll leave the below comment as-is, because from my (3rd world, generational financial instability, no health insurance, filial obligations etc.) point of view I think the perspective of a broke person ought to be represented.
But what counts as “numerous”, though? How many EAs are actually working in AI—fifty people? A hundred people? Who’s collecting data on this subgroup versus the larger EA group?
I agree that the question itself is appropriate and there’s nothing wrong with it. I was saying this question doesn’t benefit from public speculation, because, for one thing, there isn’t any reliable data for objective analysis, and for another, the logistics of an individual’s personal finance are a bigger factor in how or how much a person donates, at the non-millionaire level (in this subclass and the broader population).
My experience from the church is the salary doesn’t correlate will with likelihood of donating, although it does of course correlate with donating larger amounts of money.
If EAs working in AI policy and safety were serious about AI Doom being a near-term possibility, I would expect they would donate huge amounts towards that cause. A clear case of “revealed preferences” not just stated ones.
I think I was assuming people working in highly paid AI jobs were donating larger percentages of their income, but I haven’t seen data in either direction?
Yes, though I thought maybe among EAs there would be some correlation. 🤷
Yeah, me neither (which, again, is probably true; just not in my circles).
Update: New Version Released with Illustrative Scenarios & Cognitive Framing
Thanks again for the thoughtful feedback on my original post Cognitive Confinement by AI’s Premature Revelation.
I’ve now released Version 2 of the paper, available on OSF: 📄 Cognitive Confinement by AI’s Premature Revelation (v2)
What’s new in this version?
– A new section of concrete scenarios illustrating how AI can unintentionally suppress emergent thought
– A framing based on cold reading to explain how LLMs may anticipate user thoughts before they are fully formed
– Slight improvements in structure and flow for better accessibility
Examples included:
A student receives an AI answer that mirrors their in-progress insight and loses motivation
A researcher consults an LLM mid-theorizing, sees their intuition echoed, and feels their idea is no longer “theirs”
These additions aim to bridge the gap between abstract ethical structure and lived experience — making the argument more tangible and testable.
Feel free to revisit, comment, or share. And thank you again to those who engaged in the original thread — your input helped shape this improved version.
Japanese version also available (PDF, included in OSF link)
This post proposes a structural alternative to dark matter called the Central Tensional Return Hypothesis (CTRH). Instead of invoking unseen mass, CTRH attributes galactic rotation to directional bias from a radially symmetric tension field. The post outlines both a phenomenological model and a field-theoretic formulation, and invites epistemic scrutiny and theoretical engagement.
I’m interested in figuring out “what skill profiles are most leveraged for altruistic work after we get the first competent AI agents”?
I think this might be one of the most important questions for field-building orgs to work on (in any cause area but particularly AI safety). I think 80k and other AIS-motivated careers groups should try to think hard about this question and what it means for their strategy going forward.
I’m optimistic about productivity on the question of “what skill profiles are most leveraged after we get the first competent AI-agents” because:
few people have actually tried to think about what managing an AI workforce would look like
this doesn’t feel conceptually abstract (e.g. some people have already adopted LMs in their day-to-day workflows)
something like drop-in replacements for human remote workers are more likely than AIs less analogous to human workers if timelines are short (and if timelines are short, this work is more urgent than if timelines are long)
One important consideration is whether human managers are going to be good proxies for AI-agent managers. It’s plausible to me that the majority of object-level AI safety work that has ever be done will be done by AI agents. It could be that the current status quo persists, where people’s leverage routes through things like leadership and management skills, or it could be that ~everyone will become more leveraged proportional to their access to AI models (roughly their budget) due to AI agents having different affordances than humans, or something else.
For example, suppose that AI agents are much more tolerant, hardworking, and understanding of user intent for short tasks than human workers. I expect that having good strategic takes would become much more leveraged and having good internal stakeholder management skills would become much less leveraged (relative to today). If 80k (for example) thought that was likely, then maybe they should double down on finding people with great strategy takes and care less about typical management skills.
Maybe “junior” roles will be automated before “senior” roles almost uniformly across the economy. In that case, it’s even more valuable than it is right now to focus on getting “senior” people.
Or maybe coding will be automated before everything else, and for some reason people’s relative leverage inside the company stays pretty fixed, or it’s too hard to say which kinds of jobs will be most leveraged at software companies. Then “coding” still becomes much more leveraged per dollar than today, and it’s probably useful to find more value-aligned coders.
It would be surprising to me if the price of certain kinds of intellectual labour decreased by 100x, and this had little impact on people’s relative leverage.
(disclaimer: I’m not sure whether the above stories actually go through, they are meant to just be illustrative of the kind of thinking that seems undersupplied)