I graduated from Georgetown University in December, 2021 with degrees in economics, mathematics and a philosophy minor. There, I founded and helped to lead Georgetown Effective Altruism. Over the last few years recent years, I’ve interned at the Department of the Interior, the Federal Deposit Insurance Corporation, and Nonlinear.
Blog: aaronbergman.net
I strongly endorse this and think that there are some common norms that stand in the way of actually-productive AI assistance.
People don’t like AI writing aesthetically
AI reduces the signal value of text purportedly written by a human (i.e. because it might have been trivial to create and the “author” needn’t even endorse each claim in the writing)
Both of these are reasonable but we could really use some sort of social technology for saying “yes, this was AI-assisted, you can tell, I’m not trying to trick anyone, but also I stand by all the claims made in the text as though I had done the token generation myself.”
I think I’m more bullish on digital storage than you.
Most alignment work today exists as digital bits: arXiv papers, lab notes, GitHub repos, model checkpoints. Digital storage is surprisingly fragile without continuous power and maintenance.
SSDs store bits as charges in floating-gate cells; when unpowered, charge leaks, and consumer SSDs may start losing data after a few years. Hard drives retain magnetic data longer, but their mechanical parts degrade; after decades of disuse they often need clean-room work to spin up safely. Data centres depend on air-conditioning, fire suppression, and regular maintenance.
In a global collapse where grids are down for years, almost all unmaintained digital archives eventually succumb to bit-rot, corrosion, fire, or physical decay.
This is true but the fundamental value proposition of digital storage is extremely cheap, and easy (~free on both fronts for a few GB of PDFs) replication. So it’s true that a single physical device by default won’t last very long, but the data itself simply (“simply”—I mean the crux is how hard/common this will be) needs to hop from device to device (possibly and ideally existing on many devices at any one time) indefinitely.
By analogy you might think that human aging/death poses a major problem for the continued existence of the information necessary to recreate a human (i.e. DNA + a few other sources of info) but in fact the ability for people to replicate makes destruction of that knowledge much, much harder.
Of course a key consideration is how bad the setback is. At the limit you’re right because our ability to create the tools for info replication won’t exist, but at least naively it might take quite a bit to destroy all of society’s ability to manufacture flash drives and computers for long enough for all info to die out via the mechanisms you describe.
Wanted to bring this comment thread out to ask if there’s a good list of AI safety papers/blog posts/urls anywhere for this?
(I think local digital storage in many locations probably makes more sense than paper but also why not both)
Lightcone and Alex Bores (so far)
Edit: to say a tiny bit more, LessWrong seems instrumentally good and important and rationality is a positive influence on EA. Lightcone doesn’t have the vibes of “best charity” to me, but when I imagine my ideal funding distribution it is the immediate example of “most underfunded org” that comes to mind. Obviously related to Coefficient not supporting rationality community building anymore. Remember, we are donating on the margin, and approximately the margin created by Coefficient Giving!
Edit: And Animal Welfare Fund
Super cool—a bit hectic and I substantively disagree with one of the “fallacies” the fallacy evaluator flagged on this post but I’ll definitely be using this going forward
Thanks for the highlight! Yeah I would love better infrastructure for trying to really figure out what the best uses of money are. I don’t think it has to be as formal/quantitative as GiveWell. To quote myself from a recent comment (bolding added)
At some level, implicitly ranking charities [eg by donating to one and not another] is kind of an insane thing for an individual to do—not in an anti-EA way (you can do way better than vibes/guessing randomly) but in a “there must be better mechanisms/institutions for outsourcing donation advice than GiveWell and ACE and ad hoc posts/tweets/etc and it’s really hard and high stakes” way.
Like what I would love is a lineup of 10-100 very highly engaged and informed people (could create the list simply by number of endorsements/requests) who talk about their strategy and values in a couple pages and then I just defer to them (does this exist?)
I did something related but haven’t updated it in a couple years! If there’s a good collection of AI safety papers/other resources/anything anywhere it would be very easy for me to add it to the archive for people to download locally, or else I could try to collect stuff myself
1. ClusterFree
2. Center for Reducing Suffering
3. Arthropoda Foundation
4. Shrimp Welfare Project
5. Effective Altruism Infrastructure Fund
6. Forethought Foundation
7. Wild Animal Initiative
8. Center for Wild Animal Welfare
9. Animal Welfare Fund
10. Aquatic Life Institute
11. Longview Philanthropy’s Emerging Challenges Fund
12. Legal Impact for Chickens
13. The Humane League
14. Rethink Priorities
15. Centre for Enabling EA Learning & Research
16. MATS Research
I used AI for advice (unlike last year) with Claude-Opus-4.5 and Gemini-3-Pro-Preview. I didn’t take either of their suggested rankings of course, but they both gave me a pretty decent starting point to work with such that I’m pretty sure I’d endorse either’s list as a marginal improvement to the vote distribution
Both were prompted with a list of my values and takes in addition to roughly 200k tokens scraped from the relevant posts list.
More substantially, my final list was probably based on a combination of not-really-all-that-amazing-but-better-than-nothing heuristics: what I feel like EA at large is under-funding, what I would be most excited to see, what tentatively aligns with my literally endorsed beliefs about the nature of suffering, what cause areas make theoretical sense to be near the Pareto frontier.
Honestly I think I would (will?) almost certainly adjust my list if I look(ed) into it for just a few additional hours
At some level, implicitly ranking charities [eg by donating to one and not another] is kind of an insane thing for an individual to do—not in an anti-EA way (you can do way better than vibes/guessing randomly) but in a “there must be better mechanisms/institutions for outsourcing donation advice than GiveWell and ACE and ad hoc posts/tweets/etc and it’s really hard and high stakes” way.
Like what I would love is a lineup of 10-100 very highly engaged and informed people (could create the list simply by number of endorsements/requests) who talk about their strategy and values in a couple pages and then I just defer to them (does this exist?)
I do not accept premise 2:
For some small amount of intense suffering, there is always some sufficiently large amount of moderate suffering such that the intense suffering is preferable.
To be clear, I think this premise is one way of distilling and clarifying the (or ‘a’) crux of my argument and if I wind up convinced that the whole argument is wrong, it will probably be because I am convinced of premise 2 or something very similar
Wow, this is super exciting and thanks so much to the judges! ☺️
An interesting dynamic around this competition was that the promise of the extremely cracked + influential judging team reading (and implicitly seriously considering) my essay was a much stronger incentive for me to write/improve it than the money (which is very nice don’t get me wrong).[1]
I’m not sure what the implications of this are, if any, but it feels useful to note this explicitly as a type of incentive that could be used to elicit writing/research in the future
Insofar as I’m not totally deluding myself, I mean the altruistic impact of some chance of shaping the judges’ views as opposed to the possibility of seeming mildly clever to some high status figures
Interesting, thanks! I might actually sign up for the Arctic Archive thing! I don’t see you mention m-discs like this—any reason for that?
Also, do you have any takes on how many physical locations a typical X is stored in, for various X?
X could be:
A wikipedia page
An EA Forum post
A YouTube video
A book that’s sold 100/1k/10k/100k/1M copies
Etc
After thinking about this post (“Utilitarians Should Accept that Some Suffering Cannot be “Offset””) some more, there’s an additional, weaker claim I want to emphasize, which is: You should be very skeptical that it’s morally good to bring about worlds you wouldn’t personally want to experience all of
We can imagine a society of committed utilitarians all working to bring about a very large universe full of lots of happiness and, in an absolute sense, lots of extreme suffering. The catch is that these very utilitarians are the ones that are going to be experiencing this grand future—not future people or digital people or whomever.
Personally, they all desperately want to opt out of this—nobody wants to actually go through the extreme suffering regardless of what benefits await, and yet all work day in and day out to bring about this future anyway, condemning themselves to heaven and to hell.
Of course it’s not impossible that their stated morals are right and preferences are morally wrong, but my claim is that we should be very skeptical that the “skin in the game” answer is the wrong one.
And of course here I’m just supposing/assuming that these utilitarians have the preference to avoid this massive future altogether
The point, again, is that you should probably not work to build a future that you yourself would not want to experience just because some abstract arguments say that this is morally good This is a strong hint (not definitive, but strong) that those abstract arguments are wrong.
I disagree but think I know what you’re getting at and am sympathetic. I made the following to try to illustrate and might add it in to the post if it seems clarifying
I made it on a whim just now without thinking too hard so don’t necessarily consider the graphical representation on as solid footing as the stuff in the post
Thanks!
(Ideal): annihilation is ideally desirable in the sense that it’s better (in expectation) than any other remotely realistic alternative, including <detail broadly utopian vision here>. (After all, continued existence always has some chance of resulting in some uncompensable suffering at some point.)
Yeah I mean on the first one, I acknowledge that this seems pretty counterintuitive to me but again just don’t think it is overwhelming evidence against the truth of the view.
Perhaps a reframing is “would this still seem like a ~reductio conditional on a long reflection type scenario that results in literally everyone agreeing that it’s desirable/good?”
And I don’t mean this in the sense of just “assume that the conclusion is ground truth”—I mean it in the sense of “does this look as bad when it doesn’t involve anyone doing anything involuntary?” to try to tease apart whether intuitions around annihilation per se are to any extent “just” a proxy for guarding against the use of force/coercion/lack of consent.
Another way to flip the ‘force’ issue would be “suppose a society concludes unanimously including via some extremely deliberative process (that predicts and includes the preferences of potential future people) that annihilation is good and desired. Should some outside observer forcibly prevent them taking action to this end (assume that the observer is interested purely in ethics and doesn’t care about their own existence or have valenced experience)?”
I’ll note that I can easily dream up scenarios where we should force people, even a whole society, to do something against its will. I know some will disagree, but I think we should (at least in principle, implementation is messy) forcibly prevent people from totally voluntarily being tortured (assume away masochism—like suppose the person just has a preference for suffering that results in pure suffering with no ‘secretly liking it’ along for the ride)
(Uncompensable Monster): one being suffering uncompensable suffering at any point in history suffices to render the entire universe net-negative or undesirable on net, no matter what else happens to anyone else. We must all (when judging from an impartial point of view) regret the totality of existence.
These strike me as extremely incredible claims, and I don’t think that most of the proposed “moderating factors” do much to soften the blow.
This one I more eagerly bite the bullet on, it just straightforwardly seems true to me that this is possible in principle (i.e., such a world could/would be genuinely very bad). And relevantly, orthodox utilitarianism also endorses this in principle, some of the time (i.e. just add up the utils, in principle one suffering monster can have enough negative utility)!
Moral uncertainty might help if it resulted in the verdict that you all things considered should prefer positive-utilitarian futures (no matter their uncompensable suffering) over annihilation. But I’m not quite sure how moral uncertainty could deliver that verdict if you really regard the suffering as uncompensable. How could a lower degree of credence in ordinary positive goods rationally outweigh a higher degree of credence in uncompensable bads? It seems like you’d instead need to give enough credence to something even worse: e.g. violating an extreme deontic constraint against annihilation. But that’s very hard to credit, given the above-quoted case where annihilation is “obviously preferable”.)
I don’t have an answer to this (yet) because my sense is just that figuring out how to make overall assessments of probability distributions on various moral views is just extremely hard in general and not “solved”.
This actually reminds me of a shortform post I wrote a while back. Let me just drop a screenshot to make my life a bit easier in terms of formatting nested quotes:
I think this^ brief discussion of how the two sides might look at the same issue gets at the fundamental problem/non-obviousness of the matter pretty well.
I think a more promising form of suffering-focused ethics would explore some form of “variable value” approach, which avoids annihilationism in principle by allowing harms to be compensated (by sufficient benefits) when the alternative is no population at all, but introduces variable thresholds for various harms being specifically uncompensable by extra benefits beyond those basic thresholds. I’m not sure whether a view of this structure could be made to work, but it seems more worth exploring than pro-annihilationist principles.
I think we may just have very different background stances on like how to do ethics. I think that we should more strongly decouple the project of abstract object level truth seeking from the project of figuring out a code of norms/rules/de facto ethics that satisfies all our many constraints and preferences today. The thing you propose seems promising to me as like a coalitional bargaining proposal for guiding action in the near future, but not especially promising as a candidate for abstract moral truth.
Thanks, yeah I may have gotten slightly confused when writing.
Wikipedia screenshot:
Let P be the thing I said in the post:
If A ≻ B ≻ C, there’s some probability p ∈ (0, 1) where a guaranteed state of the world B is ex ante morally equivalent to “lottery p·A + (1-p)·C”
or, symbolically
and let
I think but not in general.
So my writing was sloppy. Super good catch (not caught by any of the various LLMs iirc!)
But for the purposes of the argument everything holds together because you need independence axiom for VNM to hold. But still, sloppy.
Me: “This arbitrariness diminishes somewhat (though, again, not entirely) when viewed through the asymptotic structure. Once we accept that compensation requirements grow without bound as suffering intensifies, some threshold becomes inevitable. The asymptote must diverge somewhere; debates about exactly where are secondary to recognizing the underlying pattern.”
You:
“Grow without bound” just means that for any M, we have f(X) > M for sufficiently large X. This is different from there being a vertical asymptote so a threshold is not inevitable. For instance one could have f(X) = X or f(X) = X^2.
Straightforward error by me, I will change the wording. Not sure how that happened
It would be confusing to call this behavior continuous, because (a) the VNM axiom you reject is called continuity and (b) we are not using any other properties of the extended reals, but we are using real-valued probabilities and x values.
Yeah, idk, English and math only provide so many words. I could have spent more words more driving home and clarifying this point or inventing and defining additional terms. My intuition is that it’s clear enough as is (evidently we disagree about this) but if a couple other people say “yeah this is misleading and confusing” then I’ll concede that I made a bad choice about clarity vs brevity as a writing decision.
Ngl I am pretty confused about everything starting here. I think I’m just reading you wrong somehow. Like the difference in those magnitudes is huge, point taken, but I don’t see why that matters for my argument.
Moving from 10^10^10 to infinity, we would then believe that suffering has a threshold t where t + epsilon intensity suffering cannot be offset by removing t—epsilon intensity suffering
Confused here because yeah clearly adding t+epsilon and removing t-epsilon gives you a net change below zero. But I sense you might be getting at the (very substantive and important) cluster of critiques I respond to in this comment (?)
also need to propose some other mechanism like lexicographic order for how to deal with suffering above the infinite badness threshold.
Yeah I’m ~totally agnostic about this in the post. There are many substantively different possibilities about what the moral world might be like when dealing above that threshold, I agree! Could be distinct levels of lexicality, perhaps some literal integer like 13 levels or perhaps arbitrarily many. Probably other solutions/models as well
Maybe I should just remove/modify the “write down”, “compute”, and “physically instantiate” bit of rhetorical flourish because it might be doing more harm than good.
(Note that it may take me some time to update the post to reflect sections 1 and 2 in this comment)
Again, sharp eye, thanks for the comment!
I am getting really excellent + thoughtful comments on this (not just saying that) - I will mention here that I have highly variable and overall reduced capacity for personal reasons at the moment, so please forgive me if it takes a little while for me to respond (and in the mean time note that I have read stuff so they’re not being ignored) 🙂
Great points both and I agree that the kind of tradeoff/scenario described by @EJT and @bruce in his comment are the strongest/best/most important objections to my view (and the thing most likely to make me change my mind)
Let me just quote Bruce to get the relevant info in one place and so this comment can serve as a dual response/update. I think the fundamentals are pretty similar (between EJT and Bruce’s examples) even though the exact wording/implementation is not:
A) 70 years of non-offsettable suffering, followed by 1 trillion happy human lives and 1 trillion happy pig lives, or
B) [70 years minus 1 hour of non-offsettable suffering (NOS)], followed by 1 trillion unhappy humans who are living at barely offsettable suffering (BOS), followed by 1 trillion pig lives that are living at the BOS,
You would prefer option B here. And it’s not at all obvious to me that we should find this deal more acceptable or intuitive than what I understand is basically an extreme form of the Very Repugnant Conclusion, and I’m not sure you’ve made a compelling case for this, or that world B contains less relevant suffering.
to which I replied:
Yeah not going to lie this is an important point, I have three semi-competing responses:
I’m much more confident about the (positive wellbeing + suffering) vs neither trade than intra-suffering trades. It sounds right that something like the tradeoff you describe follows from the most intuitive version of my model, but I’m not actually certain of this; like maybe there is a system that fits within the bounds of the thing I’m arguing for that chooses A instead of B (with no money pumps/very implausible conclusions following)
Well the question again is “what would the IHE under experiential totalization do?” Insofar as the answer is “A”, I endorse that. I want to lean on this type of thinking much more strongly than hyper-systematic quasi-formal inferences about what indirectly follows from my thesis.
I think it’s possible that the answer is just B because BOS is just radically qualitatively different from NOS.
Maybe most importantly I (tentatively?) object to the term “barely” here because under the asymptotic model I suggest, the value of subtracting arbitrarily small amount of suffering instrument from the NOS state results in no change in moral value at all because (to quote myself again) “Working in the extended reals, this is left-continuous: ”
So in order to get BOS, we need to remove something larger than , and now it’s a quasi-empirical question of how different that actually feels from the inside. Plausibly the answer is that “BOS” (scare quotes) doesn’t actually feel “barely” different—it feels extremely and categorically different
Consider “which of these responses if any is correct” a bit of an open question for me.
Plausibly I should have figured this out before writing/publishing my piece but I’ve updated nontrivially (though certainly not all the way) towards just being wrong on the metaphysical claim.
This is in part because after thinking some more since my reply to Bruce (and chatting with some LLMs), I’ve updated away from my points (1) and (2) above.
I am still struggling with (3) both at:
the conceptual level of whether it could be the case that there are fundamental qualitative discontinuities corresponding to the asymptote location at arbitrarily small but not infinitesimal (!) changes in i_s; and
the quasi-empirical level of whether that’s actually how things are
Mostly (2) though, I should add. I think (uncertain/tentative etc etc) that this is conceptually on the table.
So to respond to Ben:
I interpret OP’s point about asymptotes to mean that he indeed bites this bullet and believes that the “compensation schedule” is massively higher even when the “instrument” only feels slightly worse?
I don’t bite the bullet in the most natural reading of this, where very small changes in i_s do only result in very small changes in subjective suffering from a subjective qualitative POV. Insofar as that is conceptually and empirically correct, I (tentatively) think it’s a counterexample that more or less disproves my metaphysical claim (if true/legit).
But I feel pretty conflicted right now about whether the small but not infinitesimal change in i_s → subjectively small difference is true (again, mostly because of quasi-empirical uncertainty).
This is hard to think about largely because my model/view leaves the actual shape of the asymptote unspecified (here’s a new version of the second pic in my post), and that includes all the uncertainty associated with what instrument we are literally or conceptually talking about (since the sole criterion is that it’s monotonic)[1]
I will add that one reason I think this might be a correct “way out” is that it would just be very strange to me if “IHE preference is to refuse 70 year torture and happiness trade mentioned in post” logically entails (maybe with some extremely basic additional assumptions like transitivity) “IHE gives up divine bliss for a very small subjective amount of suffering mitigation”
I know that this could just be a failure of cognition and/or imagination on my part. Tbh this is really the thing that I’m trying to grok/wrestle with (as of now, like for the last day or so, not in the post)
I also know this is ~motivated reasoning, but idk I just do think it has some evidential weight. Hard to justify in explicit terms though.
I’m curious if others have different intuitions about how weird/plausible this [2] is from a very abstract POV
Mostly for fun I vibecoded an API to easily parse EA Forum posts as markdown with full comment details based on post URL (I think helpful mostly for complex/nested comment sections where basic copy and paste doesn’t work great)
I have tested it on about three posts and every possible disclaimer applies
Again I appreciate your serious engagement!
The positive argument for the metaphysical claim and the title of this piece relies (IMO) too heavily on a single thought experiment, that I don’t think supports the topline claim as written.
Not sure what you mean by the last clause, and to quote myself from above:
I don’t expect to convince all readers, but I’d be largely satisfied if someone reads this and says: “You’re right about the logic, right about the hidden premise, right about the bridge from IHE preferences to moral facts, but I would personally, both in real life and as an IHE, accept literally anything, including a lifetime of being boiled alive, for sufficient happiness afterward.”
Yeah it’s possible I should (have) emphasized this specific thesis (“IHE thought experiment, I claim, is an especially epistemically productive way of exploring that territory, and indeed for doing moral philosophy more broadly”) more as an explicit claim, distinct from the two I highlight as the organizing/motivating claims corresponding to each section. Maybe I will add a note or something about this.
I don’t have a rock solid response to the “too heavily” thing because, idk, I think the thought experiment is actually what matters and what corresponds to the true answer. And I’ll add that a background stance I have is that I’m trying to convey what think is the right answer, not only in terms of explicit conclusions but in terms of what evidence matters and such.
A) 70 years of non-offsettable suffering, followed by 1 trillion happy human lives and 1 trillion happy pig lives, or
B) [70 years minus 1 hour of non-offsettable suffering (NOS)], followed by 1 trillion unhappy humans who are living at barely offsettable suffering (BOS), followed by 1 trillion pig lives that are living at the BOSYou would prefer option B here. And it’s not at all obvious to me that we should find this deal more acceptable or intuitive than what I understand is basically an extreme form of the Very Repugnant Conclusion, and I’m not sure you’ve made a compelling case for this.
Yeah not going to lie this is an important point, I have three semi-competing responses:
I’m much more confident about the (positive wellbeing + suffering) vs neither trade than intra-suffering trades. It sounds right that something like the tradeoff you describe follows from the most intuitive version of my model, but I’m not actually certain of this; like maybe there is a system that fits within the bounds of the thing I’m arguing for that chooses A instead of B (with no money pumps/very implausible conclusions following)
Well the question again is “what would the IHE under experiential totalization do?” Insofar as the answer is “A”, I endorse that. I want to lean on this type of thinking much more strongly than hyper-systematic quasi-formal inferences about what indirectly follows from my thesis.
I think it’s possible that the answer is just B because BOS is just radically qualitatively different from NOS.
Maybe most importantly I (tentatively?) object to the term “barely” here because under the asymptotic model I suggest, the value of subtracting arbitrarily small amount of suffering instrument from the NOS state results in no change in moral value at all because (to quote myself again) “Working in the extended reals, this is left-continuous: ”
So in order to get BOS, we need to remove something larger than , and now it’s a quasi-empirical question of how different that actually feels from the inside. Plausibly the answer is that “BOS” (scare quotes) doesn’t actually feel “barely” different—it feels extremely and categorically different
Consider “which of these responses if any is correct” a bit of an open question for me.
And I’ll add that insofar as the answer is (2) and NOT 3, I’m pretty inclined to update towards “I just haven’t developed an explicit formalization that handles both the happiness trade case and the intra-suffering trade case yet” more strongly than towards “the whole thing is wrong, suffering is offsetable by positive wellbeing”—after all, I don’t think it directly follows from “IHE chooses A” that “IHE would choose the 70 years of torture.” But I could be wrong about this! I 100% genuinely think I’m literally not smart enough to intuit super confidently whether or a formalization that chooses both A and no torture exists. I will think about this more!
Thought experiment variations:
People’s intuitions about the suffering/bliss trade might reasonably change based on factors like:
Duration of suffering (70 minutes vs. 70 years vs. 70 billion years)
Whether experiences happen in series or parallel
Whether you can transfer the bliss to others
I agree (1) offers interesting variations. I do have a vague, vibey sense that one human lifetime seems like a pretty “fair” central case to start from but this is not well-justified.
I more strongly want to push back on (2) and (3) in the sense that I think parallel experience, while probably conceptually fine in principle, really greatly degrades the epistemic virtue of the thought experiment because this literally isn’t something human brains were/are designed to do or simulate. And likewise with (3), the self interest bit seems pretty epistemically important.
Threshold problem:
Formalizing where the lexical threshold sits is IMO pretty important, because there are reasonable pushbacks to both, but they feel like meaningfully different views
High threshold (e.g.,”worst torture”) leads to unintuitive package deals where you’d accept vast amounts of barely-offsettable suffering (BOS) to avoid small amounts of suffering that does cross the threshold
Low threshold (e.g., “broken hip” or “shrimp suffering”) seems like it functionally becomes negative utilitarianism
I agree it is imporant! Someone should figure out the right answer! Also in terms of practical implementation, probably better to model as a probability distribution than a single certain line.
Asymptotic compensation schedule:
The claim that compensation requirements grow asymptotically (rather than linearly, or some other way) isn’t well-justified, and doesn’t seem to meaningfully change the unintuitive nature of the tradeoffs your view is willing to endorse.
I disagree that it isn’t well-justified in principlle, but maybe I should have argued this more thoroughly. It just makes a ton of intuitive sense to me but possibly I am typical-minding. And I’m pretty sure you’re wrong about the second thing—see point 3 a few bullets up. It seems radically less plausible to me that the true nature of ethics involves discontinuous i_s vs i_h compensation schedules.
Ok lol your comment is pretty long so I think I will need to revisit the rest of it! Some vibes are likely to include:
“I literally don’t know what the threshold is. I agree it would be nice to formalize it! My uncertainty isn’t much evidence against the view as a whole”
I feel like we agree about continuity; the asymptote seems very intuitively like the most likely way to connect “paper cuts are acceptable to create in exchange for wellbeing” and “the IHE would refuse the torture trade”. I agree it’s not literally impossible that a discontinuous model is correct
I suspect this intended to be illustrative, but I would be surprised if there were many, if any standard utilitarians who would actually say that you need TREE(3)[5] flourishing human life years to offset a cluster headache lasting 1 hour, so this seems like a strawman?
Yes illustrative, I’m not trying to claim that these people actually exist. I don’t think it’s a strawman. Maybe this is a grammatical thing with my use of the word the to refer to a hypothetical person
New interview with Will MacAskill by @MHR🔸
Almost a year after the 2024 holiday season Twitter fundraiser, we managed to score a very exciting “Mystery EA Guest” to interview: Will MacAskill himself.
@MHR🔸 was the very talented interviewer and shrimptastic fashion icon
Thanks to @AbsurdlyMax🔹 for help behind the scenes
And of course huge thanks to Will for agreeing to do this
Summary, highlights, and transcript below video!
Summary and Highlights
(summary AI-generated)
Effective Altruism has changed significantly since its inception. With the arrival of “mega donors” and major institutional changes, does individual effective giving still matter in 2025?
Will MacAskill—co-founder of the Centre for Effective Altruism and Giving What We Can, and currently a senior research fellow at the Forethought Institute—says the answer is a resounding yes. In fact, he argues that despite the resources currently available, individuals are “systematically not ambitious enough” relative to the scale of the problems the world faces.
In this special interview for the 2025 EA Twitter Fundraiser, Will joins host Matt to discuss the evolution of the movement’s focus. They discuss why animal welfare—specifically the fight against factory farming—has risen in prominence relative to global health, and why Will believes those working on it are “on the right side of history.”
Will also shares updates from his current work at the Forethought Institute, where he is moving beyond standard AI safety concerns to focus on “model character”—the idea that as AI agents become more autonomous, their embedded ethics and personality will determine how our economy and society function.
Matt and Will discuss:
Why “mega donors” haven’t made individual giving obsolete
The “founder effect” that initially prioritized global health over animal welfare
The funniest moment from the What We Owe the Future media tour (involving Tyler Cowen)
Why Forethought is focused on the moral status of digital beings
Will’s call for the community to embrace an “impartial, altruistic scout mindset” on social media
Highlights
On earning to give:
On current, most exciting work at Forethought:
On what he’d tell EA Twitter:
Transcript
MATT
Hello everyone and welcome. I’m Matt from the 2025 EA Twitter Fundraiser. I’m here with Will MacAskill himself, who likely needs no introduction for folks watching this. But in case anyone does, Will is a senior research fellow at the Forethought Institute and former associate professor at Oxford University. He co-founded Giving What We Can and the Centre for Effective Altruism. He is the author of numerous books and articles, including Doing Good Better, What We Owe the Future, and my personal favorite, An Introduction to Utilitarianism. Thank you for joining us, Will.
WILL
Thanks for the interview.
MATT
We’re doing this in service of promoting effective giving. To start us off, we were curious to ask: What do you think the role of effective giving should be in the EA community in 2025? And maybe in particular, should people be more ambitious with trying to earn to give?
WILL
Yes. I think of effective giving as a core part of Effective Altruism. It was the start of it all, and it continues to be important. It continues to be important even if you see “mega donors” coming on the scene, because there are often a lot of things that individuals can fund that the mega donors can’t. And even taking the mega donors into account, there are still enormous problems in the world that they are not going to be able to solve.
Then, should people be more ambitious? I genuinely think yes. I think people systematically aren’t ambitious enough, so the answer is almost always yes. Again, the ambition you have should match the scale of the problems that we’re facing—and the scale of those problems is very large indeed.
MATT
Absolutely. One of those big problems—and the one that we’re raising money for in this year’s online fundraiser—is specifically farmed animal welfare. That’s something that has been really popular with highly engaged EAs recently. If you look at, for example, the donation election on the Forum, you’ll see there’s really a lot of enthusiasm around that. However, it is something that has historically played a smaller role in public EA messaging. Do you think that’s a good balance, or is that something that you would like to see change?
WILL
I definitely think that there was never some grand plan of reducing the focus on animal welfare. I and some others really got in via global health and development, so I think that really set the stage—there is a bit of a founder effect there. And then I think it’s also just the case that maybe there is naturally wider reach because more people are concerned [about human-centric causes].
But yes, I’d be down for that to change. The problem of factory farming is just so stark and so unnecessary. The arguments for [addressing] it are so good.
I remember when I became vegetarian twenty years ago—I’m getting old now—I got a lot of pushback. It was quite a weird thing to do. People thought it was “holier than thou.” It was like you were morally grandstanding or judging others at the time. That has changed. Now, regarding the people who are vegetarian or vegan who really care about animal welfare, it’s just clear that these are the good people. They are on the right side of history.
The amount of good that you can do is just absolutely enormous; the sheer amount you can affect is huge. So, yes, I’d love to see that rise in prominence.
MATT
Very cool. When I was doing some research for this, I was looking back at old versions of the 80,000 Hours and Giving What We Can websites on the Wayback Machine. I was surprised that the first-ever archive of the 80,000 Hours page has farmed animal welfare as one of the issues on there. So, you mentioned the founder effect, but I was pleasantly surprised to see just how early this was getting into EA messaging.
WILL
We all cared. I was vegetarian. Little known fact: I and others at 80,000 Hours helped to set up Animal Charity Evaluators.
MATT
Yeah, I remember reading that at some point.
WILL
That was because we felt it was a gap at that point—that was in 2012 or something.
MATT
Well, one of the ways that we are trying to promote the fundraiser is leaning into EA memes. That’s certainly something that’s always been fun about the online EA community. Do you have a favorite EA meme that you’ve run across?
WILL
I feel I have many favorites. One is the April Fools’ posts naming Giving What We Can “Naming What We Can.”
I also feel it’s a shame that Dustin Moskovitz is no longer memeing quite as hard as he used to. But something he did… there was a big debate on Reddit where people were complaining about Wytham Abbey. There was a debate in the comments between who bought it—who owned it. One person was saying it was Will MacAskill, and the other was saying it was Owen Cotton-Barratt—both of which are incorrect. And then someone came in and said, “What if it’s neither of those people, but some mystery third person?”
That commenter, like many threads down into this Reddit post, was Dustin Moskovitz himself, who actually bought it. So, I salute him. Maybe not a meme per se, but part of the same culture.
MATT
Maybe if we lean in, we can get him to participate a little bit more again.
WILL
Maybe.
MATT
You did quite the media blitz for What We Owe the Future. Was there a funniest or most unexpected story you have from that whirlwind tour?
WILL
The thing that leaps out the most is when I was being interviewed by Tyler Cowen. Podcasts normally have a bit of a chat and get a bit of a warm-up. Tyler just immediately goes in: “Okay, so what is something you like that’s ineffective?”
I just couldn’t think of an answer. I went completely blank. So this gets edited from the podcast, but there were minutes of me just being like… [silence]. It kind of threw me off for the whole episode, really. It’s not that I only like maximally efficient things, to be clear!
MATT
I sometimes listen to Conversations with Tyler, and yes, he sometimes has that energy where you really feel like he should let the guest take a breather.
WILL
That was right at the start.
MATT
Well, Will, I know you’ve been doing some really exciting work at Forethought these days. Do you want to tell us a little bit about the work that’s going on there and maybe what you’re most excited about?
WILL
Sure. Forethought in general is trying to do research that is as useful as possible in helping the world as a whole navigate the transition to extremely advanced AI—superintelligence. Distinctively, we tend to not focus on the risk of loss of control to AI systems themselves. Our work often hits on that or helps on that, and it’s a big risk—we’re glad that other people are working on it—but I personally don’t think that makes up the vast majority of the issues we face.
We are often looking at neglected issues other than that. That can include:
The risk of intense concentration of power among humans.
The idea that we should be taking seriously the moral status of digital beings themselves and what kind of rights they have.
A number of other issues, like how AI will interact with society’s ability to reason well and make good decisions.
Maybe the thing I’ll emphasize most at the moment, because I’m very excited about it from an impact perspective but I think people don’t appreciate it quite enough, is the idea of “model character” or AI personality.
There is now OpenAI, which has this “Model Spec,” and then Claude’s “soul” was leaked. Basically, these companies are having to make decisions about how these AIs behave. That affects hundreds of millions of people every day already. And then in the future, it will determine how our most active economic agents in the world act.
There’s a real spectrum that you can imagine between AI that is as maximally like a tool as possible—just an obedient servant—versus an AI that actually has its own sense of ethics and own conscience. Once you’ve got that, then take every single hairy ethical issue or important circumstance the AIs will face; we really should be thinking in advance about how we want them to behave. I’d be keen for a lot more thought to be going into that.
MATT
Absolutely. That seems incredibly important. All right, well to close, is there anything else you want to tell the people of EA Twitter?
WILL
The main thing is just that I have been loving what I’ve been seeing as a resurgence in EA activity online—on Substack and Twitter in particular. If you think I haven’t been following along with popcorn on the Andy Masley data center water usage story, you would be wrong.
Honestly, I just love seeing more people really coming out to bat for EA ideas, demonstrating EA ideas on all sorts of topics—whether that’s data center usage or the utter monstrosity of current shrimp factory farming. So I would just love for more people to feel empowered to be shouting about the giving they’re doing, the causes that they’re working on, and demonstrating the impartial, altruistic scout mindset that I think is EA at its best.
MATT
Awesome. Well, you heard it here first, folks: post more.
WILL
And thanks for the fundraiser; thanks to everybody who has contributed. It’s very, very cool to see this.
Matt
Yes, we definitely are really grateful for everyone who has donated so far and who will continue to donate, because the animal welfare fund is awesome and supporting effective animal interventions is a really, really fantastic cause.
All right. Well, thank you, Will, for joining us. It’s been a pleasure speaking with you.
WILL
It’s been a joy. Cool. Merry Christmas, Happy Holidays.
MATT
Merry Christmas.