Arepo

• Arepo 3 Apr 2026 4:48 UTC

  7 points

  1 ∶ 1

  in reply to: richard_ngo’s comment on: What is in­te­gral al­tru­ism?

  >I think it’s historically pretty incorrect that the grounding in cost-effectiveness is what made EA good

  FWIW the reasons you’re giving here are closely related to the reasons why I’m sceptical that modern AI-focused EA is in fact as good. I don’t think it’s unreasonable to support AI safety work, but I think it’s throwing away most of the epistemics that could make EA a long-term robustly positive influence. EA’s original tagline used to be ‘using evidence and reason’, but the extreme AI safety focus seems to drop the ‘evidence’ part.

  To believe you should focus on AI safety, you need to believe all of

  • short timelines

  • either

    • no trend of convergence between intelligence and morality or

    • the view that convergence wouldn’t matter or wouldn’t be enough to avoid moral disaster

  • either

    • long timelines on other GCRs or

    • other GCRs not really mattering to humanity’s long term prospects and

      • zero discounting on future people

  • that a flourishing human future is +EV

  • that trying to improve average welfare in a flourishing future is less good than trying to increase the probability of a flourishing future

  • reasonable confidence that AI safety work has learned from its past mistakes and will be reliably +EV

  • there won’t be a sufficient public shift towards AI safety to make it low enough leverage that less

  • you personally have more comparative advantage working on AI safety than any other cause

  and surely some further assumptions I’ve missed, and many ways to further unpack these premises. To advocate work on AI safety as the primary EA cause you need to believe that the final bullet applies to the majority of your audience.

  But I think there’s plenty in that list of assumptions that’s easy to disagree with, and a lot of entangled assumptions whose entanglement to my knowledge hasn’t really been explored (e.g. I find it hard to credit both that there’s no convergence between intelligence and morality and that there’s a long term equilibrium which is both stable and in some nontrivial sense positive or desirable).

  So I semiagree with @MichaelDickens’s original comment’s in-principle scepticism while wondering whether in practice int/​a might end up promoting causes that feel closer to the what I view as the original spirit of the movement.

  There’s also some practical concerns in the OP that I think EA has dropped the ball on, such as building the sort of real community that would have retained greater support/​membership over the years (my impression is the substantial majority of EAs who joined the movement more than 6 or 7 years ago have largely disengaged with it).

  So I guess I’m noncommittally hopeful that this becomes something valuable—and remains, and Euan said, symbiotic with EA. If it just gives people who would have been somewhat supportive but felt too constrained a way to stay engaged with an encouraging community, that seems like it could be high value.

• Arepo 19 Mar 2026 1:50 UTC

  3 points

  0 ∶ 0

  in reply to: Arepo’s comment on: GHD dis­cus­sion here is slowly dying

  I’m confused by the strong negative reaction to this comment. I guess it’s about the CoGi funding, which does sound like I was wrong. But it seems to be true that there’s no option to directly apply for funding for a new project (NickLaing mentions the GH funding circle, but they completed one round last year and their website doesn’t currently imply there would be any more).

  I think this helps explain the decline of GHD in the OP—AIM’s charity list notwithstanding, no-one in the movement is incentivised to come up with practical ideas in the field.

• Arepo 17 Mar 2026 8:12 UTC

  2 points

  0 ∶ 10

  in reply to: Henry Howard🔸’s comment on: GHD dis­cus­sion here is slowly dying

  Last I heard it was something like 10% of their GCR budget.

  It’s also basically impossible to apply for GHD funding. I recently decided to put my money where my mouth is and get involved in an early stage GHD project, but there’s basically no EA-aligned funder who’s willing to let you approach them.

  SFF are exclusively longtermist, EA GHD as mentioned basically shut down, and Givewell and CoGi don’t accept unsolicited applications. So as far as I can see if you think you have an idea in the GHD space and need funding for it you basically have to look outside the EA world (someone tell me if I missed something!)

• Arepo 17 Mar 2026 5:28 UTC

  2 points

  0 ∶ 1

  on: Crit­i­cisms of What We Owe The Future

  It seems like, considering how intelligent and creative our species is, we should expect that, even in very dire conditions, we would be able to re-build civilization.

  That shouldn’t necessarily be the primary concern. Though it also seems that people who’ve studied our ability to rebuild civilisation are substantially more pessimistic.

• Arepo 17 Mar 2026 5:21 UTC

  10 points

  5 ∶ 1

  on: GHD dis­cus­sion here is slowly dying

  I think the simple answer is that it’s become less prioritised by the central orgs (the EA GHD fund is on indefinite hiatus, GHD is a diminishing part of CoGi’s budget, 80k moved away from it almost entirely, Rethink seem to have shifted towards animal welfare, CEA seem to have an increasingly longtermist/​AI focus, etc). This gives a top-down cultural impetus away from the subject, and just means there’s less money in it.

  It’s also, for better or worse, as an evidence-oriented field, a subject that’s harder to have amateur conversations about. I’ve been consistently supportive of it in my time here, but have had very little to contribute to conversations about what actually works, and felt that there was little value in contributing to any others.

  I would love to see this reverse—I think EA is much richer for spanning multiple cause areas, and especially those which are well-evidenced. I don’t have any good solutions though :\

• Arepo 12 Mar 2026 2:52 UTC

  13 points

  4 ∶ 4

  in reply to: lilly’s comment on: Effec­tive Altru­ism Will Be Great Again

  I agree that the average college student encountering EA today should focus on issues related to AI safety

  I broadly nodded along to your OP, but strong disagree here. There are tonnes of people working in AI safety, to the extent that it’s already hypercompetitive and the marginal value of one more person getting in the long queue for such a job seems very low.

  Meanwhile I continue to find the case for AI safety, as least as envisioned by EA doomers, highly speculative. That’s not to say it shouldn’t get any attention, but there’s a far better evidenced path from e.g. ‘nuclear bombs or major pandemics cause the fall of civilisation’ than from ‘LLMs cause the fall of civilisation’.

  And if you’re sufficiently pessimistic on the doomer narrative, we’re all screwed and there’s likely at least as much EV in short-term improvement of the lives of existing beings as in fighting an impossible struggle to prevent AGI from ever being developed. So there’s a credence window in which AI safety as top priority belongs. That window might be reasonably wide, but I don’t think it’s anywhere near wide enough to justify abandoning all other causes.

• Arepo 21 Jan 2026 11:17 UTC

  6 points

  3 ∶ 0

  on: Every­thing is ne­glected

  I’m still highly sceptical of neglectedness as anything but a first-pass heuristic for how prioritisation organisations might spend their early research—firstly because there are so many ways a field can be ‘not neglected’ and still highly leveraged (e.g. Givewell and Giving What We Can were only able to have the comparative impact they did because the global health field had been vigorously researched but no-one had systemically done the individual-level prioritisation they did with the research results); secondly because it encourages EA to reject established learning in a way I find dangerously hubristic (FTX weren’t irresponsible—they just took a neglected approach to fundraising!).

  If we must keep using this heuristic, it helps to introduce supporting heuristics like the one you mention; to which I’d add ‘look at the amount of input relative to the amount of input needed to solve the issue’. Climate has had far more input than AI safety, but it’s unclear to me whether the proportion of input it’s had to what it needs is higher.

• Arepo 20 Jan 2026 15:29 UTC

  2 points

  0 ∶ 0

  in reply to: Nathan Young’s comment on: Ex­ag­ger­at­ing the risks (Part 20: AI 2027 timelines fore­cast, bench­marks and gaps) | Reflec­tive Altruism

  Sure, so we agree?

  Ah, sorry, I misunderstood that as criticism.

  Do you think that forecasting like this will hurt the information landscape on average?

  I’m a big fan of the development e.g. QRI’s process of making tools that make it increasingly easy to translate natural thoughts into more usable forms. In my dream world, if you told me your beliefs it would be in the form of a set of distributions that I could run a monte carlo sim on, having potentially substituted my own opinions if I felt differently confident than you (and maybe beyond that there’s still neater ways of unpacking my credences that even better tools could reveal).

  Absent that, I’m a fan of forecasting, but I worry that overnormalising the naive I-say-a-number-and-you-have-no-idea-how-I-reached-it-or-how-confident-I-am-in-it form of it might get in the way of developing it into something better.

• Arepo 3 Jan 2026 12:31 UTC

  34 points

  5 ∶ 3

  in reply to: Nathan Young’s comment on: Ex­ag­ger­at­ing the risks (Part 20: AI 2027 timelines fore­cast, bench­marks and gaps) | Reflec­tive Altruism

  I weakly disagree here. I am very much in the “make up statistics and be clear about that” camp.

  I’m sympathetic to that camp, but I think it has major epistemic issues that largely go unaddressed:

  • It systemically biases away from extreme probabilities (it’s hard to assert < than ${10}^{-3}$, for e.g., but many real-world probabilities are and post-hoc credences look like they should have been below this)

  • By focusing on very specific pathways towards some outcome, it diverts attention towards easily definable issues, and hence away from the prospects of more complex pathways of causing the same or value-equivalent outcomes.^[1]

  • It strongly emphasises point credence estimates over distributions, the latter of which are IMO well worth the extra effort, at least whenever you’re broadcasting your credences to the rest of the world.

  By the way, I find this a strange remark:

  Seems like a lot of specific, quite technical criticisms.

  This sounds like exactly the sort of criticism that’s most valuable of a project like this! If their methodology were sound it might be more valuable to present a more holistic set of criticisms and some contrary credences, but David and titotal aren’t exactly nitpicking syntactic errors—IMO they’re finding concrete reasons to be deeply suspicious of virtually every step of the AI 2027 methodology.

  1. ^

     For e.g. I think it’s a huge concern that the EA movement have been pulling people away from nonextinction global catastrophic work because they focused for so long on extinction being the only plausible way we could fail to become interstellar, subject to the latter being possible. I’ve been arguing for years now, that the extinction focus is too blunt a tool, at least for the level of investigation the question has received from longtermists and x-riskers.

• Arepo 28 Dec 2025 19:26 UTC

  3 points

  0 ∶ 0

  in reply to: Karla Still 🔸’s comment on: Some re­flec­tions on cre­at­ing an Effec­tive Cowork­ing Dis­cord Server

  Confirmed, there’s still people around :)

• Arepo 26 Dec 2025 11:48 UTC

  2 points

  0 ∶ 0

  in reply to: William_MacAskill’s comment on: Rerun­ning the Time of Perils

  I would strongly push back on the idea that a world where it’s unlikely and we can’t change that is uninteresting. In that world, all the other possible global catastrophic risks become far more salient as potential flourishing-defeaters.

• Arepo 26 Dec 2025 11:42 UTC

  2 points

  0 ∶ 0

  in reply to: Denkenberger🔸’s comment on: Rerun­ning the Time of Perils

  Thanks for the shout-out :) If you mentally replace the ‘multiplanetary’ state with ‘post-AGI’ in this calculator, I do think it models the set of concerns Will’s talking about here pretty well.

• Arepo 24 Dec 2025 13:22 UTC

  4 points

  0 ∶ 0

  on: An­nounc­ing RoastMyPost: LLMs Eval Blog Posts and More

  Thanks Ozzie! I’ll definitely try this out if I ever finish my current WIP :)

  Questions that come to mind:

  • Will it automatically improve as new versions of the underlying model families are released?

  • Will you be actively developing it?

  • Feature suggestion: could/​would you add a check for obviously relevant literature and ‘has anyone made basically this argument before’?

• Arepo 9 Dec 2025 19:22 UTC

  8 points

  0 ∶ 0

  in reply to: Mjreard’s comment on: How I hope the EA com­mu­nity will re­spond to the AI bub­ble popping

  A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well.

  Not necessarily. It depends on

  a) your credence distribution of TAI after this decade,

  b) your estimate of annual risk per year of other catastrophes, and

  c) your estimate of the comparative longterm cost of other catastrophes.

  I don’t think it’s unreasonable to think, for example, that

  • there’s a very long tail to when TAI might arrive, given that its prospects of arriving in 2-3 decades are substantially related to to its prospects of arriving this decade) it arriving this decade (e.g. if we scale current models substantially and they still show no signs of becoming TAI, that undermines the case for future scaling getting us there under same paradigm); or

  • the more pessimistic annual risk estimates I talked about in the previous essay of 1-2% per year are correct, and that future civilisations will have a sufficiently increased difficulty for a collapse to cost to have near 50% the expected cost of extinction

  And either of these beliefs (and others) would suggest we’re relatively overspending on on AI.

  It’s also important to understand that Hendrycks and Yudkowsky were simply describing/​predicting the geopolitical equilibrium that follows from their strategies, not independently advocating for the airstrikes or sabotage.

  This is grossly disingenuous. Yudkowsky frames his call for airstrikes as what we ‘need’ to do, and describes them in the context of the hypothetical ‘if I had infinite freedom to write laws’. Hendrycks is slightly less direct in actively calling for it, claiming that it’s the default, but the document clearly states the intent of supporting it ‘we outline measures to maintain the conditions for MAIM’.

  These aren’t the words of people dispassionately observing a phenomenon—they are both clearly trying to bring about the scenarios they describe when the lines they’ve personally drawn are crossed.

• Arepo 8 Dec 2025 18:28 UTC

  6 points

  0 ∶ 0

  on: Con­tra Thorstad on the Math­e­mat­ics of Ex­is­ten­tial Risks

  But the expected value of existential risk reduction is—if not infinite, which I think it clearly is in expectation—extremely massive.

  I commented something similar on your blog, but as soon as you allow that one decision is infinite in expectation you have to allow that all outcomes are, since whatever possibility of infinite value you have given that action must still be present without it.

  If you think the Bostrom number of 10^52 happy people has a .01% chance of being right, then you’ll get 10^48 expected future people if we don’t go extinct, meaning reducing odds of existential risks by 1/​10^20 creates 10^28 extra lives.

  Reasoning like this seems kind of scope insensitive to me. In the real world, it’s common to see expected payoffs declining as offered rewards get larger, and I don’t see any reason to think this pattern shouldn’t typically generalise to most such prospects, even when the offer is astronomically large.

  The odds are not trivial that if we get very advanced AI, we’ll basically eliminate any possibility of human extinction for billions of years.

  I think the stronger case is just security in numbers. Get a civilisation around multiple star systems and capable of proliferating, and the odds of its complete destruction rapidly get indistinguishable from 0.

• Arepo 7 Dec 2025 18:51 UTC

  4 points

  1 ∶ 2

  in reply to: Thomas Kwa🔹’s comment on: Noah Birn­baum’s Quick takes

  I agree with Yarrow’s anti-‘truth-seeking’ sentiment here. That phrase seems to primarily serve as an epistemic deflection device indicating ‘someone whose views I don’t want to take seriously and don’t want to justify not taking seriously’.

  I agree we shouldn’t defer to the CEO of PETA, but CEOs aren’t—often by their own admission—subject matter experts so much as people who can move stuff forwards. In my book the set of actual experts is certainly murky, but includes academics, researchers, sometimes forecasters, sometimes technical workers—sometimes CEOs but only in particular cases—anyone who’s spent several years researching the subject in question.

  Sometimes, as you say, they don’t exist, but in such cases we don’t need to worry about deferring to them. When they do, it seems foolish to not to upweight their views relative to our own unless we’ve done the same, or unless we have very concrete reasons to think they’re inept or systemically biased (and perhaps even then).

• Arepo 6 Dec 2025 13:06 UTC

  21 points

  2 ∶ 2

  in reply to: Mjreard’s comment on: How I hope the EA com­mu­nity will re­spond to the AI bub­ble popping

  I agree that the OP is too confident/​strongly worded, but IMO this

  which is more than enough to justify EA efforts here.

  could be dangerously wrong. As long as AI safety consumes resources that might have counterfactually gone to e.g. nuclear disarmament, stronger international relations, it might well be harmful in expectation.

  This is doubly true for warlike AI ‘safety’ strategies like Aschenbrenner’s call to intentionally arms race China, Hendrycks, Schmidt and Wang’s call to ‘sabotage’ countries that cross some ill-defined threshold, and Yudkowsky calling for airstrikes on data centres. I think such ‘AI safety’ efforts are very likely increasing existential risk.

• Arepo 6 Dec 2025 12:17 UTC

  17 points

  0 ∶ 0

  on: Peter Wilde­ford fea­tured on the Daily Show about the risks from AI

  Well handled, Peter! I’m curious how much of that conversation was organic, how much scripted or at least telegraphed in advance?

• Arepo 4 Dec 2025 15:14 UTC

  4 points

  1 ∶ 0

  in reply to: NickLaing’s comment on: Open Philan­thropy Is Now Coeffi­cient Giving

  I’m warming to CoGi

• Arepo 3 Dec 2025 15:34 UTC

  8 points

  1 ∶ 0

  in reply to: Duncan Sabien’s comment on: MIRI’s 2025 Fundraiser

  That makes some sense, but leaves me with questions like

  • Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?

  • Which projects were failures that you feel were justifiable given your knowledge state at the time?

  • What do these past projects demonstrate about the team’s competence to work on future projects?

  • What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?

  • Are there any other analogies you could draw between past and possible future projects that would enable us update on the latter’s probability of success?

  MIRI is hardly unique even in the EA/​rat space in having special projects—Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility.

  And many EA organisations in the space that don’t explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing.

  In order to receive funds—at least from effectiveness-minded funders—all these orgs have to put a certain amount of effort into answering questions like those above.

  And ok, you say you’re not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations—and to hope that MIRI has some concrete answers.