Arepo

Karma: 5,576

Arepo 25 Jun 2026 22:08 UTC
4 points
1 ∶ 2
in reply to: Anthony DiGiovanni 🔸’s comment on: Announcing: Cluelessness Critiques Competition
literally, you pick one precise credence over many others for no reason
At some foundational level, a credence has no deeper reason than ‘some neurons fired that way’. But you don’t need to restrict yourself to concerns about the whole future lightcone to run into this problem—at the foundational level this is true of every statement.
There are various ways one might respond to this challenge, but if we don’t view it as insurmountable elsewhere, I don’t see why we should do so with credences (which are of course usually non-foundational statements). And if we do, it undermines e.g. any argument about unawareness.
Why does “our best effort” need to be precise?
It only needs to be as precise as is necessary for decision-making. I will probably never need to forecast rain to 8 decimal places. But if you’re saying forecasting rain as ‘less than .5’ is ok, but that forecasting 0.1234567% chance of rain if the extra precision was actually decision-relevant, would be un-ok/impossible/qualitatively different, then I disagree.
(from link) It doesn’t follow from “we don’t know the net direction of the consequences we’re unaware of” that we should regard the positives and negatives as precisely symmetric. One reason symmetry is implausible: If we become aware of a new possible consequence, this should update our beliefs about the others we’re unaware of, breaking the symmetry.
If you ‘become aware’ of something, you’ve gained information and should update your priors accordingly. That doesn’t move me away from being happy to treat genuine unknowns as EV-0. Your counterpoint seems to be that in some cases that feel sort-of- equal (and about which, in the cases you describe we actually have a lot of information), we might be inclined to give equal credence. But it seems to me correct to say ‘if you have meaningful knowledge of two possible outcomes, and the weight you assign to them is decision-relevant, giving them equal credence is a mistake’, which fixes this purported problem without radically undermining our epistemology.
My claim is that we don’t have a positive argument in favor of doing what the precise EV approach recommends
The precise EV approach is well evidenced in short-term decision-making, so the positive argument is that there isn’t any principled difference between short and long-term decision-making

Arepo 25 Jun 2026 18:46 UTC
5 points
1 ∶ 0
on: Announcing: Cluelessness Critiques Competition
I don’t know how to challenge the premises because the key premise seems to be an assertion that I don’t find convincing.
In ‘Should you go with your best guess?‘, which appears to be the primary argument against the idea of ‘deterministic’ Bayesian credences, DiGiovanni repeatedly signposts that he’s going to give an argument against them… but I can’t see anything that constitutes one.
In the section ‘Background on degrees of belief and what makes them rational’, he talks about how we don’t get to find out which beliefs outperform others, but doesn’t say why this means we shouldn’t/can’t pick credences according to our best effort. It also doesn’t say why, if we can measure short term value, we shouldn’t use that as a justification for our decisionmaking process and assume EV from events that we don’t think we can assess is 0.
In the section ‘Motivating example’, he gives an example vignette, at the end of which we’re given that ‘this feels so arbitrary’. But it doesn’t seem like ‘feeling arbitrary’ is a reason not to do something—especially when we’re not given an alternative (or at least, no other decision process that seems less or equally arbitrary).
So my response is just to say ‘using credences still seems fine, if occasionally emotionally uncomfortable (maybe using distributions is sometimes empirically better, and if so I support it)’ - in which case I don’t see a problem in need of solving.

Arepo 14 Jun 2026 15:34 UTC
2 points
0 ∶ 0
in reply to: dan.pandori 🔸’s comment on: Are top existential risk estimates 50,000 times too high? An optimizer’s curse model and analysis
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don’t see why error estimates should be thin though—there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)

Arepo 14 Jun 2026 12:12 UTC
3 points
0 ∶ 0
in reply to: MaxReith’s comment on: Are top existential risk estimates 50,000 times too high? An optimizer’s curse model and analysis
I think 1) implies that you should give up some substantial optimisation for the sake of greater versatility (which seems approx titotal’s view with reference to overcommitting)
2) feels correct and important to me, also since I’ve been arguing in the post op linked and elsewhere that treating extinction as special is a heuristic that was useful for initial cause prioritisation but isn’t a valid reason for focusing on it two decades later.

Arepo 10 Jun 2026 10:46 UTC
3 points
3 ∶ 1
in reply to: Tristan Katz’s comment on: My disagreements with CEA’s approach to stewarding EA
I think there are many strong reasons to be sceptical of a lot of EA orthodoxy—many people including me have written about them at length (also keep an eye out for a post by titotal in the next few days that comes up with a magnitude estimate for such errors).
These reasons might turn out to be wrong, but I strongly disagree that e.g. classic longtermist/x-risk/AI doomer arguments are anywhere near the standard of economics—as evidenced e.g. by economists vigorously disagreeing with them.

Arepo 25 May 2026 11:30 UTC
12 points
6 ∶ 0
on: Stewardship in EA’s Most Important Era
I just want to say this
I do my best to reply to everyone who reaches out about careers at CG, even if sometimes this is just a quick note celebrating their pursuit of impactful work and sharing career resources with them. At the time of writing, this has worked out to roughly 1,200 messages and 600 calls/in-person meetings.
is really inspiring. It sounds exhausting, but as someone who’s often struggled to get feedback from decisionmakers in the EA space, I’m sure those 1800 people very much appreciated it—and I’d be amazed if it didn’t lead to some substantially positive counterfactual changes in their plans :)
(and maybe I’ll reach out to you for some feedback soon!)

Arepo 25 May 2026 11:24 UTC
5 points
0 ∶ 1
on: My Current Solution to the Repugnant Conclusion
A relatively simple way of making the repugnant conclusion more intuitive to me is to recognise that individual selves are largely an illusion, i.e. that empty individualism (or, its better marketed/more spiritual sounding but functional equivalent open individualism) is correct.
Suppose you’ve set up your parameters such that pinpricks actually involve negative utility—because (trigger warning of slightly graphical image for the second of these links) in many cases it obviously isn’t actually negative, which muddies our intuitions. Then for empty individualism a tiny amount of torture either is actually closely analogous to a large number of pinpricks (that just happen to be locally clustered). The OI equivalent is that a small amount of torture is analogous to a pinprick on the cosmic entity
Even in the case of humans, we can imagine how pinpricks could add up. A single superficial pinprick isn’t that bad—and the difference between a 1mm and 2mm insertion would be very slight. But if you insert hundreds or thousands of pins 10+mm deep, gradually you move towards an experience that seems as bad as any other torture.
To put it simply, sufficiently many pinpricks, of sufficient depth to be unpleasant, are torture. And they can be torture at a degree of virtually any level of pain a human is capable of experiencing—so I don’t see a need to introduce dramatic discontinuities in our moral evaluation to explain why other tortures are somehow still morally worse.

Arepo 21 May 2026 14:24 UTC
2 points
0 ∶ 0
on: Can EA Frameworks Unlock India’s $4.2 Billion CSR Pool?
That was fascinating—I really like the idea of reframing EA ideas as a way of saving on future research costs.
I’m a bit unclear what we’re currently to make of the the ‘3–5x’ estimate—you say it’s illustrative, but also ‘plausible’. Assuming that is your current best guess, could you say how you reached it?

Arepo 20 May 2026 17:43 UTC
4 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: The Goodhart Singularity
Have you had any takers (who you can publicly name) other than Greg?

Arepo 19 May 2026 17:12 UTC
5 points
2 ∶ 2
on: Why are longtermists so much less focused on human extinction these days?
I don’t encounter many people who still identify as longtermist, but as someone who does, I recently wrote these arguments for why longtermists should be less extinction-focused.
The tl;dr is that I think that other than extinction there are predictable patterns, with perhaps the most prominent related to entropy, and that those patterns provide more nuanced ways to estimate the cost of lesser catastrophes—and that while assessing the costs of lesser catastrophes precisely is infeasible, that’s not a basis for thinking they would be negligible compared to extinction.

Arepo 19 May 2026 11:29 UTC
4 points
0 ∶ 1
on: The Goodhart Singularity
Nice post :) This helps clarify some of my own continuing scepticism about likely speed and degree of change in the near term.
I’ve had a longer-term analogous view that AI doom futures involving interstellar expansion seem unlikely on the grounds that, after wiping out all humans such an AI would find itself in an extremely novel scenario, in which anything resembling deep learned training on the world to that point would be useless.

Arepo 3 Apr 2026 4:48 UTC
9 points
2 ∶ 1
in reply to: richard_ngo’s comment on: What is integral altruism?
>I think it’s historically pretty incorrect that the grounding in cost-effectiveness is what made EA good
FWIW the reasons you’re giving here are closely related to the reasons why I’m sceptical that modern AI-focused EA is in fact as good. I don’t think it’s unreasonable to support AI safety work, but I think doing so to the extent the movement is being guided to is throwing away most of the epistemics that could make EA a long-term robustly positive influence. EA’s original tagline used to be ‘using evidence and reason’, but the extreme AI safety focus seems to drop the ‘evidence’ part.
To believe you should focus on AI safety, you approximately need to believe all of
- short timelines; and
- either
  - no trend of convergence between intelligence and morality or
  - the view that convergence wouldn’t matter or wouldn’t be enough to avoid moral disaster; and
- either
  - long timelines on other GCRs or
  - other GCRs not really mattering to humanity’s long term prospects and
    zero discounting on future people; and
- that a flourishing human future is +EV; and
- that trying to improve average welfare in a flourishing future is less good than trying to increase the probability of a flourishing future; and
- reasonable confidence that AI safety work has learned from its past mistakes and will be reliably +EV; and
- there won’t be a sufficient public shift towards AI safety to make it low enough leverage that less; and
- you personally have more comparative advantage working on AI safety than any other cause; and
and surely some further assumptions I’ve missed, and many ways to further unpack these premises. To advocate work on AI safety as the primary EA cause you need to believe that the final bullet applies to the majority of your audience.
But I think there’s plenty in that list of assumptions that’s easy to disagree with, and a lot of entangled assumptions whose entanglement to my knowledge hasn’t really been explored (e.g. I find it hard to credit both that there’s no convergence between intelligence and morality and that there’s a long term equilibrium which is both stable and in some nontrivial sense positive or desirable).
So I semiagree with @MichaelDickens’s original comment’s in-principle scepticism while wondering whether in practice int/a might end up promoting causes that feel closer to the what I view as the original spirit of the movement.
There’s also some practical concerns in the OP that I think EA has dropped the ball on, such as building the sort of real community that would have retained greater support/membership over the years (my impression is the substantial majority of EAs who joined the movement more than 6 or 7 years ago have largely disengaged with it).
So I guess I’m noncommittally hopeful that this becomes something valuable—and remains, and Euan said, symbiotic with EA. If it just gives people who would have been somewhat supportive but felt too constrained a way to stay engaged with an encouraging community, that seems like it could be high value.

Arepo 19 Mar 2026 1:50 UTC
3 points
0 ∶ 0
in reply to: Arepo’s comment on: GHD discussion here is slowly dying
I’m confused by the strong negative reaction to this comment. I guess it’s about the CoGi funding, which does sound like I was wrong. But it seems to be true that there’s no option to directly apply for funding for a new project (NickLaing mentions the GH funding circle, but they completed one round last year and their website doesn’t currently imply there would be any more).
I think this helps explain the decline of GHD in the OP—AIM’s charity list notwithstanding, no-one in the movement is incentivised to come up with practical ideas in the field.

Arepo 17 Mar 2026 8:12 UTC
2 points
0 ∶ 10
in reply to: Henry Howard🔸’s comment on: GHD discussion here is slowly dying
Last I heard it was something like 10% of their GCR budget.
It’s also basically impossible to apply for GHD funding. I recently decided to put my money where my mouth is and get involved in an early stage GHD project, but there’s basically no EA-aligned funder who’s willing to let you approach them.
SFF are exclusively longtermist, EA GHD as mentioned basically shut down, and Givewell and CoGi don’t accept unsolicited applications. So as far as I can see if you think you have an idea in the GHD space and need funding for it you basically have to look outside the EA world (someone tell me if I missed something!)

Arepo 17 Mar 2026 5:28 UTC
2 points
0 ∶ 1
on: Criticisms of What We Owe The Future
It seems like, considering how intelligent and creative our species is, we should expect that, even in very dire conditions, we would be able to re-build civilization.
That shouldn’t necessarily be the primary concern. Though it also seems that people who’ve studied our ability to rebuild civilisation are substantially more pessimistic.

Arepo 17 Mar 2026 5:21 UTC
10 points
5 ∶ 1
on: GHD discussion here is slowly dying
I think the simple answer is that it’s become less prioritised by the central orgs (the EA GHD fund is on indefinite hiatus, GHD is a diminishing part of CoGi’s budget, 80k moved away from it almost entirely, Rethink seem to have shifted towards animal welfare, CEA seem to have an increasingly longtermist/AI focus, etc). This gives a top-down cultural impetus away from the subject, and just means there’s less money in it.
It’s also, for better or worse, as an evidence-oriented field, a subject that’s harder to have amateur conversations about. I’ve been consistently supportive of it in my time here, but have had very little to contribute to conversations about what actually works, and felt that there was little value in contributing to any others.
I would love to see this reverse—I think EA is much richer for spanning multiple cause areas, and especially those which are well-evidenced. I don’t have any good solutions though :\

Arepo 12 Mar 2026 2:52 UTC
13 points
4 ∶ 4
in reply to: lilly’s comment on: Effective Altruism Will Be Great Again
I agree that the average college student encountering EA today should focus on issues related to AI safety
I broadly nodded along to your OP, but strong disagree here. There are tonnes of people working in AI safety, to the extent that it’s already hypercompetitive and the marginal value of one more person getting in the long queue for such a job seems very low.
Meanwhile I continue to find the case for AI safety, as least as envisioned by EA doomers, highly speculative. That’s not to say it shouldn’t get any attention, but there’s a far better evidenced path from e.g. ‘nuclear bombs or major pandemics cause the fall of civilisation’ than from ‘LLMs cause the fall of civilisation’.
And if you’re sufficiently pessimistic on the doomer narrative, we’re all screwed and there’s likely at least as much EV in short-term improvement of the lives of existing beings as in fighting an impossible struggle to prevent AGI from ever being developed. So there’s a credence window in which AI safety as top priority belongs. That window might be reasonably wide, but I don’t think it’s anywhere near wide enough to justify abandoning all other causes.

Arepo 21 Jan 2026 11:17 UTC
6 points
3 ∶ 0
on: Everything is neglected
I’m still highly sceptical of neglectedness as anything but a first-pass heuristic for how prioritisation organisations might spend their early research—firstly because there are so many ways a field can be ‘not neglected’ and still highly leveraged (e.g. Givewell and Giving What We Can were only able to have the comparative impact they did because the global health field had been vigorously researched but no-one had systemically done the individual-level prioritisation they did with the research results); secondly because it encourages EA to reject established learning in a way I find dangerously hubristic (FTX weren’t irresponsible—they just took a neglected approach to fundraising!).
If we must keep using this heuristic, it helps to introduce supporting heuristics like the one you mention; to which I’d add ‘look at the amount of input relative to the amount of input needed to solve the issue’. Climate has had far more input than AI safety, but it’s unclear to me whether the proportion of input it’s had to what it needs is higher.

Arepo 20 Jan 2026 15:29 UTC
2 points
0 ∶ 0
in reply to: Nathan Young’s comment on: Exaggerating the risks (Part 20: AI 2027 timelines forecast, benchmarks and gaps) | Reflective Altruism
Sure, so we agree?
Ah, sorry, I misunderstood that as criticism.
Do you think that forecasting like this will hurt the information landscape on average?
I’m a big fan of the development e.g. QRI’s process of making tools that make it increasingly easy to translate natural thoughts into more usable forms. In my dream world, if you told me your beliefs it would be in the form of a set of distributions that I could run a monte carlo sim on, having potentially substituted my own opinions if I felt differently confident than you (and maybe beyond that there’s still neater ways of unpacking my credences that even better tools could reveal).
Absent that, I’m a fan of forecasting, but I worry that overnormalising the naive I-say-a-number-and-you-have-no-idea-how-I-reached-it-or-how-confident-I-am-in-it form of it might get in the way of developing it into something better.

Arepo 3 Jan 2026 12:31 UTC
34 points
5 ∶ 3
in reply to: Nathan Young’s comment on: Exaggerating the risks (Part 20: AI 2027 timelines forecast, benchmarks and gaps) | Reflective Altruism
I weakly disagree here. I am very much in the “make up statistics and be clear about that” camp.
I’m sympathetic to that camp, but I think it has major epistemic issues that largely go unaddressed:
- It systemically biases away from extreme probabilities (it’s hard to assert < than $10^{- 3}$ , for e.g., but many real-world probabilities are and post-hoc credences look like they should have been below this)
- By focusing on very specific pathways towards some outcome, it diverts attention towards easily definable issues, and hence away from the prospects of more complex pathways of causing the same or value-equivalent outcomes.^[1]
- It strongly emphasises point credence estimates over distributions, the latter of which are IMO well worth the extra effort, at least whenever you’re broadcasting your credences to the rest of the world.
By the way, I find this a strange remark:
Seems like a lot of specific, quite technical criticisms.
This sounds like exactly the sort of criticism that’s most valuable of a project like this! If their methodology were sound it might be more valuable to present a more holistic set of criticisms and some contrary credences, but David and titotal aren’t exactly nitpicking syntactic errors—IMO they’re finding concrete reasons to be deeply suspicious of virtually every step of the AI 2027 methodology.
1. ^
  For e.g. I think it’s a huge concern that the EA movement have been pulling people away from nonextinction global catastrophic work because they focused for so long on extinction being the only plausible way we could fail to become interstellar, subject to the latter being possible. I’ve been arguing for years now, that the extinction focus is too blunt a tool, at least for the level of investigation the question has received from longtermists and x-riskers.