Duncan Sabien

Karma: 678

Duncan Sabien 3 Dec 2025 17:42 UTC
12 points
3 ∶ 2
in reply to: titotal’s comment on: MIRI’s 2025 Fundraiser
I disagree re: motte and bailey; the above is not at all in conflict with the position of the book (which, to be clear, I endorse and agree with and is also my position).
re: “you can imagine,” I strongly encourage people to be careful about leaning too hard on their own ability to imagine things; it’s often fraught and a huge chunk of the work MIRI does is poking at those imaginings to see where they collapse.
I’ll note that core MIRI predictions about e.g. how machines will be misaligned at current levels of sophistication are being borne out—things we have been saying for years about e.g. emergent drives and deception and hacking and brittle proxies. I’m pretty sure that’s not “rooted in the actual nuts and bolts details” in the way you’re wanting, but it still feels … relevant.

Duncan Sabien 3 Dec 2025 17:40 UTC
6 points
1 ∶ 1
in reply to: Arepo’s comment on: MIRI’s 2025 Fundraiser
Noting that this is more “opinion of an employee” than “the position of MIRI overall”—I’ve held a variety of positions within the org and can’t speak for e.g. Nate or Eliezer or Malo:
- The Agent Foundations team feels, to me, like it was a slam dunk at the time; the team produced a ton of good research and many of their ideas have become foundational to discussions of agency in the broader AI sphere
- The book feels like a slam dunk
- The research push of 2020/2021 (that didn’t pan out) feels to me like it was absolutely the right bet, but resulted in (essentially) nothing; it was an ambitious, many-person project for a speculative idea that had a shot at being amazing.
I think it’s hard to generalize lessons, because various projects are championed by various people and groups within the org (“MIRI” is nearly a ship of Theseus). But some very basic lessons include:
- Things pretty much only have a shot at all when there are people with a clear and ambitious vision/when there’s an owner
- When we say to ourselves “this has an X% chance of working out” we seem to be actually pretty calibrated
- As one would expect, smaller projects and clearer projects work out more frequently than larger or vaguer ones
(Sorry, that feels sort of useless, but.)
From my limited perspective/to the best of my ability to see and descrbe, budget is essentially allocated in a “Is this worth doing? If so, how do we find the resources to make it work?” sense. MIRI’s funding situation has always been pretty odd; we don’t usually have a pie that must be divided up carefully so much as a core administrative apparatus that needs to be continually funded + a preexisting pool of resources that can be more or less freely allocated + a sense that there are allies out there who are willing to fund specific projects if we fall short and want to make a compelling pitch.
Unfortunately, I can’t really draw analogies that help an outsider evaluate future projects. We’re intending to try stuff that’s different from anything we’ve tried before, which means it’s hard to draw on the past (except insofar as the book and surrounding publicity were also something we’d never tried before, so you can at least a little bit assess our ability to pivot and succeed at stuff outside our wheelhouse by looking at the book).

Duncan Sabien 2 Dec 2025 22:52 UTC
3 points
1 ∶ 0
in reply to: MichaelDickens’s comment on: MIRI’s 2025 Fundraiser
Oh, I agree that if one feels equipped to go actually look at the arguments, one doesn’t need any argument-from-consensus. This is just, like, “if you are going to defer, defer reasonably.” Thanks for your comment; I feel similarly/endorse.
Made a small edit to reflect.

Duncan Sabien 2 Dec 2025 20:57 UTC
13 points
6 ∶ 0
in reply to: Yarrow Bouchard 🔸’s comment on: MIRI’s 2025 Fundraiser
Again speaking more for the broad audience:
“Some experts downvote Yudkowsky’s standing to opine” is not a reasonable standard; some experts think vaccines cause autism. You can usually find someone with credentials in a field who will say almost anything.
The responsible thing to do (EDIT: if you’re deferring at all, as opposed to evaluating the situation for yourself) is to go look at the balance of what experts in a field are saying, and in this case, they’re fairly split, with plenty of respected big names (including many who disagree with Eliezer on many questions) saying he knows enough of what he’s talking about to be worth listening to. I get that Yarrow is not convinced, but I trust Hinton, who has reservations of his own but not of the form “Eliezer should be dismissed out of hand for lack of some particular technical expertise.”
Also: when the experts in a field are split, and the question is one of existential danger, it seems that the splitness itself is not reassuring. Experts in nuclear physics do not drastically diverge in their predictions about what will happen inside a bomb or reactor, because we understand nuclear physics. When experts in the field of artificial intelligence have wildly different predictions and the disagreement cannot be conclusively resolved, this is a sign of looseness in everyone’s understanding, and when you ask normal people on the street “hey, if one expert says an invention will kill everyone, and another says it won’t, and you ask the one who says it won’t where their confidence comes from, and they say ‘because I’m pretty sure we’ll muddle our way through, with unproven techniques that haven’t been invented yet, the risk of killing everyone is probably under 5%,’ how do you feel?”

they tend to feel alarmed.
And that characterization is not uncharitable—the optimists in this debate do not have an actual concrete plan. You can just go check. It all ultimately boils down to handwaving and platitudes and “I’m sure we’ll stay ahead of capabilities [for no explicable reason].”
And we’re intentionally aiming at something that exceeds us along the very axis that led us to dominate the planet, so … ?
Another way of saying this: it’s very, very weird that the burden of proof on this brand-new and extremely powerful technology is “make an airtight case that it’s dangerous” instead of “make an airtight case that it’s a good idea.” Even a ⁵⁰⁄₅₀ shared burden would be better than the status quo.
I’ll note that
In response to any sort of criticism or disagreement, Yudkowsky and other folks’ default response seems to be to fly into a rage and to try to attack or humiliate the person making the criticism/expressing the disagreement.
...seems false.

Duncan Sabien 2 Dec 2025 17:58 UTC
47 points
7 ∶ 2
in reply to: Yarrow Bouchard 🔸’s comment on: MIRI’s 2025 Fundraiser
If deep learning doesn’t change things, Yudkowsky/MIRI should explain why not.
Speaking in my capacity as someone who currently works for MIRI, but who emphatically does not understand all things that Eliezer Yudkowsky understands, and can’t authoritatively represent him (or Nate, or the other advanced researchers at MIRI who are above my intellectual pay grade):
My own understanding is that Eliezer has, all along, for as long as I’ve known him and been following his work, been fairly agnostic as to questions of how AGI and ASI will be achieved, and what the underlying architectures of the systems will be.
I’ve often seen Eliezer say “I think X will not work” or “I think Y is less doomed than X,” but in my experience it’s always been with a sort of casual shrug and an attitude of “but of course these are very hard calls” and also with “and it doesn’t really matter to the ultimate outcome except insofar as some particular architecture might make reliable alignment possible at all.”
Eliezer’s take (again, as I understand it) is something like “if you have a system that is intelligent enough and powerful enough to do the actual interesting work that humans want to do, such as end all wars and invent longevity technology and get us to the stars (and achieve these goals in the real world, which involves also being competent at things like persuasion and communication), then that system is going to be very, very, very hard to make safe. It’s going to be easier by many orders of magnitude to create systems that are capable of that level of sophisticated agency that don’t care about human flourishing, than it will be to hit the narrow target of a sufficiently sophisticated system that also does in fact happen to care.”
That’s true regardless of whether you’re working with deep learning or symbolic AI. In fact, deep learning makes it worse—Eliezer was pointing at “even if you build this thing out of nuts and bolts that you thoroughly understand, alignment is a hard problem,” and instead we have ended up in a timeline where the systems are grown rather than crafted, giving us even less reason to be confident or hopeful.
(This is a trend: people often misunderstand MIRI’s attempts to underscore how hard the problem is as being concrete predictions about what will happen, c.f. the era in which people were like, well, obviously any competent lab trying to build ASI will keep their systems airgapped and secure and have a very small number of trusted and monitored employees acting as intermediaries. MIRI’s response was to demonstrate how even in such a paradigm, a sufficiently sophisticated system would have little trouble escaping the box. Now, all of the frontier labs routinely feed their systems the entire internet and let those systems interact with any human on Earth and in many cases let those systems write and deploy their own code with no oversight, and some people say “haha, look, MIRI was wrong.” Those people are confused.)
Symbolic AI vs. deep learning was never a crux, for Eliezer or the MIRI view. It was a non-crucial sidebar in which Eliezer had some intuitions and guesses, some of which he was more confident about and others less confident, and some of those guesses turned out wrong, and none of that ever mattered to the larger picture. The crucial considerations are the power/sophistication/intelligence of the system, and the degree to which its true goals can be specified/pinned-down, and being wrong about whether deep learning or symbolic AI specifically were capable of reaching the required level of sophistication is mostly irrelevant.
One could argue “well, Eliezer proved himself incapable of predicting the future with those guesses!” but this would be, in my view, disingenuous. Eliezer has long said, and continues to say, “look, guesses about how the board will look in the middle of the chess game are fraught, I’m willing to share my intuitions but they are far more likely to be wrong than right; it’s hard to know what moves Stockfish will make or how the game will play out; what matters is that it’s still easy to know with high confidence that Stockfish will win.”
That claim was compelling to me in 2015, and it remains compelling to me in 2025, and the things that have happened in the world in the ten years in between have, on the whole, made the case for concern stronger rather than weaker.
To draw up one comment from your response below:
The author of the review’s review does not demonstrate to me that they understand Collier’s point.
...Collier’s review does not even convincingly demonstrate that they read the book, since they get some extremely basic facts about it loudly, loudly wrong, in a manner that’s fairly crucial for their criticisms. I think that you should hold the reviewer and the review’s reviewer to the same standard, rather than letting the person you agree with more off the hook.
Fair warning: I wrote this response less for Yarrow specifically and more for the benefit of the EA forum userbase writ large, so I’m not promising that I will engage much beyond this reply. I might! But I also might not. I think I said the most important thing I had to say, in the above.

EDIT: oh, for more on how this:
In fact, there are plenty of reasons why the fact that AIs are grown and not crafted might cut against the MIRI argument. For one: The most advanced, generally capable AI systems around today are trained on human-generated text, encoding human values and modes of thought.
...is badly, badly wrong, see the supplemental materials for the book, particularly chapters 2, 3, and 4, which exhaustively addressed this point long before Collier ever made it, because we knew people would make it. (It’s also addressed in the book, but I guess Collier missed that in their haste to say a bunch of things it seems they already believed and were going to say regardless.)

Duncan Sabien 2 Dec 2025 16:10 UTC
21 points
3 ∶ 0
in reply to: Arepo’s comment on: MIRI’s 2025 Fundraiser
(Speaking in my capacity as someone who currently works for MIRI)
I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above. We adopted a stance of closed-by-default research years ago for that reason, but that’s not why e.g. we don’t publish concrete and exhaustive lists of outputs and budget.
We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.
But mainly, we think of ourselves as asking for money from only one of the two kinds of donors. MIRI feels that it’s pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).
(A colleague of mine once joked that “wages are for people who can demonstrate the value of their labor within a single hour; I can’t do that, which is why I’m on a salary.” A similar principle applies here.)
In the past, funding MIRI led to outputs like our alignment research publications. In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).
That’s enough for some donors—”If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn’t pan out and the occasional home run.”
Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.
(One might ask “okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?” and the answer is “mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history.” Definitely we do not feel entitled to anyone’s dollars, and the hesitations of any donor who doesn’t want to send their money toward MIRI-like efforts are valid.)

Duncan Sabien 17 Apr 2023 19:38 UTC
21 points
7 ∶ 1
in reply to: Geoffrey Miller’s comment on: We’re losing creators due to our nitpicking culture
Another way to think about this (imo) is “do you screen falsehoods immediately, such that none ever enter, or do you prune them later at leisure?”
Sometimes, assembling false things (such as rough approximations or heuristics!) can give you insight as to the general shape of a new Actually True thing, but discovering the new Actually True thing using only absolutely pure definite grounded vetted airtight parts would be way harder and wouldn’t happen in expectation.
And if you’re trying to (e.g.) go “okay, men are stronger than women, and adults are smarter than kids” and somebody interrupts to go “aCtUaLlY this is false” because they have a genuinely correct point about, e.g., the variance present in bell curves, and there being some specific women who are stronger than many men and some specific children who are smarter than many adults … this whole thing just derails the central train of thought that was trying to go somewhere.
(And if the “aCtUaLlY” happens so reliably that you can viscerally feel it coming, as you start to type out your rough premises, you get demoralized before you even begin, close your draft, and go do something else instead.)

Duncan Sabien 5 Mar 2023 21:22 UTC
5 points
19 ∶ 22
on: Nick Bostrom should step down as Director of FHI
“This post is anonymised because I don’t want to have to deal with the interpersonal consequences of the beliefs I hold; I don’t want people knowing that I hold my beliefs, and would rather trick people into associating with me in ways they might not if they actually knew my true stance.”

Duncan Sabien 19 Feb 2023 0:36 UTC
10 points
1 ∶ 0
in reply to: EliezerYudkowsky’s comment on: People Will Sometimes Just Lie About You
Selfish piggyback plug for the concept of sazen.

Duncan Sabien 18 Feb 2023 17:45 UTC
6 points
11 ∶ 5
in reply to: lilly’s comment on: People Will Sometimes Just Lie About You
The essay itself is the argument for why EAs shouldn’t steelman things like the TIME piece.
(I understand you’re disagreeing with the essay and that’s :thumbsup: but, like.)
If you set out to steelman things that were generated by a process antithetical to truth, what you end up with is something like [justifications for Christianity]; privileging-the-hypothesis is an unwise move.
If one has independent reasons to think that many of the major claims in the article are true, then I think the course most likely to not-mislead one is to follow those independent reasons, and not spend a lot of time anchored on words coming from a source that’s pretty clearly not putting truth first on the priority list.

Duncan Sabien 18 Feb 2023 17:09 UTC
80 points
31 ∶ 6
in reply to: lilly’s comment on: People Will Sometimes Just Lie About You
This language is inflammatory (“overwhelming”, “incestuous”), but we can boil this down to a more sterile sounding claim
A major part of the premise of the OP is something like “the inflammatory nature is a feature, not a bug; sure, you can boil it down to a more sterile sounding claim, but most of the audience will not; they will instead follow the connotation and thus people will essentially ‘get away’ with the stronger claim that they merely implied.”
The accuser doesn’t offer concrete behaviors, but rather leaves the badness as general associations. They don’t make explicit accusations, but rather implicit ones. The true darkness is hinted at, not named. They speculate about my bad traits without taking the risk of making a claim. They frame things in a way that increases my perceived culpability.
I think it is a mistake to steelman things like the TIME piece, for precisely this reason, and it’s also a mistake to think that most people are steelmanning as they consume it.
So pointing out that it could imply something reasonable is sort of beside the point—it doesn’t, in practice.

Duncan Sabien 18 Feb 2023 5:50 UTC
96 points
28 ∶ 2
on: People Will Sometimes Just Lie About You
I am at best 1/1000th as “famous” as the OP, but the first ten paragraphs ring ABSOLUTELY TRUE from my own personal experience, and generic credulousness on the part of people who are willing to entertain ludicrous falsehoods without any sort of skepticism has done me a lot of damage.
What links here?
- Duncan Sabien's comment on Share the burden by Emma_2 (12 Mar 2023 18:41 UTC; 38 points)

Duncan Sabien 10 Feb 2023 4:56 UTC
7 points
5 ∶ 7
in reply to: sphor’s comment on: EA Community Builders’ Commitment to Anti-Racism & Anti-Sexism
I mean, I don’t have this hypothetical document made in my head (or I would’ve posted it myself).
But an easy example is something of the shape:

[EDIT: The below was off-the-cuff and, on reflection, I endorse the specific suggestion much less. The structural thing it was trying to gesture at, though, of something clear and concrete and observable, is still the thing I would be looking for, that is a prerequisite for enduring endorsement.]
“We commit to spending at least 2% of our operational budgets on outreach to [racial group/gender group/otherwise unrepresented group] for the next 5 years.”
Maybe the number is 1%, or 10%, or something else; maybe it’s 1 year or 10 years or instead of years it’s “until X members of our group/board/whatever are from [nondominant demographic].”
The thing that I like about the above example in contrast with the OP is that it’s clear, concrete, specific, and evaluable, and not just an applause light.

Duncan Sabien 10 Feb 2023 3:08 UTC
10 points
10 ∶ 5
in reply to: Rockwell’s comment on: EA Community Builders’ Commitment to Anti-Racism & Anti-Sexism
I would like for all involved to consider this, basically, a bet, on “making and publishing this pledge” being an effective intervention on … something.
I’m not sure whether the something is “actual racism and sexism and other bigotry within EA,” or “the median EA’s discomfort at their uncertainty about whether racism and sexism are a part of EA,” or what.
But (in the spirit of the E in EA) I’d like that bet to be more clear, so since you were willing to leave a comment above: would you be willing to state with a little more detail which problem this was intended to solve, and how confident you (the group involved) are that it will be a good intervention?

Duncan Sabien 10 Feb 2023 2:07 UTC
142 points
75 ∶ 31
on: EA Community Builders’ Commitment to Anti-Racism & Anti-Sexism
I am opposed to this.
I am also not an EA leader in any sense of the word, so perhaps my being opposed to this is moot. But I figured I would lay out the basics of my position in case there are others who were not speaking up out of fear [EDIT: I now know of at least one bona fide EA leader who is not voicing their own objection, out of something that could reasonably be described as “fear”].
Here are some things that are true:
- Racism is harmful and bad
- Sexism is harmful and bad
- Other “isms” such as homophobia or religious oppression are harmful and bad.
- To the extent that people can justify their racist, sexist, or otherwise bigoted behavior, they are almost always abusing information, in a disingenuous fashion. e.g. “we showed a 1% difference in the medians of the bell curves for these two populations, thereby ‘proving’ one of those populations to be fundamentally superior!” This is bullshit from a truth-seeking perspective, and it’s bullshit from a social progress perspective, and in most circumstances it doesn’t need to be entertained or debated at all. In practice, it is already the case that the burden of proof on someone wanting to have a discussion about these things is overwhelmingly on the person starting the discussion, to demonstrate that they are both a) genuinely well-intentioned, and b) have something real to talk about.
However:
Intelligent, moral, and well-meaning people will frequently disagree about to-what-extent a given situation is explained by various bigotries as opposed to other factors. Intelligent, moral, and well-meaning people will frequently disagree about which actions are wise and appropriate to take, in response to the presence of various bigotries.
By taking anti-racism and anti-sexism and other anti-bigotry positions which are already overwhelmingly popular and overwhelmingly agreed-upon within the Effective Altruism community, and attempting to convert them to Anti-Racism™, Anti-Sexism™, and Anti-Bigotry™ applause lights with no clear content underneath them, all that’s happening is the creation of a motte-and-bailey, ripe for future abuse.
There were versions of the above proposal which were not contentless and empty, which stake out clear and specific positions, which I would’ve been glad to see and enthusiastically supported and considered concrete progress for the community. It is indeed true that EA as a whole can do better, and that there exist new norms and new commitments that would represent an improvement over the current status quo.
But by just saying “hey, [thing] is bad! We’re going to create social pressure to be vocally Anti-[thing]!” you are making the world worse, not better. Now, there is a List Of Right-Minded People Who Were Wise Enough To Sign The Thing, and all of the possible reasons to have felt hesitant to sign the thing are compressible to “oh, so you’re NOT opposed to bigotry, huh?”
Similarly, if four-out-of-five signatories of The Anti-Racist Pledge think we should take action X, but four-out-of-five non-signatories think it’s a bad idea for various pragmatic or logistical reasons, it’s pretty easy to imagine that being rounded off to “the opposition is racist.”
(I can imagine people saying “we won’t do that!” and my response is “great—you won’t. Are you claiming no one will? Because at the level of 1000+ person groups, this is how this always goes.”)
The best possible outcome from this document is that everybody recognizes it as a basically meaningless non-thing, and nobody really pays attention to it in the future, and thus having signed it means basically nothing. This is also a bad outcome, though, because it saps momentum for creating and signing useful versions of such a pledge. It’s saturating the space, and inoculating us against progress of this form; the next time someone tries to make a pledge that actually furthers equity and equality, the audience will be that much less likely to click, and that much less willing to believe that anything useful will result.
The road to hell is paved with good intentions. This is clearly a good intention. It does not manage to avoid being a pavestone.

Duncan Sabien 8 Feb 2023 19:07 UTC
1 point
0 ∶ 0
in reply to: Jason’s comment on: The number of burner accounts is too damn high
I would support that.

Duncan Sabien 8 Feb 2023 18:43 UTC
7 points
3 ∶ 2
in reply to: Jason’s comment on: The number of burner accounts is too damn high
The community is quite capable of dealing with actual “bs” by downvoting it into oblivion.
I disagree that the community is doing anything remotely close to a good job of distinguishing bs from non-bs via downvotes. [The evidence is that the community does not find the bulk of burner-poster posts to be “bs”] is a true statement, and is revealing the problem.
It is also very easy for users who do not want to engage with burner posts to skip on past them.
This is straight false; they’re showing up on all sorts of posts WAY more than they used to.

Duncan Sabien 8 Feb 2023 18:41 UTC
−22 points
0 ∶ 7
in reply to: sick_burnerer’s comment on: The number of burner accounts is too damn high
I didn’t need to post, and it’s quite unpleasant to do so.
Then … please stop.

Duncan Sabien 8 Feb 2023 18:40 UTC
8 points
2 ∶ 0
in reply to: Aptdell’s comment on: The number of burner accounts is too damn high
There are low-bs forums such as Hacker News and slatestarcodex where most people don’t use their real names.
In those forums, reputation accrues to username; little (or at least less) attention is paid to brand-new accounts.
Here, a lot of accounts are trying to recruit/use the “I’m a for real serious longtime member of this community” reputational/seriousness boost, while being who the heck knows who.

Duncan Sabien 8 Feb 2023 8:01 UTC
13 points
17 ∶ 11
in reply to: ElliotJDavies’s comment on: Why People Use Burner Accounts: A Commentary on Blacklists, the EA “Inner Circle”, and Funding
It’s also full of insinuation and implication and “X may mean Y [which is damning]” in a way that’s attempting to get the benefit of “X = Y” without having to actually demonstrate it.
In my opinion, “you have to use a burner account to put forth this kind of ‘thinking’ and ‘reasoning’ and ‘argument’” is actually a point in EA culture’s favor.