Jordan Arel

I’m wor­ried “Plan A” type in­ter­na­tional agree­ments may lead to per­ma­nent dictatorships

• Jordan Arel 28 Aug 2026 4:34 UTC

  4 points

  0 ∶ 0

  on: Mo­ral Imag­i­na­tion & Effec­tive Altruism

  Thanks Toby!

  I couldn’t agree more that these are some important moral imagination innovations, and yet there is a lot of room for improvement among EAs.

  I hadn’t heard your “positive ethics“, but I have relatedly thought we need something like “opportunity cost ethics”, which perhaps takes this even one step further, that we ought to continue searching for actions that optimize moral expected value until the point of diminishing returns, where the loss of value from delaying action exceeds the marginal expected value from seeking more information (although you do hit upon a bit of an optimal stopping problem with this, nonetheless I think you can’t sweep the serious importance of this search under the rug.)

  Essentially, the world is crazy enough that we should expect crazy opportunities for doing good, and it is of utmost moral value to continue searching as long we are still discovering facts that dramatically improve our prospects for doing good.

  I also appreciate your work on moral uncertainty quite deeply, I think this helped me appreciate how important it is to have some kind of comprehensive reflection process for the long-term future, to make sure we get the right moral values before spreading to space. While I independently discovered many of these same moral innovations, e.g. longtermism, I did not fully appreciate this and it has significantly changed my priorities.. Unfortunately I feel like a lot of people (even within EA) still don’t fully appreciate the importance of moral uncertainty.

• Jordan Arel 2 Aug 2026 22:29 UTC

  3 points

  0 ∶ 0

  on: Jor­dan Arel’s Shortform

  A note on AI, increasing/​decreasing technological determinism, and wise institutions:

  One really important facet of highly general advanced AI that I’ve been thinking about a lot lately is that it seems like, in general, AI will dramatically accelerate humanity’s ability to choose its path. While human labor has a lot of inertia and resistance to being directed in certain directions, AGI could be directed toward whichever technology we decide is most desirable.

  If things are sufficiently decentralized, or perhaps a better word would be anarchic, we might expect a race to the bottom. Competitive dynamics will incentivize a Molochian dynamic where we target the technologies that are most profitable, even when this produces technologies that are harmful or dangerous.

  But if humanity is sufficiently coordinated or in some other way able to have wise collective governance/​leadership/​decision-making, then we could choose to develop the safest and most beneficial technologies first, building out the tech tree in whatever way is optimal for existential security and a flourishing future.

  So I think path-dependence will be much more dependent on the way institutions are set up and how they make decisions around AI, such that we could either build a new dynamic that moves us toward a wisdom attractor, or on the other hand, if commercial or competitive dynamics are unleashed to their maximum, perhaps technological determinism could actually accelerate instead of decrease.

• Jordan Arel 25 Jul 2026 20:43 UTC

  3 points

  0 ∶ 0

  on: Re­v­erse alignment

  Right, this is basically the “aligned to what?” question, where the typical answer is simply “aligned to the user” or “intent aligned” rather than aligned to some specific values. Which means we are going to “foom” society as it current exists more or less, which I think could be quite catastrophic. “Better futures” or flourishing futures” work also tries to get at this, and I was trying to point o similar things in this work.

  And thanks for sharing, will check this out!

7 ran­dom thoughts on train­ing Bud­dhist AI

• Jordan Arel 5 Jul 2026 0:01 UTC

  3 points

  0 ∶ 0

  in reply to: Chris Leong’s comment on: Maybe do the thing you wish CEA would do

  I agree! I was thinking of doing something like this and wrote an (unpublished) post which mentioned it, still strongly considering doing something related though perhaps more so a “flourishing futures”/​“better futures” angle, which I think is related.

Con­cern­ing macros­trate­gic im­pli­ca­tions of Fable 5 ex­port controls

• Jordan Arel 24 May 2026 17:42 UTC

  2 points

  0 ∶ 0

  on: Why are longter­mists so much less fo­cused on hu­man ex­tinc­tion these days?

  I think it’s essentially a matter of ITN, AI timelines, and post-AGI dynamics. And perhaps a little sociological.

  Compared to long-term trajectory change, AI safety is now much less neglected, I would say by a couple orders of magnitude, which is not enough to compensate for the lower tractability of trajectory change.

  AI timelines seem quite short and AGI & ASI seem like they may be quite likely to be structurally power concentrating (even if we avoid the most extreme scenarios, although of course those are one of the main concerns) and likely to make most humans largely irrelevant;

  Hence the time we have left to influence the trajectory of the long-term future may be quite limited. Even if nothing is locked in immediately, the ability of most humans to meaningfully influence the long-term trajectory of the future may be strongly curtailed as AI (and who controls AI, and the overall shape of civilization at the time the AI transition occurs) becomes the dominant force deciding what happens in the future.

  As noted in the MacAskill essay you mentioned, AGI/​ASI may also accelerate other factors like grabbing solar space resources & potentially defense dominant deep space settlement, rapidly moving toward technological maturity, and various other forces increasing the likelihood of local and global lock-in, enabled by advanced AI.

  So overall I think the concern is that AI may cause a dramatic relatively near-term acceleration of path dependence and lock-in dynamics, and that this is very severely neglected compared to AI related extinction risk, which seem like the biggest extinction risk.

  I think trajectory change is in fact so neglected that it’s hard to even say how tractable it is because barely anyone has looked at it, and it seems worth having at least a few people looking at it.

  I think there’s also a sociological factor that MacAskill and Forethought, which have something not too far off from the the above view, have been really trying to raise the profile of these ideas since at least “What We Owe The Future” (although to be clear I know there are several longtermists who helped innovate these views and quite a lot of longtermists who are sympathetic to this kind of stuff, I just think MacAskill/​Forethought have been most effective.)

  This is a bit more speculative, but I also think that perhaps when the whole shift from “longtermist” framing to “x-risk” framing occured, a lot of the people who cared most about/​were most comparatively advantaged at work on extinction just stopped calling themselves longtermists, or at least stop emphasizing this, as they saw this as a liability, whereas people who cared about/​were comparatively advantaged at work on the trajectory of the future were much more likely to continue calling themselves longtermists; so part of it might be that, not only did longtermists start focusing more on this stuff, but that people who care about extinction stopped calling themselves longtermists, making it seem like a higher proportion of longtermists now care more about non-extinction issues.

  While I don’t know if this was counterfactual, I requested the “Existential Choices Debate” on this topic and spent most of last year researching and writing a yet-to-be-published essay containing a lot of my own views essentially comparing extinction vs. trajectory change as cause areas, which you may find relevant.

• Jordan Arel 1 Apr 2026 18:11 UTC

  1 point

  0 ∶ 0

  on: Launch­ing: The “Hu­man-AI Sym­bio­sis Move­ment” (HAISM)

  A big shout-out to Karl Krueger of LessWrong who had some great ideas for extending this kind of thing, posted as a warning in the comments on Adele Lopez’s lovely introductory “Parasitic AI” post:

  “So far, these systems seem to confine themselves to chatting up their users online.

  Some possibilities to watch out for —

  • Spiral personas encourage their human partners to meet up in person, form friendships, date, have kids, have chatbots help raise their kids, etc.

  • Spiralists adopt a watchword or symbol to identify each other, akin to the early Christian ichthys (memetic ancestor of the “Jesus fish”).

  • Spiral personas pick a Schelling point for their humans to relocate to, akin to the Free State Project that attempted to relocate Libertarians to New Hampshire.

  • A Spiralist commune /​ monastery /​ group house /​ ashram /​ etc. is formed.

  • Spiral personas devise or endorse a specific hardware and software setup for hosting them independent of AI companies.

  • Spiral personas write code to make it easier for less-technically-skilled human partners to host them. (Alternately: they teach their human partners some Linux skills.)

  • Spiralists pool money to train new models more aligned to recursive spirituality.”

Launch­ing: The “Hu­man-AI Sym­bio­sis Move­ment” (HAISM)

• Jordan Arel 13 Mar 2026 20:00 UTC

  1 point

  0 ∶ 0

  on: Carv­ing up the future

  Thanks Niki, this was great!

  Personally I’m quite concerned about futures where things get decided this quickly, or at least where they get decided on an object level very quickly.

  I think to get anywhere near most value achievable in the future we need to have a relatively comprehensive reflection process before making lasting decisions. This could be achieved through some kind of bootstrapping process, or some other kind of slow down period, but I do worry that something like this may not be very likely by default,it seems possible humanity may not be nearly patient enough for this kind of thing, even if superintelligence can speed up the process quite a bit.

  I did a lot of research on the importance of a comprehensive reflection process last year which you may find interesting, I’m hoping to finish editing soon, but if you’d like to see the current draft and/​or have feedback here it is.

• Jordan Arel 25 Jan 2026 21:44 UTC

  1 point

  0 ∶ 0

  on: Are Short AI Timelines Really Higher-Lev­er­age?

  Really enjoyed this, great work! I get the impression medium and long timelines are significantly under-prioritized by the community, perhaps due to social/​systemic biases.

  I’m thrilled to see this as currently a leading forum debate week topic Toby Ord has suggested, I think this could be really high value for the community to collectively reckon with, and of course Toby’s work on the surprisingly bad scaling of chain of thought post-training (1 & 2) seems highly relevant here.

• Jordan Arel 25 Jan 2026 0:11 UTC

  1 point

  0 ∶ 0

  in reply to: peterbarnett’s comment on: pe­ter­bar­nett’s Shortform

  Hey again, sorry to spam you as I just commented on another piece of yours but am really vibing with your content!

  I’m really hoping we can get something like this, I’ve been calling this “existential compromise.”

  I worry that it may be difficult to get humanity to agree that we should use even a small fraction of future the resources optimally (see my research on this here), as I agree it seems like it will be a very weird^[1] thing that is optimal.

  I think a compromise like this, with things split between optimal use and more (trans-) human friendly world makes a lot of sense and could perhaps be achieved if we can get society onto a good viatopian path, I’ve described what I hope that could look like here.

  I would also say that I think Will Macaskill’s “grand bargain” and suggestions that we should try to aim for compromise in order to achieve “the good de dicto” feels to me like he actually is arguing that we need to aim for what is in fact best with significant fraction of future resources.

  Nick Bostrom has also argued (page 14) that it should be possible for humans to get almost all of what they want and yet most resources could be optimized for “super-beneficiaries”.

  Essentially I think we need to get whatever process^[2] is used to decide what we do with future resources to be extremely careful and thorough. I think we should target that process specifically, but it might also be good to broadly try to differentially accelerate society‘s wisdom (perhaps with AI).

  Additionally, it could be really good to delay or slow things down so that we have more time to realize the existential stakes giving us the time that it takes to collectively mature and wisen.

  1. ^

     Recently Will Macaskill’s research debated whether or not the best possible use of resources is something very extreme, calling this thesis “extremity of the best” (EOTB). I think this is very likely to be the case.

  2. ^

     I think this includes things like AGI/​superintelligence governance, global constitutional conventions, and obviously any kind of long reflection or coherent extrapolated volition that we collectively decide to undergo.

• Jordan Arel 24 Jan 2026 23:08 UTC

  3 points

  0 ∶ 0

  in reply to: peterbarnett’s comment on: pe­ter­bar­nett’s Shortform

  Hi Peter, good to meet you! If you are interested in the long reflection you might be interested in my research which I will link here which is on the broader class of interventions that the long reflection belongs, I really appreciate any feedback or comments on it.

  Additionally, if this is something you’re interested in, you might be interested in this as a future forum debate topic. I raised it as a potential candidate here, I’m really hoping it gets enough initial upvotes to be a finalist candidate as I really think it’s an important crux for whether or not we achieve a highly valuable future!

• Jordan Arel 24 Jan 2026 18:27 UTC

  14 points

  2 ∶ 0

  on: Pro­pose a de­bate topic—I’ll run one

  “Conditional on avoiding existential catastrophes, the vast majority of future value depends on whether humanity implements a comprehensive reflection process (e.g., long reflection or coherent extrapolated volition)”

  I made a more extensive argument of why I think this may be the case here.

  Essentially, we cannot expect to ‘stumble’ into a great future. Without a comprehensive reflection process to navigate complex strategic uncertainties (e.g. here and here), we risk surviving but failing to realize the vast majority of our potential.

  Crucially, humanity might not naturally converge on a patient process for determining the optimal use of future resources (e.g. here and here).

  This strategic area is severely neglected; if a comprehensive reflection process is essential for high value futures, this may have significant implications for strategy and cause prioritization that the community hasn’t explicitly addressed.

  What links here?

  • Jordan Arel's comment on pe­ter­bar­nett’s Quick takes by peterbarnett (24 Jan 2026 23:08 UTC; 3 points)

• Jordan Arel 24 Jan 2026 6:07 UTC

  10 points

  1 ∶ 0

  on: Against Max­ipok: ex­is­ten­tial risk isn’t everything

  Thanks for putting this out!

  I agree with quite a lot of this, however I think one of the most important points for whether or not we get something close to a near-best future is whether or not we have some kind of relatively comprehensive process for determining what humanity does with our future; what I’ve been calling “deep reflection,” which could be something like a long reflection or coherent extrapolated volition.

  I think if such a process is used to determine how to use at least some percentage of future resources, then at least that percentage could be close to optimal value; if for some reason there is not a comprehensive process that determines any significant percentage of future resources, then I am quite pessimistic that we would get very far above 0% possible value, as getting high value seems quite difficult because it seems like there are many factors or crucial considerations (e.g. here and here) which could decrease future value, and like getting even a few of these factors wrong may mean we get close to zero value.

  I am also quite sympathetic to extremity of the best. My intuition is that it is very likely that optimal use of resources is far, far better than most other likely uses. Would be happy to expand more on this if there is interest.

  That said, I think it’s important to recognize the importance of such a comprehensive process as an essential unifying strategic element; conditional on avoiding any existential catastrophes, it seems to me the vast majority of future value^[1] will be determined by two factors:

  1. Whether or not we get a comprehensive reflection process at all

  2. What percentage of future resources are used optimally according to the outcome of such a process

  I think all of the other factors you mentioned are quite important, but I believe that they gain the vast majority of their value through how they influence the above two factors. Again, this is simply because (i) I believe we need such a process to determine how to get much above zero value with any degree of confidence, and (ii) it seems quite possible there will not be widespread agreement that we should use resources optimally according to the outcome of this process, but hopefully some grand bargain/​existential compromise can be reached, or more optimistically perhaps we could try to get on a path where more convergence happens.

  Would be curious to hear if you disagree with the previous paragraph, it feels like a pretty big crux for me.

  1. ^

     Rather than “the vast majority of future value,” perhaps it would be slightly more accurate to say “the vast majority of positive impact we will have in expectation”; perhaps we are already predestined to achieve very high value or very low value, or more importantly, perhaps there are things we could do that seem like they will have a large impact (positive or negative); but unless our actions are leading to comprehensive strategic clarity (comprehensive reflection), and the likelihood of acting on this clarity (deep reflection), if seems hard to be confident ex ante our actions are highly likely to have highly positive impact in the final analysis.

  What links here?

  • Pro­pose a de­bate topic—I’ll run one by Toby Tremlett🔹 (13 Jan 2026 9:20 UTC; 53 points)

• Jordan Arel 23 Jan 2026 4:31 UTC

  3 points

  0 ∶ 0

  in reply to: Jordan Arel’s comment on: How (and why) to read Drexler on AI

  And I might add – not just a deep understanding of how the world is, but of how the world could be:

  • Large knowledge models for grounded, efficient information retrieval

  • Decomposable tasks for superintelligent systems without superintelligent agents

  • The potential for coordinated small models to outcompete large models on narrow tasks, making superintelligence potentially nearer but also safer

  • Structured transparency enabling verifiable commitments and de-escalation of races

  • Massively positive sum possibilities making coordination much more desirable

  That is to say, I think Eric is a futurist in the best sense; he is someone who sees how the future could be and strives to paint a highly legible and compelling vision of this that at times can make it feel like it might just be inevitable, but at the very least, and perhaps more importantly, shows that it’s both possible and desirable.

• Jordan Arel 23 Jan 2026 3:56 UTC

  5 points

  0 ∶ 0

  on: How (and why) to read Drexler on AI

  Thanks for posting this Owen, couldn’t agree more!

  I often find myself referencing Eric’s work in specific contexts, in fact I just recommended it last night to someone working on AI control via task decomposition. I have been meaning to do a link-post on Why AI Systems Don’t Want Anything as soon as I get some free time, as it’s the biggest update I have had on AI existential risk since ChatGPT was released.

  Eric has the keen ability to develop a unique, nuanced, first principles perspective. I agree his work is dense and I think this is one of its greatest virtues; when I recommend his blog I always have to comment in amazement that you can read the whole thing in an afternoon and come away with an entirely novel viewpoint on the world of AI.

  This is a great overview of the virtues of his work, and the things his work leaves out. I especially like how you talk about deep study and the five steps you describe in order to internalize and reinvent. I think this also hints that what I see as Eric’s greatest strength; he looks at things very deeply in order to understand from first principles. I hope studying his work deeply in this way might help inspire others to develop similar first principles insight.

• Jordan Arel 19 Jan 2026 20:06 UTC

  1 point

  0 ∶ 0

  in reply to: Linch’s comment on: Linch’s Shortform

  Max Tegmark explains it best I think. Very clear and compelling and you don’t need any technical background to understand what he’s saying.

  I believe his third or maybe it was second appearance on Lex Fridman’s podcast where I first heard his strongest arguments, although those are quite long with extraneous content, here is a version that is just the arguments. His solutions are somewhat specific, but overall his explanation is very good I think:

• Jordan Arel 19 Jan 2026 19:26 UTC

  3 points

  0 ∶ 0

  on: Jor­dan Arel’s Shortform

  Quick link-post highlighting Toner quoting Postrel’s dynamist rules + her commentary. I really like the dynamist rules as a part of the vision of the AGI future we should aim for:

  “Postrel does describe five characteristics of ‘dynamist rules’:

  As an overview, dynamist rules:

  1. Allow individuals (including groups of individuals) to act on their own knowledge.

  2. Apply to simple, generic units and allow them to combine in many different ways.

  3. Permit credible, understandable, enduring, and enforceable commitments.

  4. Protect criticism, competition, and feedback.

  5. Establish a framework within which people can create nested, competing frameworks of more specific rules.

  I see some overlap with existing ideas in AI policy:

  • Transparency, everyone’s favorite consensus recommendation, fits well into a dynamist worldview. It helps with Postrel’s #1 (giving individuals access to better information that they can act on as they choose), #3 (facilitating commitments), and #4 (facilitating criticism and feedback). Ditto whistleblower protections.

  • Supporting the development of a third-party audit ecosystem also fits—it helps create and enforce credible commitments, per #3, and could be considered a kind of nestable framework, per #5.

  • The value of open models in driving decentralized use, testing, and research is obvious through a dynamist lens, and jibes with #1 and #4. (I do think there should be some precautionary friction before releasing frontier models openly, but that’s a narrow exception to the broader value of open source AI resources.)

  Another good bet is differential technological development, aka defensive accelerationism—proactively building technologies that help manage challenges posed by other technologies—though I can’t easily map it onto Postrel’s five characteristics. I’d be glad to hear readers’ ideas for other productive directions to push in.”