Oliver Sourbut

• Oliver Sourbut 24 Feb 2026 9:53 UTC

  5 points

  0 ∶ 0

  on: Which ques­tions can’t we punt?

  I like this, and it’s simultaneously exciting and bewildering to take seriously the prospect of punting difficult things.

  It could be worth emphasising more clearly that this is about (futurist) strategy, which is about as cognitive as things get. Other types of preparation and problem-solving have other critical inputs, and may face ~inherent delays. For those, ‘punting’ can look risky, especially if you expect later phases to move quite fast. This has bearing on strategy: it’s worth attempting to foretell the kinds of lead-time-constrained preparation that might be needed to face upcoming challenges.

  (A concrete example that stands out to me is bio monitoring and defenses. But in general I’d love to see more and richer work on characterising emerging threats, especially technological. Not necessarily from Forethought! Other kinds of lead-time-constrained activities might involve coalition building and spreading well-informed takes about important topics.)

• Oliver Sourbut 24 Feb 2026 9:48 UTC

  5 points

  1 ∶ 0

  in reply to: cb’s comment on: Which ques­tions can’t we punt?

  Knowing these authors, my guess on ontology is that they might say that it could be instrumental in things like

  • motivating progress in safer paradigms of AI development

  • understanding ‘hybrid’ human-AI-org opportunities and threats

  • figuring out what types of ‘post early’ conditions look favourable for dealing with the next challenges

  These all look like activities with bearing on how to tackle ‘early’ challenges.

• Oliver Sourbut 19 Feb 2026 14:32 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  Helpful, thanks, I think I understand a little bit better now (still not yet sure what the specific tuple elements are doing)!

  In case it’s inspiring or can provoke useful critique, here are some areas where I think compounding/​reuse can be really useful in epistemic activities are:

  • claim decomposition of larger artefacts

  • citation (and perhaps provenance) resolution

  • clustering related claims and evidence

  • collecting ‘topics’ (including perhaps differing perspectives on broad or narrow subjects)

  • relating (especially, backlinking) claims which weaken/​elaborate/​refute other claims

  • detecting loadbearing evidence or subclaims

  • flagging scarcity of evidence or analysis

• Oliver Sourbut 17 Feb 2026 15:31 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  See also the collective epistemics discussion, if you haven’t already, which I suspect might also be of interest to you!

• Oliver Sourbut 17 Feb 2026 15:30 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  Could you explain the community reuse thing again? I don’t understand the tuples, but is the idea that query responses (which yield something like document sets?) can be cached with some identifiers? This helps future users by...? (Thinking: it can serve as a tag to a reproducible/​amendable/​updateable query, it can save someone running the exact same query again, …)

• Oliver Sourbut 17 Feb 2026 15:27 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  That looks ambitious and awesome! I haven’t looked deeply, but a few quick qs

  • what do the costs look like to get embeddings for all those docs? How are you making choices about which embedding models to use and things like that?

  • do you have qualitative (or quantitative?) sense of how well the semantic joins work for queries like the examples on the homepage?

  • what’s your sense of how this compares to tools like elicit?

Some tools for strate­gic awareness

• Oliver Sourbut 9 Feb 2026 15:54 UTC

  3 points

  0 ∶ 0

  in reply to: Lizka’s comment on: The first type of trans­for­ma­tive AI?

  Basically +1 here. I guess some relevant considerations are the extent to which a tool can act as antidote to its own (or related) misuse—and under what conditions of effort, attention, compute, etc. If that can be arranged, then ‘simply’ making sure that access is somewhat distributed is a help. On the other hand, it’s conceivable that compute advantages or structural advantages could make misuse of a given tech harder to block, in which case we’d want to know that (without, perhaps, broadcasting it indiscriminately) and develop responses. Plausibly those dynamics might change nonlinearly with the introduction of epistemic/​coordination tech of other kinds at different times.

  In theory, it’s often cheaper and easier to verify the properties of a proposal (‘does it concentrate power?’) than to generate one satisfying given properties, which gives an advantage to a defender if proposals and activity are mostly visible. But subtlety and obfuscation and misdirection can mean that knowing what properties to check for is itself a difficult task, tilting the other way.

  Likewise, narrowly facilitating coordination might produce novel collusion with substantial negative externalities on outsiders. But then ex hypothesi those outsiders have an outsized incentive to block that collusion, if only they can foresee it and coordinate in turn.

  It’s confusing.

Some an­gel-on-the-shoulder AI tools

Some tools for col­lec­tive epistemics

AI for Hu­man Rea­son­ing for Rationalists

• Oliver Sourbut 16 Jan 2026 12:47 UTC

  1 point

  0 ∶ 0

  on: leaveno­harm.org – Offset Everything

  A nit

  lifestyle supports the planet, rather than taking from it

  appeals to me, I’m sure to some others, but (I sense) could come across with a particular political-tribal flavour, which you might want to try neutralising. (Or not! if that’d detract from the net appeal)

[Question] Another Cost Disease? We are all cap­i­tal­ists now

• Oliver Sourbut 16 Dec 2025 16:25 UTC

  1 point

  1 ∶ 0

  in reply to: Simon ’s comment on: Rerun­ning the Time of Perils

  On point 1 (space colonization), I think it’s hard and slow! So the same issue as with bio risks might apply: AGI doesn’t get you this robustness quickly for free. See other comment on this post.

• Oliver Sourbut 16 Dec 2025 16:24 UTC

  1 point

  1 ∶ 0

  in reply to: Simon ’s comment on: Rerun­ning the Time of Perils

  I like your point 2 about chancy vs merely uncertain. I guess a related point is that when the ‘runs’ of the risks are in some way correlated, having survived once is evidence that survivability is higher. (Up to an including the fully correlated ‘merely uncertain’ extreme?)

• Oliver Sourbut 16 Dec 2025 16:21 UTC

  1 point

  0 ∶ 0

  in reply to: Toby_Ord’s comment on: Rerun­ning the Time of Perils

  For clarity, you’re using ‘important’ here in something like an importance x tractability x neglectedness factoring? So yes more important (but there might be reasons to think it’s less tractable or neglected)?

• Oliver Sourbut 16 Dec 2025 16:18 UTC

  1 point

  0 ∶ 0

  on: Rerun­ning the Time of Perils

  I’ve been meaning to write something about ‘revisiting the alignment strategy’. The section 5 here (‘Won’t AGI make post-AGI catastrophes essentially irrelevant?’) makes the point very clearly:

  On this view, a post-AGI world is nearly binary—utopia or extinction—leaving little room for Sisyphean scenarios.

  But I think this is too optimistic about the speed and completeness of the transition to globally deployed, robustly aligned “guardian” systems.

  without making much of a case for it. Interested in Will and reviewers’ sense of the space and literature here.

• Oliver Sourbut 16 Dec 2025 16:16 UTC

  5 points

  1 ∶ 0

  in reply to: rosehadshar’s comment on: Rerun­ning the Time of Perils

  Yep, definitely for me ‘big civ setbacks are really bad’ was already baked in from the POV of setting bad context for pre-AGI-transition(s) (as well as their direct badness). But while I’d already agreed with Will about post-AGI not being an ‘end of history’ (in the sense that much remains uncertain re safety), I hadn’t thought through the implication that setbacks could force a rerun of the most perilous transition(s), which does add some extra concern.

• Oliver Sourbut 16 Dec 2025 16:13 UTC

  1 point

  0 ∶ 0

  on: Rerun­ning the Time of Perils

  A small aside: some put forth interplanetary civilisation as a partial defence against either of total destruction and ‘setback’. But reaching the milestone of having a really robustly interplanetary civ might itself take quite a long time after AGI—especially if (like me) you think digital uploading is nontrivial.

  (This abstractly echoes the suggestion in this piece that bio defence might take a long time, which I agree with.)

  What links here?

  • Oliver Sourbut's comment on Rerun­ning the Time of Perils by William_MacAskill (16 Dec 2025 16:25 UTC; 1 point)

• Oliver Sourbut 30 Jul 2025 13:59 UTC

  1 point

  0 ∶ 0

  on: Bet­ter than log­a­r­ith­mic re­turns to rea­son­ing?

  Some gestures which didn’t make the cut as they’re too woolly or not quite the right shape:

  • adversarial exponentials might force exponential expense per gain

    • e.g. combatting replicators

    • e.g. brute forcing passwords

  • many empirical ‘learning curve’ effects appear to consume exponential observations per increment

    • Wright’s Law (which is the more general cousin of Moore’s Law) requires exponentially many production iterations per incremental efficiency gain

    • Deep learning scaling laws appear to consume exponential inputs per incremental gain

    • AlphaCode and AlphaZero appear to make uniform gains per runtime compute doubling

    • OpenAI’s o-series ‘reasoning models’ appear to improve accuracy on many benchmarks with logarithmic returns to more ‘test time’ compute

    • (in all of these examples, there’s some choice of what scale to represent ‘output’ on, which affects whether the gains look uniform or not, so the thesis rests on whether the choices made are ‘natural’ in some way)