Oliver Sourbut

• Oliver Sourbut 8 Jun 2026 19:48 UTC

  1 point

  0 ∶ 0

  in reply to: Jmaar’s comment on: Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition

  Thanks Jim! Be sure to pop your details on the expression of interest if you haven’t already. We’ll connect people together who’d like to be in contact while participating.

Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition

A (Slightly) Mechanis­tic The­ory for Ex­po­nen­tially In­creas­ing AI Time Hori­zons?

Re­in­force­ment learn­ing scal­ing might in­cen­tivise hid­den rea­son­ing ar­chi­tec­tures for AI

• Oliver Sourbut 14 Apr 2026 9:00 UTC

  5 points

  0 ∶ 0

  on: AI for epistemics: the good, the bad and the ugly

  I appreciate this discussion a lot. Two things which stand out to me as deserving more emphasis.

  First though, quickly framing ‘good epistemic outcomes’ as something like a product of ‘people trying to understand clearly’ and ‘people can do that effectively’. (Of course these are interrelated, because people’s willingness is obviously affected by the practicalities—more on that in point 2.)

  OK, the things:

  1. It looks to me like most of the object-level task of collective epistemics is the checking and generally piecing together good ‘secondary research’ (broadly construed). i.e. looking at provenance, tracking the evidence and reasoning dependencies for a claim, proactively gathering the best arguments for and against, reasons to downweight certain testimony etc.

     • Why? Almost all our information about our environment beyond our direct sensory access is mediated through highly iterated message passing, reinterpretation, aggregation, and so on—especially in the heights of science and the depths (!) of political/​influence goings-on

     • AI enables this (The Good) not so much (directly) by ‘knowing’ more or having ‘more insights’, but rather by hugely expanding the availability of clerical checking, tracing, and knowledge mapping work!

     • You kind of talk about this in the collective epistemics discussion, but I think it warrants more

  2. Most of the overall task of collective epistemics may be in the motivating i.e. having more people more of the time actually trying to understand things with accuracy, rather than retreating into one or other alternative cognitive mode

     • The usual label I use for alternative cognitive modes is ‘tribal cognition’, where most of what’s said and recounted (and even believed), especially (but not even only) about what’s outside of the immediate sensory environment, is in service of building and maintaining allegiances and coalitions

     • When is ‘tribal cognition’ incentivised? I don’t fully know, but it has to do with

       • When people are/​feel threatened, they reach for affiliations which offer (perhaps passing or merely apparent) security

         • Abusers can play on this by a combination of bigging up threats and presenting as a effective and sympathetic

       • When the epistemic environment is difficult true perception is more difficult and less rewarded

         • Abusers can push this. In politics: flood the zone, firehose of falsehoods, FUD. In science: p-hacking, importance-hacking, conflating/​obscuring methodologies.

       • Generally adding noise and more convincing fake content undermines The Good above, the ability to check and trace, not by making people believe the fake stuff but by making them correctly recognise that it’s hard to tell at all (thus ‘retreat’)

       • Certain coalition norms can encourage epistemic insularity and discourage (genuine) scrutiny

     • I think you’re touching on this in The Ugly, ‘undermine sense-making’. To me it’s possibly ‘most of the problem’! Or at least, understanding under what conditions people mobilise one or other cognitive intents in sensemaking, and how those conditions can be influenced is a really big part of the picture here.

• Oliver Sourbut 14 Apr 2026 8:42 UTC

  1 point

  0 ∶ 0

  on: AI for epistemics: the good, the bad and the ugly

  It’s ‘Sourbut’ (one ‘t’, thankfully!)

Defense-favoured co­or­di­na­tion de­sign sketches

Orders of mag­ni­tude: use semi­tones, not decibels

• Oliver Sourbut 24 Feb 2026 9:53 UTC

  5 points

  0 ∶ 0

  on: Which ques­tions can’t we punt?

  I like this, and it’s simultaneously exciting and bewildering to take seriously the prospect of punting difficult things.

  It could be worth emphasising more clearly that this is about (futurist) strategy, which is about as cognitive as things get. Other types of preparation and problem-solving have other critical inputs, and may face ~inherent delays. For those, ‘punting’ can look risky, especially if you expect later phases to move quite fast. This has bearing on strategy: it’s worth attempting to foretell the kinds of lead-time-constrained preparation that might be needed to face upcoming challenges.

  (A concrete example that stands out to me is bio monitoring and defenses. But in general I’d love to see more and richer work on characterising emerging threats, especially technological. Not necessarily from Forethought! Other kinds of lead-time-constrained activities might involve coalition building and spreading well-informed takes about important topics.)

• Oliver Sourbut 24 Feb 2026 9:48 UTC

  5 points

  1 ∶ 0

  in reply to: cb’s comment on: Which ques­tions can’t we punt?

  Knowing these authors, my guess on ontology is that they might say that it could be instrumental in things like

  • motivating progress in safer paradigms of AI development

  • understanding ‘hybrid’ human-AI-org opportunities and threats

  • figuring out what types of ‘post early’ conditions look favourable for dealing with the next challenges

  These all look like activities with bearing on how to tackle ‘early’ challenges.

• Oliver Sourbut 19 Feb 2026 14:32 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  Helpful, thanks, I think I understand a little bit better now (still not yet sure what the specific tuple elements are doing)!

  In case it’s inspiring or can provoke useful critique, here are some areas where I think compounding/​reuse can be really useful in epistemic activities are:

  • claim decomposition of larger artefacts

  • citation (and perhaps provenance) resolution

  • clustering related claims and evidence

  • collecting ‘topics’ (including perhaps differing perspectives on broad or narrow subjects)

  • relating (especially, backlinking) claims which weaken/​elaborate/​refute other claims

  • detecting loadbearing evidence or subclaims

  • flagging scarcity of evidence or analysis

• Oliver Sourbut 17 Feb 2026 15:31 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  See also the collective epistemics discussion, if you haven’t already, which I suspect might also be of interest to you!

• Oliver Sourbut 17 Feb 2026 15:30 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  Could you explain the community reuse thing again? I don’t understand the tuples, but is the idea that query responses (which yield something like document sets?) can be cached with some identifiers? This helps future users by...? (Thinking: it can serve as a tag to a reproducible/​amendable/​updateable query, it can save someone running the exact same query again, …)

• Oliver Sourbut 17 Feb 2026 15:27 UTC

  1 point

  0 ∶ 0

  in reply to: exopriors’s comment on: Some tools for strate­gic awareness

  That looks ambitious and awesome! I haven’t looked deeply, but a few quick qs

  • what do the costs look like to get embeddings for all those docs? How are you making choices about which embedding models to use and things like that?

  • do you have qualitative (or quantitative?) sense of how well the semantic joins work for queries like the examples on the homepage?

  • what’s your sense of how this compares to tools like elicit?

Some tools for strate­gic awareness

• Oliver Sourbut 9 Feb 2026 15:54 UTC

  3 points

  0 ∶ 0

  in reply to: Lizka’s comment on: The first type of trans­for­ma­tive AI?

  Basically +1 here. I guess some relevant considerations are the extent to which a tool can act as antidote to its own (or related) misuse—and under what conditions of effort, attention, compute, etc. If that can be arranged, then ‘simply’ making sure that access is somewhat distributed is a help. On the other hand, it’s conceivable that compute advantages or structural advantages could make misuse of a given tech harder to block, in which case we’d want to know that (without, perhaps, broadcasting it indiscriminately) and develop responses. Plausibly those dynamics might change nonlinearly with the introduction of epistemic/​coordination tech of other kinds at different times.

  In theory, it’s often cheaper and easier to verify the properties of a proposal (‘does it concentrate power?’) than to generate one satisfying given properties, which gives an advantage to a defender if proposals and activity are mostly visible. But subtlety and obfuscation and misdirection can mean that knowing what properties to check for is itself a difficult task, tilting the other way.

  Likewise, narrowly facilitating coordination might produce novel collusion with substantial negative externalities on outsiders. But then ex hypothesi those outsiders have an outsized incentive to block that collusion, if only they can foresee it and coordinate in turn.

  It’s confusing.

Some an­gel-on-the-shoulder AI tools

Some tools for col­lec­tive epistemics

AI for Hu­man Rea­son­ing for Rationalists

• Oliver Sourbut 16 Jan 2026 12:47 UTC

  1 point

  0 ∶ 0

  on: leaveno­harm.org – Offset Everything

  A nit

  lifestyle supports the planet, rather than taking from it

  appeals to me, I’m sure to some others, but (I sense) could come across with a particular political-tribal flavour, which you might want to try neutralising. (Or not! if that’d detract from the net appeal)