Oliver Sourbut

• Oliver Sourbut 30 Jul 2025 13:59 UTC

  1 point

  0 ∶ 0

  on: Bet­ter than log­a­r­ith­mic re­turns to rea­son­ing?

  Some gestures which didn’t make the cut as they’re too woolly or not quite the right shape:

  • adversarial exponentials might force exponential expense per gain

    • e.g. combatting replicators

    • e.g. brute forcing passwords

  • many empirical ‘learning curve’ effects appear to consume exponential observations per increment

    • Wright’s Law (which is the more general cousin of Moore’s Law) requires exponentially many production iterations per incremental efficiency gain

    • Deep learning scaling laws appear to consume exponential inputs per incremental gain

    • AlphaCode and AlphaZero appear to make uniform gains per runtime compute doubling

    • OpenAI’s o-series ‘reasoning models’ appear to improve accuracy on many benchmarks with logarithmic returns to more ‘test time’ compute

    • (in all of these examples, there’s some choice of what scale to represent ‘output’ on, which affects whether the gains look uniform or not, so the thesis rests on whether the choices made are ‘natural’ in some way)

Bet­ter than log­a­r­ith­mic re­turns to rea­son­ing?

• Oliver Sourbut 9 Jun 2025 8:19 UTC

  5 points

  1 ∶ 0

  on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  This is lovely, thank you!

  My main concern would be that it takes the same very approximating stance as much other writing in the area, conflating all kinds of algorithmic progress into a single scalar ‘quality of the algorithms’.

  You do moderately well here, noting that the most direct interpretation of your model regards speed or runtime compute efficiency, yielding ‘copies that can be run’ as the immediate downstream consequence (and discussing in a footnote the relationship to ‘intelligence’^[1] and the distinction between ‘inference’ and training compute).

  I worry that many readers don’t track those (important!) distinctions and tend to conflate these concepts. For what it’s worth, by distinguishing these concepts, I have come to the (tentative) conclusion that a speed/​compute efficiency explosion is plausible (though not guaranteed), but an ‘intelligence’ explosion in software alone is less likely, except as a downstream effect of running faster (which might be nontrivial if pouring more effective compute into training and runtime yields meaningful gains).

  ---

  1. ↩︎

     Of course, ‘intelligence’ is also very many-dimensional! I think the most important factor in discussions like these regarding takeoff is ‘sample efficiency’, since that’s quite generalisable and feeds into most downstream applications of more generic ‘intelligence’ resources. This is relevant to R&D because sample efficiency affects how quickly you can accrue research taste, which controls the stable level of your exploration quality. Domain-knowledge and taste are obviously less generalisable, and harder to get in silico alone.

• Oliver Sourbut 20 May 2025 5:30 UTC

  1 point

  0 ∶ 0

  in reply to: Jordan Arel’s comment on: FLF Fel­low­ship on AI for Hu­man Rea­son­ing: $25-50k, 12 weeks

  Glad to hear it! Any particular thoughts or suggestions? (Consider applying, or telling colleagues and friends you think would be a good fit!)

FLF Fel­low­ship on AI for Hu­man Rea­son­ing: $25-50k, 12 weeks

• Oliver Sourbut 19 May 2025 10:57 UTC

  1 point

  0 ∶ 0

  on: What if we just…didn’t build AGI? An Ar­gu­ment Against Inevitability

  On this note, the Future of Life Foundation (headed by Anthony Aguirre, mentioned in this post) is today launching a fellowship on AI for Human Reasoning.

  Why? Whether you expect gradual or sudden AI takeoff, and whether you’re afraid of gradual or acute catastrophes, it really matters how well-informed, clear-headed, and free from coordination failures we are navigating into and through AI transitions. Just the occasion for human reasoning uplift!

  12 weeks, $25-50k stipend, mentorship, and potential pathways to future funding and impact. Applications close June 9th.

• Oliver Sourbut 23 Apr 2025 13:31 UTC

  5 points

  0 ∶ 0

  on: Knowl­edge, Rea­son­ing, and Superintelligence

  (cross-posted on LW)

  Love this!

  As presaged in our verbal discussion my top conceptual complement would be to emphasise exploration/​experimentation as central to the knowledge production loop—the cycle of ‘developing good taste to plan better experiments to improve taste (and planning model)’ is critical (indispensable?) to ‘produce new knowledge which is very helpful by the standards of human civilization’ (on any kind of meaningful timescale).

  This because just flailing, or even just ‘doing stuff’, gets you some novelty of observations, but directedly seeking informative circumstances at the boundaries of the known (which includes making novel unpredictable events happen, as well as getting equipped with richer means to observe and record them, and perhaps preparing to deliberatively extract insight) turns out to be able to mine vastly more insight per resource (time, materials, etc.). Hence science, but also hence individual human and animal playfulness, curiosity, adversarial exercises and drills (self-play ish), and whatnot.

  Said another way, maybe I’d characterise ‘the way that fluid intelligence and crystallised intelligence synergise in the knowledge production loop’ as ‘directed exploration/​experimentation’?

  Having said that, I don’t necessarily think these capacities need to reside ‘in the same mind’, just as contemporary human orgs get more of this done and more effectively than individuals. But the pieces do need to be fit to each other (like, a physicist with great physics taste can’t usually very well complement a bio lab without first becoming a person with great bio taste).

Co­op­er­a­tion and Align­ment in Del­e­ga­tion Games: You Need Both!

• Oliver Sourbut 25 Jul 2024 13:01 UTC

  5 points

  0 ∶ 0

  on: De­com­pos­ing Agency — ca­pa­bil­ities with­out desires

  I like this decomposition!

  I think ‘Situational Awareness’ can quite sensibly be further divided up into ‘Observation’ and ‘Understanding’.

  The classic control loop of ‘observe’, ‘understand’, ‘decide’, ‘act’^[1], is consistent with this discussion, where ‘observe’+‘understand’ here are combined as ‘situational awareness’, and you’re pulling out ‘goals’ and ‘planning capacity’ as separable aspects of ‘decide’.

  Are there some difficulties with factoring?

  Certain kinds of situational awareness are more or less fit for certain goals. And further, the important ‘really agenty’ thing of making plans to improve situational awareness does mean that ‘situational awareness’ is quite coupled to ‘goals’ and to ‘implementation capacity’ for many advanced systems. Doesn’t mean those parts need to reside in the same subsystem, but it does mean we should expect arbitrary mix and match to work less well than co-adapted components—hard to say how much less (I think this is borne out by observations of bureaucracies and some AI applications to date).

  ---

  1. ↩︎

     Terminology varies a lot; this is RL-ish terminology. Classic analogues might be ‘feedback’, ‘process model’/​‘inference’, ‘control algorithm’, ‘actuate’/​‘affect’…

• Oliver Sourbut 30 Apr 2024 14:46 UTC

  6 points

  0 ∶ 0

  in reply to: Oliver Sourbut’s comment on: Ca­reers Ques­tions Open Thread

  A little followup:

  I took part in the inaugural SERI MATS programme in 2021-2022 (where incidentally I interacted with Richard), and started an AI Safety PhD at Oxford in 22.

  I’m now working for the AI Safety Institute (UK Gov) since Jan 2024 as a hybrid technical expert, utilising my engineering and DS background, alongside AI/​ML research and threat modelling. Likely to continue such work, there or elsewhere. Unsure if I’ll finish my PhD in the end, as a result, but I don’t regret it: I produced a little research, met some great collaborators, and had fun while learning as a consequence!

  Between the original thread and my leaving for PhD, I’d say I grew my engineering, DS, and project management skills a little, though diminishing, while also doing a lot of AIS prep. My total income also went up while I remained FT employed. This was due for a slowdown as a consequence of stock movements and vesting, but regardless I definitely forwent a lot of money thanks to becoming a student again (and then a researcher rather than a high-paid engineer)! As far as I can tell this is the main price I paid, in terms of both personal situation and impact, and perhaps I should have made the move sooner (though having money in the bank is very freeing and enables indirect impact).

• Oliver Sourbut 28 Apr 2024 21:40 UTC

  8 points

  0 ∶ 0

  on: AI Reg­u­la­tion is Unsafe

  FWIW I work at the AI Safety Institute UK and we’re considering a range of both misuse and misalignment threats, and there are a lot of smart folks on board taking things pretty seriously. I admit I… don’t fully understand how we ended up in this situation and it feels contingent and precious, as does the tentative international consensus on the value of cooperation on safety (e.g. the Bletchley declaration). Some people in government are quite good, actually!

• Oliver Sourbut 1 Jan 2024 0:11 UTC

  1 point

  0 ∶ 0

  in reply to: Larks’s comment on: Public Fundrais­ing has Pos­i­tive Externalities

  Sure, take it or leave it! I think for the field-building benefits it can look more obviously like an externality (though I-the-fundraiser would in fact be pleased and not indifferent, presumably!), but the epistemic benefits could easily accrue mainly to me-the-fundraiser (of course they could also benefit other parties).

• Oliver Sourbut 30 Dec 2023 11:12 UTC

  12 points

  4 ∶ 0

  on: Altru­ism sharp­ens altruism

  How much of this is lost by compressing to something like: virtue ethics is an effective consequentialist heuristic?

  I’ve been bought into that idea for a long time. As Shaq says, ‘Excellence is not a singular act, but a habit. You are what you repeatedly do.’

  We can also make analogies to martial arts, music, sports, and other practice/​drills, and to aspects of reinforcement learning (artificial and natural).

• Oliver Sourbut 30 Dec 2023 11:05 UTC

  5 points

  0 ∶ 0

  on: Public Fundrais­ing has Pos­i­tive Externalities

  Simple, clear, thought-provoking model. Thanks!

  I also faintly recall hearing something similar in this vicinity: apparently some volunteering groups get zero (or less!?) value from many/​most volunteers, but engaged volunteers dominate donations, so it’s worthwhile bringing in volunteers and training them! (citation very much needed)

  Nitpick: are these ‘externalities’? I’d have said, ‘side effects’. An externality is a third-party impact from some interaction between two parties. The effects you’re describing don’t seem to be distinguished by being third-party per se (I can imagine glossing them as such but it’s not central or necessary to the model).

• Oliver Sourbut 30 Dec 2023 10:56 UTC

  5 points

  2 ∶ 0

  in reply to: SiebeRozendal’s comment on: At­ten­tion on AI X-Risk Likely Hasn’t Dis­tracted from Cur­rent Harms from AI

  Yeah. I also sometimes use ‘extinction-level’ if I expect my interlocutor not to already have a clear notion of ‘existential’.

• Oliver Sourbut 21 Dec 2023 23:22 UTC

  3 points

  1 ∶ 0

  on: OpenAI’s Su­per­al­ign­ment team has opened Fast Grants

  Point of information: at least half the funding comes from Schmidt futures (not OpenAI), though OpenAI are publicising and administrating it.

• Oliver Sourbut 1 Nov 2023 8:49 UTC

  1 point

  0 ∶ 0

  on: Thoughts on the AI Safety Sum­mit com­pany policy re­quests and responses

  Another high(er?) priority for governments:

  • start building multilateral consensus and preparations on what to do if/​when

    • AI developers go rogue

    • AI leaked to/​stolen by rogue operators

    • AI goes rogue

• Oliver Sourbut 12 Oct 2023 15:33 UTC

  31 points

  11 ∶ 1

  on: Pause For Thought: The AI Pause Debate

  I think this is a good and useful post in many ways, in particular laying out a partial taxonomy of differing pause proposals and gesturing at their grounding and assumptions. What follows is a mildly heated response I had a few days ago, whose heatedness I don’t necessarily endorse but whose content seems important to me.

  Sadly this letter is full of thoughtless remarks about China and the US/​West. Scott, you should know better. Words have power. I recently wrote an admonishment to CAIS for something similar.

  The biggest disadvantage of pausing for a long time is that it gives bad actors (eg China) a chance to catch up.

  There are literal misanthropic ‘effective accelerationists’ in San Francisco, some of whose stated purpose is to train/​develop AI which can surpass and replace humanity. There’s Facebook/​Meta, whose leaders and executives have been publicly pooh-poohing discussion of AI-related risks as pseudoscience for years, and whose actual motto is ‘move fast and break things’. There’s OpenAI, which with great trumpeting announces its ‘Superalignment’ strategy without apparently pausing to think, ‘But what if we can’t align AGI in 5 years?‘. We don’t need to invoke bogeyman ‘China’ to make this sort of point. Note also that the CCP (along with EU and UK gov) has so far been more active in AI restraint and regulation than, say, the US government, or orgs like Facebook/​Meta.

  Suppose the West is right on the verge of creating dangerous AI, and China is two years away. It seems like the right length of pause is 1.9999 years, so that we get the benefit of maximum extra alignment research and social prep time, but the West still beats China.

  Now, this was in the context of paraphrases of others’ positions on a pause in AI development, so it’s at least slightly mention-flavoured (as opposed to use). But as far as I can tell, the precise framing here has been introduced in Scott’s retelling.

  Whoever introduced this formulation, this is bonkers in at least two ways. First, who is ‘the West’ and who is ‘China’? This hypothetical frames us as hivemind creatures in a two-player strategy game with a single lever. Reality is a lot more porous than that, in ways which matter (strategically and in terms of outcomes). I shouldn’t have to point this out, so this is a little bewildering to read. Let me reiterate: governments are not currently pursuing advanced AI development, only companies. The companies are somewhat international, mainly headquartered in the US and UK but also to some extent China and EU, and the governments have thus far been unwitting passengers with respect to the outcomes. Of course, these things can change.

  Second, actually think about the hypothetical where ‘we’^[1] are ‘on the verge of creating dangerous AI’. For sufficient ‘dangerous’, the only winning option for humanity is to take the steps we can to prevent, or at least delay^[2], that thing coming into being. This includes advocacy, diplomacy, ‘aggressive diplomacy’ and so on. I put forward that the right length of pause then is ‘at least as long as it takes to make the thing not dangerous’. You don’t win by capturing the dubious accolade of nominally belonging to the bloc which directly destroys everything! To be clear, I think Scott and I agree that ‘dangerous AI’ here is shorthand for, ‘AI that could defeat/​destroy/​disempower all humans in something comparable to an extinction event’. We already have weak AI which is dangerous to lesser levels. Of course, if ‘dangerous’ is more qualified, then we can talk about the tradeoffs of risking destroying everything vs ‘us’ winning a supposed race with ‘them’.

  I’m increasingly running with the hypothesis that many anglophones are mind-killed on the inevitability of contemporary great power conflict in a way which I think wasn’t the case even, say, 5 years ago. Maybe this is how thinking people felt in the run up to WWI, I don’t know.

  I wonder if a crux here is some kind of general factor of trustingness toward companies vs toward governments—I think extremising this factor would change the way I talk and think about such matters. I notice that a lot of American libertarians seem to have a warm glow around ‘company/​enterprise’ that they don’t have around ‘government/​regulation’.

  [ In my post about this I outline some other possible cruxes and I’d love to hear takes on these ]

  Separately, I’ve got increasingly close to the frontier of AI research and AI safety research, and the challenge of ensuring these systems are safe remains very daunting. I think some policy/​people-minded discussions are missing this rather crucial observation. If you expect it to be easy (and expect others to expect that) to control AGI, I can see more why people would frame things around power struggles and racing. For this reason, I consider it worthwhile repeating: we don’t know how to ensure these systems will be safe, and there are some good reasons to expect that they won’t be by default.

  I repeat that the post as a whole is doing a service and I’m excited to see more contributions to the conversation around pause and differential development and so on.

  ---

  1. ↩︎

     Who, me? You? No! Some development team at DeepMind or OpenAI, presumably, or one of the current small gaggle of other contenders, or a yet-to-be-founded lab.

  2. ↩︎

     If it comes to it, extinction an hour later is better than an hour sooner.

• Oliver Sourbut 28 Sep 2023 10:02 UTC

  3 points

  0 ∶ 2

  on: Aim for con­di­tional pauses

  I think that the best work on AI alignment happens at the AGI labs

  Based on your other discussion e.g. about public pressure on labs, it seems like this might be a (minor?) loadbearing belief?

  I appreciate that you qualify this further in a footnote

  This is a controversial view, but I’d guess it’s a majority opinion amongst AI alignment researchers.

  I just wanted to call out that I weakly hold the opposite position, and also opposite best guess on majority opinion (based on safety researchers I know). Naturally there are sampling effects!

  This is a marginal sentiment, and I certainly wouldn’t trade all lab researchers for non-lab researchers or vice versa. Diversification of research settings seems quite precious, and the dialogue is important to preserve.

  I also question

  Reasons include: access to the best alignment talent,

  because a lot of us are very reluctant to join AGI labs, for obvious reasons! I know folks inside and outside of AGI labs, and it seems to me that the most talented are among the outsiders (but this also definitely could be an artefact of sample sizes).

• Oliver Sourbut 22 Sep 2023 4:34 UTC

  8 points

  0 ∶ 0

  in reply to: aog’s comment on: Care­less talk on US-China AI com­pe­ti­tion? (and crit­i­cism of CAIS cov­er­age)

  This is an exemplary and welcome response: concise, full-throated, actioned. Respect, thank you Aidan.

  Sincerely, I hope my feedback was all-considered good from your perspective. As I noted in this post, I felt my initial email was slightly unkind at one point, but I am overall glad I shared it—you appreciate my getting exercised about this, even over a few paragraphs!

  It’s important to discuss national AI policies which are often explicitly motivated by goals of competition without legitimizing or justifying zero-sum competitive mindsets which can undermine efforts to cooperate.

  Yes, and I repeat that CAIS newsletter has a good balance of nuance, correctness, helpfulness, reach. Hopefully your example here sets the tone for conversations in this space!