Oliver Sourbut

Karma: 387

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut Sep 20, 2023, 5:07 PM
0 points
0 ∶ 0
in reply to: Stephen Clare’s comment on: Careless talk on US-China AI competition? (and criticism of CAIS coverage)
Thanks for this thoughtful response!

this tendency leads to analysis that assumes more coordination among governments, companies, and individuals in other countries than is warranted. When people talk about “the US” taking some action… more likely to be aware of the nuance this ignores… less likely to consider such nuances when people talk about “China” doing something

This seems exactly right and is what I’m frustrated by. Though, further than you give credit (or un-credit) for, frequently I come across writing or talking about “US success in AI”, “US leading in AI”, or “China catching up to US”, etc. which are all almost nonsense as far as I’m concerned. What do those statements even mean? In good faith I hope for someone to describe what these sorts of claims mean in a way which clicks for me, but I have come to expect that there is probably none.

Do people actually think that Google+OpenAI+Anthropic (for sake of argument) are the US? Do they think the US government/military can/will appropriate those staff/artefacts/resources at some point? Are they referring to integration of contemporary ML/DS into the economy? The military? Or impacts on other indicators^[1]? What do people mean by “China” here: CCP, Alibaba, Tencent, …? If people mean these things, they should say those things, or otherwise say what they do mean. Otherwise I think people motte-and-bailey themselves (and others) into some really strange understandings. There’s not some linear scoreboard which “US” and “China” have points on but people behave/talk like they actually think in those terms.

your claim that governments don’t influence AI development [via semiconductor progress] is too strong

Thanks, this would indeed be too strong :) but it’s not what I mean. (Also thank you for the example bullets below that, for me and for other readers.)

I don’t mean to imply they have no influence on AI development and deployment^[2]. What I meant by ‘not currently meaningful players in AI development and deployment’ was that, to date, governments have had little to no say in the course or nature of AI development. Rather, they have been mostly passive or unwitting passengers, with recent interventions (to date) comprising coarse economy-level lever-pulls, like your examples of regulation on chip production and sales. Can you think of a better compression of this than what I wrote? ‘currently mainly passive except for coarse interventions at the economy-level’?

early demand for semiconductors was driven by the US government’s military and space program

The key difference between e.g. space-race or nuclear/ICBM etc. and AI is that in those cases, governments were appropriately thought of as somewhat-coherently instigating, steering and directing, and could be described as being key players with a real competition between them. With AI, none of those things are (currently) true. So ideally we would use different language to describe the different situations (especially because the misleading use of language is inflammatory).

I get exercised about this overall issue because on one model, this sort of failure of imagination and the confusion it gives rise to is exactly what leads to escalation and conflict, which I sense you agree on. We do not want sloppy foregone-conclusion thinking leading to WWIII with AI and nukes.
1. ↩︎
  What indicators? Education, unemployment, privacy, health, productivity, democracy, inequality, …?
2. ↩︎
  Ironically for a piece on bringing clarity through nuance, I evidently wasn’t clear enough about where I was drawing the boundaries in my initial post…

Oliver Sourbut Sep 20, 2023, 2:42 PM
3 points
0 ∶ 0
on: Israeli Prime Minister, Musk and Tegmark on AI Safety
The stream cut out, but there are longer versions available e.g. https://www.youtube.com/watch?v=Dg-rKXi9XYg

Oliver Sourbut Aug 5, 2023, 10:01 PM
3 points
1 ∶ 0
on: Are there diseconomies of scale in the reputation of communities?
Great read, and interesting analysis. I like encountering models for complex systems (like community dynamics)!

One factor I don’t think was discussed (maybe the gesture at possible inadequacy of $f (N)$ encompasses this) is the duration of scandal effects. E.g. imagine some group claiming to be the Spanish Inquisition or the Mongol Horde, or the Illuminati tried to get stuff done. I think (assuming taken seriously) they’d encounter lingering reputational damage more than one year after the original scandals! Not sure how this models out; I’m not planning to dive into it, but I think this stands out to me as the ‘next marginal fidelity gain’ for a model like this.

Oliver Sourbut Jul 21, 2023, 10:49 PM
9 points
3 ∶ 0
on: I’m interviewing Jan Leike, co-lead of OpenAI’s new Superalignment project. What should I ask him?
OpenAI as a whole, and individuals affiliated with or speaking for the org, appear to be largely behaving as if they are caught in an overdetermined race toward AGI.

What proportion of people at OpenAI believe this, and to what extent? What kind of observations, or actions or statements by others (and who?) would change their minds?

Oliver Sourbut Jul 13, 2023, 9:52 PM
1 point
0 ∶ 0
on: A Defense of Work on Mathematical AI Safety
Great post. I basically agree, but in a spirit of devil’s advocating, I will say: when I turn my mind to agent foundations thinking, I often find myself skirting queasily close to concepts which feel also capabilities-relevant (to the extent that I have avoided publicly airing several ideas for over a year).

I don’t know if that’s just me, but it does seem that some agent foundations content from the past has also had bearing on AI capabilities—especially if we include decision theory stuff, dynamic programming and RL, search, planning etc. which it’s arguably artificial to exclude. How would you ex ante distinguish e.g. work which explores properties and constraints of hypothetical planning routines from work which informs creation of more effective planning routines? This sort of thinking seems relevant.

Oliver Sourbut May 15, 2023, 1:54 PM
1 point
0 ∶ 0
in reply to: Oliver Sourbut’s comment on: Talking publicly about AI risk
I wrote a little here about unpluggability (and crossposted on LessWrong/AF)

Oliver Sourbut Apr 28, 2023, 2:56 PM
4 points
0 ∶ 0
on: Talking publicly about AI risk
Thank you for sharing this! Especially the points about relevant maps and Meta/FAIR/LeCun.

I was recently approached by the UK FCDO as a technical expert in AI with perspective on x-risk. We had what I think were very productive conversations, with an interesting convergence of my framings and the ones you’ve shared here—that’s encouraging! If I find time I’m hoping to write up some of my insights soon.

Oliver Sourbut Dec 2, 2022, 6:25 PM
1 point
1 ∶ 0
in reply to: lukeprog’s comment on: Why Neuron Counts Shouldn’t Be Used as Proxies for Moral Weight
I’ve given a little thought to this hidden qualia hypothesis but it remains very confusing for me.

To what extent should we expect to be able to tractably and knowably affect such hidden qualia?

Oliver Sourbut Sep 8, 2022, 12:30 PM
8 points
2 ∶ 0
on: Effective altruism in the garden of ends
This is beautiful and important Tyler, thank you for sharing.
I’ve seen a few people burn out (and come close myself), and I have made a point of gently socially making and reinforcing this sort of point (far less eloquently) myself, in various contexts.
I have a lot of thoughts about this subject.
One thing I embrace always is silliness and (often self-deprecating) humour, which are useful antidotes to stress for a lot of people. Incidentally, your tweet thread rendition of the Eqyptian spell includes
I am light heading for light. Even in the dark, a fire bums in the distance.
(emphasis mine) which I enjoyed. A case of bad keming reified?
A few friends and acquaintances have recently been working on something they’re calling Shard Theory, which considers the various parts of a human’s motivation system and their interactions. They’re interested for other reasons, but I was reminded here. See also Kaj Sotala’s Multiagent Models of Mind which is more explicitly about how to be a human.
As a firm descriptive (but undecidedly prescriptive) transhumanist, I think your piece here also touches on something we will likely one day (maybe soon?) have to grapple with, which is the fundamental relationship between (moral) agency and moral patienthood. As it happens, modern humans are quite conclusively both, by most lights, but it doesn’t look like this is a law of nature. Indeed there are likely many deserving moral patients today who are not much by way of agents. And we may bring into being agents which are not especially moral-patienty. (Further, something sufficiently agenty might render humans ~~themselves~~ ourselves to the status of ‘not much by way of agents’.)

Oliver Sourbut Jul 21, 2022, 7:59 AM
2 points
0 ∶ 0
in reply to: ekka’s comment on: Why EAs are skeptical about AI Safety
Seconded/thirded on Human Compatible being near that frontier. I did find its ending ‘overly optimistic’ in the sense of framing it like ‘but lo, there is a solution!’ while other similar resources like Superintelligence and especially The Alignment Problem seem more nuanced in presenting uncertain proposals for paths forward not as oven-ready but preliminary and speculative.

Oliver Sourbut Jul 13, 2022, 8:23 AM
1 point
0 ∶ 0
in reply to: Zach Stein-Perlman’s comment on: Announcing Non-trivial, an EA learning platform for teenagers
I think it’s a staircase? Maybe like climbing upwards to more good stuff. Plus some cool circles to make it logo ish.

Oliver Sourbut Jul 13, 2022, 8:19 AM
5 points
0 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: Announcing Non-trivial, an EA learning platform for teenagers
I’m intrigued by this thread. I don’t have an informed opinion on the particular aesthetic or choice of quiz questions, but I note some superficial similarities to Coursera, Khan Academy, and TED-Ed, which are aimed at mainly professional age adults, students of all ages, and youth/students (without excluding adults) respectively.
Fun/cute/cartoon aesthetics do seem to abound these days in all sorts of places, not just for kids.
My uninformed opinion is that I don’t see why it should put off teenagers (talented or otherwise) in particular, but I weakly agree that if something is explicitly pitched at teenagers, that might be offputting!

Oliver Sourbut Jul 4, 2022, 8:39 AM
1 point
0 ∶ 0
in reply to: Oliver Sourbut’s comment on: The Future Might Not Be So Great.
It looks like I got at least one downvote on this comment. Should I be providing tips of this kind in a different way?

Oliver Sourbut Jun 30, 2022, 3:46 PM
31 points
0 ∶ 0
on: The Future Might Not Be So Great.
I’ve considered a possible pithy framing of the Life Despite Suffering question as a grim orthogonality thesis (though I’m not sure how useful it is):

We sometimes point to the substantial majority’s revealed preference for staying alive as evidence of a ‘life worth living’. But perhaps ‘staying-aliveness’ and ‘moral patient value’ can vary more independently than that claim assumes. This is the grim orthogonality thesis.

An existence proof for the ‘high staying-aliveness x low moral patient value’ quadrant is the complex of torturer+torturee, which quite clearly can reveal a preference for staying alive, while quite plausibly being net negative value.

Can we rescue the correlation of revealed ‘staying-aliveness’ preference with ‘life worth livingness’?

We can maybe reason about value from the origin of moral patients we see, without having a physical theory of value. All the patients we see at present are presumably products of natural selection. Let’s also assume for now that patienthood comes from consciousness.

Two obvious but countervailing observations
- to the extent that conscious content is upstream of behaviour but downstream of genetic content, natural selection will operate on conscious content to produce behaviour which is fitness-correlated
  - if positive conscious content produces attractive behaviour (and vice versa), we might anticipate that an organism ‘doing well’ according to suitable fitness-correlates would be experiencing positive conscious content
  - this seems maybe true of humans?
- to the extent that behaviour is downstream of non-conscious control processes, natural selection will operate on non-conscious control processes to produce behaviour which is fitness-correlated
  - we can not rule out experiences ‘not worth living’ which nevertheless produce net revealed staying-aliveness preference, if the behaviour is sufficiently under non-conscious control, or if the selection for behaviour downstream of negative conscious experience is weak
    weak selection is especially likely in novel out-of-distribution situations
  - in general, organisms which reveal preferences for not staying alive will never be ceteris paribus fitter (though there are special cases of course)
For non-naturally-selected moral patients, I think even the above bets are basically off.

Oliver Sourbut Jun 30, 2022, 3:42 PM
64 points
0 ∶ 0
on: The Future Might Not Be So Great.
I’m shocked and somewhat concerned that your empirical finding is that so few people have encountered or thought about this crucial consideration.

My experience is different, with maybe 70% of AI x-risk researchers I’ve discussed with being somewhat au fait with the notion that we might not know the sign of future value conditional on survival. But I agree that it seems people (myself included) have a tendency to slide off this consideration or hope to defer its resolution to future generations, and my sample size is quite small (a few dozen maybe) and quite correlated.

For what it’s worth, I recall this question being explicitly posed in at least a few of the EA in-depth fellowship curricula I’ve consumed or commented on, though I don’t recall specifics and when I checked EA Cambridge’s most recent curriculum I couldn’t find it.

Oliver Sourbut Jun 30, 2022, 3:13 PM
8 points
0 ∶ 0
on: The Future Might Not Be So Great.
Typo hint:

“10<sup>38</sup>” hasn’t rendered how you hoped. You can use <dollar>10^{38}<dollar> which renders as $10^{38}$

Oliver Sourbut Jun 27, 2022, 7:46 AM
2 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: Critiques of EA that I want to read
Got it, I think you’re quite right on one reading. I should have been clearer about what I meant, which is something like
- there is a defensible reading of that claim which maps to some negative utilitarian claim (without necessarily being a central example)
- furthermore I expect many issuers of such sentiments are motivated by basically pretheoretic negative utilitarian insight
E.g. imagine a minor steelification (which loses the aesthetic and rhetorical strength) like “nobody’s positive wellbeing (implicitly stemming from their freedom) can/should be celebrated until everyone has freedom (implicitly necessary to escape negative wellbeing)” which is consistent with some kind of lexical negative utilitarianism.

You’re right that if we insist that ‘freedom’ be interpreted identically in both places (parsimonious, granted, though I think the symmetry is better explained by aesthetic/rhetorical concerns) another reading explicitly neglects the marginal benefit of lifting merely some people out of illiberty. Which is only consistent with utilitarianism if we use an unusual aggregation theory (i.e. minimising) - though I have also seen this discussed under negative utilitarianism.

Anecdata: as someone whose (past) political background and involvement (waning!) is definitely some kind of lefty, and who, if it weren’t for various x- and s-risks, would plausibly consider some form (my form, naturally!) of lefty politics to be highly important (if not highly tractable), my reading of that claim at least goes something like the first one. I might not be representative in that respect.

I have no doubt that many people expressing that kind of sentiment would still celebrate marginal ‘releases’, while considering it wrong to celebrate further the fruits of such freedom, ignoring others’ lack of freedom.

Oliver Sourbut Jun 25, 2022, 8:24 PM
−3 points
0 ∶ 1
in reply to: abrahamrowe’s comment on: Critiques of EA that I want to read
Minor nitpick: “nobody’s free until everyone is free” is precisely a (negative) utilitarian claim (albeit with unusual wording)

Oliver Sourbut Jun 25, 2022, 11:30 AM
5 points
0 ∶ 0
on: Are too many young, highly-engaged longtermist EAs doing movement-building?
It’s possible the selection bias is high, but I don’t have good evidence for this besides personal anecdata. I don’t know how many people are relevantly similar to me, and I don’t know how representative we are of the latest EA ‘freshers’, since dynamics will change and I’m reporting with several years’ lag.

Here’s my personal anecdata.

Since 2016, around when I completed undergrad, I’ve been an engaged (not sure what counts as ‘highly engaged’) longtermist. (Before that point I had not heard of EA per se but my motives were somewhat proto EA and I wanted to contribute to ‘sustainable flourishing at scale’ and ‘tech for good’.) Nevertheless, until 2020 or so I was relatively invisibly upskilling, reflecting on priorities, consuming advice and ideas etc. and figuring out (perhaps too humbly and slowly) how to orient. More recently I’ve overcome some amount of impostor syndrome and simultaneously become more ‘community engaged’ (hence visible) and started directly contributing to technical AI safety research.

If there are a lot with stories like that, they might form a large but quiet cohort countervailing your concern.

Having said that, I think what you express here is excellent to discuss, I think I may have been unusually quiet+cautious, I didn’t encounter EA during undergrad, and I suspect (without here justifying) that community dynamics have changed sufficiently that my anecdote is not IID with the cohort you’re discussing.

Oliver Sourbut Jun 25, 2022, 9:08 AM
23 points
0 ∶ 0
in reply to: RobBensinger’s comment on: On Deference and Yudkowsky’s AI Risk Estimates
I just wanted to state agreement that it seems a large number of people largely misread Death with Dignity, at least according to what seems to me the most plausible intended message: mainly about the ethical injunctions (which are very important as a finitely-rational and prone-to-rationalisation being), as Yudkowsky has written of in the past.

The additional detail of ‘and by the way this is a bad situation and we are doing badly’ is basically modal Yudkowsky schtick and I’m somewhat surprised it updated anyone’s beliefs (about Yudkowsky’s beliefs, and therefore their all-things-considered-including-deference beliefs).

I think if he had been a little more audience-aware he might have written it differently. Then again maybe not, if the net effect is more attention and investment in AI safety—and more recent posts and comments suggest he’s more willing than before to use certain persuasive techniques to spur action (which seems potentially misguided to me, though understandable).