Oliver Sourbut

Karma: 381

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut Jun 27, 2022, 7:46 AM
2 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: Critiques of EA that I want to read
Got it, I think you’re quite right on one reading. I should have been clearer about what I meant, which is something like
- there is a defensible reading of that claim which maps to some negative utilitarian claim (without necessarily being a central example)
- furthermore I expect many issuers of such sentiments are motivated by basically pretheoretic negative utilitarian insight
E.g. imagine a minor steelification (which loses the aesthetic and rhetorical strength) like “nobody’s positive wellbeing (implicitly stemming from their freedom) can/should be celebrated until everyone has freedom (implicitly necessary to escape negative wellbeing)” which is consistent with some kind of lexical negative utilitarianism.

You’re right that if we insist that ‘freedom’ be interpreted identically in both places (parsimonious, granted, though I think the symmetry is better explained by aesthetic/rhetorical concerns) another reading explicitly neglects the marginal benefit of lifting merely some people out of illiberty. Which is only consistent with utilitarianism if we use an unusual aggregation theory (i.e. minimising) - though I have also seen this discussed under negative utilitarianism.

Anecdata: as someone whose (past) political background and involvement (waning!) is definitely some kind of lefty, and who, if it weren’t for various x- and s-risks, would plausibly consider some form (my form, naturally!) of lefty politics to be highly important (if not highly tractable), my reading of that claim at least goes something like the first one. I might not be representative in that respect.

I have no doubt that many people expressing that kind of sentiment would still celebrate marginal ‘releases’, while considering it wrong to celebrate further the fruits of such freedom, ignoring others’ lack of freedom.

Oliver Sourbut Jun 25, 2022, 8:24 PM
−3 points
0 ∶ 1
in reply to: abrahamrowe’s comment on: Critiques of EA that I want to read
Minor nitpick: “nobody’s free until everyone is free” is precisely a (negative) utilitarian claim (albeit with unusual wording)

Oliver Sourbut Jun 25, 2022, 11:30 AM
5 points
0 ∶ 0
on: Are too many young, highly-engaged longtermist EAs doing movement-building?
It’s possible the selection bias is high, but I don’t have good evidence for this besides personal anecdata. I don’t know how many people are relevantly similar to me, and I don’t know how representative we are of the latest EA ‘freshers’, since dynamics will change and I’m reporting with several years’ lag.

Here’s my personal anecdata.

Since 2016, around when I completed undergrad, I’ve been an engaged (not sure what counts as ‘highly engaged’) longtermist. (Before that point I had not heard of EA per se but my motives were somewhat proto EA and I wanted to contribute to ‘sustainable flourishing at scale’ and ‘tech for good’.) Nevertheless, until 2020 or so I was relatively invisibly upskilling, reflecting on priorities, consuming advice and ideas etc. and figuring out (perhaps too humbly and slowly) how to orient. More recently I’ve overcome some amount of impostor syndrome and simultaneously become more ‘community engaged’ (hence visible) and started directly contributing to technical AI safety research.

If there are a lot with stories like that, they might form a large but quiet cohort countervailing your concern.

Having said that, I think what you express here is excellent to discuss, I think I may have been unusually quiet+cautious, I didn’t encounter EA during undergrad, and I suspect (without here justifying) that community dynamics have changed sufficiently that my anecdote is not IID with the cohort you’re discussing.

Oliver Sourbut Jun 25, 2022, 9:08 AM
23 points
0 ∶ 0
in reply to: RobBensinger’s comment on: On Deference and Yudkowsky’s AI Risk Estimates
I just wanted to state agreement that it seems a large number of people largely misread Death with Dignity, at least according to what seems to me the most plausible intended message: mainly about the ethical injunctions (which are very important as a finitely-rational and prone-to-rationalisation being), as Yudkowsky has written of in the past.

The additional detail of ‘and by the way this is a bad situation and we are doing badly’ is basically modal Yudkowsky schtick and I’m somewhat surprised it updated anyone’s beliefs (about Yudkowsky’s beliefs, and therefore their all-things-considered-including-deference beliefs).

I think if he had been a little more audience-aware he might have written it differently. Then again maybe not, if the net effect is more attention and investment in AI safety—and more recent posts and comments suggest he’s more willing than before to use certain persuasive techniques to spur action (which seems potentially misguided to me, though understandable).

Oliver Sourbut Jun 16, 2022, 10:54 AM
3 points
0 ∶ 0
on: Blake Richards on Why he is Skeptical of Existential Risk from AI
I wrote something similar (with more detail) about the Gato paper at the time.

I don’t think this is any evidence at all against AI risk though? It is maybe weak evidence against ‘scaling is all you need’ or that sort of thing.

Oliver Sourbut Jun 16, 2022, 10:46 AM
1 point
0 ∶ 0
in reply to: Rohin Shah’s comment on: Blake Richards on Why he is Skeptical of Existential Risk from AI
Thanks Rohin, I second almost all of this.

Interested to hear more about why long-term credit assignment isn’t needed for powerful AI. I think it depends how you quantify those things and I’m pretty unsure about this myself.

Is it because there is already loads of human-generated data which implicitly embody or contain enough long-term credit assignment? Or is it that long-term credit assignment is irrelevant for long-term reasoning? Or maybe long-term reasoning isn’t needed for ‘powerful AI’?

Oliver Sourbut May 9, 2022, 10:45 PM
1 point
0 ∶ 0
in reply to: Ada-Maaria Hyvärinen’s comment on: How I failed to form views on AI safety
OK, this is the terrible terrible failure mode which I think we are both agreeing on (emphasis mine)

the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”

By ‘a sceptical approach’ I basically mean ‘the thing where we don’t do that’. Because there is not enough epistemic credit in the field, yet, to expect that all (tentative, not-consensus-yet) conclusions to be definitely right.

In traditional/undergraduate mathematics, it’s different—almost always when you don’t understand or agree with the professor, she is simply right and you are simply wrong or confused! This is a justifiable perspective based on the enormous epistemic weight of all the existing work on mathematics.

I’m very glad you call out the distinction between performing skepticism and actually doing it.

Oliver Sourbut Apr 20, 2022, 7:08 PM
2 points
0 ∶ 0
on: How I failed to form views on AI safety

I feel like while “superintelligent AI would be dangerous” makes sense if you believe superintelligence is possible, it would be good to look at other risk scenarios from current and future AI systems as well.

I agree, and I think there’s a gap for thoughtful and creative folks with technical understanding to contribute to filling out the map here!

One person I think has made really interesting contributions here is Andrew Critch, for example on Multipolar Failure and Robust Agent-Agnostic Processes (I realise this is literally me sharing a link without much context which was a conversation-failure-mode discussed in the OP so feel free to pass on this). He also has made some attempts to discuss more breadth e.g. here. Critch isn’t the only one.

Oliver Sourbut Apr 20, 2022, 7:03 PM
2 points
0 ∶ 1
on: How I failed to form views on AI safety

I’m fairly sure deep learning alone will not result in AGI

How sure? :)

What about some combination of deep learning (e.g. massive self-supervised) + within-context/episodic memory/state + procedurally-generated tasks + large-scale population-based training + self-play...? I’m just naming a few contemporary ‘prosaic’ practices which, to me, seem plausibly-enough sufficient to produce AGI that it warrants attention.

Oliver Sourbut Apr 20, 2022, 6:51 PM
4 points
0 ∶ 0
on: How I failed to form views on AI safety
I was one of the facilitators in the most recent run of EA Cambridge’s AGI Safety Fundamentals course, and I also have professional DS/ML experience.

In my case I very deliberately emphasised a sceptical approach to engaging with all the material, while providing clarifications and corrections where people’s misconceptions are the source of scepticism. I believe this was well-received by my cohort, all of whom appeared to engage thoughtfully and honestly with the material.

I think this is the best way to engage, when time permits, because (in brief)
- many arguments invoke ill-defined terms, and we need to sharpen these
- many arguments are (perhaps explicitly) speculative and empirically uncertain
- even mathematically/empirically rigorous content has important modelling assumptions and experimental caveats
- scepticism often produces better creative/generative engagement
- collectively we will fail if our individual opinions are overly shaped by founder effects
I hope that this is a common perspective, but to the extent that it isn’t, I wonder if this (especially the last point) may be a source of some of your confusing experiences.

I’d also say: it seems appropriate to have ‘very messy views’ if by that you mean uncertainty about where things are going and how to make them better! I think folks who don’t are doing one of two things
- mistakenly concentrating more hypothesis weight than their observations/thinking in fact justify (which is a bad idea)
- engaging in a thinking manoeuvre something like ‘temporary MAP stance’ or ‘subjective probability matching’ (which may be a good idea, if done transparently)
‘Temporary MAP stance’ or ‘subjective probability matching’

MAP is Maximum A Posteriori i.e. your best guess after considering evidence. Probability matching is making actions/guesses proportional to your estimate of them being right (rather than picking the single MAP choice)

By this manoeuvre I’m gesturing at a kind of behaviour where you are quite unsure about what’s best (e.g. ‘should I work on interpretability or demystifying deception?’) and rather than allowing that to result in analysis paralysis, you temporarily collapse some uncertainty and make some concrete assumptions to get moving in one or other direction. Hopefully in so doing you a) make a contribution and b) grow your skills and collect new evidence to make better decisions/contributions next time.

Oliver Sourbut Apr 20, 2022, 6:29 PM
4 points
0 ∶ 0
on: How I failed to form views on AI safety
Hey, as someone who also has professional CS and DS experience, this was a really welcome and interesting read. I have all sorts of thoughts but I had one main question

So I used the AGISF Slack to find people who had already had a background in machine learning before getting into AI safety and asked them what had originally convinced them. Finally, I got answers from 3 people who fit my search criteria. They mentioned some different sources of first hearing about AI safety (80 000 Hours and LessWrong), but all three mentioned one same source that had deeply influenced them: Superintelligence.

When I read this I remembered that I was one of the folks you reached out to on Slack! But I didn’t mention Superintelligence at all (though in fact I have read it and have a generally good opinion of it, though it was several years ago). I guess I didn’t fit your criteria quite right? In my case I had CS and a little academic AI, but no professional DS/ML experience, before getting ‘into’ AI safety. Interested to know what other people you spoke to and for what reasons they didn’t fall into the criteria you were looking for.

Oliver Sourbut Dec 18, 2021, 9:44 AM
5 points
0 ∶ 0
on: What holiday songs, stories, etc. do you associate with effective altruism?
It’s not EA but I have a soft spot for Good King Wenceslas (https://en.m.wikipedia.org/wiki/Good_King_Wenceslas)

It’s a Christmas hymn about a rich prince who was busy striding around and giving to the poor, and it ends by saying all good Christians ‘wealth or rank possessing’ should do the same. It’s a cracking tune and it means that at least once per year, most Anglican churchgoers will get reminded of those words.

The story is medieval but the particular text comes out of the Victorian charity movement which, at its best, was vaguely proto EA and proto progress studies in many ways.

Oliver Sourbut Nov 11, 2021, 8:55 AM
3 points
0 ∶ 0
in reply to: Tessa A 🔸’s comment on: How do EAs deal with having a “weird” appearance?
Just seconding this. For context I work not in academia but as a software engineer and data scientist in London.

I usually have crazy sticky-up hair that sort of does different things each day especially as it grows. That’s my main superficial weirdness (unless you count the unusually big nose) though I have plenty of other quirks which are harder to label and harder to spot from a distance.

In hindsight I think the hair has made me memorable and recognisable in my workplaces (e.g. people have expressed looking forward to seeing me and my hair in meetings...), and since I’m also reasonably agreeable and competent, I suspect this memorableness has ultimately been net useful for networking (which is useful because networking hasn’t historically come naturally to me).

Oliver Sourbut Oct 30, 2021, 8:52 AM
6 points
0 ∶ 0
on: Many Undergrads Should Take Light Courseloads
Thank you, I found myself agreeing with most of this post and reflecting on how I might have optimised during my undergrad experience. On the other hand, I note that neither the post nor any comments yet contains what I consider an important caveat:

Taking extra classes is a great way to explore in the sense of dissolving known- and unknown-unknowns (what fits me? what problem-framings am I missing? what tools do other disciplines have? what concerns to people interested in X have? what even is there if I look further?)

Extra-curricular activities also enable some of this sort of exploration. But I’d emphasise that for a competent young person, appropriate exploration, one way or another, is a really key aspect of impact.

For my personal experience (UK) I blended music (!) and mathematics with later explorations in philosophy and computer science, each of which is responsible in one way or another for opening doors to impactful, challenging, rewarding, and lucrative possibilities.

Oliver Sourbut Oct 21, 2021, 10:12 AM
3 points
0 ∶ 0
on: Prioritization Research for Advancing Wisdom and Intelligence
Yes yes, more strength to this where it’s tractable and possible backfires are well understood and mitigated/avoided!

One adjacent category which I think is helpful to consider explicitly (I think you have it implicit here) is ‘well-informedness’, which I motion is distinct from ‘intelligence’ or ‘wisdom’. One could be quite wise and intelligent but crippled or even misdirected if the information available/salient is limited or biased. Perhaps this is countered by an understanding of one’s own intellectual and cognitive biases, leading to appropriate (‘wise’) choices of information-gathering behaviour to act against possible bias? But perhaps there are other levers to push which act on this category effectively.

To the extent that you think long-run trajectories will be influenced by few specific decision-making entities, it could be extremely valuable to identify, and improve the epistemics and general wisdom (and benevolence) of those entities. To the extent that you think long-run trajectories will be influenced by the interactions of many cooperating and competing decision-making entities, it could be more important to improve mechanisms for coordination, especially coordination against activities which destroy value. Well-informedness may be particularly relevant in the latter case.

Oliver Sourbut Oct 14, 2021, 9:58 AM
4 points
0 ∶ 0
in reply to: Aaron Gertler 🔸’s comment on: Ben_Snodin’s Shortform
It depends what media type you’re talking about (audio, video, display, …) - $6m/100m is $60CPM (‘cost per mille’), which is certainly above the odds for similar ‘premium video’ advertising, but only by maybe 2-5x. For other media like audio and display the CPMs can be quite a bit lower, and if you’re just looking to reach ‘someone, somewhere’ you can get a bargain via programmatic advertising.

I happen to work for a major demand-side platform in real-time ad buying and I’ve been wondering if there might be a way to efficiently do good this way. The pricing can be quite nuanced. Haven’t done any analysis at this point.

Oliver Sourbut Oct 14, 2021, 9:32 AM
1 point
0 ∶ 0
in reply to: Jackson Wagner’s comment on: How impactful is free and open source software development?
Hey, let me know if you’d like another reviewer. I’m a medium-experienced senior software engineer whose professional work and side-projects use various proportions of open-source and proprietary software. And I enjoy reviewing/proof-reading :)

Oliver Sourbut Oct 7, 2021, 8:36 AM
1 point
0 ∶ 0
on: Beyond fire alarms: freeing the groupstruck
I appreciated your detailed analysis of the fire alarm situation along with evidence and introspection notes.

I’m not sure if it opens up any action-relevant new hypothesis space, but one feature of the fire alarm situation which I think you did not analyse is that commonly people are concerned also for the welfare of their fellows, especially those who are close by. This makes sense: if you find yourself in a group, even of strangers (and you’ve reached consensus that you’re not fighting each other) it will usually pay off to look out for each other! So perhaps another social feature at play when people fail to leave a smoky room when the group shows no signs of doing so is that they aren’t willing to unilaterally secure their own safety unless they know the others are also going to be safe. Though on this hypothesis you might expect to see more ‘speaking up’ and attempts to convince others to move.

Oliver Sourbut Oct 7, 2021, 8:35 AM
1 point
0 ∶ 0
on: Beyond fire alarms: freeing the groupstruck
This was a great read, thank you—I especially valued the multiple series of illustrating/motivating examples, and the several sections laying out various hypotheses along with evidence/opinion on them.

I sometimes wonder how evolution ended up creating humans who are sometimes nonconformist, when it seems socially costly, but I think a story related to what you’ve written here makes sense: at least one kind of nonconformity can sometimes shift a group consensus from a fatal misinterpretation to an appropriate and survivable group response (and furthermore, presumably in expectation gain some prestige for the nonconforming maverick(s) who started the shift). So there’s maybe some kind of evolutionary stable meta-strategy of ‘probability of being conformist or not (maybe context-dependent)’.

Oliver Sourbut May 27, 2021, 8:02 AM
1 point
0 ∶ 0
on: EA Survey 2020: How People Get Involved in EA
Thanks for these very helpful insights! I thought the mosaic charts were particularly creative and visually insightful.

I have one minor statistical nit and one related question.

In cases where ‘only one significant difference was found’ (at a 95% c.i.), it could be worth noting that you have around 20 categories… so on average one spurious significant difference is to be expected! (If the difference is small.)

Also a question about how the significance test was carried out. $σ (a - b) = \sqrt{σ (a)^{2} + σ (b)^{2}} \leq σ (a) + σ (b)$ so for calling a difference significant at 95% it matters whether you a) check if the individual 95% confidence intervals overlap or b) check if the diff’d confidence interval noted above contains 0 (the usual approach). Which approach was used here? I ask because to my eye there might be a few more (weakly) significant results than were mentioned in the text.

Oliver Sourbut

‘Temporary MAP stance’ or ‘subjective probability matching’