Oliver Sourbut

Karma: 295

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut 17 Sep 2020 18:40 UTC
1 point
0 ∶ 0
on: You Should Write a Forum Bio
Thanks for this, Aaron. Joining a new community can be tricky online so it’s helpful to have an explicit welcome like this!

Oliver Sourbut 10 Dec 2020 15:45 UTC
4 points
0 ∶ 0
on: Careers Questions Open Thread
I’m trying to choose between doubling down on skills in software engineering or branching out with the goal of working on AI safety longer term. I get the impression that a lot of people are in a similar position.

For me, my undergrad was an unusual mix of things but included Maths, Music (!) and Computer Science. I got good grades and I think there’s a reasonable chance of my getting into a university like Oxford, Cambridge or Imperial to study a Masters and perhaps subsequently a PhD in Computer Science/AI.

Currently I’m paid well and developing a fair amount of expertise as a software engineer. I’ve been at it for ~4 years and gained a fair amount of respect and responsibility; most likely on the cusp of a ‘senior’ promotion within months. Sticking at the engineering, I might be able to give £10s of thousands/year or more, perhaps sustained over a few decades, and there’s also a chance I could find myself doing impactful direct work or being in a position to influence the direction of large forces in tech.

On the other hand, from the work I’ve done and investigations in my own time I think my temperament and working style suit research, but I’ve little direct evidence of that so far and I’d need to prove that to myself and others. I’m thinking of a Masters as being a great way to do that. I’ve enjoyed several recommended online courses in ML and statsy things as well as a little tinkering on toy projects. On this evidence my capacities appear to align well with applying and implementing machine learning but I think research and policy have higher leverage over future flourishing.

Is applying for Masters courses in AI/ML is a sensible next exploration? One catch is that some of my current compensation is in equity which vests over time, meaning making a change would sacrifice some earnings.

Oliver Sourbut 11 Dec 2020 8:30 UTC
1 point
0 ∶ 0
in reply to: navika_cookie’s comment on: Careers Questions Open Thread
As an engineer (software) myself for a few years, I can encourage you that is rewarding, challenging, and in the right position you can have quite a bit of autonomy to drive decision-making and execute on your own vision. Depending on the role and organisation, it can be far from merely technical; the outline you give of the college project sounds exactly like engineering to me!

That said, there are few or no places where engineers are completely unconstrained. But there are routes from engineering into more ‘overseeing’-type roles, e.g. architect, tech director, technical project manager. A lot of those people do much better if they have solid engineering experience of their own first.

Some different thoughts on which I have much less or no experience but seem relevant:
- management consulting. Have you heard of that? I think they solve hard problems and have some room for vision.
- entrepreneurs obviously have an opportunity to create and oversee a vision. I gather that a lot of the time it helps to have related experience in the relevant industry/field beforehand

Oliver Sourbut 16 Dec 2020 21:24 UTC
1 point
0 ∶ 0
in reply to: Michelle_Hutchinson’s comment on: Careers Questions Open Thread
I welcome the reinforcement that a) it is indeed a tough call and b) I’m sane and they’re good options! Thank you for the encouragement, and the advice.

I remain fuzzy on what shape ‘impactful direct work’ could take, and I’m not sure to what degree keeping my mind ‘open’ in that sense is rational (the better to capture emergent opportunities) vs merely comforting (specifying a path is scary and procastinating is emotionally safer)! I acknowledge that my tentative principal goal besides donations, if I continue engineering growth, is indeed working on safety. The MIRI job link is interesting. I’d be pleased and proud to work in that sort of role (though I’m likely to remain in the UK at least for the near future).

Thank you for the suggestion to talk to Richard or others. I’ve gathered a few accounts from friends I know well who have gone into further degrees in other disciplines, and I expect it would be useful to complement that with evidence from others to help better predict personal fit. I wouldn’t know whom to talk to about impact on a long-term engineering track.

Oliver Sourbut 16 Dec 2020 21:26 UTC
2 points
0 ∶ 0
in reply to: anonymous_123’s comment on: Careers Questions Open Thread
I really appreciate these data points! Actually it’s interesting you mention the networking aspect—one of the factors that would push me towards more higher education is the (real or imagined?) networking opportunities. Though I get on very well with most people I work or study with, I’m not an instinctive ‘networker’ and I think for me, improving that could be a factor with relatively high marginal return.

As for learning practical skills… I’d hope to get some from a higher degree but if that were all I wanted I might indeed stick to Coursera and the like! It’s the research aspect I’d really like to explore my fit for.

Trying to negotiate a break with the company had crossed my mind but sounds hard. Thanks for the nudge and anecdata about that possibility. It would be a big win if possible!

I’m really glad to hear that your path has been working out without regret. I hope that continues. :)

Oliver Sourbut 27 May 2021 8:02 UTC
1 point
0 ∶ 0
on: EA Survey 2020: How People Get Involved in EA
Thanks for these very helpful insights! I thought the mosaic charts were particularly creative and visually insightful.

I have one minor statistical nit and one related question.

In cases where ‘only one significant difference was found’ (at a 95% c.i.), it could be worth noting that you have around 20 categories… so on average one spurious significant difference is to be expected! (If the difference is small.)

Also a question about how the significance test was carried out. $σ (a - b) = \sqrt{σ (a)^{2} + σ (b)^{2}} \leq σ (a) + σ (b)$ so for calling a difference significant at 95% it matters whether you a) check if the individual 95% confidence intervals overlap or b) check if the diff’d confidence interval noted above contains 0 (the usual approach). Which approach was used here? I ask because to my eye there might be a few more (weakly) significant results than were mentioned in the text.

Oliver Sourbut 7 Oct 2021 8:35 UTC
1 point
0 ∶ 0
on: Beyond fire alarms: freeing the groupstruck
This was a great read, thank you—I especially valued the multiple series of illustrating/motivating examples, and the several sections laying out various hypotheses along with evidence/opinion on them.

I sometimes wonder how evolution ended up creating humans who are sometimes nonconformist, when it seems socially costly, but I think a story related to what you’ve written here makes sense: at least one kind of nonconformity can sometimes shift a group consensus from a fatal misinterpretation to an appropriate and survivable group response (and furthermore, presumably in expectation gain some prestige for the nonconforming maverick(s) who started the shift). So there’s maybe some kind of evolutionary stable meta-strategy of ‘probability of being conformist or not (maybe context-dependent)’.

Oliver Sourbut 7 Oct 2021 8:36 UTC
1 point
0 ∶ 0
on: Beyond fire alarms: freeing the groupstruck
I appreciated your detailed analysis of the fire alarm situation along with evidence and introspection notes.

I’m not sure if it opens up any action-relevant new hypothesis space, but one feature of the fire alarm situation which I think you did not analyse is that commonly people are concerned also for the welfare of their fellows, especially those who are close by. This makes sense: if you find yourself in a group, even of strangers (and you’ve reached consensus that you’re not fighting each other) it will usually pay off to look out for each other! So perhaps another social feature at play when people fail to leave a smoky room when the group shows no signs of doing so is that they aren’t willing to unilaterally secure their own safety unless they know the others are also going to be safe. Though on this hypothesis you might expect to see more ‘speaking up’ and attempts to convince others to move.

Oliver Sourbut 14 Oct 2021 9:58 UTC
4 points
0 ∶ 0
in reply to: Aaron Gertler’s comment on: Ben_Snodin’s Shortform
It depends what media type you’re talking about (audio, video, display, …) - $6m/100m is $60CPM (‘cost per mille’), which is certainly above the odds for similar ‘premium video’ advertising, but only by maybe 2-5x. For other media like audio and display the CPMs can be quite a bit lower, and if you’re just looking to reach ‘someone, somewhere’ you can get a bargain via programmatic advertising.

I happen to work for a major demand-side platform in real-time ad buying and I’ve been wondering if there might be a way to efficiently do good this way. The pricing can be quite nuanced. Haven’t done any analysis at this point.

Oliver Sourbut 21 Oct 2021 10:12 UTC
3 points
0 ∶ 0
on: Prioritization Research for Advancing Wisdom and Intelligence
Yes yes, more strength to this where it’s tractable and possible backfires are well understood and mitigated/avoided!

One adjacent category which I think is helpful to consider explicitly (I think you have it implicit here) is ‘well-informedness’, which I motion is distinct from ‘intelligence’ or ‘wisdom’. One could be quite wise and intelligent but crippled or even misdirected if the information available/salient is limited or biased. Perhaps this is countered by an understanding of one’s own intellectual and cognitive biases, leading to appropriate (‘wise’) choices of information-gathering behaviour to act against possible bias? But perhaps there are other levers to push which act on this category effectively.

To the extent that you think long-run trajectories will be influenced by few specific decision-making entities, it could be extremely valuable to identify, and improve the epistemics and general wisdom (and benevolence) of those entities. To the extent that you think long-run trajectories will be influenced by the interactions of many cooperating and competing decision-making entities, it could be more important to improve mechanisms for coordination, especially coordination against activities which destroy value. Well-informedness may be particularly relevant in the latter case.

Oliver Sourbut 30 Oct 2021 8:52 UTC
6 points
0 ∶ 0
on: Many Undergrads Should Take Light Courseloads
Thank you, I found myself agreeing with most of this post and reflecting on how I might have optimised during my undergrad experience. On the other hand, I note that neither the post nor any comments yet contains what I consider an important caveat:

Taking extra classes is a great way to explore in the sense of dissolving known- and unknown-unknowns (what fits me? what problem-framings am I missing? what tools do other disciplines have? what concerns to people interested in X have? what even is there if I look further?)

Extra-curricular activities also enable some of this sort of exploration. But I’d emphasise that for a competent young person, appropriate exploration, one way or another, is a really key aspect of impact.

For my personal experience (UK) I blended music (!) and mathematics with later explorations in philosophy and computer science, each of which is responsible in one way or another for opening doors to impactful, challenging, rewarding, and lucrative possibilities.

Oliver Sourbut 20 Apr 2022 18:29 UTC
4 points
0 ∶ 0
on: How I failed to form views on AI safety
Hey, as someone who also has professional CS and DS experience, this was a really welcome and interesting read. I have all sorts of thoughts but I had one main question

So I used the AGISF Slack to find people who had already had a background in machine learning before getting into AI safety and asked them what had originally convinced them. Finally, I got answers from 3 people who fit my search criteria. They mentioned some different sources of first hearing about AI safety (80 000 Hours and LessWrong), but all three mentioned one same source that had deeply influenced them: Superintelligence.

When I read this I remembered that I was one of the folks you reached out to on Slack! But I didn’t mention Superintelligence at all (though in fact I have read it and have a generally good opinion of it, though it was several years ago). I guess I didn’t fit your criteria quite right? In my case I had CS and a little academic AI, but no professional DS/ML experience, before getting ‘into’ AI safety. Interested to know what other people you spoke to and for what reasons they didn’t fall into the criteria you were looking for.

Oliver Sourbut 20 Apr 2022 18:51 UTC
4 points
0 ∶ 0
on: How I failed to form views on AI safety
I was one of the facilitators in the most recent run of EA Cambridge’s AGI Safety Fundamentals course, and I also have professional DS/ML experience.

In my case I very deliberately emphasised a sceptical approach to engaging with all the material, while providing clarifications and corrections where people’s misconceptions are the source of scepticism. I believe this was well-received by my cohort, all of whom appeared to engage thoughtfully and honestly with the material.

I think this is the best way to engage, when time permits, because (in brief)
- many arguments invoke ill-defined terms, and we need to sharpen these
- many arguments are (perhaps explicitly) speculative and empirically uncertain
- even mathematically/empirically rigorous content has important modelling assumptions and experimental caveats
- scepticism often produces better creative/generative engagement
- collectively we will fail if our individual opinions are overly shaped by founder effects
I hope that this is a common perspective, but to the extent that it isn’t, I wonder if this (especially the last point) may be a source of some of your confusing experiences.

I’d also say: it seems appropriate to have ‘very messy views’ if by that you mean uncertainty about where things are going and how to make them better! I think folks who don’t are doing one of two things
- mistakenly concentrating more hypothesis weight than their observations/thinking in fact justify (which is a bad idea)
- engaging in a thinking manoeuvre something like ‘temporary MAP stance’ or ‘subjective probability matching’ (which may be a good idea, if done transparently)
‘Temporary MAP stance’ or ‘subjective probability matching’

MAP is Maximum A Posteriori i.e. your best guess after considering evidence. Probability matching is making actions/guesses proportional to your estimate of them being right (rather than picking the single MAP choice)

By this manoeuvre I’m gesturing at a kind of behaviour where you are quite unsure about what’s best (e.g. ‘should I work on interpretability or demystifying deception?’) and rather than allowing that to result in analysis paralysis, you temporarily collapse some uncertainty and make some concrete assumptions to get moving in one or other direction. Hopefully in so doing you a) make a contribution and b) grow your skills and collect new evidence to make better decisions/contributions next time.

Oliver Sourbut 20 Apr 2022 19:03 UTC
2 points
0 ∶ 1
on: How I failed to form views on AI safety

I’m fairly sure deep learning alone will not result in AGI

How sure? :)

What about some combination of deep learning (e.g. massive self-supervised) + within-context/episodic memory/state + procedurally-generated tasks + large-scale population-based training + self-play...? I’m just naming a few contemporary ‘prosaic’ practices which, to me, seem plausibly-enough sufficient to produce AGI that it warrants attention.

Oliver Sourbut 20 Apr 2022 19:08 UTC
2 points
0 ∶ 0
on: How I failed to form views on AI safety

I feel like while “superintelligent AI would be dangerous” makes sense if you believe superintelligence is possible, it would be good to look at other risk scenarios from current and future AI systems as well.

I agree, and I think there’s a gap for thoughtful and creative folks with technical understanding to contribute to filling out the map here!

One person I think has made really interesting contributions here is Andrew Critch, for example on Multipolar Failure and Robust Agent-Agnostic Processes (I realise this is literally me sharing a link without much context which was a conversation-failure-mode discussed in the OP so feel free to pass on this). He also has made some attempts to discuss more breadth e.g. here. Critch isn’t the only one.

Oliver Sourbut 9 May 2022 22:45 UTC
1 point
0 ∶ 0
in reply to: Ada-Maaria Hyvärinen’s comment on: How I failed to form views on AI safety
OK, this is the terrible terrible failure mode which I think we are both agreeing on (emphasis mine)

the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”

By ‘a sceptical approach’ I basically mean ‘the thing where we don’t do that’. Because there is not enough epistemic credit in the field, yet, to expect that all (tentative, not-consensus-yet) conclusions to be definitely right.

In traditional/undergraduate mathematics, it’s different—almost always when you don’t understand or agree with the professor, she is simply right and you are simply wrong or confused! This is a justifiable perspective based on the enormous epistemic weight of all the existing work on mathematics.

I’m very glad you call out the distinction between performing skepticism and actually doing it.

Oliver Sourbut 16 Jun 2022 10:46 UTC
1 point
0 ∶ 0
in reply to: Rohin Shah’s comment on: Blake Richards on Why he is Skeptical of Existential Risk from AI
Thanks Rohin, I second almost all of this.

Interested to hear more about why long-term credit assignment isn’t needed for powerful AI. I think it depends how you quantify those things and I’m pretty unsure about this myself.

Is it because there is already loads of human-generated data which implicitly embody or contain enough long-term credit assignment? Or is it that long-term credit assignment is irrelevant for long-term reasoning? Or maybe long-term reasoning isn’t needed for ‘powerful AI’?

Oliver Sourbut 16 Jun 2022 10:54 UTC
3 points
0 ∶ 0
on: Blake Richards on Why he is Skeptical of Existential Risk from AI
I wrote something similar (with more detail) about the Gato paper at the time.

I don’t think this is any evidence at all against AI risk though? It is maybe weak evidence against ‘scaling is all you need’ or that sort of thing.

Oliver Sourbut 25 Jun 2022 9:08 UTC
23 points
0 ∶ 0
in reply to: RobBensinger’s comment on: On Deference and Yudkowsky’s AI Risk Estimates
I just wanted to state agreement that it seems a large number of people largely misread Death with Dignity, at least according to what seems to me the most plausible intended message: mainly about the ethical injunctions (which are very important as a finitely-rational and prone-to-rationalisation being), as Yudkowsky has written of in the past.

The additional detail of ‘and by the way this is a bad situation and we are doing badly’ is basically modal Yudkowsky schtick and I’m somewhat surprised it updated anyone’s beliefs (about Yudkowsky’s beliefs, and therefore their all-things-considered-including-deference beliefs).

I think if he had been a little more audience-aware he might have written it differently. Then again maybe not, if the net effect is more attention and investment in AI safety—and more recent posts and comments suggest he’s more willing than before to use certain persuasive techniques to spur action (which seems potentially misguided to me, though understandable).

Oliver Sourbut 25 Jun 2022 20:24 UTC
−5 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: Critiques of EA that I want to read
Minor nitpick: “nobody’s free until everyone is free” is precisely a (negative) utilitarian claim (albeit with unusual wording)

Oliver Sourbut

‘Temporary MAP stance’ or ‘subjective probability matching’