Oliver Sourbut

• Oliver SourbutAug 5, 2023, 10:01 PM

  3 points

  1 ∶ 0

  on: Are there dis­ec­onomies of scale in the rep­u­ta­tion of com­mu­ni­ties?

  Great read, and interesting analysis. I like encountering models for complex systems (like community dynamics)!

  One factor I don’t think was discussed (maybe the gesture at possible inadequacy of $f\left(N\right)$ encompasses this) is the duration of scandal effects. E.g. imagine some group claiming to be the Spanish Inquisition or the Mongol Horde, or the Illuminati tried to get stuff done. I think (assuming taken seriously) they’d encounter lingering reputational damage more than one year after the original scandals! Not sure how this models out; I’m not planning to dive into it, but I think this stands out to me as the ‘next marginal fidelity gain’ for a model like this.

• Oliver SourbutJul 21, 2023, 10:49 PM

  9 points

  3 ∶ 0

  on: I’m in­ter­view­ing Jan Leike, co-lead of OpenAI’s new Su­per­al­ign­ment pro­ject. What should I ask him?

  OpenAI as a whole, and individuals affiliated with or speaking for the org, appear to be largely behaving as if they are caught in an overdetermined race toward AGI.

  What proportion of people at OpenAI believe this, and to what extent? What kind of observations, or actions or statements by others (and who?) would change their minds?

• Oliver SourbutJul 13, 2023, 9:52 PM

  1 point

  0 ∶ 0

  on: A Defense of Work on Math­e­mat­i­cal AI Safety

  Great post. I basically agree, but in a spirit of devil’s advocating, I will say: when I turn my mind to agent foundations thinking, I often find myself skirting queasily close to concepts which feel also capabilities-relevant (to the extent that I have avoided publicly airing several ideas for over a year).

  I don’t know if that’s just me, but it does seem that some agent foundations content from the past has also had bearing on AI capabilities—especially if we include decision theory stuff, dynamic programming and RL, search, planning etc. which it’s arguably artificial to exclude. How would you ex ante distinguish e.g. work which explores properties and constraints of hypothetical planning routines from work which informs creation of more effective planning routines? This sort of thinking seems relevant.

• Oliver SourbutMay 15, 2023, 1:54 PM

  1 point

  0 ∶ 0

  in reply to: Oliver Sourbut’s comment on: Talk­ing pub­li­cly about AI risk

  I wrote a little here about unpluggability (and crossposted on LessWrong/​AF)

Un-un­plug­ga­bil­ity—can’t we just un­plug it?

• Oliver SourbutApr 28, 2023, 2:56 PM

  4 points

  0 ∶ 0

  on: Talk­ing pub­li­cly about AI risk

  Thank you for sharing this! Especially the points about relevant maps and Meta/​FAIR/​LeCun.

  I was recently approached by the UK FCDO as a technical expert in AI with perspective on x-risk. We had what I think were very productive conversations, with an interesting convergence of my framings and the ones you’ve shared here—that’s encouraging! If I find time I’m hoping to write up some of my insights soon.

• Oliver SourbutDec 2, 2022, 6:25 PM

  1 point

  1 ∶ 0

  in reply to: lukeprog’s comment on: Why Neu­ron Counts Shouldn’t Be Used as Prox­ies for Mo­ral Weight

  I’ve given a little thought to this hidden qualia hypothesis but it remains very confusing for me.

  To what extent should we expect to be able to tractably and knowably affect such hidden qualia?

• Oliver SourbutSep 8, 2022, 12:30 PM

  8 points

  2 ∶ 0

  on: Effec­tive al­tru­ism in the gar­den of ends

  This is beautiful and important Tyler, thank you for sharing.

  I’ve seen a few people burn out (and come close myself), and I have made a point of gently socially making and reinforcing this sort of point (far less eloquently) myself, in various contexts.

  I have a lot of thoughts about this subject.

  One thing I embrace always is silliness and (often self-deprecating) humour, which are useful antidotes to stress for a lot of people. Incidentally, your tweet thread rendition of the Eqyptian spell includes

  I am light heading for light. Even in the dark, a fire bums in the distance.

  (emphasis mine) which I enjoyed. A case of bad keming reified?

  A few friends and acquaintances have recently been working on something they’re calling Shard Theory, which considers the various parts of a human’s motivation system and their interactions. They’re interested for other reasons, but I was reminded here. See also Kaj Sotala’s Multiagent Models of Mind which is more explicitly about how to be a human.

  As a firm descriptive (but undecidedly prescriptive) transhumanist, I think your piece here also touches on something we will likely one day (maybe soon?) have to grapple with, which is the fundamental relationship between (moral) agency and moral patienthood. As it happens, modern humans are quite conclusively both, by most lights, but it doesn’t look like this is a law of nature. Indeed there are likely many deserving moral patients today who are not much by way of agents. And we may bring into being agents which are not especially moral-patienty. (Further, something sufficiently agenty might render humans themselves ourselves to the status of ‘not much by way of agents’.)

• Oliver SourbutJul 21, 2022, 7:59 AM

  2 points

  0 ∶ 0

  in reply to: ekka’s comment on: Why EAs are skep­ti­cal about AI Safety

  Seconded/​thirded on Human Compatible being near that frontier. I did find its ending ‘overly optimistic’ in the sense of framing it like ‘but lo, there is a solution!’ while other similar resources like Superintelligence and especially The Alignment Problem seem more nuanced in presenting uncertain proposals for paths forward not as oven-ready but preliminary and speculative.

• Oliver SourbutJul 13, 2022, 8:23 AM

  1 point

  0 ∶ 0

  in reply to: Zach Stein-Perlman’s comment on: An­nounc­ing Non-triv­ial, an EA learn­ing plat­form for teenagers

  I think it’s a staircase? Maybe like climbing upwards to more good stuff. Plus some cool circles to make it logo ish.

• Oliver SourbutJul 13, 2022, 8:19 AM

  5 points

  0 ∶ 0

  in reply to: Owen Cotton-Barratt’s comment on: An­nounc­ing Non-triv­ial, an EA learn­ing plat­form for teenagers

  I’m intrigued by this thread. I don’t have an informed opinion on the particular aesthetic or choice of quiz questions, but I note some superficial similarities to Coursera, Khan Academy, and TED-Ed, which are aimed at mainly professional age adults, students of all ages, and youth/​students (without excluding adults) respectively.

  Fun/​cute/​cartoon aesthetics do seem to abound these days in all sorts of places, not just for kids.

  My uninformed opinion is that I don’t see why it should put off teenagers (talented or otherwise) in particular, but I weakly agree that if something is explicitly pitched at teenagers, that might be offputting!

Two tongue-in-cheek EA anthems

• Oliver SourbutJul 4, 2022, 8:39 AM

  1 point

  0 ∶ 0

  in reply to: Oliver Sourbut’s comment on: The Fu­ture Might Not Be So Great.

  It looks like I got at least one downvote on this comment. Should I be providing tips of this kind in a different way?

• Oliver SourbutJun 30, 2022, 3:46 PM

  31 points

  0 ∶ 0

  on: The Fu­ture Might Not Be So Great.

  I’ve considered a possible pithy framing of the Life Despite Suffering question as a grim orthogonality thesis (though I’m not sure how useful it is):

  We sometimes point to the substantial majority’s revealed preference for staying alive as evidence of a ‘life worth living’. But perhaps ‘staying-aliveness’ and ‘moral patient value’ can vary more independently than that claim assumes. This is the grim orthogonality thesis.

  An existence proof for the ‘high staying-aliveness x low moral patient value’ quadrant is the complex of torturer+torturee, which quite clearly can reveal a preference for staying alive, while quite plausibly being net negative value.

  Can we rescue the correlation of revealed ‘staying-aliveness’ preference with ‘life worth livingness’?

  We can maybe reason about value from the origin of moral patients we see, without having a physical theory of value. All the patients we see at present are presumably products of natural selection. Let’s also assume for now that patienthood comes from consciousness.

  Two obvious but countervailing observations

  • to the extent that conscious content is upstream of behaviour but downstream of genetic content, natural selection will operate on conscious content to produce behaviour which is fitness-correlated

    • if positive conscious content produces attractive behaviour (and vice versa), we might anticipate that an organism ‘doing well’ according to suitable fitness-correlates would be experiencing positive conscious content

    • this seems maybe true of humans?

  • to the extent that behaviour is downstream of non-conscious control processes, natural selection will operate on non-conscious control processes to produce behaviour which is fitness-correlated

    • we can not rule out experiences ‘not worth living’ which nevertheless produce net revealed staying-aliveness preference, if the behaviour is sufficiently under non-conscious control, or if the selection for behaviour downstream of negative conscious experience is weak

      • weak selection is especially likely in novel out-of-distribution situations

    • in general, organisms which reveal preferences for not staying alive will never be ceteris paribus fitter (though there are special cases of course)

  For non-naturally-selected moral patients, I think even the above bets are basically off.

• Oliver SourbutJun 30, 2022, 3:42 PM

  64 points

  0 ∶ 0

  on: The Fu­ture Might Not Be So Great.

  I’m shocked and somewhat concerned that your empirical finding is that so few people have encountered or thought about this crucial consideration.

  My experience is different, with maybe 70% of AI x-risk researchers I’ve discussed with being somewhat au fait with the notion that we might not know the sign of future value conditional on survival. But I agree that it seems people (myself included) have a tendency to slide off this consideration or hope to defer its resolution to future generations, and my sample size is quite small (a few dozen maybe) and quite correlated.

  For what it’s worth, I recall this question being explicitly posed in at least a few of the EA in-depth fellowship curricula I’ve consumed or commented on, though I don’t recall specifics and when I checked EA Cambridge’s most recent curriculum I couldn’t find it.

• Oliver SourbutJun 30, 2022, 3:13 PM

  8 points

  0 ∶ 0

  on: The Fu­ture Might Not Be So Great.

  Typo hint:

  “10<sup>38</​sup>” hasn’t rendered how you hoped. You can use <dollar>10^{38}<dollar> which renders as ${10}^{38}$

• Oliver SourbutJun 27, 2022, 7:46 AM

  2 points

  0 ∶ 0

  in reply to: abrahamrowe’s comment on: Cri­tiques of EA that I want to read

  Got it, I think you’re quite right on one reading. I should have been clearer about what I meant, which is something like

  • there is a defensible reading of that claim which maps to some negative utilitarian claim (without necessarily being a central example)

  • furthermore I expect many issuers of such sentiments are motivated by basically pretheoretic negative utilitarian insight

  E.g. imagine a minor steelification (which loses the aesthetic and rhetorical strength) like “nobody’s positive wellbeing (implicitly stemming from their freedom) can/​should be celebrated until everyone has freedom (implicitly necessary to escape negative wellbeing)” which is consistent with some kind of lexical negative utilitarianism.

  You’re right that if we insist that ‘freedom’ be interpreted identically in both places (parsimonious, granted, though I think the symmetry is better explained by aesthetic/​rhetorical concerns) another reading explicitly neglects the marginal benefit of lifting merely some people out of illiberty. Which is only consistent with utilitarianism if we use an unusual aggregation theory (i.e. minimising) - though I have also seen this discussed under negative utilitarianism.

  Anecdata: as someone whose (past) political background and involvement (waning!) is definitely some kind of lefty, and who, if it weren’t for various x- and s-risks, would plausibly consider some form (my form, naturally!) of lefty politics to be highly important (if not highly tractable), my reading of that claim at least goes something like the first one. I might not be representative in that respect.

  I have no doubt that many people expressing that kind of sentiment would still celebrate marginal ‘releases’, while considering it wrong to celebrate further the fruits of such freedom, ignoring others’ lack of freedom.

• Oliver SourbutJun 25, 2022, 8:24 PM

  −3 points

  0 ∶ 0

  in reply to: abrahamrowe’s comment on: Cri­tiques of EA that I want to read

  Minor nitpick: “nobody’s free until everyone is free” is precisely a (negative) utilitarian claim (albeit with unusual wording)

• Oliver SourbutJun 25, 2022, 11:30 AM

  5 points

  0 ∶ 0

  on: Are too many young, highly-en­gaged longter­mist EAs do­ing move­ment-build­ing?

  It’s possible the selection bias is high, but I don’t have good evidence for this besides personal anecdata. I don’t know how many people are relevantly similar to me, and I don’t know how representative we are of the latest EA ‘freshers’, since dynamics will change and I’m reporting with several years’ lag.

  Here’s my personal anecdata.

  Since 2016, around when I completed undergrad, I’ve been an engaged (not sure what counts as ‘highly engaged’) longtermist. (Before that point I had not heard of EA per se but my motives were somewhat proto EA and I wanted to contribute to ‘sustainable flourishing at scale’ and ‘tech for good’.) Nevertheless, until 2020 or so I was relatively invisibly upskilling, reflecting on priorities, consuming advice and ideas etc. and figuring out (perhaps too humbly and slowly) how to orient. More recently I’ve overcome some amount of impostor syndrome and simultaneously become more ‘community engaged’ (hence visible) and started directly contributing to technical AI safety research.

  If there are a lot with stories like that, they might form a large but quiet cohort countervailing your concern.

  Having said that, I think what you express here is excellent to discuss, I think I may have been unusually quiet+cautious, I didn’t encounter EA during undergrad, and I suspect (without here justifying) that community dynamics have changed sufficiently that my anecdote is not IID with the cohort you’re discussing.

• Oliver SourbutJun 25, 2022, 9:08 AM

  23 points

  0 ∶ 0

  in reply to: RobBensinger’s comment on: On Defer­ence and Yud­kowsky’s AI Risk Estimates

  I just wanted to state agreement that it seems a large number of people largely misread Death with Dignity, at least according to what seems to me the most plausible intended message: mainly about the ethical injunctions (which are very important as a finitely-rational and prone-to-rationalisation being), as Yudkowsky has written of in the past.

  The additional detail of ‘and by the way this is a bad situation and we are doing badly’ is basically modal Yudkowsky schtick and I’m somewhat surprised it updated anyone’s beliefs (about Yudkowsky’s beliefs, and therefore their all-things-considered-including-deference beliefs).

  I think if he had been a little more audience-aware he might have written it differently. Then again maybe not, if the net effect is more attention and investment in AI safety—and more recent posts and comments suggest he’s more willing than before to use certain persuasive techniques to spur action (which seems potentially misguided to me, though understandable).