Sam Clarke

Karma: 1,360

Sam Clarke 19 Nov 2019 10:13 UTC
27 points
0 ∶ 0
on: I’m Buck Shlegeris, I do research and outreach at MIRI, AMA
My sense of the current general landscape of AI Safety is: various groups of people pursuing quite different research agendas, and not very many explicit and written-up arguments for why these groups think their agenda is a priority (a notable exception is Paul’s argument for working on prosaic alignment). Does this sound right? If so, why has this dynamic emerged and should we be concerned about it? If not, then I’m curious about why I developed this picture.
What links here?
- Resources for AI Alignment Cartography by Gyrodiot (LessWrong; 4 Apr 2020 14:20 UTC; 45 points)

Sam Clarke 3 Mar 2022 11:56 UTC
23 points
0 ∶ 0
in reply to: Linch’s comment on: You are probably underestimating how good self-love can be
After practising some self-love I am now noticeably less stressed about work in general. I sleep better, have more consistent energy, enjoy having conversations about work-related stuff more (so I just talk about EA and AI risk more than I used to, which was a big win on my previous margin). I think I maybe work fewer hours than I used to because before it felt like there was a bear chasing me and if I wasn’t always working then it was going to eat me, whereas now that isn’t the case. But my working patterns feel healthy and sustainable now; before, I was going through cycles of half-burning out every 3 months or so (which was bad enough for my near-term productivity, not to mention long-term producitivity and health). I also spend relatively less time just turning the handle on my mainline tasks (vs zooming out, having random conversations that feel useful but won’t pay off immediately, reading more widely), which again I think was a win on my previous margin (maybe reduced it from ~90% to ~80% of my research hours).

I’m confused about how this happened. My model is that before there were two parts of me that strongly disagreed about whether work is good, and that these parts have now basically resolved (they agree that doing sensible amounts of work is good), because both feel understood and loved. Basically the part that didn’t think work was good just needed its needs to be understood and taken into account.

I think this model is quite different from Charlie’s main model of what happens (which is to do with memory consolidation), so I’m especially confused.

I haven’t attained persistent self-love of the sort described here.

Sam Clarke 19 Nov 2019 10:00 UTC
23 points
0 ∶ 0
on: I’m Buck Shlegeris, I do research and outreach at MIRI, AMA
What do you think are the biggest mistakes that the AI Safety community is currently making?

Sam Clarke 19 Nov 2019 9:59 UTC
23 points
0 ∶ 0
on: I’m Buck Shlegeris, I do research and outreach at MIRI, AMA
Paul Christiano is a lot more optimistic than MIRI about whether we could align a Prosaic AGI. In a relatively recent interview with AI Impacts he said he thinks “probably most of the disagreement” about this lies in the question of “can this problem [alignment] just be solved on paper in advance” (Paul thinks there’s “at least a third chance” of this, but suggests MIRI’s estimate is much lower). Do you have a sense of why MIRI and Paul disagree so much on this estimate?

Sam Clarke 9 Jun 2023 14:56 UTC
16 points
2 ∶ 0
on: SamClarke’s Shortform
(Post 3/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)
Some hot takes on AI governance field-building strategy
- More people should consciously upskill as ‘founders’, i.e. people who form and lead new teams/centres/etc. focused on making AI go well
  - A case for more founders: plausibly in crunch time there will be many more people/teams within labs/govs/think-tanks/etc. that will matter for how AI goes. Would be good if those teams were staffed with thoughtful and risk-conscious people.
  - What I think is required to be a successful founder:
    Strong in strategy (to steer their team in useful directions), management (for obvious reasons) and whatever object level work their team is doing
    Especially for teams within existing institutions, starting a new team requires skill in stakeholder management and consensus building.
  - Concrete thing you might consider doing: if you think you might want to be a founder, and you agree with the above list of skills, think above how to close your skill gaps
- More people should consciously upskill for the “AI endgame” (aka “acute risk period” aka “crunch time”). What might be different in the endgame and what does this imply about what people should do now?
  - Lots of ‘task force-style advising’ work
    → people should practise it now
  - Everyone will be very busy, especially senior people, so it won’t work as well to just defer
    → build your own models
  - More possible to mess things up real bad
    → start thinking harder about worst-case scenarios, red-teaming, etc. now, even if it seems a bit silly to e.g. spend time tightening up your personal infosec
  - The world may well be changing scarily fast
    → practice decision-making under pressure and uncertainty. Strategy might get even harder in the endgame
  - Being able to juggle 6 different kinds of things might be more valuable than being able to do one thing really well, because there might just be lots of different kinds of things to do (cf. ‘task force-style advising’)
    → specialise less? But specialisation tends to be pretty valuable, so I’m not sure this carries much weight overall
  - Relationships and reputation matter
    → build them now

Sam Clarke 4 Oct 2021 8:10 UTC
16 points
0 ∶ 0
on: SamClarke’s Shortform
An effective mental health intervention, for me, is listening to a podcast which ideally (1) discusses the thing I’m struggling with and (2) has EA, Rationality or both in the background. I gain both in-the-moment relief, and new hypotheses to test or tools to try.

Esp since it would be scalable, this makes me think that creating an EA mental health podcast would be an intervention worth testing—I wonder if anyone is considering this?

In the meantime, I’m on the look out for good mental health podcasts in general.
What links here?
- MichaelA's comment on Is there anyone working full-time on helping EAs address mental health problems? by richard_ngo (3 Nov 2021 8:26 UTC; 2 points)

Sam Clarke 11 Jun 2021 13:56 UTC
15 points
0 ∶ 0
in reply to: RobBensinger’s comment on: Survey on AI existential risk scenarios
Thanks Rob, interesting question. Here are the correlation coefficients between pairs of scenarios (sorted from max to min):

So it looks like there are only weak correlations between some scenarios.

It’s worth bearing in mind that we asked respondents not to give an estimate for any scenarios they’d thought about for less than 1 hour. The correlations could be stronger if we didn’t have this requirement.

Sam Clarke 13 Dec 2021 19:13 UTC
14 points
0 ∶ 0
on: EU AI Act now has a section on general purpose AI systems
I think equally important for longtermists is the new requirement for the Commission to consider updating the definition of AI, and the list of high-risk systems, every 1 year. If you buy that adaptive/flexible/future-proof governance will be important for regulating AGI, then this looks good.

(The basic argument for this instance of adaptive governance is something like: AI progress is fast and will only get faster, so having relevant sections of regulation come up for mandatory review every so often is a good idea, especially since policymakers are busy so this doesn’t tend to happen by default.)

Relevant part of the doc:
1. As regards the modalities for updates of Annexes I and III, the changes in Article 84 introduce a new reporting obligation for the Commission whereby it will be obliged to assess the need for amendment of the lists in these two annexes every 24 months following the entry into force of the AIA.

Sam Clarke 9 Jun 2023 14:59 UTC
13 points
0 ∶ 0
on: SamClarke’s Shortform
(Post 4/N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)
Some exercises for developing good judgement
I’ve spent a bit of time over the last year trying to form better judgement. Dumping some notes here on things I tried or considered trying, for future reference.
- Jump into the mindset of “the buck stops at me” for working out whether some project takes place, as if you were the grantmaker having to make the decision. Ask yourself: “wait, should this actually happen?”^[1]
  - (Rather than “does anything jump out as incorrect” or “do I have any random comments/ideas”—which are often helpful mindsets to be in when giving feedback to people, but don’t really train the core skill of good judgement.)
- I think forecasting trains a similar skill to this. I got some value from making some forecasts in the Metaculus Beginners’ Tournament.
- Find Google Docs where people (whose judgement you respect) have left comments and an overall take on the promisingness of the idea. Hide their comments and form your own take. Compare. (To make this a faster process, pick a doc/idea where you have enough background knowledge to answer without looking up loads of things).
- Ask people/orgs for things along the lines of [minimal trust investigations | grant reports | etc.] that they’ve written up. Do it yourself. Compare.
- Do any of the above with a friend; write your timeboxed answers then compare reasoning.
1. ^
  I think this framing of the exercise might have been mentioned to me by Michael Aird.

Sam Clarke 22 Nov 2019 16:22 UTC
13 points
0 ∶ 0
in reply to: Jan_Kulveit’s comment on: I’m Buck Shlegeris, I do research and outreach at MIRI, AMA
Thanks for the reply! Could you give examples of:
a) two agendas that seem to be “reflecting” the same underlying problem despite appearing very different superficially?
b) a “deep prior” that you think some agenda is (partially) based on, and how you would go about working out how deep it is?

Sam Clarke 3 Jun 2021 10:59 UTC
12 points
0 ∶ 0
in reply to: Lukas_Gloor’s comment on: “Existential risk from AI” survey results
I helped run the other survey mentioned , so I’ll jump in here with the relevant results and my explanation for the difference. The full results will be coming out this week.
Results
We asked participants to estimate the probability of an existential catastrophe due to AI (see definitions below). We got:
- mean: 0.23
- median: 0.1
Our question isn’t directly comparable with Rob’s, because we don’t condition on the catastrophe being “as a result of humanity not doing enough technical AI safety research” or “as a result of AI systems not doing/optimizing what the people deploying them wanted/intended”. However, that means that our results should be even higher than Rob’s.
Also, we operationalise existential catastophe/risk differently, though I think the operationalisations are similar to the point that they wouldn’t effect my estimate. Nonetheless:
- it’s possible that some respondents mistook “existential catastrophe” for “extinction” in our survey, despite our clarifications (survey respondents often don’t read the clarifications!)
- while “the overall value of the future will be drastically less than it could have been” and “existential catastrophe” are intended to be basically the same, the former intuitively “sounds” more likely than the latter, which might have affected some responses.
My explanation
I think it’s probably a combination of things, including this difference in operationalisation, random noise, and Rob’s suggestion that “respondents who were following the forum discussion might have been anchored in some way by that discussion, or might have had a social desirability effect from knowing that the survey-writer puts high probability on AI risk. It might also have made a difference that I work at MIRI.”
I can add a bit more detail to how it might have made a difference that Rob works at MIRI:
- In Rob’s survey, ⁵⁄₂₇ of respondents who specified an affiliation said they work at MIRI (~19%)
- In our survey, ¹⁄₄₃ respondents who specified an affiliation said they work at MIRI (~2%)
- (Rob’s survey had 44 respondents in total, ours had 75)
Definitions from our survey
Define an existential catastrophe as the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development (Bostrom, 2013).
Define an existential catastrophe due to AI as an existential catastrophe that could have been avoided had humanity’s development, deployment or governance of AI been otherwise. This includes cases where:
- AI directly causes the catastrophe.
- AI is a significant risk factor in the catastrophe, such that no catastrophe would have occurred without the involvement of AI.
- Humanity survives but its suboptimal use of AI means that we fall permanently and drastically short of our full potential.
Other results from our survey
We also asked participants to estimate the probability of an existential catastrophe due to AI under two other conditions.
Within the next 50 years
- mean: 0.12
- median: 0.05
In a counterfactual world where AI safety and governance receive no further investment or work from people aligned with the ideas of “longtermism”, “effective altruism” or “rationality” (but there are no other important changes between this counterfactual world and our world, e.g. changes in our beliefs about the importance and tractability of AI risk issues).
- mean: 0.32
- median: 0.25
What links here?
- Rob Bensinger's comment on “Existential risk from AI” survey results by Rob Bensinger (LessWrong; 4 Jun 2021 0:58 UTC; 5 points)

Sam Clarke 6 Oct 2021 15:02 UTC
11 points
0 ∶ 0
on: We’re Redwood Research, we do applied alignment research, AMA
What might be an example of a “much better weird, theory-motivated alignment research” project, as mentioned in your intro doc? (It might be hard to say at this point, but perhaps you could point to something in that direction?)

Sam Clarke 30 Mar 2023 23:14 UTC
10 points
2 ∶ 0
on: Deference on AI timelines: survey results
Things that surprised me about the results
- There’s more variety than I expected in the group of people who are deferred to
  - I suspect that some of the people in the “everyone else” cluster defer to people in one of the other clusters—in which case there is more deference happening than these results suggest.
- There were more “inside view” responses than I expected (maybe partly because people who have inside views were incentivised to respond, because it’s cool to say you have inside views or something). Might be interesting to think about whether it’s good (on the community level) for this number of people to have inside views on this topic.
- Metaculus was given less weight than I expected (but as per Eli (see footnote 2), I think that’s a good thing).
- Grace et al. AI expert surveys (1, 2) were deferred to less than I expected (but again, I think that’s good—many respondents to those surveys seem to have inconsistent views, see here for more details. And also there’s not much reason to expect AI experts to be excellent at forecasting things like AGI—it’s not their job, it’s probably not a skill they spend time training).
- It seems that if you go around talking to lots of people about AI timelines, you could move the needle on community beliefs more than I expected.

Sam Clarke 14 Mar 2022 15:48 UTC
10 points
0 ∶ 0
on: AI Risk is like Terminator; Stop Saying it’s Not
Thanks for writing this!

There are yet other views about about what exactly AI catastrophe will look like, but I think it is fair to say that the combined views of Yudkowsky and Christiano provide a fairly good representation of the field as a whole.

I disagree with this.

We ran a survey of prominent AI safety and governance researchers, where we asked them to estimate the probability of five different AI x-risk scenarios.

Arguably, the “terminator-like” scenarios are the “Superintelligence” scenario, and part 2 of “What failure looks like” (as you suggest in your post).^[1]

Conditional on an x-catastrophe due to AI occurring, the median respondent gave those scenarios 10% and 12% probability (mean 16% each). The other three scenarios^[2] got median 12.5%, 10% and 10% (means 18%, 17% and 15%).

So I don’t think that the “field as a whole” thinks terminator-like x-risk scenarios are the most likely. Accordingly, I’d prefer if the central claim of this post was “AI risk could actually be like terminator; stop saying it’s not”.
1. ↩︎
  Part 1 of “What failure looks like” probably doesn’t look that much like Terminator (disaster unfolds more slowly and is caused by AI systems just doing their jobs really well)
2. ↩︎
  That is, the following three secanrios: Part 1 of “What failure looks like”, existentially catastrophic AI misuse, and existentially catastrophic war between humans exacerbated by AI. See the post for full scenario descriptions.

Sam Clarke 18 Oct 2021 13:07 UTC
10 points
0 ∶ 0
in reply to: Rohin Shah’s comment on: General vs specific arguments for the longtermist importance of shaping AI development

The problem with general arguments is that they tell you very little about how to solve the problem

Agreed!

If I were producing key EA content/fellowships/etc, I would be primarily interested in getting people to solve the problem

I think this is true for some kinds of content/fellowships/etc, but not all. For those targeted at people who aren’t already convinced that AI safety/governance should be prioritised (which is probably the majority), it seems more important to present them with the strongest arguments for caring about AI safety/governance in the first place. This suggests presenting more general arguments.

Then, I agree that you want to get people to help solve the problem, which requires talking about specific failure modes. But I think that doing this prematurely can lead people to dismiss the case for shaping AI development for bad reasons.

Another way of saying this: for AI-related EA content/fellowships/etc, it seems worth separating motivation (“why should I care?”) and action (“if I do care, what should I do?”). This would get you the best of both worlds: people are presented with the strongest arguments, allowing them to make an informed decision about how much AI stuff should be prioritised, and then also the chance to start to explore specific ways to solve the problem.

I think this maybe applies to longtermism in general. We don’t yet have that many great ideas of what to do if longtermism is true, and I think that people sometimes (incorrectly) dismiss longtermism for this reason.

Sam Clarke 6 Oct 2021 13:37 UTC
10 points
0 ∶ 0
on: We’re Redwood Research, we do applied alignment research, AMA
Some questions that aren’t super related to Redwood/applied ML AI safety, so feel free to ignore if not your priority:
1. Assuming that it’s taking too long to solve the technical alignment problem, what might be some of our other best interventions to reduce x-risk from AI? E.g., regulation, institutions for fostering cooperation and coordination between AI labs, public pressure on AI labs/other actors to slow deployment, …
2. If we solve the technical alignment problem in time, what do you think are the other major sources of AI-related x-risk that remain? How likely do you think these are, compared to x-risk from not solving the technical alignment problem in time?

Sam Clarke 2 Mar 2022 14:29 UTC
9 points
0 ∶ 0
in reply to: Sam Clarke’s comment on: How I Formed My Own Views About AI Safety
Also, nitpick, but I find the “inside view” a more confusing and jargony way of just saying “independent impressions” (okay, also jargon to some extent, but closer to plain English), which also avoids the problem you point out: inside view is not the opposite of the Tetlockian sense of outside view (and the other ambiguities with outside view that another commenter pointed out).

Sam Clarke 18 Jan 2022 20:02 UTC
9 points
0 ∶ 0
on: The longtermist AI governance landscape: a basic overview
Something that would improve this post but I didn’t have time for:

For each kind of work, give a sense of:
- The amount of effort currently going into it
- What the biggest gaps/bottlenecks/open questions are
- What kinds of people might be well-suited to it

Sam Clarke 17 Aug 2022 0:43 UTC
8 points
0 ∶ 0
on: Feedback I’ve been giving to junior x-risk researchers
Love it, thanks for the post!

“Reading ‘too much’ is possibly the optimal strategy if you’re mainly trying to skill up (e.g., through increased domain knowledge), rather than have direct impact now. But also bear in mind that becoming more efficient at direct impact is itself a form of skilling up, and this pushes back toward ‘writing early’ as the better extreme.”

Two thoughts on this section:
1. Additional (obvious) arguments for writing early: producing stuff builds career capital, and is often a better way to learn than just reading.
2. I want to disentangle ‘aiming for direct impact’ and ‘writing early’. You can write without optimising hard for direct impact, and I claim that more junior people should do so (on the current margin). There’s some failure mode (which I fell into myself) where junior researchers try to solve hugely important problems, because they really want to make direct impact. But this leads them to work on problems that are wicked, poorly scoped, or methodologically fraught. Which ends with them getting stuck, demoralised, and not producing anything.
Often, I think it’s better for junior researchers to still aim to write/produce stuff (bc of the arguments above/in your piece), but not be optimising hard for direct impact with that writing. Picking more tractable problems and less important ones.

Sam Clarke 9 Aug 2022 10:00 UTC
8 points
0 ∶ 0
in reply to: Karthik Tadepalli’s comment on: Why does no one care about AI?

they basically see AGI as very unlikely

Certainly some people you talk to in the fairness/bias crowd think AGI is very unlikely, but that’s definitely not a consensus view among AI researchers. E.g. see this survey of AI researchers (at top conferences in 2015, not selecting for AI safety folk), which finds that:

Researchers believe there is a 50% chance of AI outperforming humans in all tasks in 45 years and of automating all human jobs in 120 years

Sam Clarke

Some hot takes on AI governance field-building strategy

Some exercises for developing good judgement

Results

My explanation

Definitions from our survey

Other results from our survey