Sam Clarke

• Sam Clarke 19 Jun 2023 14:20 UTC

  2 points

  0 ∶ 0

  in reply to: JP Addison’s comment on: Some tal­ent needs in AI governance

  Thanks JP!

Ap­ti­tudes for AI gov­er­nance work

Some tal­ent needs in AI governance

• Sam Clarke 9 Jun 2023 15:25 UTC

  4 points

  1 ∶ 0

  on: SamClarke’s Shortform

  (Post 6/​N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

  Some heuristics for prioritising between talent pipeline interventions

  Explicit backchaining is one way to do prioritisation. I sometimes forget that there are other useful heuristics, like:

  • Cheap to pilot

    • E.g. doesn’t require new infrastructure or making a new hire

  • Cost is easier to estimate than benefit, so lower cost things tend to be more likely to actually happen

  • Visualise some person or org has been actually convinced to trial the thing. Imagine the conversation with that decision-maker. What considerations actually matter for them?

  • Is there someone else who would do most of the heavy lifting?

• Sam Clarke 9 Jun 2023 15:18 UTC

  2 points

  0 ∶ 0

  on: SamClarke’s Shortform

  (Post 5/​N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

  Laundry list of talent pipeline interventions

  • More AI governance groups/​programs at universities

  • Run workshops on the most marginally valuable aptitudes

    • E.g. a macrostrategy workshop could look like: people tell stories about, concretely, things could go badly, and backchaining to what we should do

  • Run bootcamps on particularly important topics, e.g. compute

    • Help bring more people up to speed in most important areas

  • Create better resources for teaching low-hanging fruit skills

    • E.g. resources for learning how to do reasoning transparency—currently this is pretty hard to learn, but seems like one of the more teachable skills

    • E.g. compile zero-shot tips that occasionally people just miss, like “don’t read books from cover to cover”

    • Tag those resources onto existing programs

  • AI-governance-careers.com

  • “Ambitious talent search”

  • The best way of predicting happiness and success on the job is to actually do the job → hackathon type stuff. “This is the problem. Here’s the internet. Here’s some resources. Go nuts.”

  Note to self: more detailed but less structured version of these notes here.

• Sam Clarke 9 Jun 2023 14:59 UTC

  13 points

  0 ∶ 0

  on: SamClarke’s Shortform

  (Post 4/​N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

  Some exercises for developing good judgement

  I’ve spent a bit of time over the last year trying to form better judgement. Dumping some notes here on things I tried or considered trying, for future reference.

  • Jump into the mindset of “the buck stops at me” for working out whether some project takes place, as if you were the grantmaker having to make the decision. Ask yourself: “wait, should this actually happen?”^[1]

    • (Rather than “does anything jump out as incorrect” or “do I have any random comments/​ideas”—which are often helpful mindsets to be in when giving feedback to people, but don’t really train the core skill of good judgement.)

  • I think forecasting trains a similar skill to this. I got some value from making some forecasts in the Metaculus Beginners’ Tournament.

  • Find Google Docs where people (whose judgement you respect) have left comments and an overall take on the promisingness of the idea. Hide their comments and form your own take. Compare. (To make this a faster process, pick a doc/​idea where you have enough background knowledge to answer without looking up loads of things).

  • Ask people/​orgs for things along the lines of [minimal trust investigations | grant reports | etc.] that they’ve written up. Do it yourself. Compare.

  • Do any of the above with a friend; write your timeboxed answers then compare reasoning.

  1. ^

     I think this framing of the exercise might have been mentioned to me by Michael Aird.

• Sam Clarke 9 Jun 2023 14:56 UTC

  16 points

  2 ∶ 0

  on: SamClarke’s Shortform

  (Post 3/​N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

  Some hot takes on AI governance field-building strategy

  • More people should consciously upskill as ‘founders’, i.e. people who form and lead new teams/​centres/​etc. focused on making AI go well

    • A case for more founders: plausibly in crunch time there will be many more people/​teams within labs/​govs/​think-tanks/​etc. that will matter for how AI goes. Would be good if those teams were staffed with thoughtful and risk-conscious people.

    • What I think is required to be a successful founder:

      • Strong in strategy (to steer their team in useful directions), management (for obvious reasons) and whatever object level work their team is doing

      • Especially for teams within existing institutions, starting a new team requires skill in stakeholder management and consensus building.

    • Concrete thing you might consider doing: if you think you might want to be a founder, and you agree with the above list of skills, think above how to close your skill gaps

  • More people should consciously upskill for the “AI endgame” (aka “acute risk period” aka “crunch time”). What might be different in the endgame and what does this imply about what people should do now?

    • Lots of ‘task force-style advising’ work

      • → people should practise it now

    • Everyone will be very busy, especially senior people, so it won’t work as well to just defer

      • → build your own models

    • More possible to mess things up real bad

      • → start thinking harder about worst-case scenarios, red-teaming, etc. now, even if it seems a bit silly to e.g. spend time tightening up your personal infosec

    • The world may well be changing scarily fast

      • → practice decision-making under pressure and uncertainty. Strategy might get even harder in the endgame

    • Being able to juggle 6 different kinds of things might be more valuable than being able to do one thing really well, because there might just be lots of different kinds of things to do (cf. ‘task force-style advising’)

      • → specialise less? But specialisation tends to be pretty valuable, so I’m not sure this carries much weight overall

    • Relationships and reputation matter

      • → build them now

• Sam Clarke 9 Jun 2023 14:54 UTC

  4 points

  0 ∶ 0

  on: SamClarke’s Shortform

  (Post 2/​N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

  Misc things it seems useful to do/​find out

  • To inform talent development activities: talk with relevant people who have skilled up. How did they do it? What could be replicated via talent pipeline infrastructure? Generally talk through their experience.

    • Kinds of people to prioritise: those who are doing exceptionally well; those who have grown quite recently (might have better memory of what they did)

  • To inform talent search activities: talk with relevant people—especially senior folks—about what got them involved. This could feed into earlier stage talent pipeline activities

  • Case studies of important AI governance ideas (e.g. model evals, importance of infosec) and/​or pipeline wins. How did they come about? What could be replicated?

  • How much excess demand is there for fellowship programs? Look into the strength of applications over time. This would inform how much value there is in scaling fellowships.

  • Figure out whether there is a mentorship bottleneck.

    • More concretely: would it be overall better if some of the more established AI governance folk spent a few more hours per month on mentorship?

    • Thing to do: very short survey asking established AI governance people how many hours per month they spend on mentorship.

    • Benefits of mentorship:

      • For the mentee: fairly light touch involvement can go a long way towards bringing them up to speed and giving them encouragement.

      • For the mentor: learn about fit for mentorship/​management. Can be helpful for making object-level progress on work.

    • These benefits are often illegible and delayed in time, so a priori likely to be undersupplied.

    • If there’s a mentorship bottleneck, it might be important to solve ~now. The number of AI governance jobs is likely going to rise dramatically over the coming years—so having more thoughtful, risk-conscious people who are better placed to land those roles is more urgent than you might think if only considering having enough people by the acute risk period.

    • If there is a mentorship bottleneck how might one actually go about solving that? Obvious idea is to nudge potential mentors to consider:

      • Asking GovAI, who have been collecting expressions of interest for RAs, whether there’s anyone who might be a good fit as your mentee

      • Posting on the forum or otherwise broadcasting what kind of thing a potential mentee could do that might make you excited about mentoring them (e.g. write a review of report X, or write a memo on topic Y), and that (e.g.) if they do that they should send it to you, and you’ll at least take a 30 minute call with them

      • Mentoring someone on a summer program (e.g. GovAI Fellowship, ERIs, SERI MAGS, HAIST/​MAIA programs, AI Safety Camp, …)

• Sam Clarke 9 Jun 2023 14:51 UTC

  5 points

  0 ∶ 0

  on: SamClarke’s Shortform

  (Post 1/​N with some rough notes on AI governance field-building strategy. Posting here for ease of future reference, and in case anyone else thinking about similar stuff finds this helpful.)

  Some key uncertainties in AI governance field-building

  According to me, these are some of the key uncertainties in AI governance field-building—questions which, if we had better answers to them, might significantly influence decisions about how field-building should be done.

  How best to find/​upskill more people to do policy development work?

  • I think there are three main skillsets involved in policy development work:

    • Macrostrategy

    • “Traditional” policy development work (e.g. detailed understanding of how policymaking works in institution, to devise feasible policy action)

    • Impact focus (i.e. working to improve lasting impacts of AI in a scope-sensitive way)

  • A more concrete question is: which pillars to prioritise in talent search/​selection vs upskilling. E.g. do you take people already skilled in X and Y and give them Z; X and Z and give them Y; etc. etc. ?

  What are the most important profiles that aren’t currently being hired for, but nonetheless might matter?

  Reasons why this seems important to get clarity on:

  • Focus on neglected aspects of the talent pipeline. People want to get hired, so will be trying to skill up for positions that are currently being hired for. Whereas for future positions—especially for positions that will never be “hired for” per se (e.g. leading a policy team that wouldn’t exist unless you pitched it), and “positions with a deadline”^[1]—the standard career incentives to skill up for them aren’t as strong. Also, some people currently hiring are already trying to solve their own bottlenecks (e.g. designing efficient hiring processes to identify talent) whereas future people aren’t.^[2]

  • Avoid myopically optimising the talent pipeline. The world will probably change a lot in the run up to advanced AI. This will affect the value of different talent pipeline interventions in three ways:
    

    • There will likely be more people interested in AI (governance). So, more people who want to do things, and hence more value in work that usefully leverages a large amount of labour.

    • The people who become interested may have different skills and inclinations, compared to the current talent pool. This will change the future comparative advantage of people we can currently find/​upskill.
      

      • More concretely, you might think that people currently working on AI governance are disproportionately inclined towards macro/​strategy, relative to the talent pool in, say, 5 years’ time. Optimising for ticking all the talent boxes by the end of the year might look like finding/​upskilling people with deep knowledge in certain areas that we’re lacking (e.g. more lawyers). But if you instead think these people will just be drawn to the field once there are important questions they can answer, and that the community can usefully leverage their knowledge, this could suggest instead doubling down on building a community that’s excellent at strategy [I’m very uncertain about this particular line of reasoning, but think there might be some important thread to pull on here, more generally]

    • The nature of important work changes as we move into the AI endgame. E.g. probably less field-building, more public comms, more founding of new institutions, more policy development, etc.

  To what extent should talent pipeline efforts treat AI governance as a (pre-)paradigmatic field?

  • More concrete versions of this question, that are all trying to get at the same thing:
    

    • On the current margin (now and in the future), how much of the most valuable work is crank-turn-y? By “crank-turn-y” I mean “work which can be delegated to sensible people even if they aren’t deeply integrated into the existing field”.

    • On the current margin (now and in the future), how high a premium should talent search/​development efforts put on the macro/​strategy aptitude?

    • On the current margin (now and in the future), how much of the most valuable work looks like contributing to an intellectual project that has been laid out (rather than doing the initial charting out of that intellectual project)?

  • Answers to these questions seem like they should affect: how quickly the field scales up, and how much we are trying to attract people who are excellent at crank-turn-y work vs strategy work. I lightly hold the intuition that erring on this question is a main way that this field could mess up.

  1. ^

     Re: “positions with a deadline”: it seems plausible to me that there will be these windows of opportunity when important positions come up, and if you haven’t built the traits you need by that time, it’s too late. E.g. more talent very skilled at public comms would probably have been pretty useful in Q1-2 2023.

  2. ^

     Counterpoint: the strongest version of this consideration assumes a kind of “efficient market hypothesis” for people building up their own skills. If people aren’t building up their own skills efficiently, then there could still be significant gains from helping them to do so, even for positions that are currently being hired for. Still, I think this consideration carries some weight.

• Sam Clarke 24 Apr 2023 18:11 UTC

  2 points

  0 ∶ 0

  in reply to: Diego Oliveira’s comment on: The longter­mist AI gov­er­nance land­scape: a ba­sic overview

  Thanks, fixed!

• Sam Clarke 5 Apr 2023 17:05 UTC

  3 points

  0 ∶ 0

  in reply to: NunoSempere’s comment on: Sur­vey on AI ex­is­ten­tial risk scenarios

  Thanks for this, I won’t use “bet” in this context in the future

• Sam Clarke 30 Mar 2023 23:35 UTC

  4 points

  0 ∶ 0

  in reply to: MichaelA’s comment on: When re­port­ing AI timelines, be clear who you’re (not) defer­ring to

  Finally posted

• Sam Clarke 30 Mar 2023 23:14 UTC

  10 points

  2 ∶ 0

  on: Defer­ence on AI timelines: sur­vey results

  Things that surprised me about the results

  • There’s more variety than I expected in the group of people who are deferred to

    • I suspect that some of the people in the “everyone else” cluster defer to people in one of the other clusters—in which case there is more deference happening than these results suggest.

  • There were more “inside view” responses than I expected (maybe partly because people who have inside views were incentivised to respond, because it’s cool to say you have inside views or something). Might be interesting to think about whether it’s good (on the community level) for this number of people to have inside views on this topic.

  • Metaculus was given less weight than I expected (but as per Eli (see footnote 2), I think that’s a good thing).

  • Grace et al. AI expert surveys (1, 2) were deferred to less than I expected (but again, I think that’s good—many respondents to those surveys seem to have inconsistent views, see here for more details. And also there’s not much reason to expect AI experts to be excellent at forecasting things like AGI—it’s not their job, it’s probably not a skill they spend time training).

  • It seems that if you go around talking to lots of people about AI timelines, you could move the needle on community beliefs more than I expected.

Defer­ence on AI timelines: sur­vey results

• Sam Clarke 29 Mar 2023 23:29 UTC

  4 points

  0 ∶ 0

  in reply to: Maxime_Riche’s comment on: Sur­vey on AI ex­is­ten­tial risk scenarios

  Thanks for your comment!

  I doubt that it’s reasonable to draw these kinds of implications from the survey results, for a few reasons:

  • respondents were very uncertain

  • there’s overlap between the scenarios

  • there’s no 1-1 mapping between “fields” and risk scenarios (e.g. I’d strongly bet that improved cooperation of certain kinds would make both catastrophic misalignment and war less likely) (though maybe your model tries to account for this, I didn’t look at it)

  A broader point: I think making importance comparisons (between interventions) on the level of abstraction of “improving cooperation”, “hard AI alignment” and “partial/​naive alignment” doesn’t make much sense. I expect comparing specific plans/​interventions to be much more useful.

• Sam Clarke 14 Mar 2023 14:55 UTC

  6 points

  0 ∶ 0

  on: In­for­ma­tion se­cu­rity con­sid­er­a­tions for AI and the long term future

  In a following post, we will explore:

  1. How you could orient your career toward working on security

  Did you end up writing this, or have a draft of it you’d be willing to share?

• Sam Clarke 17 Feb 2023 0:53 UTC

  6 points

  0 ∶ 0

  in reply to: MichaelA’s comment on: When re­port­ing AI timelines, be clear who you’re (not) defer­ring to

  Will get them written up this month—sorry for the delay!

• Sam Clarke 3 Feb 2023 12:24 UTC

  1 point

  0 ∶ 0

  in reply to: David Krueger’s comment on: Clar­ifi­ca­tions about struc­tural risk from AI

  In fact, one I am writing this comment because I think this post itself endorses that framing to too great an extent.

  Probably agree with you there

  I do not think it is appropriate to describe this [the Uber crash] simply as an accident

  Also agree with that. I wasn’t trying to claim it is simply an accident—there are also structural causes (i.e. bad incentives). As I wrote:

  Note that this could also be well-described as an “accident risk” (there was some incompetence on behalf of the engineers, along with the structural causes). [emphasis added]

  If I were writing this again, I wouldn’t use the word “well-described” (unclear what I actually mean; sounds like I’m making a stronger claim than I was). Maybe I’d say “can partly be described as an accident”.

  But today, I think this mostly just introduces unnecessary/​confusing abstraction. The main important point in my head now is: when stuff goes wrong, it can be due to malintent, incompetence, or the incentives. Often it’s a complicated mixture of all three. Make sure your thinking about AI risk takes that into account.

  And sure, you could carve up risks into categories, where you’re like:

  • if it’s mostly incompetence, call it an accident

  • if it’s mostly malintent, call it misuse

  • if it’s mostly incentives, call it structural

  But it’s pretty unclear what “mostly” means, and moreover it just feels kind of unnecessary/​confusing.

The ‘Old AI’: Les­sons for AI gov­er­nance from early elec­tric­ity regulation

• Sam Clarke 10 Oct 2022 17:36 UTC

  5 points

  0 ∶ 0

  in reply to: Emrik’s comment on: When re­port­ing AI timelines, be clear who you’re (not) defer­ring to

  Unfortunately, when someone tells you “AI is N years away because XYZ technical reasons,” you may think you’re updating on the technical reasons, but your brain was actually just using XYZ as excuses to defer to them.

  I really like this point. I’m guilty of having done something like this loads myself.

  When someone gives you gears-level evidence, and you update on their opinion because of that, that still constitutes deferring. What you think of as gears-level evidence is nearly always disguised testimonial evidence. At least to some, usually damning, degree. And unless you’re unusually socioepistemologically astute, you’re just lost to the process.

  If it’s easy, could you try to put this another way? I’m having trouble making sense of what exactly you mean, and it seems like an important point if true.