Sam Clarke

• Sam ClarkeOct 10, 2022, 5:31 PM

  2 points

  0 ∶ 0

  in reply to: aaron_mai’s comment on: When re­port­ing AI timelines, be clear who you’re (not) defer­ring to

  Thanks for your comment! I agree that the concept of deference used in this community is somewhat unclear, and a separate comment exchange on this post further convinced me of this. It’s interesting to know how the word is used in formal epistemology.

  Here is the EA Forum topic entry on epistemic deference. I think it most closely resembles your (c). I agree there’s the complicated question of what your priors should be, before you do any deference, which leads to the (b) /​ (c) distinction.

• Sam ClarkeOct 10, 2022, 5:19 PM

  6 points

  1 ∶ 0

  on: When re­port­ing AI timelines, be clear who you’re (not) defer­ring to

  Thanks for your comment!

  Asking “who do you defer to?” feels like a simplification

  Agreed! I’m not going to make any changes to the survey at this stage, but I like the suggestion and if I had more time I’d try to clarify things along these lines.

  I like the distinction between deference to people/​groups and deference to processes.

  deference to good ideas

  [This is a bit of a semantic point, but seems important enough to mention] I think “deference to good ideas” wouldn’t count as “deference”, in the way that this community has ended up using it. As per the forum topic entry on epistemic deference:

  Epistemic deference is the process of updating one’s beliefs in response to what others appear to believe, even if one ignores the reasons for those beliefs or do not find those reasons persuasive. (emphasis mine)

  If you find an argument persuasive and incorporate it into your views, I think that doesn’t qualify as “deference”. Your independent impressions don’t (and in most cases won’t) be the views you formed in isolation. When forming your independent impressions, you can and should take other people’s arguments into account, to the extent that you find them convincing. Deference occurs when you take into account knowledge about what other people believe, and how trustworthy you find them, without engaging with their object level arguments.

  non-defensible original ideas

  A similar point applies to this one, I think.

  (All of the above makes me think that the concept of deference is even less clear in the community than I thought it was—thanks for making me aware of this!)

  What links here?

  • Sam Clarke's comment on When re­port­ing AI timelines, be clear who you’re defer­ring to by Sam Clarke (Oct 10, 2022, 5:31 PM; 2 points)

• Sam ClarkeOct 10, 2022, 5:00 PM

  1 point

  0 ∶ 0

  in reply to: L Rudolf L’s comment on: Assess­ing SERI/​CHERI/​CERI sum­mer pro­gram im­pact by sur­vey­ing fellows

  Cool, makes sense.

  The main way to answer this seems to be getting a non-self-rated measure of research skill change.

  Agreed. Asking mentors seems like the easiest thing to do here, in the first instance.

When re­port­ing AI timelines, be clear who you’re defer­ring to

• Sam ClarkeOct 5, 2022, 11:09 AM

  1 point

  0 ∶ 0

  in reply to: Sam Clarke’s comment on: Assess­ing SERI/​CHERI/​CERI sum­mer pro­gram im­pact by sur­vey­ing fellows

  Somewhat related comment: next time, I think it could be better to ask “What percentage of the value of the fellowship came from these different components?”* instead of “What do you think were the most valuable parts of the programme?”. This would give a bit more fine-grained data, which could be really important.

  E.g. if it’s true that most of the value of ERIs comes from networking, this would suggest that people who want to scale ERIs should do pretty different things (e.g. lots of retreats optimised for networking).

  *and give them several buckets to select from, e.g. <3%, 3-10%, 10-25%, etc.

• Sam ClarkeOct 5, 2022, 11:01 AM

  6 points

  0 ∶ 0

  on: Assess­ing SERI/​CHERI/​CERI sum­mer pro­gram im­pact by sur­vey­ing fellows

  Thanks for putting this together!

  I’m surprised by the combination of the following two survey results:

  Fellows’ estimate of how comfortable they would be pursuing a research project remains effectively constant. Many start out very comfortable with research. A few decline.

  and

  Networking, learning to do research, and becoming a stronger candidate for academic (but not industry) jobs top the list of what participants found most valuable about the programs. (emphasis mine)

  That is: on average, fellows claim they learned to do better research, but became no more comfortable pursuing a research project.

  Do you think this is mostly explained by most fellows already being pretty comfortable with research?

  A scatter plot of comfort against improvement in research skill could be helpful to examine different hypotheses (though won’t be possible with the current data, given how the “greatest value adds” question was phrased.

• Sam ClarkeSep 21, 2022, 3:12 PM

  3 points

  0 ∶ 0

  in reply to: ben.smith’s comment on: Sur­vey on AI ex­is­ten­tial risk scenarios

  Re (1) See When Will AI Exceed Human Performance? Evidence from AI Experts (2016) and the 2022 updated version. These surveys don’t ask about x-risk scenarios in detail, but do ask about the overall probability of very bad outcomes and other relevant factors.

  Re (1) and (3), you might be interested in various bits of research that GovAI has done on the American public and AI researchers.

  You also might want to get in touch with Noemi Dreksler, who is working on surveys at GovAI.

• Sam ClarkeSep 5, 2022, 5:32 PM

  2 points

  0 ∶ 0

  on: Strate­gic Per­spec­tives on Long-term AI Gover­nance: Introduction

  A potentially useful subsection for each perspective could be: evidence that should change your mind about how plausible this perspective is (including things you might observe over the coming years/​decades). This would be kinda like the future-looking version of the “historical analogies” subsection.

• Sam ClarkeAug 17, 2022, 12:53 AM

  3 points

  0 ∶ 0

  on: Feed­back I’ve been giv­ing to ju­nior x-risk researchers

  Another random thought: a bunch of these lessons seem like the kind of things that general writing and research coaching can teach. Maybe summer fellows and similar should be provided with that? (Freeing up time for you/​other people in your reference class to play to your comparative advantage.)

  (Though some of these lessons are specific to EA research and so seem harder to outsource.)

• Sam ClarkeAug 17, 2022, 12:43 AM

  8 points

  0 ∶ 0

  on: Feed­back I’ve been giv­ing to ju­nior x-risk researchers

  Love it, thanks for the post!

  “Reading ‘too much’ is possibly the optimal strategy if you’re mainly trying to skill up (e.g., through increased domain knowledge), rather than have direct impact now. But also bear in mind that becoming more efficient at direct impact is itself a form of skilling up, and this pushes back toward ‘writing early’ as the better extreme.”

  Two thoughts on this section:

  1. Additional (obvious) arguments for writing early: producing stuff builds career capital, and is often a better way to learn than just reading.

  2. I want to disentangle ‘aiming for direct impact’ and ‘writing early’. You can write without optimising hard for direct impact, and I claim that more junior people should do so (on the current margin). There’s some failure mode (which I fell into myself) where junior researchers try to solve hugely important problems, because they really want to make direct impact. But this leads them to work on problems that are wicked, poorly scoped, or methodologically fraught. Which ends with them getting stuck, demoralised, and not producing anything.

  Often, I think it’s better for junior researchers to still aim to write/​produce stuff (bc of the arguments above/​in your piece), but not be optimising hard for direct impact with that writing. Picking more tractable problems and less important ones.

• Sam ClarkeAug 17, 2022, 12:13 AM

  1 point

  0 ∶ 0

  in reply to: Lukas_Gloor’s comment on: Gen­eral vs spe­cific ar­gu­ments for the longter­mist im­por­tance of shap­ing AI development

  Thanks, I agree with all of this

• Sam ClarkeAug 10, 2022, 4:18 PM

  5 points

  0 ∶ 0

  on: Slow­ing down AI progress?

  Relevant discussion from a couple of days ago: https://​​astralcodexten.substack.com/​​p/​​why-not-slow-ai-progress

• Sam ClarkeAug 9, 2022, 10:49 AM

  2 points

  0 ∶ 0

  in reply to: Karthik Tadepalli’s comment on: Why does no one care about AI?

  The question was:

  Assume for the purpose of this question that HLMI* will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run?

  So it doesn’t presuppose some agentic form of AGI—but rather asks about the same type of technology that the median respondant gave a 50% chance of arriving within 45 years.

  *HLMI was defined in the survey as:

  “High-level machine intelligence” (HLMI) is achieved when unaided machines can accomplish every task better and more cheaply than human workers.

• Sam ClarkeAug 9, 2022, 10:30 AM

  1 point

  0 ∶ 0

  in reply to: Karthik Tadepalli’s comment on: Why does no one care about AI?

  Right, I just wanted to point out that the average AI researcher who dismisses AI x-risk doesn’t do so because they think AGI is very unlikely. But I admit to often being confused about why they do dismiss AI x-risk.

  The same survey asked AI researchers about the outcome they expect from AGI:

  The median probability was 25% for a “good” outcome and 20% for an “extremely good” outcome. By contrast, the probability was 10% for a bad outcome and 5% for an outcome described as “Extremely Bad (e.g., human extinction).”

  If I learned that there was some scientific field where the median researcher assigned a 5% probability that we all die due to advances in their field, I’d be incredibly worried. Going off this data alone, it seems hard to make a case that x-risk from AI is some niche thing that almost no AI researchers think is real.

  The median researcher does think it’s somewhat unlikely, but 5% extinction risk is more than enough to take it very seriously and motivate a huge research and policy effort.

• Sam ClarkeAug 9, 2022, 10:15 AM

  3 points

  0 ∶ 0

  in reply to: Mau’s comment on: Clas­sify­ing sources of AI x-risk

  Thanks, I agree with most of these suggestions.

  “Other (AI-enabled) dangerous tech” feels to me like it clearly falls under “exacerbating other x-risk factors”

  I was trying to stipulate that the dangerous tech was a source of x-risk in itself, not just a risk factor (admittedly the boundary is fuzzy). The wording was “AI leads to deployment of technology that causes extinction or unrecoverable collapse” and the examples (which could have been clearer) were intended to be “a pathogen kills everyone” or “full scale nuclear war leads to unrecoverable collapse”

• Sam ClarkeAug 9, 2022, 10:00 AM

  8 points

  0 ∶ 0

  in reply to: Karthik Tadepalli’s comment on: Why does no one care about AI?

  they basically see AGI as very unlikely

  Certainly some people you talk to in the fairness/​bias crowd think AGI is very unlikely, but that’s definitely not a consensus view among AI researchers. E.g. see this survey of AI researchers (at top conferences in 2015, not selecting for AI safety folk), which finds that:

  Researchers believe there is a 50% chance of AI outperforming humans in all tasks in 45 years and of automating all human jobs in 120 years

Clas­sify­ing sources of AI x-risk

• Sam Clarke 21 Jul 2022 15:29 UTC

  1 point

  0 ∶ 0

  in reply to: Algo_Law’s comment on: A Sur­vey of the Po­ten­tial Long-term Im­pacts of AI

  Thanks, I’m glad this was helpful to you!

• Sam Clarke 18 Jul 2022 10:21 UTC

  1 point

  0 ∶ 0

  in reply to: technicalities’s comment on: How would a lan­guage model be­come goal-seek­ing?

  I’m also still a bit confused about what exactly this concept refers to. Is a ‘consequentialist’ basically just an ‘optimiser’ in the sense that Yudkowsky uses in the sequences (e.g. here), that has later been refined by posts like this one (where it’s called ‘selection’) and this one?

  In other words, roughly speaking, is a system a consequentialist to the extent that it’s trying to take actions that push its environment towards a certain goal state?

A Sur­vey of the Po­ten­tial Long-term Im­pacts of AI