Sam Clarke

• Sam ClarkeDec 13, 2021, 7:13 PM

  14 points

  0 ∶ 0

  on: EU AI Act now has a sec­tion on gen­eral pur­pose AI systems

  I think equally important for longtermists is the new requirement for the Commission to consider updating the definition of AI, and the list of high-risk systems, every 1 year. If you buy that adaptive/​flexible/​future-proof governance will be important for regulating AGI, then this looks good.

  (The basic argument for this instance of adaptive governance is something like: AI progress is fast and will only get faster, so having relevant sections of regulation come up for mandatory review every so often is a good idea, especially since policymakers are busy so this doesn’t tend to happen by default.)

  Relevant part of the doc:

  1. As regards the modalities for updates of Annexes I and III, the changes in Article 84 introduce a new reporting obligation for the Commission whereby it will be obliged to assess the need for amendment of the lists in these two annexes every 24 months following the entry into force of the AIA.

• Sam ClarkeNov 24, 2021, 11:20 AM

  3 points

  0 ∶ 0

  in reply to: Eevee🔹’s comment on: What is most con­fus­ing to you about AI stuff?

  On the margin, should donors prioritize AI safety above other existential risks and broad longtermist interventions?

  To the extent that this question overlaps with Mauricio’s question 1.2 (i.e. A bunch of people seem to argue for “AI stuff is important” but believe /​ act as if “AI stuff is overwhelmingly important”—what are arguments for the latter view?), then you might find his answer helpful.

  other x-risks and longtermist areas seem rather unexplored and neglected, like s-risks

  Only a partial answer, but worth noting that I think the most plausible source of s-risk is messing up on AI stuff

• Sam ClarkeNov 24, 2021, 11:15 AM

  3 points

  0 ∶ 0

  in reply to: Eevee🔹’s comment on: What is most con­fus­ing to you about AI stuff?

  Is “intelligence” … really enough to make an AI system more powerful than humans (individuals, groups, or all of humanity combined)?

  Some discussion of this question here: https://​​www.alignmentforum.org/​​posts/​​eGihD5jnD6LFzgDZA/​​agi-safety-from-first-principles-control

• Sam ClarkeNov 24, 2021, 11:12 AM

  11 points

  0 ∶ 0

  in reply to: Miranda_Zhang’s comment on: What is most con­fus­ing to you about AI stuff?

  Do we need to decide on a moral principle(s) first? How would it be possible to develop beneficial AI without first ‘solving’ ethics/​morality?

  Good question! The answer is no: ‘solving’ ethics/​morality first is one thing that we probably eventually need to do, but we could first solve a narrower, simpler form of AI alignment, and use those aligned systems to help us solve ethics/​morality and the other trickier problems (like the control problem for more general, capable systems). This is more or less what is discussed in ambitious vs narrow value learning. Narrow value learning is one narrower, simpler form of AI alignment. There are others, discussed here under the heading “Alternative solutions”.

• Sam ClarkeNov 24, 2021, 10:52 AM

  3 points

  0 ∶ 0

  in reply to: Yonatan Cale’s comment on: What is most con­fus­ing to you about AI stuff?

  A timely post: https://​​forum.effectivealtruism.org/​​posts/​​DDDyTvuZxoKStm92M/​​ai-safety-needs-great-engineers

  (The focus is software engineering not development, but should still be informative.)

[Question] What is most con­fus­ing to you about AI stuff?

CGP Grey: The Fable of the Dragon-Tyrant

• Sam ClarkeNov 5, 2021, 2:44 PM

  2 points

  0 ∶ 0

  on: Why AI al­ign­ment could be hard with mod­ern deep learning

  I love this post and also expect it to be something that I point people towards in the future!

  I was wondering about what kind of alignment failure—i.e. outer or inner alignment—you had in mind when describing sycophant models (for schemer models, it’s obviously an inner alignment failure).

  It seems you could get sycophant models via inner alignment failure, because you could train them a sensible, well-specified objective functions, and yet the model learns to pursue human approval anyway (because “pursuing human approval” turned out to be more easily discovered by SGD).

  It also seems you could also sycophant models them via outer alignment failure, because e.g. a model trained using naive reward modelling (which would be an obviously terrible objective) seems very likely to yield a model that is pursuing approval from the humans whose feedback is used in training the reward model.

  Does this seem right to you, and if so, which kind of alignment failure did you have in mind?

  (Paul has written most explicitly about what a world full of advanced sycophants looks like/​how it leads to existential catastrophe, and his stories are about outer alignment, so I’d be especially curious if you disagreed with that.)

• Sam ClarkeNov 3, 2021, 3:27 PM

  3 points

  0 ∶ 0

  in reply to: Rohin Shah’s comment on: Gen­eral vs spe­cific ar­gu­ments for the longter­mist im­por­tance of shap­ing AI development

  (Apologies for my very slow reply.)

  I feel like something has gone wrong in this conversation; you have tricked Bob into working on learning from human feedback, rather than convincing him to do so.

  I agree with this. If people become convinced to work on AI stuff by specific argument X, then they should definitely go and try to fix X, not something else (e.g. what other people tell them needs doing in AI safety/​governance).

  I think when I said I wanted a more general argument to be the “default”, I was meaning something very general, that doesn’t clearly imply any particular intervention—like the one in the most important century series, or the “AI is a big deal” argument (I especially like Max Daniel’s version of this).

  Then, it’s very important to think clearly about what will actually go wrong, and how to actually fix that. But I think it’s fine to do this once you’re already convinced that you should work on AI, by some general argument.

  I’d be really curious if you still disagree with this?

• Sam ClarkeOct 18, 2021, 1:07 PM

  10 points

  0 ∶ 0

  in reply to: Rohin Shah’s comment on: Gen­eral vs spe­cific ar­gu­ments for the longter­mist im­por­tance of shap­ing AI development

  The problem with general arguments is that they tell you very little about how to solve the problem

  Agreed!

  If I were producing key EA content/​fellowships/​etc, I would be primarily interested in getting people to solve the problem

  I think this is true for some kinds of content/​fellowships/​etc, but not all. For those targeted at people who aren’t already convinced that AI safety/​governance should be prioritised (which is probably the majority), it seems more important to present them with the strongest arguments for caring about AI safety/​governance in the first place. This suggests presenting more general arguments.

  Then, I agree that you want to get people to help solve the problem, which requires talking about specific failure modes. But I think that doing this prematurely can lead people to dismiss the case for shaping AI development for bad reasons.

  Another way of saying this: for AI-related EA content/​fellowships/​etc, it seems worth separating motivation (“why should I care?”) and action (“if I do care, what should I do?”). This would get you the best of both worlds: people are presented with the strongest arguments, allowing them to make an informed decision about how much AI stuff should be prioritised, and then also the chance to start to explore specific ways to solve the problem.

  I think this maybe applies to longtermism in general. We don’t yet have that many great ideas of what to do if longtermism is true, and I think that people sometimes (incorrectly) dismiss longtermism for this reason.

• Sam ClarkeOct 15, 2021, 3:34 PM

  3 points

  0 ∶ 0

  in reply to: David_Moss’s comment on: Les­sons learned run­ning the Sur­vey on AI ex­is­ten­tial risk scenarios

  Thanks for the detailed reply, all of this makes sense!

  I added a caveat to the final section mentioning your disagreements with some of the points in the “Other small lessons about survey design” section

Gen­eral vs spe­cific ar­gu­ments for the longter­mist im­por­tance of shap­ing AI development

Les­sons learned run­ning the Sur­vey on AI ex­is­ten­tial risk scenarios

• Sam ClarkeOct 6, 2021, 3:02 PM

  11 points

  0 ∶ 0

  on: We’re Red­wood Re­search, we do ap­plied al­ign­ment re­search, AMA

  What might be an example of a “much better weird, theory-motivated alignment research” project, as mentioned in your intro doc? (It might be hard to say at this point, but perhaps you could point to something in that direction?)

• Sam ClarkeOct 6, 2021, 3:00 PM

  8 points

  0 ∶ 0

  on: We’re Red­wood Re­search, we do ap­plied al­ign­ment re­search, AMA

  How crucial a role do you expect x-risk-motivated AI alignment will play in making things go well? What are the main factors you expect will influence this? (e.g. the occurrence of medium-scale alignment failures as warning shots)

• Sam ClarkeOct 6, 2021, 1:37 PM

  10 points

  0 ∶ 0

  on: We’re Red­wood Re­search, we do ap­plied al­ign­ment re­search, AMA

  Some questions that aren’t super related to Redwood/​applied ML AI safety, so feel free to ignore if not your priority:

  1. Assuming that it’s taking too long to solve the technical alignment problem, what might be some of our other best interventions to reduce x-risk from AI? E.g., regulation, institutions for fostering cooperation and coordination between AI labs, public pressure on AI labs/​other actors to slow deployment, …

  2. If we solve the technical alignment problem in time, what do you think are the other major sources of AI-related x-risk that remain? How likely do you think these are, compared to x-risk from not solving the technical alignment problem in time?

• Sam ClarkeOct 6, 2021, 1:28 PM

  1 point

  0 ∶ 0

  on: How to suc­ceed as an early-stage re­searcher: the “lean startup” approach

  The ‘lean startup’ approach reminds me of Jacob Steinhardt’s post about his approach to research, of which the key takeaways are:

  • When working on a research project, you should basically either be in “de-risking mode” (determining if the project is promising as quickly as possible) OR “execution mode” (assuming the project is promising and trying to do it quickly). This probably looks like trying to do an MVP version of the project quickly, and then iterating on that if it’s promising.

  • If a project doesn’t work out, ask why. That way you:

    • avoid trying similar things that will fail for the same reasons.

    • will find out whether it didn’t work because your implementation was broken, or the high-level approach you were taking isn’t promising.

  • Try hard, early, to try to show that your project won’t solve the problem.

• Sam ClarkeOct 6, 2021, 1:27 PM

  8 points

  0 ∶ 0

  on: How to suc­ceed as an early-stage re­searcher: the “lean startup” approach

  Write out at least 10 project ideas, and ask somebody more senior to rank the best few

  For bonus points, try to understand how they did the ranking. That way, you can start building up a model of how senior researchers think about evaluating project ideas, and refining your own research taste explicitly.

• Sam ClarkeOct 6, 2021, 1:26 PM

  4 points

  0 ∶ 0

  on: How to suc­ceed as an early-stage re­searcher: the “lean startup” approach

  Thanks for writing this, I found it helpful and really clearly written!

  One reaction: if you’re testing research as a career (rather than having committed and now aiming to maximise your chances of success), your goal isn’t exactly to succeed as an early stage researcher. It might be that trying your best to succeed is approximately the best way to test your fit—but it seems like there are a few differences:

  • “Going where there’s supervision” might be especially important, since a supervisor who comes to know you very well is a big and reliable source of information about your fit for research—which seems esp. important given that feedback in the form of “how much other people like your ideas” is often biased (e.g. because most of your early ideas are bad) or noisy (e.g. because some factors that influence the success of your research aren’t under your control).

  • It might be important to test your fit for different fields or flavours (e.g. quantitative vs qualitative, empircal vs theoretical) of research. This can come apart from the goal of trying to succeed as an early-stage researcher—since moving into unfamiliar territory might mean your outputs are less good in the short term.

  • Relatedly, it might be important to select at least some of your projects based on the skills or knowledge gaps they help you fill. Again, this goal might come apart from short term success (e.g. you pick a forecasting project to improve those skills, despite not expecting it to generate interesting findings)

  • Probably you want to spend less energy marketing your work, except to the extent that it’s helpful in getting more people to give you feedback on your fit for a research career.

  • [most uncertain] “Someone senior tells you what to work on” might actually not be the ideal solution to your problem 1. If the skills of research execution and research planning are importantly different, then you might fail to get enough info about your competence/​enjoyment/​fit for research planning skills (but I’m pretty uncertain if they are importantly different).

  I’d be curious how much you agree with any of these points :)

• Sam ClarkeOct 4, 2021, 3:54 PM

  8 points

  0 ∶ 0

  in reply to: MichaelA🔸’s comment on: SamClarke’s Shortform

  Incidentally, that 80k episode and some from Clearer Thinking are the exact examples I had in mind!

  Maybe one could promote specific podcast episodes of this type, see if people found them useful in that way, and if so then encourage those podcasts to have more such eps or a new such podcast to start?

  As a step towards this, and in case any other find it independently useful, here are the episodes of Clearer Thinking that I recall finding helpful for my mental health (along with the issues they helped with).

  • #11 Comfort Languages and Nuanced Thinking (for thinking through I what need, and what loved ones need, in difficult times)

  • #21 Antagonistic Learning and Civilization (had some useful thoughts about how education has taught me that breaking rules makes me bad, whereas in reality, breaking rules is just a cost to include in my calculation of what the best action is)

  • #22 Self-Improvement and Research Ethics (getting more traction on why my attempts at self-improvement often don’t work)

  • #25 Happiness and Hedonic Adaptation (hedonic adaptation seems like a very important concept for living a happier life, and this is the best discussion of it that I’ve heard)

  • #26 Past /​ Future Selves and Intrinsic Values (I recall something being useful about how I relate to past and future me)

  • #43 Online and IRL Relationships (getting relationships are a big part of my happiness and this had a very dense collection of insights about how to do relationships well—other dense insights have come from reading Nonviolent Communication and doing Circling with partners)

  • #54 Self-Improvement and Behavior Change (lots of stuff, most important was realising that many “negative” behaviour patterns are actually bringing you some benefit in a convoluted way, and until you identify find a substitute for that benefit, they’ll be very hard to change)

  • #60 Heaven and hell on earth (thinking about the value of “bad” mental states like anxiety and depression)

  • #65 Utopia on earth and morality without guilt (thinking through how I relate to my desire to do good, guilt vs bright desire; the handle of “clingy-ness” for a certain flavour of mental experiences)

  • #68 How to communicate better with the people in your life (getting more traction on why some social interactions leave me feeling disconnected/​isolated)