Max Nadeau

Karma: 203

Max Nadeau 8 Nov 2024 1:42 UTC
3 points
0 ∶ 2
in reply to: Austin’s comment on: Has your organisation lost funding due to the Good Ventures funding shift? Have you managed to replace it?
https://www.openphilanthropy.org/focus/global-aid-policy/

“Build right-of-center support for aid, such as Civita’s work to create and discuss development policy recommendations with conservative Norwegian lawmakers.”

Max Nadeau 27 Jun 2024 21:51 UTC
12 points
3 ∶ 0
on: Detecting Genetically Engineered Viruses With Metagenomic Sequencing
I love seeing posts from people making tangible progress towards preventing catastrophes—it’s very encouraging!
I know nothing about this area, so excuse me if my question doesn’t make sense or was addressed in your post. I’m curious what the returns are on spending more money on sequencing, e.g. running the machine more than one a week or running it on more samples. If we were spending $10M a year instead of $1.5M on sequencing, how much less than 0.2% of people would have to be infected before an alert was raised?
Some other questions:
- How should I feel about 0.2%? Where is 0.2% on the value spectrum from no alert system and an alert system that triggered on a single infection?
- How many people’s worth of wastewater can be tested with $1.5M of sequencing?
Thanks for the update; it was interesting even as a layperson.

Max Nadeau 12 Jan 2024 20:18 UTC
6 points
0 ∶ 0
on: I’m interviewing Vitalik Buterin about ‘my techno-optimism’, E/acc and D/acc. What should I ask him?
I’d love to hear his thoughts on defensive measures for “fuzzier” threats from advanced AI, e.g. manipulation, persuasion, “distortion of epistemics”, etc. Since it seems difficult to delineate when these sorts of harms are occuring (as opposed to benign forms of advertising/rhetoric/expression), it seems hard to construct defenses.
This is a related concept mechanisms for collective epistemics like prediction markets or community notes, which Vitalik praises here. But the harms from manipulation are broader, and could route through “superstimuli”, addictive platforms, etc. beyond just the spread of falsehoods. See manipulation section here for related thoughts.

Max Nadeau 24 Oct 2023 18:27 UTC
8 points
0 ∶ 0
on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Disclaimer: I joined OP two weeks ago in the Program Associate role on the Technical AI Safety team. I’m leaving some comments describing questions I wanted to know to assess whether I should take the job (which, obviously, I ended up doing).
What sorts of personal/career development does the PA role provide? What are the pros and cons of this path over e.g. technical research (which has relatively clear professional development in the form of published papers, academic degrees, high-status job titles that bring public credibility)?

Max Nadeau 24 Oct 2023 18:23 UTC
1 point
0 ∶ 0
on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Disclaimer: I joined OP two weeks ago in the Program Associate role on the Technical AI Safety team. I’m leaving some comments describing questions I wanted to know to assess whether I should take the job (which, obviously, I ended up doing).
How inclined are you/would the OP grantmaking strategy be towards technical research with theories of impact that aren’t “researcher discovers technique that makes the AI internally pursue human values” → “labs adopt this technique”. Some examples of other theories of change that technical research might have:
- Providing evidence for the dangerous capabilities of current/future models (should such capabilities emerge) that can more accurately inform countermeasures/policy/scaling decisions.
- Detecting/demonstrating emergent misalignment from normal training procedures. This evidence would also serve to more accurately inform countermeasures/policy/scaling decisions.
- Reducing the ease of malicious misuse of AIs by humans.
- Limiting the reach/capability of models instead of ensuring their alignment.

Max Nadeau 24 Oct 2023 17:43 UTC
1 point
0 ∶ 0
on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Disclaimer: I joined OP two weeks ago in the Program Associate role on the Technical AI Safety team. I’m leaving some comments describing questions I wanted to know to assess whether I should take the job (which, obviously, I ended up doing).
How much do the roles on the TAIS team involve engagement with technical topics? How do the depth and breadth of “keeping up with” AI safety research compare to being an AI safety researcher?

Max Nadeau 24 Oct 2023 1:37 UTC
8 points
1 ∶ 0
on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Disclaimer: I joined OP two weeks ago in the Program Associate role on the Technical AI Safety team. I’m leaving some comments describing questions I wanted to know to assess whether I should take the job (which, obviously, I ended up doing).
What does OP’s TAIS funding go to? Don’t professors’ salaries already get paid by their universities? Can (or can’t) PhD students in AI get no-strings-attached funding (at least, can PhD students at prestigious universities)?

Max Nadeau 24 Oct 2023 1:36 UTC
7 points
0 ∶ 0
on: AMA: Six Open Philanthropy staffers discuss OP’s new GCR hiring round
Disclaimer: I joined OP two weeks ago in the Program Associate role on the Technical AI Safety team. I’m leaving some comments describing questions I wanted to know to assess whether I should take the job (which, obviously, I ended up doing).
Is it way easier for researchers to do AI safety research within AI scaling labs (due to: more capable/diverse AI models, easier access to them (i.e. no rate limits/usage caps), better infra for running experiments, maybe some network effects from the other researchers at those labs, not having to deal with all the logistical hassle that comes from being a professor/independent researcher)?
Does this imply that the research ecosystem OP is funding (which is ~all external to these labs) isn’t that important/cutting-edge for AI safety?

Max Nadeau 13 Sep 2023 21:59 UTC
45 points
7 ∶ 0
on: Who should we interview for The 80,000 Hours Podcast?
Sampled from my areas of personal interest, and not intended to be at all thorough or comprehensive:

AI researchers (in no particular order):
- Prof. Jacob Steinhardt: author of multiple fascinating pieces on forecasting AI progress and contributor/research lead on numerous AI safety-relevant papers.
- Dan Hendrycks: director of the multi-faceted and hard-to-summarize research and field-building non-profit CAIS.
- Prof. Sam Bowman: has worked on many varieties of AI safety research at Anthropic and NYU
- Ethan Perez: researcher doing fascinating work to display and address misalignments in today’s AIs.
- Toby Shevlane: Model Evaluations for Extreme Risks
- Jess Whittlestone: head of AI policy at Center for Long-Term Resilience, much research here
- Plenty of others: Jade Leung (AI governance and evaluations at OpenAI), Prof. David Krueger (varied AI safety research), Prof. Percy Liang (evaluating models), Prof. Roger Grosse (influence functions for interpretability), many others listed here.
Economists who have written (esp. but not only deflationary arguments contra Davidson) on AI’s economic impact:
- Chad Jones (see here)
- Ben Jones (see e.g. this, but also all his research)
- Matt Clancy (see this debate, though an episode with him should also address his non-AI work as well!)
- Daron Acemoglu (see Power and Progress)
- Maybe other reviewers here?
Ethicists:
- Iason Gabriel: has worked both on critiques of effective altruism, AI evaluations (extreme risks, representation), and normative questions related to AI alignment. This excellent FLI interview had so many ideas that would be great to explore in more depth.
- David Thorstad: has written critiques of existential risk reduction and longtermism.
- Emma Curran: author of contractualist reply to longtermism
The three I would personally be most excited to listen to: Toby Shevlane, Matt Clancy, Iason Gabriel.

Max Nadeau 16 May 2023 20:49 UTC
5 points
4 ∶ 0
in reply to: Habiba Banu’s comment on: Habiba’s Shortform
Best of luck with your new gig; excited to hear about it! Also, I really appreciate the honesty and specificity in this post.

Max Nadeau 4 Nov 2022 2:10 UTC
1 point
0 ∶ 0
in reply to: Eli Rose’s comment on: Apply to the Redwood Research Mechanistic Interpretability Experiment (REMIX), a research program in Berkeley
From the post: “We plan to have some researchers arrive early, with some people starting as soon as possible. The majority of researchers will likely participate during the months of December and/or January.”

Max Nadeau 28 Oct 2022 17:18 UTC
13 points
0 ∶ 0
on: What should I ask Ajeya Cotra — senior researcher at Open Philanthropy, and expert on AI timelines and safety challenges?
Artir Kel (aka José Luis Ricón Fernández de la Puente) at Nintil wrote an essay broadly sympathetic to AI risk scenarios but doubtful of a particular step in the power-seeking stories Cotra, Gwern, and others have told. In particular, he has a hard time believing that a scaled-up version of present systems (e.g. Gato) would learn facts about itself (e.g. that it is an AI in a training process, what its trainers motivations would be, etc) and incorporate those facts into its planning (Cotra calls this “situational awareness”). Some AI safety researchers I’ve spoken to personally agree with Kel’s skepticism on this point.
Since incorporating this sort of self-knowledge into one’s plans is necessary for breaking out of training, initiating deception, etc, this seems like a pretty important disagreement. In fact, Kel claims that if he came around on this point, he would agree almost entirely with Cotra’s analysis.
Can she describe in more detail what situational awareness means? Could it be demonstrated with current/nearterm models? Why does she think that Kel (and others) think it’s so unlikely?

Max Nadeau 27 Oct 2022 18:00 UTC
3 points
0 ∶ 0
in reply to: Dan Valentine’s comment on: Apply to the Redwood Research Mechanistic Interpretability Experiment (REMIX), a research program in Berkeley
No concrete plans one way or the other.

Apply to the Redwood Research Mechanistic Interpretability Experiment (REMIX), a research program in Berkeley

Max Nadeau27 Oct 2022 1:39 UTC

95 points

5 comments12 min readEA link

Max Nadeau 9 May 2022 2:32 UTC
2 points
0 ∶ 0
in reply to: SeanEngelhart’s comment on: Apply to the second ML for Alignment Bootcamp (MLAB 2) in Berkeley [Aug 15 - Fri Sept 2]
It is possible but unlikely that such a person would be a TA. Someone with little prior ML experience would be a better fit as a participant.

Max Nadeau 8 May 2022 0:27 UTC
2 points
0 ∶ 0
in reply to: SeanEngelhart’s comment on: Apply to the second ML for Alignment Bootcamp (MLAB 2) in Berkeley [Aug 15 - Fri Sept 2]
We intended that sentence to be read as: “In addition to people who plan on doing technical alignment, MLAB can be valuable to other sorts of people (e.g. theoretical researchers)”.

Apply to the second ML for Alignment Bootcamp (MLAB 2) in Berkeley [Aug 15 - Fri Sept 2]

Buck6 May 2022 0:19 UTC

111 points

7 comments6 min readEA link

Max Nadeau

Ap­ply to the Red­wood Re­search Mechanis­tic In­ter­pretabil­ity Ex­per­i­ment (REMIX), a re­search pro­gram in Berkeley

Ap­ply to the sec­ond ML for Align­ment Boot­camp (MLAB 2) in Berkeley [Aug 15 - Fri Sept 2]

Apply to the Redwood Research Mechanistic Interpretability Experiment (REMIX), a research program in Berkeley

Apply to the second ML for Alignment Bootcamp (MLAB 2) in Berkeley [Aug 15 - Fri Sept 2]