aog

Karma: 3,356

aog Oct 17, 2022, 8:28 PM
6 points
0 ∶ 0
on: AI Safety Ideas: An Open AI Safety Research Platform
The application form is showing up as private for me. Very cool idea though, the success of Eleuther and Stability suggests that this is a viable model. Excited to see it unfold and hopefully contribute!

aog Oct 17, 2022, 4:25 PM
4 points
1 ∶ 2
in reply to: Ofer’s comment on: The US expands restrictions on AI exports to China. What are the x-risk effects?
Fair point, the answer is unclear and could change. The most important fact IMO is that two of the leading AGI companies in the US, OpenAI and Deepmind, are explicitly concerned with x-risk and have invested seriously in safety. (Not as much as I’d like, but significant investments.) I’d rather those companies reach AGI than others who don’t care about safety. They’re US-based and benefit relative to Chinese companies from US policy that slows China.

Second, while I don’t think Joe Biden thinks or cares about AI x-risk, I do think US policymakers are more likely to be convinced of the importance of AI x-risk. Most of the people arguing for AI risk are English speaking, and I think they’re gaining some traction. Some evidence:

The Catastrophic Risk Management Act introduced by Senators Portman and Peters is clearly longtermist in motivation. From the act: “Not later than 1 year after the date of enactment of this Act, the President, with support from the committee, shall conduct and submit to Congress a detailed assessment of global catastrophic and existential risk.” Several press releases explicitly mentioned risks from advanced AI, though not the alignment problem. This seems indicative of longtermism and EAs gaining traction in DC.

https://www.congress.gov/bill/117th-congress/senate-bill/4488/text

https://www.hsgac.senate.gov/media/minority-media/portman-peters-introduce-bipartisan-bill-to-ensure-federal-government-is-prepared-for-catastrophic-risks-to-national-security-

The National Security Commission on AI commissioned by Congress in 2018 did not include x-risk in their report, which is disappointing. That group, led by Eric Schmidt former CEO of Google, has continued their policy advocacy as the Special Competitive Studies Project. They are evidently aware of x-risk concerns, as they cited Holden Karnofsky’s writeup of the most important century hypothesis. Groups like these seem like they could be persuaded of the x-risk hypothesis, and could successfully advocate sensible policy to the US government.

https://www.scsp.ai/reports/mid-decade-challenges-for-national-competitiveness/preface/

Finally, there are think tanks who explicitly care about AI x-risk. My understanding is that CSET and CNAS are the two leaders, but the strong EA grantmaking system could easily spur more and more successful advocacy.

On the other hand, I’m unaware of a single major group in China that professes to care about x-risk from AI. I might not know if they did exist, so if there’s any evidence I’d love to hear it. China does seem to have much stronger regulatory skills, and would probably be better at implementing compute controls and other “pivotal acts”. But without a channel to communicate why they should do so, I’m skeptical that they will.

aog Oct 17, 2022, 1:50 AM
3 points
1 ∶ 0
in reply to: RobBensinger’s comment on: What does it mean for an AGI to be ‘safe’?
Thanks for the thoughtful response. My original comment was simply to note that some people disagree with the pivotal act framing, but it didn’t really offer an alternative and I’d like to engage with the problem more.

I think we have a few worldview differences that drive disagreement on how to limit AI risk given solutions to technical alignment challenges. Maybe you’d agree with me in some of these places, but a few candidates:
1. Stronger AI can protect us against weaker AI. When you imagine a world where anybody can train an AGI at home, you conclude that anybody will be able to destroy the world from home. I would expect that governments and corporations will maintain a sizable lead over individuals, meaning that individuals cannot take over the world. They wouldn’t necessarily need to preempt the creation of an AGI; they could simply contain it afterwards, by denying it access to resources and exposing its plans for world destruction. This is especially true in worlds where intelligence alone cannot take over the world, and instead requires resources or cooperation between entities, as argued in Section C of Katja Grace’s recent post. I could see somw of these proposals overlapping with your definition of a pivotal act, though I have more of a preference for multilateral and government action.
2. Government AI policy can be competent. Our nuclear non-proliferation regime is strong, only 8 countries have nuclear capabilities. Gain-of-function research is a strong counter example, but the Biden administration’s export controls on selling advanced semiconductors to China for national security purpose again support the idea of government competence. Strong government action seems possible with either (a) significant AI warning shots or (b) convincing mainstream ML and policy leaders of the danger of AI risk. When Critch suggested that governments build weapons to monitor and disable rogue AGI projects, Eliezer said it’s not realistic but would be incredible if accomplished. Those are the kinds of proposals I’d want to popularize early.
3. I have longer timelines, expect a more distributed takeoff, and have a more optimistic view of the chances of human survival than I’d expect you do. My plan for preventing AI x-risk is to solve the technical problems, and to convince influential people in ML and policy that the solutions must be implemented. They can then build aligned AI, and employ measures like compute controls and monitoring of large projects to ensure widespread implementation. If it turns out that my worldview is wrong and an AI lab invents a single AGI that could destroy the world relatively soon, I’d be much more open to dramatic pivotal acts that I’m not excited about in my mainline scenario.
Three more targeted replies to your comments:

Your proposed pivotal act in your reply to Critch seems much more reasonable to me than “burn all GPUs”. I’m still fuzzy on the details of how you would uncover all potential AGI projects before they get dangerous, and what you would do to stop them. Perhaps more crucially, I wouldn’t be confident that we’ll have AI that can run whole brain emulation of humans before we have AI that brings x-risk, because WBE would likely require experimental evidence from human brains that early advanced AI will not have.

I strongly agree with the need for more honest discussions about pivotal acts / how to make AI safe. I’m very concerned by the fact that people have opinions they wouldn’t share, even within the AI safety community. One benefit of more open discussion could be reduced stigma around the term — my negative association comes from the framing of a single dramatic action that forever ensures our safety, perhaps via coercion. “Burn all GPUs” exemplifies these failure modes, but I might be more open to alternatives.

I really like “don’t leave you fingerprints on the future.” If more dramatic pivotal acts are necessary, I’d endorse that mindset.

This was interesting to think about and I’d be curious to answer any other questions. In particular, I’m trying to think how to ensure ongoing safety in Ajeya’s HFDT world. The challenge is implementation, assuming somebody has solved deceptive alignment using e.g. interpretability, adversarial training, or training strategies that exploit inductive biases. Generally speaking, I think you’d have to convince the heads of Google, Facebook, and other organizations that can build AGI that these safety procedures are technically necessary. This is a tall order but not impossible. Once the leading groups are all building aligned AGIs, maybe you can promote ongoing safety either with normal policy (e.g. compute controls) or AI-assisted monitoring (your proposal or Critch’s EMPs). I’d like to think about this more but have to run.

aog Oct 16, 2022, 4:01 PM
4 points
1 ∶ 0
on: Is interest in alignment worth mentioning for grad school applications?
This might be helpful context: https://www.lesswrong.com/posts/SqjQFhn5KTarfW8v7/lessons-learned-from-talking-to-greater-than-100-academics

In general my sense is that most potential advisors would be put off discussion of AGI or x-risk. Talking about safety in more mainstream terms might get a better reception, for example Unsolved Problems in ML Safety and X-Risk Analysis for AI Research by Hendrycks et al both present AI safety concerns in a way that might have broader appeal. Another approach would be presenting specific technical challenges that you want to work on, such as ELK or interpretability or OOD robustness, which can interest people on technical grounds even if they don’t share your motivations.

I don’t mean to totally discourage x-risk discussion, I’m actually writing a thesis proposal right now and trying to figure out how to productively mention my real motivations. I think it’s tough but hopefully you can find a way to make it work.

aog Oct 15, 2022, 9:48 PM
6 points
0 ∶ 0
in reply to: Geoffrey Miller’s comment on: Alignment 201 curriculum
You might be well aware of this, but there is a great line of research on machine ethics that tries to build AI with a sophisticated understanding of human values. The ETHICS benchmark for example measures language model understanding of various moral theories: https://arxiv.org/abs/2008.02275

aog Oct 14, 2022, 7:42 PM
6 points
0 ∶ 2
in reply to: aog’s comment on: The US expands restrictions on AI exports to China. What are the x-risk effects?
Great podcast on it from Jordan Schnieder, the document itself, and the press release.

aog Oct 14, 2022, 7:38 PM
14 points
6 ∶ 3
on: The US expands restrictions on AI exports to China. What are the x-risk effects?
Great writeup! I also wrote about the restrictions here, with some good discussion. A few thoughts:
- I think this slows China’s AI progress by a few years. Losing Nvidia GPUs alone is a serious hit to ML researchers in China. They are building their own alternatives, for example CodeGeeX is a GPT-sized language model trained entirely on Chinese GPUs. But this makes GPUs more scarce.
- It probably also reduces US influence over China and Chinese AI in the future. We’re making them less reliant on us now, meaning we can’t use GPUs as leverage to force safety standards or other kinds of cooperation in the future.
  - I agree with your concern about cooperation on other existential risks. If we want to work together on climate change or banning research on dangerous pathogens, this hurts us.
- China more or less does not care about AI safety from existential risks. Therefore, slowing their timelines is good, but sacrificing US influence over China is bad. It’s unclear how these two balance out. If you have longer timelines, you’d probably prioritize long-term influence.
- I think this definitely increases the chances of war with China, because it’s explicitly designed to prepare for a possible war. It’s Tom Cotton’s strategy of economic decoupling to “Beat China”.
  - From the standard US foreign policy viewpoint, I think this shift is well-warranted. Cooperation with China on trade has not given us the soft power we hoped it would. They’re as anti-democratic as ever, still committing human rights abuses and arguably only growing more aggressive. It’s time to move from the carrot to the stick.
  - From an EA standpoint placing higher value on existential risk and lower value on typical US foreign policy interests, conflict with China definitely looks worse. The US stance looks like it would rather start World War III than allow China to become the top global superpower. I do believe that US democratic values are much better for the world than authoritarianism, and I’m scared of long-term authoritarianism, but solving it with global war doesn’t help.
  - This is analogous to the question of Ukraine: Do you support a democratic nation attacked by a despot at risk of nuclear war? Avoiding armageddon has to be the top priority, but over the years we’ve been able to pursue our other interests without spiraling into nuclear war.
  - Important operationalization: Do we defend Taiwan with US troops? Biden says yes. Taiwan is very important to defend (not least for TSMC and its semiconductors), but I think it’s probably better to lose Taiwan than raise the chances of nuclear war by 0.1%.

aog Oct 7, 2022, 7:24 PM
14 points
4 ∶ 3
on: What does it mean for an AGI to be ‘safe’?
I don’t think it’s a good plan to build an AI that enacts some pivotal act ensuring that nobody ever builds a misaligned AGI. See Critch here and here. When I think about building AI that is safe, I think about multiple layers of safety including monitoring, robustness, alignment, and deployment. Safety is not a single system that doesn’t destroy the world; it’s an ongoing process that prevents bad outcomes. See Hendrycks here and here.
What links here?
- Greg_Colbourn ⏸️ 's comment on What does it mean for an AGI to be ‘safe’? by So8res (Oct 10, 2022, 10:00 AM; 2 points)

aog Oct 3, 2022, 2:42 AM
32 points
14 ∶ 2
in reply to: Zach Stein-Perlman’s comment on: To those who contributed to Carrick Flynn in Oregon CD-6 - Please help now
I think it’s worth engaging with Carol, the Salinas campaign, and more generally people who have been adversely affected by EA efforts. If EA wants win elections in party politics, it will require working together with people who run those parties. Narrowly speaking, you might think that they’re not focused on the most important issues or that you have better policy ideas, and you might be right. But the ability to build coalitions, working together despite disagreements to accomplish common goals, is a central challenge of party politics.
I’m not convinced that EAs should donate to the Salinas campaign. FiveThirtyEight gives her a 78% chance of winning her race, meaning that closer races would offer a better chance for donations to tip the scales. Salinas also doesn’t list pandemic preparedness on her Issues page, which was the key issue of the Flynn campaign and I believe an important and neglected cause. But if the argument for the cost-effectiveness of donations to the Salinas campaign were to change, or if EAs found a more cost-effective way to offset possible harms of the Flynn campaign by continuing to engage with Oregonian or Democratic politics, I would consider supporting such an effort.
More simply, EAs should be kind and understanding in our discussions with Carol and others affected by our work. Maybe they’re interested in the EA mindset, but they’re unsure how to interpret our actions. We should show them good examples of how we think.
I believe we should think in terms of marginal effectiveness rather than offsetting particular harms we (individually or as a community) cause (see the author’s “you will have contributed in a small way to this failure” argument). If you want to offset harm that you have done or if you feel guilty, there’s little reason to do good in that particular domain (in this case, by donating to Salinas) rather than doing good in a more effective manner.
I think many people would disagree, and I expect that they’ll interpret your unwillingness to offset direct harms as a moral failure and an inability to cooperate with others. There are some domains that call for ruthless cost-effectiveness, and others that call for building relationships and trust with people with whom you might not always agree. I think politics is the latter.

aog Oct 1, 2022, 5:11 PM
5 points
3 ∶ 0
on: Google could build a conscious AI in three months
I would find this more persuasive if it had thorough references to the existing science of consciousness. I was under the impression that we still don’t know what the necessary conditions for consciousness are, and there are many competing theories on the topic. Stating that one theory is correct doesn’t answer that key question for me.

aog Oct 1, 2022, 2:44 AM
19 points
5 ∶ 0
on: Winners of the EA Criticism and Red Teaming Contest
The BioAnchors review by Jennifer Lin is incredible. Has it ever been shared as an independent post to LessWrong or EAF? I think many people might learn from it, and further discussion could be productive.

aog Sep 30, 2022, 5:21 AM
2 points
0 ∶ 0
in reply to: dEAsign’s comment on: Biosecurity Dual Use Filtering—Project Proposal (seeking vetting & project lead)
+1 to contacting Nuclear Threat Initiative, they seem to be active and well connected across many relevant areas.

aog Sep 28, 2022, 10:03 PM
2 points
0 ∶ 0
in reply to: aog’s comment on: How Open Source Machine Learning Software Shapes AI
New open source implementation of factored cognition / IDA techniques from Ought! https://www.lesswrong.com/posts/X5L9g4fXmhPdQrBCA/a-library-and-tutorial-for-factored-cognition-with-language

aog Sep 28, 2022, 6:56 PM
6 points
1 ∶ 0
on: How Open Source Machine Learning Software Shapes AI
Really cool topic, thanks for sharing. One of the ways that alignment techniques could gain adoption and reduce the alignment tax is by integrating them into popular open source libraries.
For example, the library TRL allows researchers to implement RLHF techniques which can benefit safety, but can also contribute to dangerous capabilities. On the other hand, I’m not aware of any open-source implementation of the techniques described in Red Teaming LMs with LMs, which could be used to filter or fine-tune the outputs of a generative language model.
Hopefully we’ll see more open source contributions of safety techniques, which could bring more interest to safety topics. Some might argue that implementing safety techniques in current models doesn’t reduce x-risk, and they’re probably right that current models aren’t directly posing x-risks, but early adoption of safety techniques seems useful for ensuring further adoption in the years to come.

aog Sep 28, 2022, 8:20 AM
5 points
0 ∶ 0
on: What should Future Fund regrantors fund?
I’m actively seeking funding for direct research and community building in technical AI safety. I‘ve been working at Lionel Levine’s lab on language model honesty research since the summer, and I recently founded USC AI Safety with more than 30 members with ML background in our first fellowship. Happy to provide much more detail upon request, send me a DM. Alternatively, if this isn’t an appropriate request I’ll take this down.

aog Sep 26, 2022, 9:01 PM
6 points
0 ∶ 0
on: Stress Externalities More in AI Safety Pitches
I like this framing a lot. My 60 second pitch for AI safety often includes something like this. “It’s all about making sure AI benefits humanity. We think AI could develop really quickly and shape our society, and the big corporations building it are thinking more about profits than about safety. We want to do the research they should be doing to make sure this technology helps everyone. It’s like working on online privacy in the 1990s and 2000s: Companies aren’t going to have the incentive to care, so you could make a lot of progress on a neglected problem by bringing early attention to the issue.”

aog Sep 23, 2022, 10:23 PM
6 points
3 ∶ 2
on: What are people’s thoughts on working for DeepMind as a general software engineer?
It’s tough to turn down an opportunity for career growth, but I would consider what kind of growth you’d get here. Building organizational tech isn’t directly related to research on AI safety, so it’s not a quick path to working on AI x-risk. I’m not sure that the more x-risk focused AI organizations are hiring for organizational tech, though perhaps you could get hired for a more general software position. A better opportunity for career growth might come from applying to LTFF or FTX regrantors to fund a Master’s in ML or independent reskilling.

That said, if you’re not certain about wanting to work on AI safety, there’s plenty of organizations in global poverty, alternative protein, public policy advocacy, and more than need organizational tech. While DeepMind does care about safety, I think their contribution to hastening the onset of AGI is ultimately very dangerous, and I would caution against supporting them in a general organizational capacity.

aog Sep 20, 2022, 6:28 PM
27 points
5 ∶ 6
on: What Do AI Safety Pitches Not Get About Your Field?
From an economics perspective, I think claims of double-digit GDP growth are dubious and undermine the credibility of the short AI timelines crowd. Here is a good summary of why it seems so implausible to me. To be clear, I think AI risk is a serious problem and I’m open to short timelines. But we shouldn’t be forecasting GDP growth, we should be forecasting the thing we actually care about: the possibility of catastrophic risk.
(This is a point of active disagreement where I’d expect e.g. some people at OpenPhil to believe double-digit GDP growth is plausible. So it’s more of a disagreement than a communication problem, but one that I think will particularly push away people with backgrounds in economics.)

aog Sep 17, 2022, 10:31 PM
2 points
0 ∶ 0
on: Open Problems in AI X-Risk [PAIS #5]
On the neglectedness of Cooperative AI, what do you think of the growing body of literature on multi-agent reinforcement learning? This paper notes that Google Scholar indexed nearly 10,000 papers on the topic in 2020 alone. Deepmind’s research agenda for cooperative AI has been widely cited, including by this review of the field that discusses four groups of recent research: analysis of emergent behaviors from existing RL algorithms in the multi-agent setting, learning to communicate, learning to cooperate, and agents modeling other agents.
Do you think this work productively contributes to x-risk motivated cooperative AI? What are the most helpful of these lines of research, and how could they better focus on x-risk safety?

aog Sep 16, 2022, 6:19 PM
5 points
0 ∶ 0
on: Katja Grace on Slowing Down AI, AI Expert Surveys And Estimating AI Risk
[...] We describe something the alignment problem and ask, “Do you think this is an important problem? Is it a hard problem? Is it a valuable problem to work on at the moment?” And I think, for all of those answers, the distribution shifted toward it being important and valuable and hard[...] They had five options for how important it is, say. For importance, the top evaluation of importance went from 5% to 20% of people who thought it was the most important. For the value of working on it today, the top category went from 8% to 27%. And for how hard it is, much harder than other things, went from 9% to 26%.

That’s really great news! Hopefully it’s not all talk and we get more mainstream ML research on safety over time.