What do you think are the main reasons behind wanting to deploy your own model instead of training an API? Some reasons I can think of:
cost savings
data privacy, not wanting usage to be tracked
interpretability research (this is EleutherAI’s justification for releasing an open-source large language model)
wanting to do things that are prohibited by the API
For anyone interested, the Center for AI Safety is offering up to $500,000 in prizes for benchmark ideas: SafeBench (mlsafety.org)
Where do you draw the line between AI startups that do vs don’t contribute excessively to capabilities externalities and existential risk? I think you’re right that your particular startup wouldn’t have a significant effect of accelerating timelines. But if we’re thinking AI startups in general, this could be another OpenAI or Adept, which probably have more of an effect on timelines.
I could imagine that even if one’s startup doesn’t working on scaling and making models generally smarter, a relatively small amount of applications work to make them more useful could put them at notably greater risk of having dangerous capabilities or intent. As an example, imagine if OpenAI only made GPT-3 and never produced InstructGPT or ChatGPT. It feels a lot harder to steer GPT-3 to do useful things, so I think that there would have been noticeably less adoption of LLMs and interest in advancing their capabilities, at least for a while. (For clarification, my claim isn’t that InstructGPT and ChatGPT necessarily contributed to existential risk, but they do have capabilities externalities and I think affected timelines, in part due to the hype they generated.)
Related: Risks of space colonization (Kovic, 2020).
Just so I understand, are all four of these quotes arguing against preference utilitarianism?
I’m curious whether the reason why EA may be perceived as a cult while, e.g., environmentalist and social justice activism are not, is primarily that the concerns of EA are much less mainstream.
I appreciate the suggestions on how to make EA less cultish, and I think they are valuable to implement, but I don’t think they would have a significant effect on public perception of whether EA is a cult.
I think AI Risk Intro 1: Advanced AI Might Be Very Bad is great.
I agree, that seems concerning. Ultimately, since the AI developers are designing the AIs, I would guess that they would try to align the AI to be helpful to the users/consumers or to the concerns of the company/government, if they succeed at aligning the AI at all. As for your suggestions “Alignment with whoever bought the AI? Whoever users it most often? Whoever might be most positively or negatively affected by its behavior? Whoever the AI’s company’s legal team says would impose the highest litigation risk?” – these all seem plausible to me.
On the separate question of handling conflicting interests: there’s some work on this (e.g., “Aligning with Heterogeneous Preferences for Kidney Exchange” and “Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning”), though perhaps not as much as we would like.
But I sometimes have a fear in the back of my mind that some of the attendees who are intrigued by these ideas are later going to look up effective altruism, get the impression that the movement’s focus is just about existential risks these days, and feel duped. Since EA pitches don’t usually start with longtermist ideas, it can feel like a bait and switch.
To avoid the feeling of a bait and switch, I think one solution is to introduce existential risk in the initial pitch. For example, when introducing my student group Effective Altruism at Georgia Tech, I tend to say something like: “Effective Altruism at Georgia Tech is a student group which aims to empower students to pursue careers tackling the world’s most pressing problems, such as global poverty, animal welfare, or existential risk from climate change, future pandemics, or advanced AI.” It’s totally fine to mention existential risk – students still seem pretty interested and happy to sign up for our mailing list.
I think AI alignment isn’t really about designing AI to maximize for the preference satisfaction of a certain set of humans. I think an aligned AI would look more like an AI which:
is not trying to cause an existential catastrophe or take control of humanity
has had undesirable behavior trained out or adversarially filtered
learns from human feedback about what behavior is more or less preferable
In this case, we would hope the AI would be aligned to the people who are allowed to provide feedback
has goals which are corrigible
is honest, non-deceptive, and non-power-seeking
Thanks for writing this! There’s been a lot of interest of EA community building, but I think the one of the most valuable parts of EA community building is basically just recruiting – e.g., notifying interested people about relevant opportunities and inspiring people to apply for impactful opportunities. A lot of potential talent isn’t looped in with a local EA group or the EA community at all, however, so I think more professional recruiting could help a lot with solving organizational bottlenecks.
I was excited to read this post! At EA at Georgia Tech, some of our members are studying industrial engineering or operations research. Should we encourage them to reach out to you if they’re interested in getting involved with operations research for top causes?
What are some common answers you hear for Question #4: “What are the qualities you look for in promising AI safety researchers? (beyond general intelligence)”
Technical note: I think we need to be careful to note the difference in meaning between extinction and existential catastrophe. When Joseph Carlsmith talks about existential catastrophe, he doesn’t necessarily mean all humans dying; in this report, he’s mainly concerned about the disempowerment of humanity. Following Toby Ord in The Precipice, Carlsmith defines an existential catastrophe as “an event that drastically reduces the value of the trajectories along which human civilization could realistically develop”. It’s not straightforward to translate his estimates of existential risk to estimates of extinction risk.
Of course, you don’t need to rely on Joseph Carlsmith’s report to believe that there’s a ≥7.9% chance of human extinction conditioning on AGI.
Upvoted. This is what longtermism is already doing (relying heavily on non-quantitative, non-objective evidence) and the approach can make sense for more standard local causes as well.