I think another promising intervention would be to persuade God to be a conditional annihilationist or support universal reconciliation with Christ. Abraham successfully negotiated conditions with God regarding the destruction of Sodom and Gomorrah with just a few sentences. Imagine what we could do with rigorous and prayerful BOTEC analyses! Even if there is a small chance of this succeeding, the impact could be incredible in expectation.
mic
Karma: 1,942
Enhancing biosecurity with language models: defining research directions
Are there any readings about how a long reflection could be realistically and concretely achieved?
Solving alignment isn’t enough for a flourishing future
Great post! I’ve written a paper along similar lines for the SERI Conference in April 2023 here, titled “AI Alignment Is Not Enough to Make the Future Go Well.” Here is the abstract:
AI alignment is commonly explained as aligning advanced AI systems with human values. Especially when combined with the idea that AI systems aim to optimize their world based on their goals, this has led to the belief that solving the problem of AI alignment will pave the way for an excellent future. However, this common definition of AI alignment is somewhat idealistic and misleading, as the majority of alignment research for cutting-edge systems is focused on aligning AI with task preferences (training AIs to solve user-provided tasks in a helpful manner), as well as reducing the risk that the AI would have the goal of causing catastrophe.
We can conceptualize three different targets of alignment: alignment to task preferences, human values, or idealized values.
Extrapolating from the deployment of advanced systems such as GPT-4 and from studying economic incentives, we can expect AIs aligned with task preferences to be the dominant form of aligned AIs by default.
Aligning AI to task preferences will not by itself solve major problems for the long-term future. Among other problems, these include moral progress, existential security, wild animal suffering, the well-being of digital minds, risks of catastrophic conflict, and optimizing for ideal values. Additional efforts are necessary to motivate society to have the capacity and will to solve these problems.
I don’t necessarily think of humans as maximizing economic consumption, but I argue that power-seeking entities (e.g., some corporations or hegemonic governments using AIs) will have predominant influence, and these will not have altruistic goals to optimize for impartial value, by default.
Congrats on launching GWWC Local Groups! Community building infrastructure can be hard to set up, so I appreciate the work here.
It would be bad to create significant public pressure for a pause through advocacy, because this would cause relevant actors (particularly AGI labs) to spend their effort on looking good to the public, rather than doing what is actually good.
I think I can reasonably model the safety teams at AGI labs as genuinely trying to do good. But I don’t know that the AGI labs as organizations are best modeled as trying to do good, rather than optimizing for objectives like outperforming competitors, attracting investment, and advancing exciting capabilities – subject to some safety-related concerns from leadership. That said, public pressure could manifest itself in a variety of ways, some of which might work toward more or less productive goals.
I agree that conditional pauses better than unconditional pauses, due to pragmatic factors. But I worry about AGI labs specification gaming their way through dangerous-capability evaluations, using brittle band-aid fixes that don’t meaningfully contribute to safety.
I think GiveWell shouldn’t be modeled as wanting to recommend organizations that save as many current lives as possible. I think a more accurate way to model them is “GiveWell recommends organizations that are [within the Overton Window]/[have very sound data to back impact estimates] that save as many current lives as possible.”
This is correct if you look at GiveWell’s criteria for evaluating donation opportunities. GiveWell’s highly publicized claim “We search for the charities that save or improve lives the most per dollar” is somewhat misleading given that they only consider organizations with RCT-style evidence backing their effectiveness.
SPAR seeks advisors and students for AI safety projects (Second Wave)
Ideas for improving epistemics in AI safety outreach
Upvoted. This is what longtermism is already doing (relying heavily on non-quantitative, non-objective evidence) and the approach can make sense for more standard local causes as well.
Supervised Program for Alignment Research (SPAR) at UC Berkeley: Spring 2023 summary
EU’s AI ambitions at risk as US pushes to water down international treaty (linkpost)
OpenAI introduces function calling for GPT-4
Navigating the Open-Source AI Landscape: Data, Funding, and Safety
What do you think are the main reasons behind wanting to deploy your own model instead of training an API? Some reasons I can think of:
cost savings
data privacy, not wanting usage to be tracked
interpretability research (this is EleutherAI’s justification for releasing an open-source large language model)
wanting to do things that are prohibited by the API
For anyone interested, the Center for AI Safety is offering up to $500,000 in prizes for benchmark ideas: SafeBench (mlsafety.org)
Where do you draw the line between AI startups that do vs don’t contribute excessively to capabilities externalities and existential risk? I think you’re right that your particular startup wouldn’t have a significant effect of accelerating timelines. But if we’re thinking AI startups in general, this could be another OpenAI or Adept, which probably have more of an effect on timelines.
I could imagine that even if one’s startup doesn’t working on scaling and making models generally smarter, a relatively small amount of applications work to make them more useful could put them at notably greater risk of having dangerous capabilities or intent. As an example, imagine if OpenAI only made GPT-3 and never produced InstructGPT or ChatGPT. It feels a lot harder to steer GPT-3 to do useful things, so I think that there would have been noticeably less adoption of LLMs and interest in advancing their capabilities, at least for a while. (For clarification, my claim isn’t that InstructGPT and ChatGPT necessarily contributed to existential risk, but they do have capabilities externalities and I think affected timelines, in part due to the hype they generated.)
Related: Risks of space colonization (Kovic, 2020).
Thanks for your comment, Jackson! I’ve removed my post since it seems that it was too confusing. One message that I meant to convey is that the imaginary nuclear company essentially does not have any safety commitments currently in effect (“we aren’t sure yet how to operate our plant safely”) and is willing to accept any number of deaths less than <10,000 people, despite adopting this “responsible nuclear policy.”