Formerly a software engineer at Google, now I’m doing independent AI alignment research.
Because of my focus on AI alignment, I tend to post more on LessWrong and AI Alignment Forum than I do here.
I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message!
You could avoid such infohazards by drawing up the scenarios in a private message or private doc that’s only shared with select people.
I love the idea of reading silently together, but I don’t think it wouldn’t have occurred to me without this post.
In particular, the analogy with alchemy seems apropos given that concepts like sentience are very ill posed.
I took another look at that section, interesting to learn more about the alchemists.
I think most AI alignment researchers consider ‘sentience’ to be unimportant for questions of AI existential risk—it doesn’t turn out to matter whether or not an AI is conscious or has qualia or anything like that. [1] What matters a lot more is whether AI can model the world and gain advanced capabilities, and AI systems today are making pretty quick progress along both these dimensions.
What would you say are good places to get up to speed on what we’ve learned about AI risk and the alignment problem in the past 8 years?
My favorite overview of the general topic is the AGI Safety Fundamentals course from EA Cambridge. I found taking the actual course to be very worthwhile, but they also make the curriculum freely available online. Weeks 1-3 are mostly about AGI risk and link to a lot of great readings on the topic. The weeks after that are mostly about looking at different approaches to solving AI alignment.
As for what has changed specifically in the last 8 years. I probably can’t do the topic justice, but a couple things that jump out at me:
The “inner alignment” problem has been identified and articulated. Most of the problems from Bostrom’s Superintelligence (2014) fall under the category of what we now call “outer alignment”, as the inner alignment problem wasn’t really known at that time. Outer alignment isn’t solved yet, but substantial work has been done on it. Inner alignment, on the other hand, is something many researchers consider to be more difficult.
Links on inner alignment: Canonical post on inner alignment, Article explainer, Video explainer
AI has advanced more rapidly than many people anticipated. People used to point to many things that ML models and other computer programs couldn’t do yet as evidence that we were a long way from having anything resembling AI. But AI has now passed many of those milestones.
Here I’ll list out some of those previously unsolved problems along with AI advances since 2015 that have solved them: Beating humans at Go (AlphaGo), beating humans at StarCraft (AlphaStar), biological protein folding (AlphaFold), having advanced linguistic/conversational abilities (GPT-3, PaLM), generalizing knowledge to competence in new tasks (XLand), artistic creation (DALL·E 2), multi-modal capabilities like combined language + vision + robotics (SayCan, Socratic Models, Gato).
Because of these rapid advances, many people have updated their estimates of when transformative AI will arrive to many years sooner than they previously thought. This cuts down on the time we have to solve the alignment problem.
--
[1]: It matters a lot whether the AI is sentient for moral questions around how we should treat advanced AI. But those are separate questions from AI x-risk.
That’s a very good point.
With the assumption of longtermist ethics which I mentioned in the post, I think the difference in likelihoods has to be very large to make a difference though. Because placing equal value on future human lives to present ones makes extinction risks astronomically worse than catastrophic non-extinction risks.
(I don’t 100% subscribe to longtermist ethics, but that was the frame I was taking for this post.)
You may have better luck getting responses to this posting on LessWrong with the ‘AI’ and ‘AI Governance’ (https://www.lesswrong.com/tag/ai-governance) tags, and/or on the AI Alignment Slack.
I skimmed the article. IMO it looks like a piece from circa 2015 dismissive of AI risk concerns. I don’t have time right now to go through each argument, but it looks pretty easily refutable esp. with all that we’ve continued to learn about AI risk and the alignment problem in the past 8 years.
Was there a particular part from that link you found particularly compelling?
Agreed. The trend of writing “Epistemic status” as one of the first things in a post without a definition or explanation (kudos to Lizka for including one) has bothered me for some time. It immediately and unnecessarily alienates readers by making them feel like they need to be familiar with the esoteric word “epistemic”, which usually has nothing to do with the rest of the post.
Would be happy to see this frequent jargon replaced with something like “How much you should trust me”, “Author confidence” or “Post status” (maybe there’s a better phrase, just some examples that come to mind).
Welcome to the field! Wow, I can imagine this post would be an intense crash course! :-o
There are some people who spend time on these questions. It’s not something I’ve spent a ton of time on, but I think you’ll find interesting posts related to this on LessWrong and AI Alignment Forum, e.g. using the value learning tag. Posts discussing ‘ambitious value learning’ and ‘Coherent Extrapolated Volition’ should be pretty directly related to your two questions.
Great post, thank you.
Really interesting observations.
I would say the conversion rate is actually shockingly low. Maybe CEA has more information on this, but I would be surprised if more than 5% of people who do Introductory EA fellowships make a high impact career change.
Do you have any sense of how many of those people are earning to give or end up making donation to effective causes play a significant role in their lives? I wonder if 5% is at least a little pessimistic for the “retention” of effective altruists if it’s not accounting for people who take this path to making an impact.
My first 2 posts for this project went live on the Alignment Forum today:
1. Introduction to the sequence: Interpretability Research for the Most Important Century
2. (main post) Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios
I learned a lot from reading this post and some of the top comments, thanks for the useful analysis.
Throughout the post and comments people are tending to classify AI safety as a “longtermist” cause. This isn’t wrong, but for anyone less familiar with the topic, I just want to point out that there are many of us who work in the field and consider AI to be a near-to-medium term existential risk.
Just in case “longtermism” gave anyone the wrong impression that AI x-risk is something we definitely won’t be confronted with for 100+ years. Many of us think it will be much sooner than that (though there is still considerable uncertainty and disagreement about timelines).
See the related post “Long-Termism” vs. “Existential Risk” by Scott Alexander.
You’re right, that paragraph was confusing. I just edited it to try and make it more clear.
These are thoughtful data points, but consider that they may just be good evidence for hard takeoff rather than soft takeoff.
What I mean is that most of these examples show a failure of narrow AIs to deliver on some economic goals. In soft takeoff, we expect to see things like broad deployment of AIs contributing to massive economic gains and GDP doublings in short periods of time well before we get to anything like AGI.
But in hard takeoff, failure to see massive success from narrow AIs could happen due to regulations and other barriers (or it could just be limitations of the narrow AI). In fact, these limitations could even point more forcefully to the massive benefits of an AI that can generalize. And having the recipe for that AGI discovered and deployed in a lab doesn’t depend on the success of prior narrow AIs in the regulated marketplace. AGI is a different breed and may also become powerful enough that it doesn’t have to play by the rules of the regulated marketplace and national legal systems.
Machines will need to learn in open-ended play with the world, where today they mostly learn from labeled examples.
Have you seen DeepMind’s Generally capable agents emerge from open-ended play? I think it is a powerful demonstration of learning from open-ended play actually working in a lab (not just a possible future approach). Though it is still in a virtual environment rather than the real physical world.
I think very, very few people really believe that superintelligence systems will be that influential.
A lot of prominent scientists, technologists and intellectuals outside of EA have warned about advanced artificial intelligence too. Stephen Hawking, Elon Musk, Bill Gates, Sam Harris, everyone on this open letter back in 2015 etc.
I agree that the number of people really concerned about this is strikingly small given the emphasis longtermist EAs put on it. But I think these many counter-examples warn us that it’s not just EAs and the AGI labs being overconfident or out of left field.
What relatively well-scoped research activities are particularly likely to be useful for longtermism-oriented AI alignment?
[...]
(3) Activity that is likely to be relevant for the hardest and most important parts of the problem, while also being the sort of thing that researchers can get up to speed on and contribute to relatively straightforwardly (without having to take on an unusual worldview, match other researchers’ unarticulated intuitions to too great a degree, etc.)
I’m planning to spend some time working on this question, or rather part of it. In particular I’m going to explore the argument that interpretability research falls into this category, with some attention to which specific aspects or angles of interpretability research seem most useful.
Since I don’t plan to spend much time thoroughly examining other research directions besides interpretability, I don’t expect to have a complete comparative answer to the question. But by answering the question for interpretability, I hope to at least put together a fairly comprehensive argument for (or perhaps against, we’ll see after I look at the evidence!) interpretability research that could be used by those considering it as a target for their funding or their time. I also hope that then someone trying to answer the larger question could use my work on interpretability as part of a comparative analysis across different research activities.
If someone is already working on this particular question and I’m duplicating effort, please let me know and perhaps we can sync up. Otherwise, I hope to have something to show on this question in a few/several weeks!
Why is this being severely downvoted? Is it because it’s a scam or something, or because Red Cross is just not considered an effective charity?
Fourthly, I am kinda worried about health effects, especially on short-to-medium-term energy levels.
I’ve been mostly (~98%) vegan since 2013. This concern really surprised me, because the health benefits of eating plant based are clearly extraordinary in my view. It was actually the primary driver for me going from vegetarian to vegan, with concern for the farmed animals growing later.
I would say that the very short term effects (1-2 weeks) can be disruptive as the bacteria in your gut become re-selected from bacteria that feed on meat or dairy to ones that feed on fibrous foods—veggies, fruit, legumes, nuts etc. But the 2-weeks-to-rest-of-life health benefits are massive. Ranging from higher energy levels to prevention of many kinds of diseases, greatly lower risk of heart disease, lower cholesterol levels, lower blood pressure, easier to lose/maintain weight… the list goes on. It does require actually eating more cruciferous and other veg, berries and other fruits, legumes, spices to etc. though and not just being a junk food vegan who eats cookies and crackers all the time. And you do need to supplement with vitamin B12 because this essential nutrient gets stripped from the fruits and veg when we wash them (and it is a good idea to still wash them).
Highly recommend exploring Dr. Michael Greger’s work (https://nutritionfacts.org/) if anyone is curious about health questions around plant-based/vegan diet. He’s an evidence-based/altruistic M.D. nutritionist and does great work separating out the quality nutrition research from ones with methodological errors, bias from funding sources etc. (I am in no way affiliated and his work as I understand it is not for profit.)
That is high-value work. Holden Karnofsky’s list of “important, actionable research questions” about AI alignment and strategy includes one about figuring out what should be done in deployment of advanced AI and leading up to it (1):