7 traps that (we think) new alignment researchers often fall into
We’ve noticed that new alignment researchers (and sometimes experienced alignment researchers) often fall into similar traps.
Here are 7 traps that we often see in new alignment researchers:
1. They think they need to be up-to-date on the literature before they can start contributing their own ideas.
Suggestion: Staying up-to-to-date on the literature is useful. But you don’t need to read everything before you contribute your own ideas. You can write naive hypotheses and try to generate new ideas. For many people, it’s easier to come up with (certain types of) new ideas before reading all of the literature. Once you’ve read the literature, it can be harder to ignore the ideas and frames of others. (see also Cached Thoughts)
2. They end up pursuing proxy goals.
A. They end up starting projects (often projects that take 3+ months) without a clear theory of change. They forget the terminal goal (e.g., reducing x-risk) and end up pursuing a proxy goal that looks like it’s helping (e.g., skill-up in ML). To be clear, many proxy goals (like skilling up in ML) are not inherently bad. But they don’t have a sense of why they’re learning ML, which subproblems they’re hoping to solve with ML, and which specific subskills in ML (and other fields) might be helpful. They don’t have a sense of how long they should spend skilling up, what else they should be learning, or what evidence they should be looking for to tell them to stop (or keep going).
B. They lose sight of the terminal goal. The real goal is not to skill-up in ML. The real goal is not to replicate the results of a paper. The real goal is not even to “solve inner alignment.” The real goal is to not die & not lose the value of the far-future.
Suggestion: Keep the terminal goal in mind. Try to have a clear idea of how your actions are getting you closer to the terminal goal. Sometimes, you won’t have clear answers (and indeed if you rely on having a clear end-to-end impact story before doing anything, you may fall into the trap of doing nothing). But notice when you’re doing things that don’t have a clear path toward the terminal goal. Occasionally ask yourself if there are projects that seem useful but don’t actually matter. Be cautious about spending many months on projects unless they have a justifiable theory of change.
3. They assume that if they don’t understand something, it’s because they are dumb (as opposed to thinking that the writer explained it unclearly, that the writer doesn’t understand it, or that the claim is wrong).
Suggestion: If you don’t understand something, have some probability on “This thing actually makes sense and I simply don’t understand it.” But do not discard hypotheses like “this thing is poorly written”, “this thing is confusing”, “the author doesn’t even understand this thing fully yet”, or “this thing is wrong.” Ask people you respect if they understand the thing. Ask them to explain it to you. Ask them to explain any jargon they use. Ask them to explain it as if they were talking to an intelligent high school student. If there are ideas that consistently fail the “high school student” check, stay open to the possibility that the idea hasn’t been properly understood, explained, or justified. If you like writing, you can also post comments on LessWrong, but I find talking to usually be ~10x better because it is so much higher bandwidth.
4. They rarely challenge the ideas and frames of authority figures.
For example, people hear that there is “inner alignment and outer alignment” or they hear that “inner alignment is the most important problem”. And then they start trying to solve inner alignment and outer alignment. And they don’t realize that this is a model. This is not the truth. This is a model.
Suggestion: Remember that the things you read are claims and the frames you hear are models. Some of these frames are unhelpful. Some of these frames are helpful but suboptimal. Notice when you are relying on the same frames as others.
5. They don’t distinguish between “intuitions” and “hypotheses”.
If Alice and Bob disagree (e.g., about whether or not evolution analogies are useful), it’s easy for them to say “ah, we just have different intuitions about the problem” and then they move on. This is unacceptable in other scientific fields and often serves as a curiosity-stopper.
Suggestion: Intuitions should be processed through reasoning and logic to figure out if intuitions are fallacies or hypotheses. If you realize that you have “intuitions” about a topic, take that as an opportunity to examine these intuitions more clearly. Where do they come from? Are there any hypotheses you can make based on them?
6. They end up working on a specific research agenda given by a senior researcher (e.g., Circuits or Infra-Bayesianism), without understanding why this is useful for solving alignment as a whole.
Ending up in this situation is a) not helpful for building up your inside views & b) makes it harder to research, because you don’t understand the constraints on your solution.
Suggestion: Try to solve the whole alignment problem. In doing this, think about 1) the key barriers that all of your proposed solutions are running into, and 2) the tools that you have. These solutions (probably) won’t be good, but they are super useful for building inside views. A useful exercise is to build an ‘alignment game tree’, where you (and maybe a few friends) propose solutions to alignment, then break those solutions, then create patches, iteratively.
7. They spend too much time on the fundamental math and CS behind alignment (e.g., trying to complete all of MIRI’s course recommendations or John Wentworth’s study guide) or getting degrees in Math/CS.
These normally take multiple years, and yet I claim that you can get near the frontier of alignment knowledge in ~6 months to a year.
Suggestion: Definitely learn linear algebra, multivariable calculus (the differentiation part, integration doesn’t come up often), probability theory, and basic ML very well. Past that, I recommend learning things as they come up in the course of working on alignment.
While we believe these traps are common, and we want more researchers looking out for them, we also encourage you to consider the law of equal and opposite advice. For each of these traps, it is possible to fall too far in the opposite direction (e.g., never spending time learning relevant math/CS concepts).
How do you think people should do this?
Akash—very nice post, and helpful for (relative) AI alignment newbies like me.
I would add that many AI alignment experts (and many EAs, actually) seem to assume that everyone getting interested in alignment is in their early 20s and doesn’t know much about anything, with no expertise in any other domain.
This might often be true, but there are also some people who get interested in alignment who have already had successful careers in other fields, and who can bring new interdisciplinary perspectives that alignment research might lack. Such people might be a lot less likely to fall into traps 3, 4, 5, and 6 that you mention. But they might fall into other kinds of traps that you don’t mention, such as thinking ‘If only these alignment kids understood my pet field X as well as I do, most of their misconceptions would evaporate and alignment research would progress 5x faster....’ (I’ve been guilty of this on occasion).
In my perspective, new and useful innovations in the past, especially in new fields, came from people with a wide and deep education and skillset that takes years to learn; and from fragmented research where no-one is necessarily thinking of a very high level terminal goal.
How sure are you that advice like “don’t pursue proxy goals” or “don’t spend years getting a degree” are useful for generating a productive field of AI alignment research, and not just for generating people who are vaguely similar to existing researchers who are thought of as successful? Or who can engage with existing research but will struggle with stepping outside its box?
After all:
Many existing researchers who have made interesting and important contributions do have PhDs,
And it doesn’t seem like we’re anywhere close to “solving alignment”, so we don’t actually know that being able to engage with their research without a much broader understanding is really that useful.
I like this, it’s a higher resolution description of what I think of as “not staying stuck in other people’s boxes”, or at least part of it. I plan to try avoiding these in the next few months, and live blog about it, so that people can correct mistakes I make early on, as opposed to me keeping doing wrong things for long. Any chance you’d like to follow?
Cool, thanks. Sorry for sounding a bit hostile, I’m just really freaked out by my strongly held inside view that we have less than 10 years until some really critical tipping point stuff happens. I’m trying to be reasonable and rational about this, but sometimes I react emotionally to comments that seem to be arguing for a ‘things will stay status quo for a good while, don’t worry about the short term ’ view.
I’d argue that if they solved inner alignment totally, then the rest of the alignment problems becomes far easier if not trivial to solve.
Calling my strongly held inside view ‘fringe’ doesn’t carry much weight as an argument for me. Do you have actual evidence of your longer than 10 years timelines view?
I hold the view that important scientific advancements tend to come disproportionately from the very smartest and most thoughtful people. My hope would be that students smart enough to be meaningfully helpful on the AGI alignment problem would be able to think through and form correct inside views on this.
If we’ve got maybe 2-3 years left before AGI, then 2 years before starting is indeed a large percentage of that remaining time. Even if we have more like 5-10… maybe better to just starting trying to work directly on the problem as best you can than let yourself get distracted by acquiring general background knowledge.