Wei Dai
3c. Other research, especially “learning to reason from humans,” looks more promising than HRAD (75%?)
From the perspective of an observer who can only judge from what’s published online, I’m worried that Paul’s approach only looks more promising than MIRI’s because it’s less “mature”, having received less scrutiny and criticism from others. I’m not sure what’s happening internally in various research groups, but the amount of online discussion about Paul’s approach has to be at least an order of magnitude less than what MIRI’s approach has received.
(Looking at the thread cited by Rob Bensinger, various people including MIRI people have apparently looked into Paul’s approach but have not written down their criticisms. I’ve been trying to better understand Paul’s ideas myself and point out some difficulties that others may have overlooked, but this is hampered by the fact that Paul seems to be the only person who is working on the approach and can participate on the other side of the discussion.)
I think Paul’s approach is certainly one of the most promising approaches we currently have, and I wish people paid more attention to it (and/or wrote down their thoughts about it more), but it seems much too early to cite it as an example of an approach that is more promising than HRAD and therefore makes MIRI’s work less valuable.
the gap seems much smaller to me when it comes to the justification for thinking HRAD is promising vs justification for Paul’s approach being promising
This seems wrong to me. For example, in the “learning to reason from human” approaches, the goal isn’t just to learn to reason from humans, but to do it in a way that maintains competitiveness with unaligned AIs. Suppose a human overseer disapproves of their AI using some set of potentially dangerous techniques, how can we then ensure that the resulting AI is still competitive? Once someone points this out, proponents of the approach, to continue thinking their approach is promising, would need to give some details about how they intend to solve this problem. Subsequently, justification for thinking the approach is promising is more subtle and harder to understand. I think conversations like this have occurred for MIRI’s approach far more than Paul’s, which may be a large part of why you find Paul’s justifications easier to understand.
And as far as I can tell, the most promising approaches to this problem apply both to MIRI’s version and the mainstream ML version.
I’m not sure which approaches you’re referring to. Can you link to some details on this?
Capability amplification requires breaking cognitive work down into smaller steps. MIRI’s approach also requires such a breakdown. Capability amplification is easier in a simple formal sense (that if you solve the agent foundations you will definitely solve capability amplification, but not the other way around).
I don’t understand how this is true. I can see how solving FAI implies solving capability amplification (just emulate the FAI at a low level *), but if all you had was a solution that allows a specific kind of agent (e.g., with values well-defined apart from its implementation details) keep those values as it self-modifies, how does that help a group of short-lived humans who don’t know their own values break down an arbitrary cognitive task and perform it safely and as well as an arbitrary competitor?
(* Actually, even this isn’t really true. In MIRI’s approach, an FAI does not need to be competitive in performance with every AI design in every domain. I think the idea is to either convert mainstream AI research into using the same FAI design, or gain a decisive strategic advantage via superiority in some set of particularly important domains.)
My understanding is, MIRI’s approach is to figure out how to safely increase capability by designing a base agent that can make safe use of arbitrary amounts of computing power and can safely improve itself by modifying its own design/code. The capability amplification approach is to figure out how to safely increase capability by taking a short-lived human as the given base agent, making copies of it and and organize how the copies work together. These seem like very different problems with their own difficulties.
I think CEV has avoided those criticisms not because it solves the problem, but because it is sufficiently vague that it’s hard to criticize along these lines (and there are sufficiently many other problems that this one isn’t even at the top of the list).
I agree that in this area MIRI’s approach and yours face similar difficulties. People (including me) have criticized CEV for being vague and likely very difficult to define/implement though, so MIRI is not exactly getting a free pass by being vague. (I.e., I assume Daniel already took this into account.)
But I’m not sure there are fewer such problems than for the MIRI agenda, since I think that being closer to concreteness may more than outweigh the smaller amount of discussion.
This seems like a fair point, and I’m not sure how to weight these factors either. Given that discussion isn’t particularly costly relative to the potential benefits, an obvious solution is just to encourage more of it. Someone ought to hold a workshop to talk about your ideas, for example.
I think it would also be a good reason to focus on the difficulties that are common to both approaches
This makes sense.
Shouldn’t this cut both ways? Paul has also spent far fewer words justifying his approach to others, compared to MIRI.
The fact that Paul hasn’t had a chance to hear from many of his (would-be) critics and answer them means we don’t have a lot of information about how promising his approach is, hence my “too early to call it more promising than HRAD” conclusion.
I actually do have some objections to it, but I feel it is likely to be significantly useful even if (as I, obviously, expect) my objections end up having teeth.
Have you written down these objections somewhere? My worry is basically that different people looked at Paul’s approach and each thought of a different set of objections, and they think, “that’s not so bad”, without knowing that there’s actually a whole bunch of other objections out there, including additional ones that people would find if they thought and talked about Paul’s ideas more.
I can talk in more detail about the reduction from (capability amplification --> agent foundations) if it’s not clear whether it is possible and it would have an effect on your view.
Yeah, this is still not clear. Suppose we had a solution to agent foundations, I don’t see how that necessarily helps me figure out what to do as H in capability amplification. For example the agent foundations solution could say, use (some approximation of) exhaustive search in the following way, with your utility function as the objective function, but that doesn’t help me because I don’t have a utility function.
When comparing difficulty of two approaches you should presumably compare the difficulty of achieving a fixed goal with one approach or the other.
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
On the agent foundations side, it seems like plausible approaches involve figuring out how to peer inside the previously-opaque hypotheses, or understanding what characteristic of hypotheses can lead to catastrophic generalization failures and then excluding those from induction.
Oh I see. In my mind the problems with Solomonoff Induction means that it’s probably not the right way to define how induction should be done as an ideal, so we should look for something kind of like Solomonoff Induction but better, not try to patch it by doing additional things on top of it. (Like instead of trying to figure out exactly when CDT would make wrong decisions and add more complexity on top of it to handle those cases, replace it with UDT.)
That actually didn’t cross my mind before, so thanks for pointing it out. After reading your comment, I decided to look into Open Phil’s recent grants to MIRI and OpenAI, and noticed that of the 4 technical advisors Open Phil used for the MIRI grant investigation (Paul Christiano, Jacob Steinhardt, Christopher Olah, and Dario Amodei), all either have a ML background or currently advocate a ML-based approach to AI alignment. For the OpenAI grant however, Open Phil didn’t seem to have similarly engaged technical advisors who might be predisposed to be critical of the potential grantee (e.g., HRAD researchers), and in fact two of the Open Phil technical advisors are also employees of OpenAI (Paul Christiano and Dario Amodei). I have to say this doesn’t look very good for Open Phil in terms of making an effort to avoid potential blind spots and bias.
I can’t really think of anyone more familiar with MIRI’s work than Paul who isn’t already at MIRI (note that Paul started out pursuing MIRI’s approach and shifted in an ML direction over time).
The Agent Foundations Forum would have been a good place to look for more people familiar with MIRI’s work. Aside from Paul, I see Stuart Armstrong, Abram Demski, Vadim Kosoy, Tsvi Benson-Tilsen, Sam Eisenstat, Vladimir Slepnev, Janos Kramar, Alex Mennen, and many others. (Abram, Tsvi, and Sam have since joined MIRI, but weren’t employees of it at the time of the Open Phil grant.)
That being said, I agree that the public write-up on the OpenAI grant doesn’t reflect that well on OpenPhil, and it seems correct for people like you to demand better moving forward
I had previously seen some complaints about the way the OpenAI grant was made, but until your comment, hadn’t thought of a possible group blind spot due to a common ML perspective. If you have any further insights on this and related issues (like why you’re critical of deep learning but still think the grant to OpenAI was a pretty good idea, what are your objections to Paul’s AI alignment approach, how could Open Phil have done better), would you please write them down somewhere?
Daniel, while re-reading one of Paul’s posts from March 2016, I just noticed the following:
[ETA: By the end of 2016 this problem no longer seems like the most serious.] … [ETA: while robust learning remains a traditional AI challenge, it is not at all clear that it is possible. And meta-execution actually seems like the ingredient furthest from existing ML practice, as well as having non-obvious feasibility.]
My interpretation of this is that between March 2016 and the end of 2016, Paul updated the difficulty of his approach upwards. (I think given the context, he means that other problems, namely robust learning and meta-execution, are harder, not that informed oversight has become easier.) I wanted to point this out to make sure you updated on his update. Clearly Paul still thinks his approach is more promising than HRAD, but perhaps not by as much as before.
What lazy solutions will look like seems unpredictable to me. Suppose someone in the future wants to realistically roleplay a historical or fantasy character. The lazy solution might be to simulate a game world with conscious NPCs. The universe contains so much potential for computing power (which presumably can be turned into conscious experiences), that even if a very small fraction of people do this (or other things whose lazy solutions happen to involve suffering), that could create an astronomical amount of suffering.
What would you say are the philosophical or other premises that FRI does accept (or tends to assume in its work), which distinguishes it from other people/organizations working in a similar space such as MIRI, OpenAI, and QRI? Is it just something like “preventing suffering is the most important thing to work on (and the disjunction of assumptions that can lead to this conclusion)”?
It seems to me that a belief in anti-realism about consciousness explains a lot of Brian’s (near) certainty about his values and hence his focus on suffering. People who are not so sure about consciousness anti-realism tend to be less certain about their values as a result, and hence don’t focus on suffering as much. Does this seem right, and if so, can you explain what premises led you to work for FRI?
I’m a bit surprised to find that Brian Tomasik attributes his current views on consciousness to his conversations with Carl Shulman, since in my experience Carl is a very careful thinker and the case for accepting anti-realism as the answer to the problem of consciousness seems pretty weak, at least as explained by Brian. I’m very curious to read Carl’s own explanation of his views, if he has written one down. I scanned Carl Shulman’s list of writings but was unable to find anything that addressed this.
The one view that seems unusually prevalent within FRI, apart from people self-identifying with suffering-focused values, is a particular anti-realist perspective on morality and moral reasoning where valuing open-ended moral reflection is not always regarded as the by default “prudent” thing to do.
Thanks for pointing this out. I’ve noticed this myself in some of FRI’s writings, and I’d say this, along with the high amount of certainty on various object-level philosophical questions that presumably cause the disvaluing of reflection about them, are what most “turns me off” about FRI. I worry a lot about potential failures of goal preservation (i.e., value drift) too, but because I’m highly uncertain about just about every meta-ethical and normative question, I see no choice but to try to design some sort of reflection procedure that I can trust enough to hand off control to. In other words, I have nothing I’d want to “lock in” at this point and since I’m by default constantly handing off control to my future self with few safeguards against value drift, doing something better than that default is one of my highest priorities. If other people are also uncertain and place high value on (safe/correct) reflection as a result, that helps with my goal (because we can then pool resources together to work out what safe/correct reflection is), so it’s regrettable to see FRI people sometimes argue for more certainty than I think is warranted and especially to see them argue against reflection.
I’m also worried about the related danger of AI persuasion technology being “democratically” deployed upon open societies (i.e., by anyone with an agenda, not necessarily just governments and big corporations), with the possible effect that in the words of Paul Christiano, “we’ll live to see a world where it’s considered dicey for your browser to uncritically display sentences written by an untrusted party.” This is arguably already true today for those especially vulnerable to conspiracy theories, but eventually will affect more and more people as the technology improves. How will we solve our collective problems when the safety of discussions are degraded to such an extent?
This is a bit tangential, but do you know if anyone has done an assessment of the impact of HPMoR? Cousin_it (Vladimir Slepnev) recently wrote:
The question then becomes, how do we set up a status economy that will encourage research? Peer review is one way, because publications and citations are a status badge desired by many people. Participating in a forum like LW when it’s “hot” and frequented by high status folks is another way, but unfortunately we don’t have that anymore. From that perspective it’s easy to see why the massively popular HPMOR didn’t attract many new researchers to AI risk, but attracted people to HPMOR speculation and rational fic writing. People do follow their interests sometimes, but mostly they try to find venues to show off.
Taking this one step further, it seems to me that HPMoR may have done harm by directing people’s attentions (including Eliezer’s own) away from doing the hard work of making philosophical and practical progress in AI alignment and rationality, towards discussion/speculation of the book and rational fic writing, thereby contributing to the decline of LW. Of course it also helped bring new people into the rationalist/EA communities. What would be a fair assessment of its net impact?
Beyond Astronomical Waste
I find it unfortunate that people aren’t using a common scale for estimating AI risk, which makes it hard to integrate different people’s estimates, or even figure out who is relatively more optimistic or pessimistic. For example here’s you (Tobias):
My inside view puts ~90% probability on successful alignment (by which I mean narrow alignment as defined below). Factoring in the views of other thoughtful people, some of which think alignment is far less likely, that number comes down to ~80%.
Robert Wiblin, based on interviews with Nick Bostrom, an anonymous leading professor of computer science, Jaan Tallinn, Jan Leike, Miles Brundage, Nate Soares, Daniel Dewey:
We estimate that the risk of a serious catastrophe caused by machine intelligence within the next 100 years is between 1 and 10%.
I think there is a >1/3 chance that AI will be solidly superhuman within 20 subjective years, and that in those scenarios alignment destroys maybe 20% of the total value of the future
It seems to me that Robert’s estimate is low relative to your inside view and Paul’s, since you’re both talking about failures of narrow alignment (“intent alignment” in Paul’s current language), while Robert’s “serious catastrophe caused by machine intelligence” seems much broader. But you update towards much higher risk based on “other thoughtful people” which makes me think that either your “other thoughtful people” or Robert’s interviewees are not representative, or I’m confused about who is actually more optimistic or pessimistic. Either way it seems like there’s some very valuable work to be done in coming up with a standard measure of AI risk and clarifying people’s actual opinions.
- 's comment on AlphaGo Zero and capability amplification by (LessWrong; 11 Jan 2019 4:33 UTC; 7 points)
- 's comment on Rohin Shah on reasons for AI optimism by (LessWrong; 1 Nov 2019 19:20 UTC; 6 points)
- 's comment on AlphaGo Zero and capability amplification by (LessWrong; 11 Jan 2019 23:38 UTC; 3 points)
What do you think about technical interventions on these problems, and “moral uncertainty expansion” as a more cooperative alternative to “moral circle expansion”?
Good point, I’ll send a message to Robert Wiblin asking for clarification.
I’m a bit surprised by the 1-10% estimate. This seems very low, especially given that “serious catastrophe caused by machine intelligence” is broader than narrow alignment failure.
Yeah, it’s also much lower than my inside view, as well as what I thought a group of such interviewees would say. Aside from Lukas’s explanation, I think maybe 1) the interviewees did not want to appear too alarmist (either personally or for EA as a whole) or 2) they weren’t reporting their inside views but instead giving their estimates after updating towards others who have much lower risk estimates. Hopefully Robert Wiblin will see my email at some point and chime in with details of how the 1-10% figure was arrived at.
It seems easy to imagine scenarios where MIRI’s work is either irrelevant (e.g., mainstream AI research keeps going in a neuromorphic or heuristic trial-and-error direction and eventually “succeeds” that way) or actively harmful (e.g., publishes ideas that eventually help others to build UFAIs). I don’t know how to tell whether MIRI’s current strategy overall has positive expected impact. What’s your approach to this problem?