Under whatever constraints Open Phil provided, I’d have sent the ‘best by academic lights’ papers I had.
We originally sent Nick Beckstead what we considered our four most important 2015 results, at his request; these were (1) the incompatibility of the “Inductive Coherence” framework and the “Asymptotic Convergence in Online Learning with Unbounded Delays” framework; (2) the demonstration in “Proof-Producing Reflection for HOL” that a non-pathological form of self-referential reasoning is possible in a certain class of theorem-provers; (3) the reflective oracles result presented in “A Formal Solution to the Grain of Truth Problem,” “Reflective Variants of Solomonoff Induction and AIXI,” and “Reflective Oracles”; (4) and Vadim Kosoy’s “Optimal Predictors” work. The papers we listed under 1, 2, and 4 then got used in an external review process they probably weren’t very well-suited for.
I think this was more or less just an honest miscommunication. I told Nick in advance that I only assigned an 8% probability to external reviewers thinking the “Asymptotic Convergence…” result was “good” on its own (and only a 20% probability for “Inductive Coherence”). My impression of what happened is that Open Phil staff interpreted my pushback as saying that I thought the external reviews wouldn’t carry much Bayesian evidence (but that the internal reviews still would), where what I was trying to communicate was that I thought the papers didn’t carry very much Bayesian evidence about our technical output (and that I thought the internal reviewers would need to speak to us about technical specifics in order to understand why we thought they were important). Thus, we were surprised when their grant decision and write-up put significant weight on the internal reviews of those papers (and they were surprised that we were surprised). This is obviously really unfortunate, and another good sign that I should have committed more time and care to clearly communicating my thinking from the outset.
Regarding picking better papers for external review: We only put out 10 papers directly related to our technical agendas between Jan 2015 and Mar 2016, so the option space is pretty limited, especially given the multiple constraints Open Phil wanted to meet. Optimizing for technical impressiveness and non-obviousness as a stand-alone result, I might have instead gone with Critch’s bounded Löb paper and the grain of truth problem paper over the AC/IC results. We did submit the grain of truth problem paper to Open Phil, but they decided not to review it because it didn’t meet other criteria they were interested in.
If MIRI is unable to convince someone like Dewey, the prospects of it making the necessary collaborations or partnerships with the wider AI community look grim.
I’m less pessimistic about building collaborations and partnerships, in part because we’re already on pretty good terms with other folks in the community, and in part because I think we have different models of how technical ideas spread. Regardless, I expect that with more and better communication, we can (upon re-evaluation) raise the probability of Open Phil staff that the work we’re doing is important.
More generally, though, I expect this task to get easier over time as we get better at communicating about our research. There’s already a body of AI alignment research (and, perhaps, methodology) that requires the equivalent of multiple university courses to understand, but there aren’t curricula or textbooks for teaching it. If we can convince a small pool of researchers to care about the research problems we think are important, this will let us bootstrap to the point where we have more resources for communicating information that requires a lot of background and sustained scholarship, as well as more of the institutional signals that this stuff warrants a time investment.
I can maybe make the time expenditure thus far less mysterious if I mention a couple more ways I erred in trying to communicate my model of MIRI’s research agenda:
My early discussion with Daniel was framed around questions like “What specific failure mode do you expect to be exhibited by advanced AI systems iff their programmers don’t understand logical uncertainty?” I made the mistake of attempting to give straight/non-evasive answers to those sorts of questions and let the discussion focus on that evaluation criterion, rather than promptly saying “MIRI’s research directions mostly aren’t chosen to directly address a specific failure mode in a notional software system” and “I don’t think that’s a good heuristic for identifying research that’s likely to be relevant to long-run AI safety.”
I fell prey to the transparency illusion pretty hard, and that was completely my fault. Mid-way through the process, Daniel made a write-up of what he had gathered so far; this write-up revealed a large number of miscommunications and places where I thought I had transmitted a concept of mine but Daniel had come away with a very different concept. It’s clear in retrospect that we should have spent a lot more time with me having Daniel try to explain what he thought I meant, and I had all the tools to predict this in foresight; but I foolishly assumed that wouldn’t be necessary in this case.
(I plan to blog more about the details of these later.)
I think these are important mistakes that show I hadn’t sufficiently clarified several concepts in my own head, or spent enough time understanding Daniel’s position. My hope is that I can do a much better job of avoiding these sorts of failures in the next round of discussion, now that I have a better model of where Open Phil’s staff and advisors are coming from and what the review process looks like.
(I am correct in that Yuan previously worked for you, right?)
Yeah, though that was before my time. He did an unpaid internship with us in the summer of 2013, and we’ve occasionally contracted him to tutor MIRI staff. Qiaochu’s also a lot socially closer to MIRI; he attended three of our early research workshops.
Unless and until then, I remain sceptical about MIRI’s value.
I think that’s a reasonable stance to take, and that there are other possible reasonable stances here too. Some of the variables I expect EAs to vary on include “level of starting confidence in MIRI’s mathematical intuitions about complicated formal questions” and “general risk tolerance.” A relatively risk-intolerant donor is right to wait until we have clearer demonstrations of success; and a relatively risk-tolerant donor who starts without a very high confidence in MIRI’s intuitions about formal systems might be pushed under a donation threshold by learning that an important disagreement has opened up between us and Daniel Dewey (or between us and other people at Open Phil).
Also, thanks for laying out your thinking in so much detail—I suspect there are other people who had more or less the same reaction to Open Phil’s grant write-up but haven’t spoken up about it. I’d be happy to talk more about this over email, too, including answering Qs from anyone else who wants more of my thoughts on this.
We originally sent Nick Beckstead what we considered our four most important 2015 results, at his request; these were (1) the incompatibility of the “Inductive Coherence” framework and the “Asymptotic Convergence in Online Learning with Unbounded Delays” framework; (2) the demonstration in “Proof-Producing Reflection for HOL” that a non-pathological form of self-referential reasoning is possible in a certain class of theorem-provers; (3) the reflective oracles result presented in “A Formal Solution to the Grain of Truth Problem,” “Reflective Variants of Solomonoff Induction and AIXI,” and “Reflective Oracles”; (4) and Vadim Kosoy’s “Optimal Predictors” work. The papers we listed under 1, 2, and 4 then got used in an external review process they probably weren’t very well-suited for.
I think this was more or less just an honest miscommunication. I told Nick in advance that I only assigned an 8% probability to external reviewers thinking the “Asymptotic Convergence…” result was “good” on its own (and only a 20% probability for “Inductive Coherence”). My impression of what happened is that Open Phil staff interpreted my pushback as saying that I thought the external reviews wouldn’t carry much Bayesian evidence (but that the internal reviews still would), where what I was trying to communicate was that I thought the papers didn’t carry very much Bayesian evidence about our technical output (and that I thought the internal reviewers would need to speak to us about technical specifics in order to understand why we thought they were important). Thus, we were surprised when their grant decision and write-up put significant weight on the internal reviews of those papers (and they were surprised that we were surprised). This is obviously really unfortunate, and another good sign that I should have committed more time and care to clearly communicating my thinking from the outset.
Regarding picking better papers for external review: We only put out 10 papers directly related to our technical agendas between Jan 2015 and Mar 2016, so the option space is pretty limited, especially given the multiple constraints Open Phil wanted to meet. Optimizing for technical impressiveness and non-obviousness as a stand-alone result, I might have instead gone with Critch’s bounded Löb paper and the grain of truth problem paper over the AC/IC results. We did submit the grain of truth problem paper to Open Phil, but they decided not to review it because it didn’t meet other criteria they were interested in.
I’m less pessimistic about building collaborations and partnerships, in part because we’re already on pretty good terms with other folks in the community, and in part because I think we have different models of how technical ideas spread. Regardless, I expect that with more and better communication, we can (upon re-evaluation) raise the probability of Open Phil staff that the work we’re doing is important.
More generally, though, I expect this task to get easier over time as we get better at communicating about our research. There’s already a body of AI alignment research (and, perhaps, methodology) that requires the equivalent of multiple university courses to understand, but there aren’t curricula or textbooks for teaching it. If we can convince a small pool of researchers to care about the research problems we think are important, this will let us bootstrap to the point where we have more resources for communicating information that requires a lot of background and sustained scholarship, as well as more of the institutional signals that this stuff warrants a time investment.
I can maybe make the time expenditure thus far less mysterious if I mention a couple more ways I erred in trying to communicate my model of MIRI’s research agenda:
My early discussion with Daniel was framed around questions like “What specific failure mode do you expect to be exhibited by advanced AI systems iff their programmers don’t understand logical uncertainty?” I made the mistake of attempting to give straight/non-evasive answers to those sorts of questions and let the discussion focus on that evaluation criterion, rather than promptly saying “MIRI’s research directions mostly aren’t chosen to directly address a specific failure mode in a notional software system” and “I don’t think that’s a good heuristic for identifying research that’s likely to be relevant to long-run AI safety.”
I fell prey to the transparency illusion pretty hard, and that was completely my fault. Mid-way through the process, Daniel made a write-up of what he had gathered so far; this write-up revealed a large number of miscommunications and places where I thought I had transmitted a concept of mine but Daniel had come away with a very different concept. It’s clear in retrospect that we should have spent a lot more time with me having Daniel try to explain what he thought I meant, and I had all the tools to predict this in foresight; but I foolishly assumed that wouldn’t be necessary in this case.
(I plan to blog more about the details of these later.)
I think these are important mistakes that show I hadn’t sufficiently clarified several concepts in my own head, or spent enough time understanding Daniel’s position. My hope is that I can do a much better job of avoiding these sorts of failures in the next round of discussion, now that I have a better model of where Open Phil’s staff and advisors are coming from and what the review process looks like.
Yeah, though that was before my time. He did an unpaid internship with us in the summer of 2013, and we’ve occasionally contracted him to tutor MIRI staff. Qiaochu’s also a lot socially closer to MIRI; he attended three of our early research workshops.
I think that’s a reasonable stance to take, and that there are other possible reasonable stances here too. Some of the variables I expect EAs to vary on include “level of starting confidence in MIRI’s mathematical intuitions about complicated formal questions” and “general risk tolerance.” A relatively risk-intolerant donor is right to wait until we have clearer demonstrations of success; and a relatively risk-tolerant donor who starts without a very high confidence in MIRI’s intuitions about formal systems might be pushed under a donation threshold by learning that an important disagreement has opened up between us and Daniel Dewey (or between us and other people at Open Phil).
Also, thanks for laying out your thinking in so much detail—I suspect there are other people who had more or less the same reaction to Open Phil’s grant write-up but haven’t spoken up about it. I’d be happy to talk more about this over email, too, including answering Qs from anyone else who wants more of my thoughts on this.