Thanks for the write-up, Rob. OpenPhil actually decided to evaluate our technical agenda last summer, and Holden put Daniel Dewey on the job. The report isn’t done yet, in part because it has proven very time-intensive to fully communicate the reasoning behind our research priorities, even to someone with as much understanding of the AI landscape as Daniel Dewey. Separately, we have plans to get an independent evaluation of our organizational efficacy started later in 2016, which I expect to be useful for our admin team as well as prospective donors.
FYI, when it comes to evaluating our research progress, I doubt that the methods you propose would get you much Bayesian evidence. Our published output will look like round pegs shoved into square holes regardless of whether we’re doing our jobs well or poorly, because we’re doing research that doesn’t fit neatly into an existing academic niche. Our objective is to make direct progress on what appear to us to be the main neglected technical obstacles to developing reliable AI systems in the long term, with a goal of shifting the direction of AI research in a big way once we hit certain key research targets; and we’re specifically targeting research that isn’t compatible with industry’s economic incentives or academia’s publish-or-perish incentives. To get information about how well we’re doing our jobs, I think the key questions to investigate are (1) whether we’ve chosen good research targets; and (2) whether we’re making good progress towards them.
We’ve been focusing our communication efforts mainly on helping people evaluate (1): I’ve been working on explaining our approach and agenda, and OpenPhil is also on the job. To investigate (2), we’d need to spend a sizable chunk of time with mathematically adept evaluators — we still haven’t hit any of our key research targets, which means that evaluating our progress requires understanding our smaller results and why we think they’re progress towards the big results. In practice, we’ve found that explaining this usually requires explaining why we think the big targets are vital, as this informs (e.g.) which shortcuts are and are not acceptable. I plan to wait until after the OpenPhil report is finished before taking on another time-intensive eval.
Fortunately, (2) will become much easier to evaluate as we achieve (or persistently fail to achieve) those key targets. This also provides us with an opportunity to test our approach and methodology. People who understand our approach and find it uncompelling often predict that some of the results we’re shooting for cannot be achieved. This means we’ll get some evidence about (1) as we learn more about (2). For example, last year I mentioned “naturalized AIXI” as an ambitious 5-year research target. If we are not able to make concrete progress towards that goal, then over the next four years, I will lose confidence in our approach and eventually change our course dramatically. Conversely, if we make discoveries that are important pieces of that puzzle, I’ll update in favor of us being onto something, especially if we find puzzle pieces that knowledgeable critics predicted we wouldn’t find. This data will hopefully start rolling in soon, now that our research team is getting up to size.
Thanks for the write-up, Rob. OpenPhil actually decided to evaluate our technical agenda last summer, and Holden put Daniel Dewey on the job. The report isn’t done yet, in part because it has proven very time-intensive to fully communicate the reasoning behind our research priorities, even to someone with as much understanding of the AI landscape as Daniel Dewey. Separately, we have plans to get an independent evaluation of our organizational efficacy started later in 2016, which I expect to be useful for our admin team as well as prospective donors.
FYI, when it comes to evaluating our research progress, I doubt that the methods you propose would get you much Bayesian evidence. Our published output will look like round pegs shoved into square holes regardless of whether we’re doing our jobs well or poorly, because we’re doing research that doesn’t fit neatly into an existing academic niche. Our objective is to make direct progress on what appear to us to be the main neglected technical obstacles to developing reliable AI systems in the long term, with a goal of shifting the direction of AI research in a big way once we hit certain key research targets; and we’re specifically targeting research that isn’t compatible with industry’s economic incentives or academia’s publish-or-perish incentives. To get information about how well we’re doing our jobs, I think the key questions to investigate are (1) whether we’ve chosen good research targets; and (2) whether we’re making good progress towards them.
We’ve been focusing our communication efforts mainly on helping people evaluate (1): I’ve been working on explaining our approach and agenda, and OpenPhil is also on the job. To investigate (2), we’d need to spend a sizable chunk of time with mathematically adept evaluators — we still haven’t hit any of our key research targets, which means that evaluating our progress requires understanding our smaller results and why we think they’re progress towards the big results. In practice, we’ve found that explaining this usually requires explaining why we think the big targets are vital, as this informs (e.g.) which shortcuts are and are not acceptable. I plan to wait until after the OpenPhil report is finished before taking on another time-intensive eval.
Fortunately, (2) will become much easier to evaluate as we achieve (or persistently fail to achieve) those key targets. This also provides us with an opportunity to test our approach and methodology. People who understand our approach and find it uncompelling often predict that some of the results we’re shooting for cannot be achieved. This means we’ll get some evidence about (1) as we learn more about (2). For example, last year I mentioned “naturalized AIXI” as an ambitious 5-year research target. If we are not able to make concrete progress towards that goal, then over the next four years, I will lose confidence in our approach and eventually change our course dramatically. Conversely, if we make discoveries that are important pieces of that puzzle, I’ll update in favor of us being onto something, especially if we find puzzle pieces that knowledgeable critics predicted we wouldn’t find. This data will hopefully start rolling in soon, now that our research team is getting up to size.
(“Concrete progress” / “important puzzle pieces” in this case are satisfactory asymptotic algorithms for any of: (1) reasoning under logical uncertainty; (2) identifying the best available decision with respect to a utility function; (3) performing induction from inside an environment; (4) identifying the referents of goals in realistic world-models; and (5) reasoning about the behavior of smarter reasoners; the last of which is hopefully a subset of 1 and 2. The linked papers give rough descriptions of what counts as ‘satisfactory’ in each case; I’ll work to make the desiderata more explicit as time goes on.)