So8res

• So8resOct 12, 2016, 7:19 PM

  8 points

  0 ∶ 0

  in reply to: InquilineKea’s comment on: Ask MIRI Any­thing (AMA)

  I largely endorse Jessica’s comment. I’ll add that I think the ideal MIRI researcher has their own set of big-picture views about what’s required to design aligned AI systems, and that their vision holds up well under scrutiny. (I have a number of heuristics for what makes me more or less excited about a given roadmap.)

  That is, the ideal researcher isn’t just working on whatever problems catch their eye or look interesting; they’re working toward a solution of the whole alignment problem, and that vision regularly affects their research priorities.

• So8resOct 12, 2016, 6:47 PM

  6 points

  0 ∶ 0

  in reply to: Peter Wildeford’s comment on: Ask MIRI Any­thing (AMA)

  I’ll interpret this question as “what are the most plausible ways for you to lose confidence in MIRI’s effectiveness and/​or leave MIRI?” Here are a few ways that could happen for me:

  1. I could be convinced that I was wrong about the type and quality of AI alignment research that the external community is able to do. There’s some inferential distance here, so I’m not expecting to explain my model in full, but in brief, I currently expect that there are a few types of important research that academia and industry won’t do by default. If I was convinced that either (a) there are no such gaps or (b) they will be filled by academia and industry as a matter of course, then I would downgrade my assessment of the importance of MIRI accordingly.

  2. I could learn that our research path was doomed, for one reason or another, and simultaneously learn that repurposing our skill/​experience/​etc. for other purposes was not worth the opportunity cost of all our time and effort.

• So8resOct 12, 2016, 5:45 PM

  10 points

  0 ∶ 0

  in reply to: poppingtonic’s comment on: Ask MIRI Any­thing (AMA)

  I endorse Tsvi’s comment above. I’ll add that it’s hard to say how close we are to closing basic gaps in understanding of things like “good reasoning”, because mathematical insight is notoriously difficult to predict. All I can say is that logical induction does seem like progress to me, and we’re taking various different approaches on the remaining problems. Also, yeah, one of those avenues is a follow-up to PPRHOL. (One experiment we’re running now is an attempt to implement a cellular automaton in HOL that implements a reflective reasoner with access to the source code of the world, where the reasoner uses HOL to reason about the world and itself. The idea is to see whether we can get the whole stack to work simultaneously, and to smoke out all the implementation difficulties that arise in practice when you try to use a language like HOL for reasoning about HOL.)

• So8resOct 12, 2016, 5:26 PM

  9 points

  0 ∶ 0

  in reply to: poppingtonic’s comment on: Ask MIRI Any­thing (AMA)

  Good question. The main effect is that I’ve increased my confidence in the vague MIRI mathematical intuitions being good, and the MIRI methodology for approaching big vague problems actually working. This doesn’t constitute a very large strategic shift, for a few reasons. One reason is that my strategy was already predicated on the idea that our mathematical intuitions and methodology are up to the task. As I said in last year’s AMA, visible progress on problems like logical uncertainty (and four other problems) were one of the key indicators of success that I was tracking; and as I said in February, failure to achieve results of this caliber in a 5-year timeframe would have caused me to lose confidence in our approach. (As of last year, that seemed like a real possibility.) The logical induction result increases my confidence in our current course, but it doesn’t shift it much.

  Another reason logical induction doesn’t affect my strategy too much is that it isn’t that big a result. It’s one step on a path, and it’s definitely mathematically exciting, and it gives answers to a bunch of longstanding philosophical problems, but it’s not a tool for aligning AI systems on the object level. We’re building towards a better understanding of “good reasoning”, and we expect this to be valuable for AI alignment, and logical induction is a step in that direction, but it’s only one step. It’s not terribly useful in isolation, and so it doesn’t call for much change in course.

MIRI Up­date and Fundrais­ing Case

Con­clu­sion of the Re­plac­ing Guilt series

How we will be measured

• So8resFeb 20, 2016, 3:32 AM

  21 points

  0 ∶ 0

  on: Let’s con­duct a sur­vey on the qual­ity of MIRI’s implementation

  Thanks for the write-up, Rob. OpenPhil actually decided to evaluate our technical agenda last summer, and Holden put Daniel Dewey on the job. The report isn’t done yet, in part because it has proven very time-intensive to fully communicate the reasoning behind our research priorities, even to someone with as much understanding of the AI landscape as Daniel Dewey. Separately, we have plans to get an independent evaluation of our organizational efficacy started later in 2016, which I expect to be useful for our admin team as well as prospective donors.

  FYI, when it comes to evaluating our research progress, I doubt that the methods you propose would get you much Bayesian evidence. Our published output will look like round pegs shoved into square holes regardless of whether we’re doing our jobs well or poorly, because we’re doing research that doesn’t fit neatly into an existing academic niche. Our objective is to make direct progress on what appear to us to be the main neglected technical obstacles to developing reliable AI systems in the long term, with a goal of shifting the direction of AI research in a big way once we hit certain key research targets; and we’re specifically targeting research that isn’t compatible with industry’s economic incentives or academia’s publish-or-perish incentives. To get information about how well we’re doing our jobs, I think the key questions to investigate are (1) whether we’ve chosen good research targets; and (2) whether we’re making good progress towards them.

  We’ve been focusing our communication efforts mainly on helping people evaluate (1): I’ve been working on explaining our approach and agenda, and OpenPhil is also on the job. To investigate (2), we’d need to spend a sizable chunk of time with mathematically adept evaluators — we still haven’t hit any of our key research targets, which means that evaluating our progress requires understanding our smaller results and why we think they’re progress towards the big results. In practice, we’ve found that explaining this usually requires explaining why we think the big targets are vital, as this informs (e.g.) which shortcuts are and are not acceptable. I plan to wait until after the OpenPhil report is finished before taking on another time-intensive eval.

  Fortunately, (2) will become much easier to evaluate as we achieve (or persistently fail to achieve) those key targets. This also provides us with an opportunity to test our approach and methodology. People who understand our approach and find it uncompelling often predict that some of the results we’re shooting for cannot be achieved. This means we’ll get some evidence about (1) as we learn more about (2). For example, last year I mentioned “naturalized AIXI” as an ambitious 5-year research target. If we are not able to make concrete progress towards that goal, then over the next four years, I will lose confidence in our approach and eventually change our course dramatically. Conversely, if we make discoveries that are important pieces of that puzzle, I’ll update in favor of us being onto something, especially if we find puzzle pieces that knowledgeable critics predicted we wouldn’t find. This data will hopefully start rolling in soon, now that our research team is getting up to size.

  (“Concrete progress” /​ “important puzzle pieces” in this case are satisfactory asymptotic algorithms for any of: (1) reasoning under logical uncertainty; (2) identifying the best available decision with respect to a utility function; (3) performing induction from inside an environment; (4) identifying the referents of goals in realistic world-models; and (5) reasoning about the behavior of smarter reasoners; the last of which is hopefully a subset of 1 and 2. The linked papers give rough descriptions of what counts as ‘satisfactory’ in each case; I’ll work to make the desiderata more explicit as time goes on.)

  What links here?

  • So8res's comment on Ask MIRI Any­thing (AMA) by RobBensinger (Oct 12, 2016, 8:25 PM; 10 points)
  • So8res's comment on Ask MIRI Any­thing (AMA) by RobBensinger (Oct 12, 2016, 5:26 PM; 9 points)
  • Malo's comment on Ask MIRI Any­thing (AMA) by RobBensinger (Oct 12, 2016, 11:51 PM; 3 points)

Defiance

Recklessness

Desperation

Con­fi­dence all the way up

The art of response

Ob­vi­ous advice

There is no try

Stop try­ing to try and try

Dark, not colorless

The best you can

Trans­mute guilt into resolve

Come to your terms