1) Your current technical agenda involves creating a math of logical uncertainty and forming world-models out of this. When (if possible) do you predict that such a math will be worked out, and will MIRI’s focus move to the value learning problem then?
2) How long do you estimate that formal logic will be the arena in which MIRI’s technical work takes place—that is, how long will knowing formal logic be of use to a potential researcher before the research moves to new places?
(1) That’s not quite how I’d characterize the current technical agenda. Rather, I’d say that in order to build an AI aligned with human interests, you need to do three things: (a) understand how to build an AI that’s aligned with anything (could you build an AI that reliably builds as much diamond as possible?), (b) understand how to build an AI that assists you in correcting things-you-perceive-as-flaws (this doesn’t come for free, but it’s pretty important, because humans are bad at getting software right on the first try), and (c) figure out how to build a machine that can safely learn human values & intentions from training data.
We’re currently splitting our time between all these problems. It’s not that we haven’t focused on the value learning problem yet, rather, it’s that the value learning problem is only a fraction of the whole problem. We’ll keep working on all the parts, and I’m not sure which parts will yield first. I can’t give you a timeline on how long various parts will take; scientific progress is very hard to predict.
(2) I wouldn’t currently say that “formal logic is the arena in which MIRI’s technical work takes place”—if anything, “math in general” is the arena, and that will probably remain the case until we have a much better understanding of the problems we’re trying to solve (and how to solve simplified versions of them), at which point computer programming will become much more essential. Again, it’s hard to say how long it will take to get there, because scientific progress is hard to predict.
Formal logic is one of many tools useful in mathematics (alongside probability theory, statistics, linear algebra, etc.) that shows up fairly frequently in our work, but I don’t think of our work as “focused on formal logic.” I don’t think we’ll “move away from formal logic” at a particular time; rather, we’ll just use whichever mathematical tools look useful for the problems at hand. That will change as the problems change :-)
It seems a bit like the question behind the question might be “I’d like to help, but I don’t know formal logic, when will that stop being a barrier”. In which case it’s worth saying that I’m attending a MIRI decision theory workshop at the moment, and I don’t really know formal logic, but it isn’t proving too much of a barrier; I can think about the assertion “Suppose PA proves that A implies B” without really understanding exactly what PA is.
1) Your current technical agenda involves creating a math of logical uncertainty and forming world-models out of this. When (if possible) do you predict that such a math will be worked out, and will MIRI’s focus move to the value learning problem then?
2) How long do you estimate that formal logic will be the arena in which MIRI’s technical work takes place—that is, how long will knowing formal logic be of use to a potential researcher before the research moves to new places?
(1) That’s not quite how I’d characterize the current technical agenda. Rather, I’d say that in order to build an AI aligned with human interests, you need to do three things: (a) understand how to build an AI that’s aligned with anything (could you build an AI that reliably builds as much diamond as possible?), (b) understand how to build an AI that assists you in correcting things-you-perceive-as-flaws (this doesn’t come for free, but it’s pretty important, because humans are bad at getting software right on the first try), and (c) figure out how to build a machine that can safely learn human values & intentions from training data.
We’re currently splitting our time between all these problems. It’s not that we haven’t focused on the value learning problem yet, rather, it’s that the value learning problem is only a fraction of the whole problem. We’ll keep working on all the parts, and I’m not sure which parts will yield first. I can’t give you a timeline on how long various parts will take; scientific progress is very hard to predict.
(2) I wouldn’t currently say that “formal logic is the arena in which MIRI’s technical work takes place”—if anything, “math in general” is the arena, and that will probably remain the case until we have a much better understanding of the problems we’re trying to solve (and how to solve simplified versions of them), at which point computer programming will become much more essential. Again, it’s hard to say how long it will take to get there, because scientific progress is hard to predict.
Formal logic is one of many tools useful in mathematics (alongside probability theory, statistics, linear algebra, etc.) that shows up fairly frequently in our work, but I don’t think of our work as “focused on formal logic.” I don’t think we’ll “move away from formal logic” at a particular time; rather, we’ll just use whichever mathematical tools look useful for the problems at hand. That will change as the problems change :-)
Thank you for the response; it was helpful :^)
It seems a bit like the question behind the question might be “I’d like to help, but I don’t know formal logic, when will that stop being a barrier”. In which case it’s worth saying that I’m attending a MIRI decision theory workshop at the moment, and I don’t really know formal logic, but it isn’t proving too much of a barrier; I can think about the assertion “Suppose PA proves that A implies B” without really understanding exactly what PA is.