Hum, I think I wrote my point badly on the comment above. What I mean isn’t that formal methods will never be useful, just that they’re not really useful yet, and will require more pure AI safety research to be useful.
The general reason is that all formal methods try to show that a program follows a specification on a model of computation. Right now, a lot of the work on formal methods applied to AI focus on adapting known formal methods to the specific programs (say Neural Networks) and the right model of computation (in what contexts do you use these programs, how can you abstract their execution to make it simpler). But one point they fail to address is the question of the specification.
Note that when I say specification, I mean a formal specification. In practice, it’s usually a modal logic formula, in LTL for example. And here we get at the crux of my argument: nobody knows the specification for almost all AI properties we care about. Nobody knows the specification for “Recognizing kittens” or “Answering correctly a question in English”. And even for safety questions, we don’t have yet a specification of “doesn’t manipulate us” or “is aligned”. That’s the work that still needs to be done, and that’s what people like Paul Christiano and Evan Hubinger, among others, are doing. But until we have such properties, the formal methods will not be really useful to either AI capability or AI safety.
Lastly, I want to point out that working on AI for formal methods is also a means to get money and prestige. I’m not going to go full Hanson and say that’s the only reason, but it’s still a part of the international situation. I have examples of people getting AI related funding in France, for a project that is really, but really useless for AI.
Hum, I think I wrote my point badly on the comment above. What I mean isn’t that formal methods will never be useful, just that they’re not really useful yet, and will require more pure AI safety research to be useful.
The general reason is that all formal methods try to show that a program follows a specification on a model of computation. Right now, a lot of the work on formal methods applied to AI focus on adapting known formal methods to the specific programs (say Neural Networks) and the right model of computation (in what contexts do you use these programs, how can you abstract their execution to make it simpler). But one point they fail to address is the question of the specification.
Note that when I say specification, I mean a formal specification. In practice, it’s usually a modal logic formula, in LTL for example. And here we get at the crux of my argument: nobody knows the specification for almost all AI properties we care about. Nobody knows the specification for “Recognizing kittens” or “Answering correctly a question in English”. And even for safety questions, we don’t have yet a specification of “doesn’t manipulate us” or “is aligned”. That’s the work that still needs to be done, and that’s what people like Paul Christiano and Evan Hubinger, among others, are doing. But until we have such properties, the formal methods will not be really useful to either AI capability or AI safety.
Lastly, I want to point out that working on AI for formal methods is also a means to get money and prestige. I’m not going to go full Hanson and say that’s the only reason, but it’s still a part of the international situation. I have examples of people getting AI related funding in France, for a project that is really, but really useless for AI.