This is a start, but just a start. Optimization/maximization isn’t actually the problem. Any highly competent agent with goals that don’t match ours is the problem.
A world that’s 10% paperclips and the rest composed of other stuff we don’t care about is no better than a true optimizer.
The idea “just don’t optimize” has a surprising amount of support in AGI safety, including quantilizers and satisficing. But they seem like only a bare start on taking the points off of the tiger’s teeth to me. The tiger will still gnaw you to death if it wants to even a little.
I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.
I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.
Regarding the last point:
Quantilizers are optimizing (namely a certain “regularized” reward function)
By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)
I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.
This is a start, but just a start. Optimization/maximization isn’t actually the problem. Any highly competent agent with goals that don’t match ours is the problem.
A world that’s 10% paperclips and the rest composed of other stuff we don’t care about is no better than a true optimizer.
The idea “just don’t optimize” has a surprising amount of support in AGI safety, including quantilizers and satisficing. But they seem like only a bare start on taking the points off of the tiger’s teeth to me. The tiger will still gnaw you to death if it wants to even a little.
Hi Seth, thank you for your thoughts!
I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.
I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.
Regarding the last point:
Quantilizers are optimizing (namely a certain “regularized” reward function)
By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)
I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.