I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.
I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.
Regarding the last point:
Quantilizers are optimizing (namely a certain “regularized” reward function)
By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)
I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.
Hi Seth, thank you for your thoughts!
I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.
I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.
Regarding the last point:
Quantilizers are optimizing (namely a certain “regularized” reward function)
By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)
I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.