Why do you think that corrigibility is mathematically impossible for AGI? Because you think it would necessarily have a predefined utility function, or some other reason?
I do not know if has a predefined utility function, or if the functions simply have similar forms. If there is a utility function that provides utility for the AI to shutdown if some arbitrary âshutdown buttonâ is pressed, then there exists a state where the âshutdown buttonâ is being pressed at a very high probability (e.g. an office intern is in the process of pushing the âshutdown buttonâ) that provides more expected utility than the current state. There is therefore an incentive for the AI to move towards that state (e.g. by convincing the office intern to push the âshutdown buttonâ). If instead there was negative utility in the âshutdown buttonâ being pressed, the AI is incentivized to prevent the button from being pressed. If instead the AI had no utility function for whether the âshutdown buttonâ was pressed or not, but there somehow existed a code segment that caused the shutdown process to happen if the âshutdown buttonâ was pressed, then there existed a daughter AGI that has slightly more efficient code if this code segment is omitted. An AGI that has a utility function that provides utility for producing daughter AGIs that are more efficient versions of itself, is incentivized to produce such a daughter that has the âshutdown buttonâ code segment removed.
Hi William! Welcome to the Forum :)
Why do you think that corrigibility is mathematically impossible for AGI? Because you think it would necessarily have a predefined utility function, or some other reason?
Hi Robi Rahman, thanks for the welcome.
I do not know if has a predefined utility function, or if the functions simply have similar forms. If there is a utility function that provides utility for the AI to shutdown if some arbitrary âshutdown buttonâ is pressed, then there exists a state where the âshutdown buttonâ is being pressed at a very high probability (e.g. an office intern is in the process of pushing the âshutdown buttonâ) that provides more expected utility than the current state. There is therefore an incentive for the AI to move towards that state (e.g. by convincing the office intern to push the âshutdown buttonâ). If instead there was negative utility in the âshutdown buttonâ being pressed, the AI is incentivized to prevent the button from being pressed. If instead the AI had no utility function for whether the âshutdown buttonâ was pressed or not, but there somehow existed a code segment that caused the shutdown process to happen if the âshutdown buttonâ was pressed, then there existed a daughter AGI that has slightly more efficient code if this code segment is omitted. An AGI that has a utility function that provides utility for producing daughter AGIs that are more efficient versions of itself, is incentivized to produce such a daughter that has the âshutdown buttonâ code segment removed.
There is a more detailed version of this description in https://ââintelligence.org/ââfiles/ââCorrigibility.pdf
I could be wrong about my conclusion about corrigiblity (and probably am), however it is my best intuition at this point.