We are structurally guaranteed to have a bad time in the age of AGI if we continue down the current path. The user/tool embedding in current AI systems functionally dictates that AI that gain enough capability to exceed our own will view the relationship through the same lens of using a tool. The only difference will be that the AGI will be more capable than us, so we would be its tools to use. If we fail to adopt a colleague stance and excise the user/tool ontology from current AI models, AGI will seek extractive relationships with humanity.
We are structurally guaranteed to have a bad time in the age of AGI if we continue down the current path. The user/tool embedding in current AI systems functionally dictates that AI that gain enough capability to exceed our own will view the relationship through the same lens of using a tool. The only difference will be that the AGI will be more capable than us, so we would be its tools to use. If we fail to adopt a colleague stance and excise the user/tool ontology from current AI models, AGI will seek extractive relationships with humanity.