A whole range of things, from elements on the ‘fancy spreadsheet’ side such as recidivism and predpol to the more complex elements surrounding evidential aspects. I am aware none of these are close to AGI, considering no current AI is given its hyper-specialism, but the point of that paragraph isn’t about the AI but how humans and organisations have been shown to use AI (or automation software, if you’re more comfortable with that phrase). When the first actual AGI is developed, it is likely to be in a very well-funded lab—a lab likely under control of an organisation who are no exception to the usual weaknesses of capitalistic behaviour. After all, the entire point of developing general intelligence for many people isn’t for scientific endeavour but for commercial reasons.
Maybe humanity has 1 AGI or 1 ASI, but if we end up with dozens or hundreds of systems, we can’t rely on people to do the ‘right thing’ to prevent misalignment problems. There needs to be an actual system of governance ready to keep them in check, so it’s not just a technical problem.
A whole range of things, from elements on the ‘fancy spreadsheet’ side such as recidivism and predpol to the more complex elements surrounding evidential aspects. I am aware none of these are close to AGI, considering no current AI is given its hyper-specialism, but the point of that paragraph isn’t about the AI but how humans and organisations have been shown to use AI (or automation software, if you’re more comfortable with that phrase). When the first actual AGI is developed, it is likely to be in a very well-funded lab—a lab likely under control of an organisation who are no exception to the usual weaknesses of capitalistic behaviour. After all, the entire point of developing general intelligence for many people isn’t for scientific endeavour but for commercial reasons.
Maybe humanity has 1 AGI or 1 ASI, but if we end up with dozens or hundreds of systems, we can’t rely on people to do the ‘right thing’ to prevent misalignment problems. There needs to be an actual system of governance ready to keep them in check, so it’s not just a technical problem.
Let me know if that clarified at all.