My main takeaway was that building AI systems to have self-interest is dangerous because that has the potential to explicitly conflict with humanity’s own interest, leading to a major existential risk with super-intelligent AIs.
I wonder if there’s any advantage of self-interest in AI though. Is there any way self-interest could possibly make an AI more effective at accomplishing its goals? In biological entities, self-interest obviously helps with e.g. avoiding threats, seeking more favourable living conditions, etc. I wonder if this applies in a similar manner to AIs, or if self-interest in an AI is inconsequential at best.
I’m curious, what exactly is the worry with AGI development in e.g. Russia and China? Is the concern that they are somehow less invested in building safe AGI (which seems to strongly conflict with their own self-interest)?
Or is the concern that they could somehow build AGI which selectively harms people/countries of their choosing? In this latter case it seems to me that the problem is exclusively a human one, and isn’t ethically different from concerns about super-lethal computer viruses or bio/nuclear weapons. It’s not clear how this precise risk is specific to AI/AGI.
This isn’t exactly the same as self-interest, though. I think a better analogy for this might be human domestication of animals for agriculture. It’s not in the self-interest of a factory farmed chicken to be on a factory farm, but humans have power over which animals exist so we’ll make sure there are lots of animals who serve our interests. AI systems will be selected for to the extent they serve the interests of people making and buying them.
RE international development: competition between states undercut arguments for domestic safety regulations/practices. These are exacerbated by beliefs that international rivals will behave less safely/responsibly, but you don’t actually need to believe that to justify cutting corners domestically. If China or Russia built an AGI that was totally safe in the sense that it is aligned with its creators interests, that would be seen as a big threat by the US govt.
If you think that building AGI is extremely dangerous no matter who does it, then having more well-resourced players in the space increases the overall risk.
Interesting podcast—I read the transcript.
My main takeaway was that building AI systems to have self-interest is dangerous because that has the potential to explicitly conflict with humanity’s own interest, leading to a major existential risk with super-intelligent AIs.
I wonder if there’s any advantage of self-interest in AI though. Is there any way self-interest could possibly make an AI more effective at accomplishing its goals? In biological entities, self-interest obviously helps with e.g. avoiding threats, seeking more favourable living conditions, etc. I wonder if this applies in a similar manner to AIs, or if self-interest in an AI is inconsequential at best.
I’m curious, what exactly is the worry with AGI development in e.g. Russia and China? Is the concern that they are somehow less invested in building safe AGI (which seems to strongly conflict with their own self-interest)?
Or is the concern that they could somehow build AGI which selectively harms people/countries of their choosing? In this latter case it seems to me that the problem is exclusively a human one, and isn’t ethically different from concerns about super-lethal computer viruses or bio/nuclear weapons. It’s not clear how this precise risk is specific to AI/AGI.
I think building AI systems with some level of autonomy/agency would make them much more useful, provided they are still aligned with the interests of their users/creators. There’s already evidence that companies are moving in this direction based on the business case: https://jacobin.com/2024/01/can-humanity-survive-ai#:~:text=Further%2C%20academics%20and,is%20pretty%20good.%E2%80%9D
This isn’t exactly the same as self-interest, though. I think a better analogy for this might be human domestication of animals for agriculture. It’s not in the self-interest of a factory farmed chicken to be on a factory farm, but humans have power over which animals exist so we’ll make sure there are lots of animals who serve our interests. AI systems will be selected for to the extent they serve the interests of people making and buying them.
RE international development: competition between states undercut arguments for domestic safety regulations/practices. These are exacerbated by beliefs that international rivals will behave less safely/responsibly, but you don’t actually need to believe that to justify cutting corners domestically. If China or Russia built an AGI that was totally safe in the sense that it is aligned with its creators interests, that would be seen as a big threat by the US govt.
If you think that building AGI is extremely dangerous no matter who does it, then having more well-resourced players in the space increases the overall risk.