I appreciate your read and the engagement, thanks.
The issue with assuming AGI will develop morality the way humans did is that humans don’t act with strict logical efficiency—we are shaped by a chaotic evolutionary process, not a clean optimisation function. We don’t always prioritise survival, and often behave irrationally—see: the Darwin Awards.
But AGI is not a product of evolution—it’s designed to pursue a goal as efficiently as possible. Morality emerged in humans as a byproduct of messy, competing survival mechanisms, not because it was the most efficient way to achieve a single goal. An AGI, by contrast, will be ruthlessly efficient in whatever it’s designed to optimise.
Hoping that AGI develops morality despite its inefficiency—and gambling all of human existence on it—seems like a terrible wager to make.
The key difference is that SGD is not evolution—it’s a guided optimisation process. Evolution has no goal beyond survival and reproduction, while SGD explicitly optimises toward a defined function chosen by human designers. Yes, the search process is stochastic, but the selection criteria are rigidly defined in a way that natural selection is not.
The fact that current AI systems don’t act with strict efficiency is not evidence that AGI will behave irrationally—it’s just a reflection of their current limitations. If anything, their errors today are an argument for why they won’t develop morality by accident: their behaviour is driven entirely by the training data and reward signals they are given. When they improve, they will become better at pursuing those goals, not more human-like.
Yes, if AGI emerges from simply trying to create it for the sake of it, then it has no real objectives. If it emerges as a result of an AI tool that is being used to optimise something within a business, or as part of a government or military, then it will. I argue in my first essay that this is the real threat AGI poses: when developed in a competitive system, it will disregard safety and morality in order to get a competitive edge.
The crux of the issue is this: humans evolved morality as an unintended byproduct of thousands of competing pressures over millions of years. AGI, by contrast, will be shaped by a much narrower and more deliberate selection process. The randomness in training doesn’t mean AGI will stumble into morality—it just means it will be highly optimised for whatever function we define, whether that aligns with human values or not.