“The risk posed by AI, however, is remote and hypothetical. The risk posed by misaligned corporations, on the other hand, is ever-present and real. Every public company in America has a legally-mandated obligation to maximize shareholder returns; every corporation, in other words, has a duty to churn all the life on this earth into corporate profit, if it is able to do so. The corporate code is like a hostile AI, run amok. And almost all of us are deeply affected by this code.”
This is false. (The analogy between corporations and unaligned AGI is misleading for many other reasons, of course, not the least of which is that corporations are not actually coherent singleton agents, but are made of people.)
Profit is residual income after inputs, wages and interests are paid. It is a signal that says “this firm is successfully deploying capital to serve the welfare of the consumer” (consumer=person weighted by wealth).
Of course, if the society does not give rights to some sentient beings, and torturing them is an intermediate step in the production of valued goods, it will make profits. In the past, when slave labour was a legal input, capitalists used it, and when the input became unavailable, they simply made profits using other available inputs.
Capitalism is a social mechanism. It allows social coordination in a massive and extremely efficient way. It empowers, multiplies and carry to the extreme existing social values and preferences.