Just as a safety-performance tradeoff, in the presence of intense competition, pushes decision-makers to cut corners on safety, so can a tradeoff between any human value and competitive performance incentivize decision makers to sacrifice that value. Contemporary examples of values being eroded by global economic competition could include non-monopolistic markets, privacy, and relative equality.
I found this section very confusing. It is definitely not the case that competition always pushes towards cutting corners on safety. In many cases consumers prize safety and are willing to pay a premium for it, for example with automobiles or baby-related products, where companies actively advertise on this basis. In some cases it does reduce safety—typically when consumers think the increase in safety is too marginal to be worth the cost. And I’m not sure that ‘non-monopolistic markets’ count as a contemporary example of a value? Nor do I think privacy is really a good example here: privacy has largely been undermined by 1) governments, who are not directly subject to strong competitive pressures and 2) consumers voluntarily giving up privacy in return for better/cheaper services.
The conclusion is, in my view, correct—competitive pressures should be expected to degrade safety for AI. But this is ultimately due to externalities: an unsafe AI imposes costs (in expectation) on third parties. If externalities were not present, as they are not (to a significant degree) in your examples, competition would not cause an undesirable reduction in safety. If AGI only threatened extinction for AGI developers and their customers, AI safety would not be a big problem.
Yes, I was compressing a lot of ideas in a small space!
We have conceptual slippage here. By safety I’m referring to the safety of the developer (and maybe the world), not of a consumer who decides which product to buy. As you note, if safety is a critical component of winning a competition, then safety will be incentivized by the competition.
It is generally the case that it is hard to separate out (self) safety from competitive performance; we might best conceptualize “safety” as the residual of safety (the neglected part) after conditioning on performance (ie we focus on the parts of safety that are not an input to performance).
Increased competitive pressure will: (1) induce some actors to drop out of the competition, since they can’t keep up; (2) those who stay in will in equilibrium invest more in winning the competition, investing less in other goods. This happens when there is some “tradeoff” or fungibility across anything of value, and winning the competition. You can think of “competitive pressure” as the slope of the function of the probability that one wins the competition, given investment in winning (and holding the other’s investment fixed). If the contest is a lottery, then the slope is flat. If one is so far ahead or behind that effort doesn’t make a difference, the slope is flat. If we are neck in neck, and investment maps into performance, then the marginal incentive to invest can get high.
You are right that in the presence of a safety externality this race to the bottom can get much worse. I think of Russell’s setting “the remaining unconstrained variables to extreme values”. In the Cold War, nuclear brinkmanship yielded advantage, and so world leaders gambled with nuclear annihilation.
But this dynamic can still happen even if the risk is only to oneself. An individual can rationally choose to accept a greater risk of harm, in exchange for keeping up in a now more intense contest. The “war of attrition” game (or equivalently “all pay auction”) exemplifies this, where the player only pays what they bid, and yet in symmetric equilibria, actors do burn resources competing for the prize (more so the more valuable is the prize).
The externality in a race is that anything I do to increase my chance of winning, reduces your chance of winning, and so “investment in winning” will be oversupplied relative to the socially optimal level.
My understanding is a lot of that is just that consumers didn’t want them. From the first source I found on this:
Safety-conscious car buyers could seek out—and pay extra for—a Ford with seatbelts and a padded dashboard, but very few did: only 2 percent of Ford buyers took the $27 seatbelt option.
This is not surprising to me given that, even after the installation of seatbelts became mandatory, it was decades until most Americans actually used them. Competition encouraged manufacturers to ‘cut corners’ on safety in this instance precisely because that was what consumers wanted them to do.
I found this section very confusing. It is definitely not the case that competition always pushes towards cutting corners on safety. In many cases consumers prize safety and are willing to pay a premium for it, for example with automobiles or baby-related products, where companies actively advertise on this basis. In some cases it does reduce safety—typically when consumers think the increase in safety is too marginal to be worth the cost. And I’m not sure that ‘non-monopolistic markets’ count as a contemporary example of a value? Nor do I think privacy is really a good example here: privacy has largely been undermined by 1) governments, who are not directly subject to strong competitive pressures and 2) consumers voluntarily giving up privacy in return for better/cheaper services.
The conclusion is, in my view, correct—competitive pressures should be expected to degrade safety for AI. But this is ultimately due to externalities: an unsafe AI imposes costs (in expectation) on third parties. If externalities were not present, as they are not (to a significant degree) in your examples, competition would not cause an undesirable reduction in safety. If AGI only threatened extinction for AGI developers and their customers, AI safety would not be a big problem.
Yes, I was compressing a lot of ideas in a small space!
We have conceptual slippage here. By safety I’m referring to the safety of the developer (and maybe the world), not of a consumer who decides which product to buy. As you note, if safety is a critical component of winning a competition, then safety will be incentivized by the competition.
It is generally the case that it is hard to separate out (self) safety from competitive performance; we might best conceptualize “safety” as the residual of safety (the neglected part) after conditioning on performance (ie we focus on the parts of safety that are not an input to performance).
Increased competitive pressure will: (1) induce some actors to drop out of the competition, since they can’t keep up; (2) those who stay in will in equilibrium invest more in winning the competition, investing less in other goods. This happens when there is some “tradeoff” or fungibility across anything of value, and winning the competition. You can think of “competitive pressure” as the slope of the function of the probability that one wins the competition, given investment in winning (and holding the other’s investment fixed). If the contest is a lottery, then the slope is flat. If one is so far ahead or behind that effort doesn’t make a difference, the slope is flat. If we are neck in neck, and investment maps into performance, then the marginal incentive to invest can get high.
You are right that in the presence of a safety externality this race to the bottom can get much worse. I think of Russell’s setting “the remaining unconstrained variables to extreme values”. In the Cold War, nuclear brinkmanship yielded advantage, and so world leaders gambled with nuclear annihilation.
But this dynamic can still happen even if the risk is only to oneself. An individual can rationally choose to accept a greater risk of harm, in exchange for keeping up in a now more intense contest. The “war of attrition” game (or equivalently “all pay auction”) exemplifies this, where the player only pays what they bid, and yet in symmetric equilibria, actors do burn resources competing for the prize (more so the more valuable is the prize).
The externality in a race is that anything I do to increase my chance of winning, reduces your chance of winning, and so “investment in winning” will be oversupplied relative to the socially optimal level.
Overall, thinking clearly through the strategic dynamics of models like these can be very complex, and one can often generate a counter-intuitive result. Robert Trager, formerly myself, and others have a lot of work exploring the various strategic wrinkles. I’m not sure what is the best link, here are some places to start that I’m more familiar with: https://forum.effectivealtruism.org/posts/c73nsggC2GQE5wBjq/announcing-the-spt-model-web-app-for-ai-governance
Eoghan Stafford, Robert Trager and Allan Dafoe, Safety Not Guaranteed: International Strategic Dynamics of Risky Technology Races, Working paper, July 2022. https://drive.google.com/file/d/1zYLALn3u8AhuXA4zheWmrD7VHmI6zTvD/view?usp=sharing
Good points, to play devil’s advocate it did take many decades for auto manufacturers to include seat belts by default.
It seems safety policy sometimes meets resistance by production / profit motive, despite clear evidence of cause of death.
My understanding is a lot of that is just that consumers didn’t want them. From the first source I found on this:
This is not surprising to me given that, even after the installation of seatbelts became mandatory, it was decades until most Americans actually used them. Competition encouraged manufacturers to ‘cut corners’ on safety in this instance precisely because that was what consumers wanted them to do.
Ah yeah thanks for informing me.