Nice post! Re the competitive pressures, this seems especially problematic in long-timelines worlds where TAI is really hard to build, as (toy model) if company A spends all its cognitive assets on capabilities (including generating profit to fund this research), while company B spends half its cognitive assets at any given time on safety work with no capabilities overflows, then if there is a long time over which this exponential growth continues, company A will likely reach the lead even if it starts well behind. Whereas if there is a relatively smaller amount of cognitive assets ever deployed before TAI, safety-oriented companies being in the lead should be the dominant factor and safety-ignoring companies wouldn’t be able to catch up even by ‘defecting’.
Nice post! Re the competitive pressures, this seems especially problematic in long-timelines worlds where TAI is really hard to build, as (toy model) if company A spends all its cognitive assets on capabilities (including generating profit to fund this research), while company B spends half its cognitive assets at any given time on safety work with no capabilities overflows, then if there is a long time over which this exponential growth continues, company A will likely reach the lead even if it starts well behind. Whereas if there is a relatively smaller amount of cognitive assets ever deployed before TAI, safety-oriented companies being in the lead should be the dominant factor and safety-ignoring companies wouldn’t be able to catch up even by ‘defecting’.