Thanks for the quibble, seems big if true! And agreed it is not something that I was tracking when writing the article.
A few thoughts:
I am fairly unsure if the economies of scale point is actually right. Some reasons for doubt:
Partly I’m thinking of Drexler’s CAIS arguments and intuitions that ecosystems of different specialised systems will outcompete monoculture
Partly I’m looking at AI development today
Partly the form of the economies of scale argument seems to be ‘one constraint on human economies of scale is coordination costs between humans. So if those are removed, economies of scale will go to infinity!’ But there may well be other trade offs that you reach at higher levels. For example, I’d expect that you lose out on things like creativity/innovation, and that you run higher risks of correlated failures, vulnerabilities etc.
Assuming it is true, it doesn’t seem like the most important argument within economic dominance to me:
The most natural way of thinking about it for me is that AGI increasing economies of scale is a subset of outgrowing the world (where the general class is ‘having better AI enables growing to most of the economy’, and the economies of scale sub-class is ‘doing that via using copies of literally the same AI, such that you get more economies of scale’
Put another way, I think the economies of scale thing only leads to extreme power concentration in combination with a big capabilities gap. If lots of people have similarly powerful AI systems, and can use them to figure out that they’d be best off by using a single system to do everything, then I don’t see any reason why one country would dominate. So it doesn’t seem like an independent route to me, it’s a particular form of a route that is causally driven by another factor.
Interested in your takes here!
I guess my prior coming into this is that non-existential catastrophes are still pretty existentially important, because:
they are bad in and of themselves
they are destabilising and make it more likely that we end up with existential catastrophes
I definitely wasn’t thinking explicitly about post-ASI catastrophes meaning we’d have to rerun the time of perils
But I was thinking about stuff like ‘a big war would probably set back AI development and could also make culture and selection pressures a fair bit worse, such that I feel worse about the outcome of AI development after that’. And similarly for bio
It sounds like your prior was that non-existential catastrophes are much much less important than existential ones, and then these considerations are a big update for you.
So I think part of why I’m less interested in this than you are is just having different priors where this update is fairly small/doesn’t change my prioritisation that much?