I should like to clarify that I also support FRI’s approach to reducing AI s-risks. The issue is more how big a fraction of our resources approaches of this kind deserve relative to other things. My view is that, relatively speaking, we very much underinvest in addressing other risks, by which I roughly mean “risks not stemming primarily from FOOM or sub-optimally written software” (which can still involve AI plenty, of course). I would like to see a greater investment in broad explorative research on s-risk scenarios and how we can reduce them.
In terms of explaining the (IMO) skewed focus, it seems to me that we mostly think about AI futures in far mode, see https://www.overcomingbias.com/2010/06/near-far-summary.html and https://www.overcomingbias.com/2010/10/the-future-seems-shiny.html. The perhaps most significant way in which this shows is that we intuitively think the future will be determined by a single or a few agents and what they want, as opposed to countless different agents, cooperating and competing with many (for those future agents) non-intentional factors influencing the outcomes.
I’d argue scenarios of the latter kind are far more likely given not just the history of life and civilization, but also in light of general models of complex systems and innovation (variation and specialization seem essential, and the way these play out is unlikely to conform to a singular will in anything like the neat way far mode would portray it). Indeed, I believe such a scenario would be most likely to emerge even if a single universal AI ancestor took over and copied itself (specialization would be adaptive, and significant uncertainty about the exact information and (sub-)aims possessed by conspecifics would emerge).
In short, I think we place too much weight on simplistic toy models of the future, in turn neglecting scenarios that don’t conform neatly to these, and the ways these could come about.
as opposed to countless different agents, cooperating and competing with many (for those future agents) non-intentional factors influencing the outcomes.
I think there are good reasons to think this isn’t likely, aside from the possibility of FOOM:
Interesting posts. Yet I don’t see how they support that what I described is unlikely. In particular, I don’t see how “easy coordination” is in tension with what I wrote.
To clarify, competition that determines outcomes can readily happen within a framework of shared goals, and as instrumental to some overarching final goal. If the final goal is, say, to maximize economic growth (or if that is an important instrumental goal), this would likely lead to specialization and competition among various agents that try out different things, and which, by the nature of specialization, have imperfect information about what other agents know (not having such specialization would be much less efficient). In this, a future AI economy would resemble ours more than far-mode thinking suggests (this does not necessarily contradict your claim about easier coordination, though).
A reason I consider what I described likely is not least that I find it more likely that future software systems will consist in a multitude of specialized systems with quite different designs, even in the presence of AGI, as opposed to most everything being done by copies of some singular AGI system. This “one system will take over everything” strikes me as far-mode thinking, and not least unlikely given the history of technology and economic growth. I’ve outlined my view on this in the following e-book (though it’s a bit dated in some ways): https://www.smashwords.com/books/view/655938 (short summary and review by Kaj Sotala: https://kajsotala.fi/2017/01/disjunctive-ai-scenarios-individual-or-collective-takeoff/)
A reason I consider what I described likely is not least that I find it more likely that future software systems will consist in a multitude of specialized systems with quite different designs, even in the presence of AGI, as opposed to most everything being done by copies of some singular AGI system.
Can you explain why this is relevant to how much effort we should put into AI alignment research today?
Thanks for sharing and for the kind words. :-)
I should like to clarify that I also support FRI’s approach to reducing AI s-risks. The issue is more how big a fraction of our resources approaches of this kind deserve relative to other things. My view is that, relatively speaking, we very much underinvest in addressing other risks, by which I roughly mean “risks not stemming primarily from FOOM or sub-optimally written software” (which can still involve AI plenty, of course). I would like to see a greater investment in broad explorative research on s-risk scenarios and how we can reduce them.
In terms of explaining the (IMO) skewed focus, it seems to me that we mostly think about AI futures in far mode, see https://www.overcomingbias.com/2010/06/near-far-summary.html and https://www.overcomingbias.com/2010/10/the-future-seems-shiny.html. The perhaps most significant way in which this shows is that we intuitively think the future will be determined by a single or a few agents and what they want, as opposed to countless different agents, cooperating and competing with many (for those future agents) non-intentional factors influencing the outcomes.
I’d argue scenarios of the latter kind are far more likely given not just the history of life and civilization, but also in light of general models of complex systems and innovation (variation and specialization seem essential, and the way these play out is unlikely to conform to a singular will in anything like the neat way far mode would portray it). Indeed, I believe such a scenario would be most likely to emerge even if a single universal AI ancestor took over and copied itself (specialization would be adaptive, and significant uncertainty about the exact information and (sub-)aims possessed by conspecifics would emerge).
In short, I think we place too much weight on simplistic toy models of the future, in turn neglecting scenarios that don’t conform neatly to these, and the ways these could come about.
I think there are good reasons to think this isn’t likely, aside from the possibility of FOOM:
Strategic implications of AIs’ ability to coordinate at low cost, for example by merging
AGI will drastically increase economies of scale
Interesting posts. Yet I don’t see how they support that what I described is unlikely. In particular, I don’t see how “easy coordination” is in tension with what I wrote.
To clarify, competition that determines outcomes can readily happen within a framework of shared goals, and as instrumental to some overarching final goal. If the final goal is, say, to maximize economic growth (or if that is an important instrumental goal), this would likely lead to specialization and competition among various agents that try out different things, and which, by the nature of specialization, have imperfect information about what other agents know (not having such specialization would be much less efficient). In this, a future AI economy would resemble ours more than far-mode thinking suggests (this does not necessarily contradict your claim about easier coordination, though).
A reason I consider what I described likely is not least that I find it more likely that future software systems will consist in a multitude of specialized systems with quite different designs, even in the presence of AGI, as opposed to most everything being done by copies of some singular AGI system. This “one system will take over everything” strikes me as far-mode thinking, and not least unlikely given the history of technology and economic growth. I’ve outlined my view on this in the following e-book (though it’s a bit dated in some ways): https://www.smashwords.com/books/view/655938 (short summary and review by Kaj Sotala: https://kajsotala.fi/2017/01/disjunctive-ai-scenarios-individual-or-collective-takeoff/)
Can you explain why this is relevant to how much effort we should put into AI alignment research today?
In brief: the less of a determinant specific AGI structure is of future outcomes, the less relevant/worthy of investment it is.