1. I interpreted the original claimāāwouldnāt you expectāāas being basically one in which observation X was evidence against hypothesis Y. Not conclusive evidence, just an update. I didnāt interpret it as āruling things outā (in a strong way) or saying that there arenāt any unknown risks without common mini-versions (just that itās less likely that there are than one would otherwise think). Note that his point seemed to be in defence of āOrdās estimates seem too high to meā, rather than āthe risks are 0ā³.
2. I do think that Tobiasās point, even interpreted that way, was probably too strong, or missing a key detail, in that the key sources of risks are probably emerging or new things, so we wouldnāt expect to have observed their mini-versions yet. Though I do tentatively think Iād expect to see mini-versions before the āfull thingā, once the new things do start arising. (Iām aware this is all pretty hand-wavey phrasing.)
3i. As I went into more in my other comment, I think the general expectation that weāll expect to see very small versions before and more often than small ones, which we expect to see before and more often than medium, which we expect to see before and more often than large, etc., probably wouldāve served well in the past. There was progressively more advanced tech before AI, and AI is progressively advancing more. There were progressively more advanced weapons, progressively more destructive wars, progressively larger numbers of nukes, etc. Iād guess the biggest pandemics and asteroid strikes werenāt the first, because the biggest are rare.
3ii. AI is the least clear of those examples, because:
(a) it seems like destruction from AI so far has been very minimal (handful of fatalities from driverless cars, the āflash crashā, etc.), yet it seems plausible major destruction could occur in future
(b) we do have specific arguments, though of somewhat unclear strength, that the same AI might actively avoid causing any destruction for a while, and then suddenly seize decisive strategic advantage etc.
But on (a), I do think most relevant researchers would say the risk this month from AI is extremely low; the risks will rise in future as systems become more capable. So thereās still time in which we may see mini-versions.
And on (b), Iād consider that a case where a specific argument updates us away from a generally pretty handy prior that weāll see small things earlier and more often than extremely large things. And we also donāt yet have super strong reason to believe that those arguments are really painting the right picture, as far as Iām aware.
3iii. I think if we interpreted Tobiasās point as something like āWeāll never see anything thatās unlike the pastā, then yes, of course thatās ridiculous. So as I mentioned elsewhere, I think it partly depends on how we carve up reality, how we define things, etc. E.g., do we put nukes in a totally new bucket, or consider it a part of trends in weaponry/āwarfare/āexplosives?
But in any case, my interpretation of Tobiasās point, where itās just about it being unlikely to see extreme things before smaller versions, would seem to work with e.g. nukes, even if we put them in their own special categoryāweād be surprise by the first nuke, but weād indeed see thereās one nuke before there are thousands, and there are two detonations on cities before thereās a full-scale nuclear war (if there ever is one, which hopefully and plausibly there wonāt be).
In general I think youāve thought this through more carefully than me so without having read all your points Iām just gonna agree with you.
So yeah, I think the main problem with Tobiasā original point was that unknown risks are probably mostly new things that havenāt arisen yet and thus the lack of observed mini-versions of them is no evidence against them. But I still think itās also true that some risks just donāt have mini-versions, or rather are as likely or more likely to have big versions than mini-versions. I agree that most risks are not like this, including some of the examples I reached for initially.
1. I interpreted the original claimāāwouldnāt you expectāāas being basically one in which observation X was evidence against hypothesis Y. Not conclusive evidence, just an update. I didnāt interpret it as āruling things outā (in a strong way) or saying that there arenāt any unknown risks without common mini-versions (just that itās less likely that there are than one would otherwise think). Note that his point seemed to be in defence of āOrdās estimates seem too high to meā, rather than āthe risks are 0ā³.
2. I do think that Tobiasās point, even interpreted that way, was probably too strong, or missing a key detail, in that the key sources of risks are probably emerging or new things, so we wouldnāt expect to have observed their mini-versions yet. Though I do tentatively think Iād expect to see mini-versions before the āfull thingā, once the new things do start arising. (Iām aware this is all pretty hand-wavey phrasing.)
3i. As I went into more in my other comment, I think the general expectation that weāll expect to see very small versions before and more often than small ones, which we expect to see before and more often than medium, which we expect to see before and more often than large, etc., probably wouldāve served well in the past. There was progressively more advanced tech before AI, and AI is progressively advancing more. There were progressively more advanced weapons, progressively more destructive wars, progressively larger numbers of nukes, etc. Iād guess the biggest pandemics and asteroid strikes werenāt the first, because the biggest are rare.
3ii. AI is the least clear of those examples, because:
(a) it seems like destruction from AI so far has been very minimal (handful of fatalities from driverless cars, the āflash crashā, etc.), yet it seems plausible major destruction could occur in future
(b) we do have specific arguments, though of somewhat unclear strength, that the same AI might actively avoid causing any destruction for a while, and then suddenly seize decisive strategic advantage etc.
But on (a), I do think most relevant researchers would say the risk this month from AI is extremely low; the risks will rise in future as systems become more capable. So thereās still time in which we may see mini-versions.
And on (b), Iād consider that a case where a specific argument updates us away from a generally pretty handy prior that weāll see small things earlier and more often than extremely large things. And we also donāt yet have super strong reason to believe that those arguments are really painting the right picture, as far as Iām aware.
3iii. I think if we interpreted Tobiasās point as something like āWeāll never see anything thatās unlike the pastā, then yes, of course thatās ridiculous. So as I mentioned elsewhere, I think it partly depends on how we carve up reality, how we define things, etc. E.g., do we put nukes in a totally new bucket, or consider it a part of trends in weaponry/āwarfare/āexplosives?
But in any case, my interpretation of Tobiasās point, where itās just about it being unlikely to see extreme things before smaller versions, would seem to work with e.g. nukes, even if we put them in their own special categoryāweād be surprise by the first nuke, but weād indeed see thereās one nuke before there are thousands, and there are two detonations on cities before thereās a full-scale nuclear war (if there ever is one, which hopefully and plausibly there wonāt be).
In general I think youāve thought this through more carefully than me so without having read all your points Iām just gonna agree with you.
So yeah, I think the main problem with Tobiasā original point was that unknown risks are probably mostly new things that havenāt arisen yet and thus the lack of observed mini-versions of them is no evidence against them. But I still think itās also true that some risks just donāt have mini-versions, or rather are as likely or more likely to have big versions than mini-versions. I agree that most risks are not like this, including some of the examples I reached for initially.