I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why “time of Carols” is a poor analogy to “time of perils”:
“Time of carols” is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths “X-risk could briefly be high then very low” than “everyone is physically tied up and made to listen to carols”. To me it’s like comparing “there will be cars on Mars in 2120″ to “there will be a humanoid crate-stacking robot on Mars in 2120 that is nicknamed Carol”.
Actually, longtermists argue for the “current X-risk is high” claim, making Thorstad’s point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols.
(Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like “not have massive disasters while I am alive.” This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally don’t want humans to die. In comparison there’s little reason to think decent numbers of people will always desire time of carols.
4. Maybe this isn’t an independent point from 1., but I actually do think it is relevant that “time of carols” just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to people’s gut reactions here.
I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why “time of Carols” is a poor analogy to “time of perils”:
“Time of carols” is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths “X-risk could briefly be high then very low” than “everyone is physically tied up and made to listen to carols”. To me it’s like comparing “there will be cars on Mars in 2120″ to “there will be a humanoid crate-stacking robot on Mars in 2120 that is nicknamed Carol”.
Actually, longtermists argue for the “current X-risk is high” claim, making Thorstad’s point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols.
(Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like “not have massive disasters while I am alive.” This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally don’t want humans to die. In comparison there’s little reason to think decent numbers of people will always desire time of carols.
4. Maybe this isn’t an independent point from 1., but I actually do think it is relevant that “time of carols” just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to people’s gut reactions here.