Sorry about the delayed reply, I saw this and accidentally removed the notification (and I guess didn’t receive an email notification, contrary to my expectations) but forgot to reply. Responding to some of your points/questions:
One can note that AIXR is definitely falsifiable, the hard part is falsifying it and staying alive.
I mostly agree with the sentiment that “if someone predicts AIXR and is right then they may not be alive”, although I do now think it’s entirely plausible that we could survive long enough during a hypothetical AI takeover to say “ah yeah, we’re almost certainly headed for extinction”—it’s just too late to do anything about it. The problem is how to define “falsify”: if you can’t 100% prove anything, you can’t 100% falsify anything; can the last person alive say with 100% confidence “yep, we’re about to go extinct?” No, but I think most people would say that this outcome basically “falsifies” the claim “there is no AIXR,” even prior to the final person being killed.
Knightian uncertainty makes more sense in some restricted scenarios especially related to self-confirming/self-denying predictions.
This is interesting; I had not previously considered the interaction between self-affecting predictions and (Knightian) “uncertainty.” I’ll have to think more about this, but as you say I do still think Knightian uncertainty (as I was taught it) does not make much sense.
This can apply as well to different people: If I believe that X has a very good reasoning process based on observations on Xs past reasoning, I might not want to/have to follow Xs entire train of thought before raising my probability of their conclusion.
Yes, this is the point I’m trying to get at with forecast legibility, although I’m a bit confused about how it builds on the previous sentence.
Some people have talked about probability distributions on probability distributions, in the case of a binary forecast that would be a function f:[0;1]→([0;1]→{0,1}), which is…weird. Do I need to tack on the resilience to the distribution? Do I compute it out of the probability distribution on probability distributions? Perhaps the people talking about imprecise probabilities/infrabayesianism are onto something when they talk about convex sets of probability distributions as the correct objects instead of probability distributions per se.
Unfortunately I’m not sure I understand this paragraph (including the mathematical portion). Thus, I’m not sure how to explain my view of resilience better than what I’ve already written and the summary illustration: someone who says “my best estimate is currently 50%, but within 30 minutes I think there is a 50% chance that my best estimate will become 75% and a 50% chance that my best estimate becomes 25%” has a less-resilient belief compared to someone who says “my best estimate is currently 50%, and I do not think that will change within 30 minutes.” I don’t know how to calculate/quantify the level of resilience between the two, but we can obviously see there is a difference.
Sorry about the delayed reply, I saw this and accidentally removed the notification (and I guess didn’t receive an email notification, contrary to my expectations) but forgot to reply. Responding to some of your points/questions:
I mostly agree with the sentiment that “if someone predicts AIXR and is right then they may not be alive”, although I do now think it’s entirely plausible that we could survive long enough during a hypothetical AI takeover to say “ah yeah, we’re almost certainly headed for extinction”—it’s just too late to do anything about it. The problem is how to define “falsify”: if you can’t 100% prove anything, you can’t 100% falsify anything; can the last person alive say with 100% confidence “yep, we’re about to go extinct?” No, but I think most people would say that this outcome basically “falsifies” the claim “there is no AIXR,” even prior to the final person being killed.
This is interesting; I had not previously considered the interaction between self-affecting predictions and (Knightian) “uncertainty.” I’ll have to think more about this, but as you say I do still think Knightian uncertainty (as I was taught it) does not make much sense.
Yes, this is the point I’m trying to get at with forecast legibility, although I’m a bit confused about how it builds on the previous sentence.
Unfortunately I’m not sure I understand this paragraph (including the mathematical portion). Thus, I’m not sure how to explain my view of resilience better than what I’ve already written and the summary illustration: someone who says “my best estimate is currently 50%, but within 30 minutes I think there is a 50% chance that my best estimate will become 75% and a 50% chance that my best estimate becomes 25%” has a less-resilient belief compared to someone who says “my best estimate is currently 50%, and I do not think that will change within 30 minutes.” I don’t know how to calculate/quantify the level of resilience between the two, but we can obviously see there is a difference.