This is a post that rings in my heart, thank you so much. I think people very often conflate these concepts, and I also think we’re in a very complex space here (especially if you look at Bayesianism from below and see that it grinds on reality/boundedness pretty hard and produces some awful screeching sounds while applied in the world).
I agree that these concepts you’ve presented are at least antidotes to common confusions about forecasts, and I have some more things to say.
I feel confused about credal resilience:
The example you state appears correct, but I don’t know how that would look like as a mathematical object. Some people have talked about probability distributions on probability distributions, in the case of a binary forecast that would be a function f:[0;1]→([0;1]→{0,1}), which is…weird. Do I need to tack on the resilience to the distribution? Do I compute it out of the probability distribution on probability distributions? Perhaps the people talking about imprecise probabilities/infrabayesianism are onto something when they talk about convex sets of probability distributions as the correct objects instead of probability distributions per se.
One can note that AIXR is definitely falsifiable, the hard part is falsifying it and staying alive.
There will be a state of the world confirming or denying the outcome, there’s just a correlation between our ability to observe those outcomes and the outcomes themselves.
Knightian uncertainty makes more sense in some restricted scenarios especially related to self-confirming/self-denying predictions. If one can read the brain state of a human and construct their predictions of the environment out of that, then one can construct an environment where the human has Knightian uncertainty by constructing outcomes that the human assigned the smallest probability to. (Even a uniform belief gets fooled: We’ll pick one option and make that happen many times in a row, but as soon as our poor subject starts predicting that outcome we shift to the ones less likely in their belief).
It need not be such a fanciful scenario: It could be that my buddy James made a very strong prediction that he will finish cleaning his car by noon, so he is too confident and procrastinates until the bell tolls for him. (Or the other way around, where his high confidence makes him more likely to finish the cleaning early, in that case we’d call it Knightian certainty).
This is a very different case than the one people normally state when talking about Knightian uncertainty, but an (imho) much more defensible one. I agree that the common reasons named for Knightian uncertainty are bad.
Another common complaint I’ve heard is about forecasts with very wide distributions, a case which evoked especially strong reactions was the Cotra bio-anchors report with (iirc) non-negligible probabilities on 12 orders of magnitude. Some people apparently consider such models worse than useless, harkening back to forecast legibility. Apparently both very wide and very narrow distributions are socially punished, even though having a bad model allows for updating & refinement.
Another point touched on very shortly in the post is on forecast precision. We usually don’t report forecasts with six or seven digits of precision, because at that level our forecasts are basically noise. But I believe that some of the common objections are about (perceived) undue precision; someone who reports 7 digits of precision is scammy, so reporting about 2 digits of precision is… fishy. Perhaps. I know there’s a Tetlock paper on the value of precision in geopolitical forecasting, but it uses a method of rounding to probabilities instead of odds or log-odds. (Approaches based on noising probabilities and then tracking score development do not work—I’m not sure why, though.). It would be cool to know more about precision in forecasting and how it relates to other dimensions.
I think that also probabilities reported by humans are weird because we do not have the entire space of hypothesis in our mind at once, and instead can shift our probabilities during reflection (without receiving evidence). This can apply as well to different people: If I believe that X has a very good reasoning process based on observations on Xs past reasoning, I might not want to/have to follow Xs entire train of thought before raising my probability of their conclusion.
Sorry about the long comment without any links, I’m currently writing this offline and don’t have my text notes file with me. I can supply more links if that sounds interesting/relevant.
Sorry about the delayed reply, I saw this and accidentally removed the notification (and I guess didn’t receive an email notification, contrary to my expectations) but forgot to reply. Responding to some of your points/questions:
One can note that AIXR is definitely falsifiable, the hard part is falsifying it and staying alive.
I mostly agree with the sentiment that “if someone predicts AIXR and is right then they may not be alive”, although I do now think it’s entirely plausible that we could survive long enough during a hypothetical AI takeover to say “ah yeah, we’re almost certainly headed for extinction”—it’s just too late to do anything about it. The problem is how to define “falsify”: if you can’t 100% prove anything, you can’t 100% falsify anything; can the last person alive say with 100% confidence “yep, we’re about to go extinct?” No, but I think most people would say that this outcome basically “falsifies” the claim “there is no AIXR,” even prior to the final person being killed.
Knightian uncertainty makes more sense in some restricted scenarios especially related to self-confirming/self-denying predictions.
This is interesting; I had not previously considered the interaction between self-affecting predictions and (Knightian) “uncertainty.” I’ll have to think more about this, but as you say I do still think Knightian uncertainty (as I was taught it) does not make much sense.
This can apply as well to different people: If I believe that X has a very good reasoning process based on observations on Xs past reasoning, I might not want to/have to follow Xs entire train of thought before raising my probability of their conclusion.
Yes, this is the point I’m trying to get at with forecast legibility, although I’m a bit confused about how it builds on the previous sentence.
Some people have talked about probability distributions on probability distributions, in the case of a binary forecast that would be a function f:[0;1]→([0;1]→{0,1}), which is…weird. Do I need to tack on the resilience to the distribution? Do I compute it out of the probability distribution on probability distributions? Perhaps the people talking about imprecise probabilities/infrabayesianism are onto something when they talk about convex sets of probability distributions as the correct objects instead of probability distributions per se.
Unfortunately I’m not sure I understand this paragraph (including the mathematical portion). Thus, I’m not sure how to explain my view of resilience better than what I’ve already written and the summary illustration: someone who says “my best estimate is currently 50%, but within 30 minutes I think there is a 50% chance that my best estimate will become 75% and a 50% chance that my best estimate becomes 25%” has a less-resilient belief compared to someone who says “my best estimate is currently 50%, and I do not think that will change within 30 minutes.” I don’t know how to calculate/quantify the level of resilience between the two, but we can obviously see there is a difference.
This is a post that rings in my heart, thank you so much. I think people very often conflate these concepts, and I also think we’re in a very complex space here (especially if you look at Bayesianism from below and see that it grinds on reality/boundedness pretty hard and produces some awful screeching sounds while applied in the world).
I agree that these concepts you’ve presented are at least antidotes to common confusions about forecasts, and I have some more things to say.
I feel confused about credal resilience:
The example you state appears correct, but I don’t know how that would look like as a mathematical object. Some people have talked about probability distributions on probability distributions, in the case of a binary forecast that would be a function f:[0;1]→([0;1]→{0,1}), which is…weird. Do I need to tack on the resilience to the distribution? Do I compute it out of the probability distribution on probability distributions? Perhaps the people talking about imprecise probabilities/infrabayesianism are onto something when they talk about convex sets of probability distributions as the correct objects instead of probability distributions per se.
One can note that AIXR is definitely falsifiable, the hard part is falsifying it and staying alive.
There will be a state of the world confirming or denying the outcome, there’s just a correlation between our ability to observe those outcomes and the outcomes themselves.
Knightian uncertainty makes more sense in some restricted scenarios especially related to self-confirming/self-denying predictions. If one can read the brain state of a human and construct their predictions of the environment out of that, then one can construct an environment where the human has Knightian uncertainty by constructing outcomes that the human assigned the smallest probability to. (Even a uniform belief gets fooled: We’ll pick one option and make that happen many times in a row, but as soon as our poor subject starts predicting that outcome we shift to the ones less likely in their belief).
It need not be such a fanciful scenario: It could be that my buddy James made a very strong prediction that he will finish cleaning his car by noon, so he is too confident and procrastinates until the bell tolls for him. (Or the other way around, where his high confidence makes him more likely to finish the cleaning early, in that case we’d call it Knightian certainty).
This is a very different case than the one people normally state when talking about Knightian uncertainty, but an (imho) much more defensible one. I agree that the common reasons named for Knightian uncertainty are bad.
Another common complaint I’ve heard is about forecasts with very wide distributions, a case which evoked especially strong reactions was the Cotra bio-anchors report with (iirc) non-negligible probabilities on 12 orders of magnitude. Some people apparently consider such models worse than useless, harkening back to forecast legibility. Apparently both very wide and very narrow distributions are socially punished, even though having a bad model allows for updating & refinement.
Another point touched on very shortly in the post is on forecast precision. We usually don’t report forecasts with six or seven digits of precision, because at that level our forecasts are basically noise. But I believe that some of the common objections are about (perceived) undue precision; someone who reports 7 digits of precision is scammy, so reporting about 2 digits of precision is… fishy. Perhaps. I know there’s a Tetlock paper on the value of precision in geopolitical forecasting, but it uses a method of rounding to probabilities instead of odds or log-odds. (Approaches based on noising probabilities and then tracking score development do not work—I’m not sure why, though.). It would be cool to know more about precision in forecasting and how it relates to other dimensions.
I think that also probabilities reported by humans are weird because we do not have the entire space of hypothesis in our mind at once, and instead can shift our probabilities during reflection (without receiving evidence). This can apply as well to different people: If I believe that X has a very good reasoning process based on observations on Xs past reasoning, I might not want to/have to follow Xs entire train of thought before raising my probability of their conclusion.
Sorry about the long comment without any links, I’m currently writing this offline and don’t have my text notes file with me. I can supply more links if that sounds interesting/relevant.
Sorry about the delayed reply, I saw this and accidentally removed the notification (and I guess didn’t receive an email notification, contrary to my expectations) but forgot to reply. Responding to some of your points/questions:
I mostly agree with the sentiment that “if someone predicts AIXR and is right then they may not be alive”, although I do now think it’s entirely plausible that we could survive long enough during a hypothetical AI takeover to say “ah yeah, we’re almost certainly headed for extinction”—it’s just too late to do anything about it. The problem is how to define “falsify”: if you can’t 100% prove anything, you can’t 100% falsify anything; can the last person alive say with 100% confidence “yep, we’re about to go extinct?” No, but I think most people would say that this outcome basically “falsifies” the claim “there is no AIXR,” even prior to the final person being killed.
This is interesting; I had not previously considered the interaction between self-affecting predictions and (Knightian) “uncertainty.” I’ll have to think more about this, but as you say I do still think Knightian uncertainty (as I was taught it) does not make much sense.
Yes, this is the point I’m trying to get at with forecast legibility, although I’m a bit confused about how it builds on the previous sentence.
Unfortunately I’m not sure I understand this paragraph (including the mathematical portion). Thus, I’m not sure how to explain my view of resilience better than what I’ve already written and the summary illustration: someone who says “my best estimate is currently 50%, but within 30 minutes I think there is a 50% chance that my best estimate will become 75% and a 50% chance that my best estimate becomes 25%” has a less-resilient belief compared to someone who says “my best estimate is currently 50%, and I do not think that will change within 30 minutes.” I don’t know how to calculate/quantify the level of resilience between the two, but we can obviously see there is a difference.