I was curious about the origins of this concept in the EA community since I think it’s correct, insightful, and I personally had first noticed it in conversation among people at Open Phil. On Twitter, @alter_ego_42 pointed out the existence of the Credal Resilience page in the “EA concepts” section of this website. That page cites
which is the earliest thorough academic reference to this idea that I know of. With apologies to Greg, this seems like the appropriate place to post a couple comments on that paper so others don’t have to trudge through it.
I didn’t find Skyrms’s critique of frequentism at the beginning, or his pseudo-formalization of resilency on page 705 (see for instance the criticism “Some Remarks on the Concept of Resiliency” by Patrick Suppes in the very next article, pages 713-714), to be very insightful, so I recommend the time-pressed reader concentrate on
The bottom of p. 705 (“The concept of probabilistic resiliency is nicely illustrated...”) to the top of p. 708 (”… well confirmed to its degree of instantial resiliency, as specified above..”).
The middle of p. 712 (“The concept of resiliency has connections with...”) to p. 713 (the end).
Skyrms quotes Savage (1954) as musing about the possibility of introducing “second-order probabilities”. This is grounded in a relative-frequency intuition: when I say that there is a (first-order) probability p of X occurring but that I am uncertain, what I really mean is something like that there is some objective physical process that generates X with (second-order) probability q, but I am uncertain about the details of that process (i.e., about what q is), so my value of p is obtained by integrating over some pdf f (q).
There is, naturally, a Bayesian version of the same idea: We shouldn’t concern ourselves with a hypothetical giant (second-order) ensemble of models, each of which generates a hypothetical (first-order) ensemble of individual trials. Resilience about probabilities is best measured by our bets on how future evidence would change those probabilities, just as probabilities is best measured by our bets on future outcomes.
(Unfortunately, and unlike the case for standard credences, there seems to be multiple possible formulations depending on which sorts of evidence we are supposing: what I expect to learn in the actual future, what I could learn if I thought about it hard, what a superforecaster would say in my shoes, etc.)
(In a similar spirit of posting things somewhat related to this general topic while apologising to Greg for doing so...)
A few months ago, I collected on LessWrong a variety of terms I’d found for describing something like the “trustworthiness” of probabilities, along with quotes and commentary about those terms. Specifically, the terms included:
Epistemic credentials
Resilience (of credences)
Evidential weight (balance vs weight of evidence)
Probability distributions (and confidence intervals)
Precision, sharpness, vagueness
Haziness
Hyperpriors, credal sets, and other things I haven’t really learned about
It’s possible that some readers of this post would find that collection interesting/useful.
To add to your list—Subjective Logic represents opinions with three values: degree of belief, degree of disbelief, and degree of uncertainty. One interpretation of this is as a form of second-order uncertainty. It’s used for modelling trust. A nice summary here with interactive tools for visualising opinions and a trust network.
I was curious about the origins of this concept in the EA community since I think it’s correct, insightful, and I personally had first noticed it in conversation among people at Open Phil. On Twitter, @alter_ego_42 pointed out the existence of the Credal Resilience page in the “EA concepts” section of this website. That page cites
Skyrms, Brian. 1977. Resiliency, propensities, and causal necessity. The journal of philosophy 74(11): 704-713. [PDF]
which is the earliest thorough academic reference to this idea that I know of. With apologies to Greg, this seems like the appropriate place to post a couple comments on that paper so others don’t have to trudge through it.
I didn’t find Skyrms’s critique of frequentism at the beginning, or his pseudo-formalization of resilency on page 705 (see for instance the criticism “Some Remarks on the Concept of Resiliency” by Patrick Suppes in the very next article, pages 713-714), to be very insightful, so I recommend the time-pressed reader concentrate on
The bottom of p. 705 (“The concept of probabilistic resiliency is nicely illustrated...”) to the top of p. 708 (”… well confirmed to its degree of instantial resiliency, as specified above..”).
The middle of p. 712 (“The concept of resiliency has connections with...”) to p. 713 (the end).
Skyrms quotes Savage (1954) as musing about the possibility of introducing “second-order probabilities”. This is grounded in a relative-frequency intuition: when I say that there is a (first-order) probability p of X occurring but that I am uncertain, what I really mean is something like that there is some objective physical process that generates X with (second-order) probability q, but I am uncertain about the details of that process (i.e., about what q is), so my value of p is obtained by integrating over some pdf f (q).
There is, naturally, a Bayesian version of the same idea: We shouldn’t concern ourselves with a hypothetical giant (second-order) ensemble of models, each of which generates a hypothetical (first-order) ensemble of individual trials. Resilience about probabilities is best measured by our bets on how future evidence would change those probabilities, just as probabilities is best measured by our bets on future outcomes.
(Unfortunately, and unlike the case for standard credences, there seems to be multiple possible formulations depending on which sorts of evidence we are supposing: what I expect to learn in the actual future, what I could learn if I thought about it hard, what a superforecaster would say in my shoes, etc.)
(In a similar spirit of posting things somewhat related to this general topic while apologising to Greg for doing so...)
A few months ago, I collected on LessWrong a variety of terms I’d found for describing something like the “trustworthiness” of probabilities, along with quotes and commentary about those terms. Specifically, the terms included:
Epistemic credentials
Resilience (of credences)
Evidential weight (balance vs weight of evidence)
Probability distributions (and confidence intervals)
Precision, sharpness, vagueness
Haziness
Hyperpriors, credal sets, and other things I haven’t really learned about
It’s possible that some readers of this post would find that collection interesting/useful.
To add to your list—Subjective Logic represents opinions with three values: degree of belief, degree of disbelief, and degree of uncertainty. One interpretation of this is as a form of second-order uncertainty. It’s used for modelling trust. A nice summary here with interactive tools for visualising opinions and a trust network.