Giving a range of probabilities when you should give a probability + giving confidence intervals over probabilities + failing to realize that probabilities of probabilities just reduce to simple probabilities
When I do this, it’s because I’m unable or unwilling to assign a probability distribution over the probabilities, so it won’t reduce to simple (precise) probabilities. Actually, in general, I think precise probabilities are epistemically unjustified (e.g. Schoenfield, 2012, section 3), but I’m willing to use more or less precise probabilities depending on the circumstances.
Unstable beliefs about stuff like AI timelines in the sense of I’d be pretty likely to say something pretty different if you asked tomorrow
I’m not sure if I’d claim to have such unstable beliefs myself, but if you’re trying to be very precise with very speculative, subjective and hard-to-specifically-defend probabilities, then I’d imagine they could be very unstable, and influenced by things like your mood, e.g. optimism and pessimism bias. That is, unless you commit to your credences even if you’d had formed different ones if you had started from scratch or you make arbitrary choices in forming them that could easily have gone differently. You might weigh the same evidence or arguments differently from one day to the next.
I’d guess most people would also have had at least slightly different credences on AI timelines if they had seen the same evidence or arguments in a different order, or were in a different mood when they were forming their credences or building models, or for many other different reasons. Some number or parameter choices will come down to intuition, and intuition can be unstable.
fluctuating predictably (dutch-book-ably) is not
I don’t think people are fluctuating predictably (dutch-book-ably). How exactly they’d change their minds or even the direction is not known to them ahead of time.
(But maybe you could Dutch book people by predicting their moods and so optimism and pessimism bias?)
Some people say things like “my doom-credence fluctuates between 10% and 25% day to day”; this is dutch-book-able and they’d make better predictions if they reported what they feel like on average rather than what they feel like today, except insofar as they have new information.
This is dutch-book-able only if there is no bid-ask spread. A rational choice in this case would be to have a very wide bid-ask spread. E.g. when Holden Karnofsky writes that his P(doom) is between 10% and 90%, I assume he would bet for doom at 9% or less, bet against doom at 91% or more, and not bet for 0.11<p<0.89. This seems a very rational choice in a high-volatility situation where information changes extremely quickly. (As an example, IIRC the bid-ask spread in financial markets increases right before earnings are released).
(I agree it is reasonable to have a bid-ask spread when betting against capable adversaries. I think the statements-I-object-to are asserting something else, and the analogy to financial markets is mostly irrelevant. I don’t really want to get into this now.)
Hmm, okay. So, for example, when they’re below 15%, you bet that it will happen at odds matching 15% against them, and when they’re above 20%, you bet that it won’t happen at 20% against them. And just make sure to size the bets right so that if you lose one bet, your payoff is higher in the other, which you’d win. They “give up” the 15-20% range for free to you.
Still, maybe they just mean to report the historical range or volatility of their estimates? This would be like reporting the historical volatility of a stock. They may not intend to imply, say, that they’ll definitely fall below 15% at some point and above 20% at another.
Plus, picking one way to average may seem unjustifiably precise to them. The average over time is one way, but another is the average over relatively unique (clusters) of states of mind, e.g. splitting weight equally between good, ~neutral and bad moods, averages over possible sets of value assignments for various parameters. There are many different reasonable choices they can make, all pretty arbitrary.
When I do this, it’s because I’m unable or unwilling to assign a probability distribution over the probabilities, so it won’t reduce to simple (precise) probabilities. Actually, in general, I think precise probabilities are epistemically unjustified (e.g. Schoenfield, 2012, section 3), but I’m willing to use more or less precise probabilities depending on the circumstances.
I’m not sure if I’d claim to have such unstable beliefs myself, but if you’re trying to be very precise with very speculative, subjective and hard-to-specifically-defend probabilities, then I’d imagine they could be very unstable, and influenced by things like your mood, e.g. optimism and pessimism bias. That is, unless you commit to your credences even if you’d had formed different ones if you had started from scratch or you make arbitrary choices in forming them that could easily have gone differently. You might weigh the same evidence or arguments differently from one day to the next.
I’d guess most people would also have had at least slightly different credences on AI timelines if they had seen the same evidence or arguments in a different order, or were in a different mood when they were forming their credences or building models, or for many other different reasons. Some number or parameter choices will come down to intuition, and intuition can be unstable.
I don’t think people are fluctuating predictably (dutch-book-ably). How exactly they’d change their minds or even the direction is not known to them ahead of time.
(But maybe you could Dutch book people by predicting their moods and so optimism and pessimism bias?)
Thanks.
Some people say things like “my doom-credence fluctuates between 10% and 25% day to day”; this is dutch-book-able and they’d make better predictions if they reported what they feel like on average rather than what they feel like today, except insofar as they have new information.
This is dutch-book-able only if there is no bid-ask spread. A rational choice in this case would be to have a very wide bid-ask spread. E.g. when Holden Karnofsky writes that his P(doom) is between 10% and 90%, I assume he would bet for doom at 9% or less, bet against doom at 91% or more, and not bet for 0.11<p<0.89. This seems a very rational choice in a high-volatility situation where information changes extremely quickly. (As an example, IIRC the bid-ask spread in financial markets increases right before earnings are released).
(I agree it is reasonable to have a bid-ask spread when betting against capable adversaries. I think the statements-I-object-to are asserting something else, and the analogy to financial markets is mostly irrelevant. I don’t really want to get into this now.)
Hmm, okay. So, for example, when they’re below 15%, you bet that it will happen at odds matching 15% against them, and when they’re above 20%, you bet that it won’t happen at 20% against them. And just make sure to size the bets right so that if you lose one bet, your payoff is higher in the other, which you’d win. They “give up” the 15-20% range for free to you.
Still, maybe they just mean to report the historical range or volatility of their estimates? This would be like reporting the historical volatility of a stock. They may not intend to imply, say, that they’ll definitely fall below 15% at some point and above 20% at another.
Plus, picking one way to average may seem unjustifiably precise to them. The average over time is one way, but another is the average over relatively unique (clusters) of states of mind, e.g. splitting weight equally between good, ~neutral and bad moods, averages over possible sets of value assignments for various parameters. There are many different reasonable choices they can make, all pretty arbitrary.