Followup question: it seems like these likelihood ratios are fairly subjective. (like, why is the LR for the chicago survey 5:1 and not 10:1 or 20:1? How can you calibrate the likelihood ratio when there is no “right answer”?
It’s the same as with probabilities. How can probabilities be calibrated, given that they are fairly subjective? The LR can be calibrated the same way given that it’s just a function of two probabilities.
You can check probability estimates against outcomes. If you make 5 different predictions and estimate a 20% probability of each, then if you are well calibrated then you expect 1 out of the 5 to happen. If all of them happened, you probably made a mistake in your predictions. I don’t think this is perfect (it’s impractical to test very low probability predictions like 1 in a million), but there is at least some level of empiricism available.
There is no similar test for likliehood ratios. A question like “what is the chance that the chicago survey said minimum wages are fine if they actually aren’t” can’t be empirically tested.
Yeah. Here’s the example in more detail:
Prior odds: 1:1
Theoretical arguments that minimum wages increase unemployment, LR = 1:3 → posterior odds 1:3
Someone sends an empirical paper and the abstract says it improved the situation, LR = 1.2:1 → posterior odds 1.2:3
IGM Chicago Survey results, LR = 5:1 → posterior odds 6:3 (or 2:1)
Ah yes, thank you, that clear it up.
Followup question: it seems like these likelihood ratios are fairly subjective. (like, why is the LR for the chicago survey 5:1 and not 10:1 or 20:1? How can you calibrate the likelihood ratio when there is no “right answer”?
It’s the same as with probabilities. How can probabilities be calibrated, given that they are fairly subjective? The LR can be calibrated the same way given that it’s just a function of two probabilities.
You can check probability estimates against outcomes. If you make 5 different predictions and estimate a 20% probability of each, then if you are well calibrated then you expect 1 out of the 5 to happen. If all of them happened, you probably made a mistake in your predictions. I don’t think this is perfect (it’s impractical to test very low probability predictions like 1 in a million), but there is at least some level of empiricism available.
There is no similar test for likliehood ratios. A question like “what is the chance that the chicago survey said minimum wages are fine if they actually aren’t” can’t be empirically tested.