It’s the same as with probabilities. How can probabilities be calibrated, given that they are fairly subjective? The LR can be calibrated the same way given that it’s just a function of two probabilities.
You can check probability estimates against outcomes. If you make 5 different predictions and estimate a 20% probability of each, then if you are well calibrated then you expect 1 out of the 5 to happen. If all of them happened, you probably made a mistake in your predictions. I don’t think this is perfect (it’s impractical to test very low probability predictions like 1 in a million), but there is at least some level of empiricism available.
There is no similar test for likliehood ratios. A question like “what is the chance that the chicago survey said minimum wages are fine if they actually aren’t” can’t be empirically tested.
It’s the same as with probabilities. How can probabilities be calibrated, given that they are fairly subjective? The LR can be calibrated the same way given that it’s just a function of two probabilities.
You can check probability estimates against outcomes. If you make 5 different predictions and estimate a 20% probability of each, then if you are well calibrated then you expect 1 out of the 5 to happen. If all of them happened, you probably made a mistake in your predictions. I don’t think this is perfect (it’s impractical to test very low probability predictions like 1 in a million), but there is at least some level of empiricism available.
There is no similar test for likliehood ratios. A question like “what is the chance that the chicago survey said minimum wages are fine if they actually aren’t” can’t be empirically tested.