Thanks for sharing the confidence intervals. I guess it might be reasonable to conclude from your experience that the interview scores have not been informative enough to justify their cost.
What I am saying is that it doesn’t seem (to me) that the data and evidence presented allows you to say that. (But maybe other analysis or inference from your experience might in fact drive that conclusion, the ‘other people in San Francisco’ in your example.)
But if I glance at just the evidence/confidence intervals it suggests to me that there may be a substantial probability that in fact there is a strongly positive relationship and the results are a fluke.
On the other hand I might be wrong. I hope to get a chance to follow up on this:
We could simulate a case where the measure has ‘the minimum correlation to the outcome to make it worth using for selecting on’, and see how likely it would be, in such a case, to observe the correlations as low as you observed
Or we could start with a minimally informative ‘prior’ over our beliefs about the measure, and do a Bayesian updating exercise in light of your observations; we could then consider the posterior probability distribution and consider whether it might justify discontinuing the use of these scores
Thanks for sharing the confidence intervals. I guess it might be reasonable to conclude from your experience that the interview scores have not been informative enough to justify their cost.
What I am saying is that it doesn’t seem (to me) that the data and evidence presented allows you to say that. (But maybe other analysis or inference from your experience might in fact drive that conclusion, the ‘other people in San Francisco’ in your example.)
But if I glance at just the evidence/confidence intervals it suggests to me that there may be a substantial probability that in fact there is a strongly positive relationship and the results are a fluke.
On the other hand I might be wrong. I hope to get a chance to follow up on this:
We could simulate a case where the measure has ‘the minimum correlation to the outcome to make it worth using for selecting on’, and see how likely it would be, in such a case, to observe the correlations as low as you observed
Or we could start with a minimally informative ‘prior’ over our beliefs about the measure, and do a Bayesian updating exercise in light of your observations; we could then consider the posterior probability distribution and consider whether it might justify discontinuing the use of these scores