Even though only the people who did best on the initial application were invited to interviews, there was some positive correlation between initial application and interview scores. If this correlation was very strong, that would be some reason not to do an interview at all and just select people based on their application. This was not the case: the interview changed our ordering of applicants considerably.
You might want to look into range restriction. You can dive deep into via Range Restriction in employment interviews: An influence too big to ignore. But if you just want the simple explanation: correlation is artificially lowered when you only sample the people that pass the initial screen, making the interview appear less effective than it really is. So it is possible (likely?) that you could reasonably not do an interview at all.
Can I check my understanding here? It sounds like:
In a hiring process, later screens are less correlated with applicant future performance than earlier screens (because in general, applicants who do well in your first form screen will do more similarly in the interviews than your applicant pool overall). This does seem like an interesting phenomenon that I hadn’t thought about before, thanks :)
I guess I’m confused what hiring managers are supposed to do about this effect, though. I skimmed the chapter you link but it seemed more relevant for correcting for this when e.g. trying to correct for this when measuring the effectiveness of different interview tools in the wild.
Would be curious for your off the cuff thoughts (but this is mostly just for my curiosity so no worries if you’re busy :) )
I think you are roughly correct: the chapter is mainly about using statistics to correct for range restriction.
My interpretation/takeaway (and my memory from reading this paper a while back) is that many selection methods are less predictive than we think due to using a pre-selected sample (the people that passed some other screen/test). I wish I had some easy actions that a hiring manager could take, but I’m afraid that I don’t. I only have vague concepts, like “be more humble about your methods” and “try not to have too much confidence that you have a good system,” which are not very actionable.
I’ve barely started to touch this type of topic and consider myself quite a novice, but there are some people in/connected EA that (I’m guessing) know more about it than I do if you ever want to get more context on it within an EA context. I’m guessing that Bret at Charity Entrepreneurship and Veronica at Rethink Priorities both know more about range restriction than I do, and would have more context of the research and how to apply it. You could also reach out to Brennan Wilkerson; he studied some related stuff as part of his master’s degree in Management. Let me know if you want intros to any of these people.
You might want to look into range restriction. You can dive deep into via Range Restriction in employment interviews: An influence too big to ignore. But if you just want the simple explanation: correlation is artificially lowered when you only sample the people that pass the initial screen, making the interview appear less effective than it really is. So it is possible (likely?) that you could reasonably not do an interview at all.
Can I check my understanding here? It sounds like:
In a hiring process, later screens are less correlated with applicant future performance than earlier screens (because in general, applicants who do well in your first form screen will do more similarly in the interviews than your applicant pool overall). This does seem like an interesting phenomenon that I hadn’t thought about before, thanks :)
I guess I’m confused what hiring managers are supposed to do about this effect, though. I skimmed the chapter you link but it seemed more relevant for correcting for this when e.g. trying to correct for this when measuring the effectiveness of different interview tools in the wild.
Would be curious for your off the cuff thoughts (but this is mostly just for my curiosity so no worries if you’re busy :) )
I think you are roughly correct: the chapter is mainly about using statistics to correct for range restriction.
My interpretation/takeaway (and my memory from reading this paper a while back) is that many selection methods are less predictive than we think due to using a pre-selected sample (the people that passed some other screen/test). I wish I had some easy actions that a hiring manager could take, but I’m afraid that I don’t. I only have vague concepts, like “be more humble about your methods” and “try not to have too much confidence that you have a good system,” which are not very actionable.
I’ve barely started to touch this type of topic and consider myself quite a novice, but there are some people in/connected EA that (I’m guessing) know more about it than I do if you ever want to get more context on it within an EA context. I’m guessing that Bret at Charity Entrepreneurship and Veronica at Rethink Priorities both know more about range restriction than I do, and would have more context of the research and how to apply it. You could also reach out to Brennan Wilkerson; he studied some related stuff as part of his master’s degree in Management. Let me know if you want intros to any of these people.
Interesting! I wasn’t familiar with this literature, but makes sense now that I see it.