Great post. I have had multiple conversations lately about people’s disappointing experiences with some EA org’s hiring processes, so this is a timely contribution to the Forum.
A lot of your recommendations (structured interviews, work samples, carefully selecting interview questions for relevance to the role) are in line with my prior understanding of good hiring practices. I am less convinced, however, that behavioural interviewing is clearly better than asking about hypothetical scenarios. I can see arguments both for and against them but am unsure about which one is the net winner.
For instance, behavioural interviews may be favourable because past behaviour tends to be a strong predictor of future behaviour and because they ask about actual behaviour, not idealised situations and behavioural intentions. On the other hand, they may introduce noise, as some capable applicants will not have very good examples of “past situation X” (and vice versa), and they may introduce bias because they judge applicants based on their past behaviour, not their current capabilities.
What are your top arguments in favour of behavioural interviews? Are you aware of any high-quality studies on this topic? Your section on behavioural interviews doesn’t have any references...
some capable applicants will not have very good examples of “past situation X”
I agree with you completely. I view the main problem of evaluating someone based on the last time they had to do X, is that perhaps this person has never had to do X (even though they are capable of doing X well).
I was always told that an applicant is less likely to make up an answer if we ask about past behavior than if we ask about how he or she would act in a future hypothetical scenario, but I actually don’t have any sources for that. As a result of you asking this question I poked around a few PDFs that I have on my to-read list and I’ve updated my beliefs a bit: I used to strongly recommend behavioral interview questions, but know I’m going to lessen the strength of that recommendation and only moderately recommend. I’m really glad that you brought this up, because now my view of reality is ever-so-slightly more accurate. Thank you. I’ll add a sentence in the body of the post to clarify.
The reviewed research suggests that SQs have slightly lower criterion- validity than PBQs, especially for high-complexity jobs. Yet, this finding does not necessarily mean that only PBQs should be used, as both SQs and PBQs have good validity for most jobs. Instead, researchers commonly recommend the use of both question types, in part because SQs and PBQs tend to measure different constructs and may complement each other (e.g., Campion et al., 1997; Day& Carroll, 2003; Huffcutt,Weekley et al., 2001; Krajewski et al., 2006) to increase overall criterion-related validity and also because they allow some variety in question format.
I will provide a caveat by saying that there seem to be different ups and downs according to different research. Regarding biases in interviews, according to one journal article when applicants of different racial groups apply, behavioral interviews seem to had lower group differences than situational interviews: the average of the differences between the two groups was .10 with behavioral interviews and .20 with situational interviews (this is from a paper called Racial group differences in employment interview evaluations). Although there are many ways to interpret this, my rough interpretation would be that behavioral questions allow less bias to creep in. However, this is only a single journal article, and I have to admit that I am not as well-read on behavioral interviews as I would like.
Great post. I have had multiple conversations lately about people’s disappointing experiences with some EA org’s hiring processes, so this is a timely contribution to the Forum.
A lot of your recommendations (structured interviews, work samples, carefully selecting interview questions for relevance to the role) are in line with my prior understanding of good hiring practices. I am less convinced, however, that behavioural interviewing is clearly better than asking about hypothetical scenarios. I can see arguments both for and against them but am unsure about which one is the net winner.
For instance, behavioural interviews may be favourable because past behaviour tends to be a strong predictor of future behaviour and because they ask about actual behaviour, not idealised situations and behavioural intentions. On the other hand, they may introduce noise, as some capable applicants will not have very good examples of “past situation X” (and vice versa), and they may introduce bias because they judge applicants based on their past behaviour, not their current capabilities.
What are your top arguments in favour of behavioural interviews? Are you aware of any high-quality studies on this topic? Your section on behavioural interviews doesn’t have any references...
I agree with you completely. I view the main problem of evaluating someone based on the last time they had to do X, is that perhaps this person has never had to do X (even though they are capable of doing X well).
I was always told that an applicant is less likely to make up an answer if we ask about past behavior than if we ask about how he or she would act in a future hypothetical scenario, but I actually don’t have any sources for that. As a result of you asking this question I poked around a few PDFs that I have on my to-read list and I’ve updated my beliefs a bit: I used to strongly recommend behavioral interview questions, but know I’m going to lessen the strength of that recommendation and only moderately recommend. I’m really glad that you brought this up, because now my view of reality is ever-so-slightly more accurate. Thank you. I’ll add a sentence in the body of the post to clarify.
According to The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature, it turns out that:
I will provide a caveat by saying that there seem to be different ups and downs according to different research. Regarding biases in interviews, according to one journal article when applicants of different racial groups apply, behavioral interviews seem to had lower group differences than situational interviews: the average of the differences between the two groups was .10 with behavioral interviews and .20 with situational interviews (this is from a paper called Racial group differences in employment interview evaluations). Although there are many ways to interpret this, my rough interpretation would be that behavioral questions allow less bias to creep in. However, this is only a single journal article, and I have to admit that I am not as well-read on behavioral interviews as I would like.
Thank you for a very thoughtful reply! (and for the sharing those papers :))