Great post—identifying experts and, in particular, comparing expertise between similar candidates is exceptionally difficult, using even a rough model seems likely to greatly improve our ability to undertake this task.
While it seems possible to make some progress on the problem of independently assessing expertise, I want to stress that we should still expect to fail if we proceed to do so entirely independently, without consulting a domain expert—Great! - now we have a simpler problem—how do we identify the best domain expert who can help us build a framework for assessing candidates?
Tyler’s model seems somewhat helpful here, and adding the components from John’s model improves it again. My prior approach was a simpler one, but shares some characteristics. I usually look for evidence of exceptional accomplishments that are rare or unprecedented, and ignore most examples of accomplishments which are difficult or competitive but common. Peer recognition is also a good barometer, more so if you ask people who are field insiders but have a merely casual acquaintance with the person in question. In the case of picking an expert who can help me identify predictors of expertise in their field, I’m less concerned with my ability to rate and compare their level of expertise with other top-level experts, as it’s fairly low cost to seek out the opinions of multiple experts.
When we were considering hiring a digital marketer, I sought input from 4 people who I will call experts, doing so dramatically improved my ability to pick the best candidates from the pool. I tested my predictions against the experts by rating applications for the top 5 candidates myself, then getting the domain expert to rank them and compare scores, watching them doing so. Watching the expert evaluate other candidates helped me pick out further elements which were not in their original verbal model. This part seems qute different than Tyler’s approach, as it is about identifying domain-specific expertise, rather than searching for domain general predictors of expertise, however it seems important to mention. I worry that neglecting to seek out domain-specific predictors would lead to a poorer outcome.
I also want to tease apart the question of attaining domain level expertise versus having a good process for generating expertise. I imagine that it is possible for those who have a good process (these people would, I imagine, score well using Tyler’s model) to become experts more quickly. I imagine there is another class of experts who have decades of experience, rich implicit models and impressive achievements, but who would struggle to present concise, detailed answers if you asked them to share their wisdom. I suspect that quiet observation of such a person in their work environment, rather than asking them questions, would yield a better measure of their level of expertise, but this requires considerable skill on the part of the observer.
I’d love to think about this more, looking forward to trying on your framework and playing around with it.
The method of observing experts and turning their heuristics into a simple scale is well supported in the forecasting literature (don’t have a quick cite handy unfortunately).
While it seems possible to make some progress on the problem of independently assessing expertise, I want to stress that we should still expect to fail if we proceed to do so entirely independently, without consulting a domain expert
Right, I should have mentioned this. Your job is much, much easier if you can identify a solid “seed” expert in the domain with a few caveats:
If the seed expert becomes your primary input to expertise identification, you should be confident that their expertise checks are good. I’m tempted to think that the skill of domain-specific expertise identification correlates strongly with expertise in that domain, but not perfectly. This will be especially true in fields where there are lots of persuaders who have learned how to mimic signs of expertise.
Keep domain-specific expertise base-rates in mind, as mentioned above. In domains where the expertise base-rate is low (e.g., sociology), you will need to run many more expertise checks on the seed expert than usual, and will have a harder time finding a passable expert in the first place.
In fields where results are not easily verifiable (e.g., sociology again), it will be more difficult to identify a seed expert. Also, these seed experts will often have a hard time identifying revolutionary forms of expertise, since they might look like crackpots. (As opposed to, say, math, where there are cases of people who prima facie look like crackpots being nonetheless hired as professors, since their results are reliably verifiable.)
In fields with high variance, you may be able to find a passable seed expert who cannot consistently identify experts who are much, much better than they are.
In fields with poorly networked knowledge, seed experts will be much less helpful. I can imagine this being the case for fields like massage therapy, where I expert there to be fewer journals and conferences.
I imagine there is another class of experts who have decades of experience, rich implicit models and impressive achievements, but who would struggle to present concise, detailed answers if you asked them to share their wisdom. I suspect that quiet observation of such a person in their work environment, rather than asking them questions, would yield a better measure of their level of expertise, but this requires considerable skill on the part of the observer.
Indeed: tacit experts. The way I assess this now is basically by looking at indirect signs around the potential tacit expert (e.g., achievements is a good one, as is evidence of them having made costly tradeoffs in the past to develop their expertise (a weaker sign).) If anyone develops tools for directly assessing tacit experts, please let me know.
I’d also be very interested if anyone has ideas for how to learn the skills of tacit experts, once you’ve identified them.
One idea for learning the skills of tacit experts that I found works is to copy their behaviors regarding the domain, without necessarily understanding the reasons behind their behaviors.
It sounds strange to us as people who are very intellectually-oriented and seek to understand the reasons behind why something works. I know it did to me when I first tried to do it. Moreover, there is a danger of copying behaviors that are incidental and do not lead to the desired outcome. Still, given that tacit experts often don’t know themselves why they do well at what they do, simply copying their behaviors seems to work.
One domain is social behavior. Emulating the social behavior of people who have high charisma has proved beneficial for me in improving my own charisma, even if the people with high charisma could not explain their own charisma.
I tested my predictions against the experts by rating applications for the top 5 candidates myself, then getting the domain expert to rank them and compare scores, watching them doing so.
Ah! This sounds like a great feedback mechanism for one’s expert assessment abilities. I’m going to steal this. =)
Tyler’s model seems somewhat helpful here, and adding the components from John’s model improves it again.
+1 - you definitely want to use more signs than the ones I mentioned above to be confident that you have identified sufficient marker of expertise. The ones listed above are only intended to be necessary markers. A good way of generating markers beyond the necessary ones: think about a few people who you can confidently say are experts. What do they have in common? (Please send me any cool markers you’ve come up with! My own list has over 30 now, and it doesn’t seem like ceiling has been hit.)
Great post—identifying experts and, in particular, comparing expertise between similar candidates is exceptionally difficult, using even a rough model seems likely to greatly improve our ability to undertake this task.
While it seems possible to make some progress on the problem of independently assessing expertise, I want to stress that we should still expect to fail if we proceed to do so entirely independently, without consulting a domain expert—Great! - now we have a simpler problem—how do we identify the best domain expert who can help us build a framework for assessing candidates?
Tyler’s model seems somewhat helpful here, and adding the components from John’s model improves it again. My prior approach was a simpler one, but shares some characteristics. I usually look for evidence of exceptional accomplishments that are rare or unprecedented, and ignore most examples of accomplishments which are difficult or competitive but common. Peer recognition is also a good barometer, more so if you ask people who are field insiders but have a merely casual acquaintance with the person in question. In the case of picking an expert who can help me identify predictors of expertise in their field, I’m less concerned with my ability to rate and compare their level of expertise with other top-level experts, as it’s fairly low cost to seek out the opinions of multiple experts.
When we were considering hiring a digital marketer, I sought input from 4 people who I will call experts, doing so dramatically improved my ability to pick the best candidates from the pool. I tested my predictions against the experts by rating applications for the top 5 candidates myself, then getting the domain expert to rank them and compare scores, watching them doing so. Watching the expert evaluate other candidates helped me pick out further elements which were not in their original verbal model. This part seems qute different than Tyler’s approach, as it is about identifying domain-specific expertise, rather than searching for domain general predictors of expertise, however it seems important to mention. I worry that neglecting to seek out domain-specific predictors would lead to a poorer outcome.
I also want to tease apart the question of attaining domain level expertise versus having a good process for generating expertise. I imagine that it is possible for those who have a good process (these people would, I imagine, score well using Tyler’s model) to become experts more quickly. I imagine there is another class of experts who have decades of experience, rich implicit models and impressive achievements, but who would struggle to present concise, detailed answers if you asked them to share their wisdom. I suspect that quiet observation of such a person in their work environment, rather than asking them questions, would yield a better measure of their level of expertise, but this requires considerable skill on the part of the observer.
I’d love to think about this more, looking forward to trying on your framework and playing around with it.
The method of observing experts and turning their heuristics into a simple scale is well supported in the forecasting literature (don’t have a quick cite handy unfortunately).
Some related research.
This as well.
Right, I should have mentioned this. Your job is much, much easier if you can identify a solid “seed” expert in the domain with a few caveats:
If the seed expert becomes your primary input to expertise identification, you should be confident that their expertise checks are good. I’m tempted to think that the skill of domain-specific expertise identification correlates strongly with expertise in that domain, but not perfectly. This will be especially true in fields where there are lots of persuaders who have learned how to mimic signs of expertise.
Keep domain-specific expertise base-rates in mind, as mentioned above. In domains where the expertise base-rate is low (e.g., sociology), you will need to run many more expertise checks on the seed expert than usual, and will have a harder time finding a passable expert in the first place.
In fields where results are not easily verifiable (e.g., sociology again), it will be more difficult to identify a seed expert. Also, these seed experts will often have a hard time identifying revolutionary forms of expertise, since they might look like crackpots. (As opposed to, say, math, where there are cases of people who prima facie look like crackpots being nonetheless hired as professors, since their results are reliably verifiable.)
In fields with high variance, you may be able to find a passable seed expert who cannot consistently identify experts who are much, much better than they are.
In fields with poorly networked knowledge, seed experts will be much less helpful. I can imagine this being the case for fields like massage therapy, where I expert there to be fewer journals and conferences.
Indeed: tacit experts. The way I assess this now is basically by looking at indirect signs around the potential tacit expert (e.g., achievements is a good one, as is evidence of them having made costly tradeoffs in the past to develop their expertise (a weaker sign).) If anyone develops tools for directly assessing tacit experts, please let me know.
I’d also be very interested if anyone has ideas for how to learn the skills of tacit experts, once you’ve identified them.
One idea for learning the skills of tacit experts that I found works is to copy their behaviors regarding the domain, without necessarily understanding the reasons behind their behaviors.
It sounds strange to us as people who are very intellectually-oriented and seek to understand the reasons behind why something works. I know it did to me when I first tried to do it. Moreover, there is a danger of copying behaviors that are incidental and do not lead to the desired outcome. Still, given that tacit experts often don’t know themselves why they do well at what they do, simply copying their behaviors seems to work.
What domains have you found this to work in?
One domain is social behavior. Emulating the social behavior of people who have high charisma has proved beneficial for me in improving my own charisma, even if the people with high charisma could not explain their own charisma.
Ah! This sounds like a great feedback mechanism for one’s expert assessment abilities. I’m going to steal this. =)
+1 - you definitely want to use more signs than the ones I mentioned above to be confident that you have identified sufficient marker of expertise. The ones listed above are only intended to be necessary markers. A good way of generating markers beyond the necessary ones: think about a few people who you can confidently say are experts. What do they have in common? (Please send me any cool markers you’ve come up with! My own list has over 30 now, and it doesn’t seem like ceiling has been hit.)