Very happy to see an article on this, since I think EA will have to rely a lot on assessing expertise in domains we don’t know very well.
I think it’s outside of the scope of this article, and seems pretty hard to do, but I would be interested in how this breakdown stacks against empirical data (maybe we have something from the forecasting and expertise literature?), and also just in general see a bit more justification on why you chose this specific set of markers to look out for.
But in general, I am happy to see concrete instructional posts on important topics on the forum.
In general, you’ve set yourself up for us to give you a hard time on this article, since it’s putting us in a frame of mind to question expertise, and even suggesting some tools for analysing that. But if we try to use the tools on you for the question of expertise in assessing expertise, it looks like you’re okay on ‘P’ and that we don’t have enough evidence on the rest.
Well-observed! Here’s my guess on where I rank on the various conditions above:
P—Process: Medium. I think my explicit process is still fairly decent, but my implicit processes still need work. E.g., I might perform well at identifying an expert if you gave me a decent amount of time to check markers with my framework, but I’m not fluent enough in my explicit models to do expertise assessments on the fly very well, Sherlock Holmes-style.
I—Interaction: Medium. I’ve spent dozens of hours interacting with expertise assessment tasks, as mentioned in the article. However, for much of this interaction with the data, I did not have strong explicit models (I only developed the expert assessment framework last month.) Since my interaction with the data was not very model-guided for the majority of the time, it’s likely that I often didn’t pay attention to the right features of the data. So I may have been rather like Bob above:
Bob, a graphic design novice, pays no attention to the signs and advertisements along the side of the street, even though they are within his field of vision.
It may have been that lots of data relating to expertise was literally and metaphorically in my field of vision, but that I wasn’t focusing on it very well, or wasn’t focusing on the proper features.
F—Feedback: Low. Since I’ve only had well-developed explicit models for about a month, I still have only gotten minor feedback on my predictive power. I have run a few predictive exercises—they went well by the n is still small. My primary feedback method has been to generate lots of examples of people I am confident have expertise and check whether each marker can be found in all the examples. I also did the opposite: generate lots of examples of people I am confident lack expertise, and check whether each marker is absent from all the examples. I also used normal proxy methods that one can apply to check the robustness of theories without knowing much about them. (E.g., are there logical contradictions?) I used a couple other methods (e.g., running simulations and checking whether my system 1 yielded error signals), but I’d need to write a full-length article about them for these to make sense. For now, I will just say that they were weak feedback processes, but useful ones. Overall, I looked for correlation between the various feedback methods.
T—Time: Low-medium. I have probably spent more time training in specifically domain-general expertise assessment relative to most people in the world. But this is not saying much, since domain-general expertise assessment is not a thriving or even recognized field, as far as I can tell. Also, I have been only a small amount of time on the skill relative to the amount of training required to become skilled in domains falling into a similar reference class. (e.g., I think expertise assessment could be it’s own scientific discipline, and people spend years in order to gain sufficient expertise in scientific disciplines.)
Very happy to see an article on this, since I think EA will have to rely a lot on assessing expertise in domains we don’t know very well.
I think it’s outside of the scope of this article, and seems pretty hard to do, but I would be interested in how this breakdown stacks against empirical data (maybe we have something from the forecasting and expertise literature?), and also just in general see a bit more justification on why you chose this specific set of markers to look out for.
But in general, I am happy to see concrete instructional posts on important topics on the forum.
Agree with this question.
In general, you’ve set yourself up for us to give you a hard time on this article, since it’s putting us in a frame of mind to question expertise, and even suggesting some tools for analysing that. But if we try to use the tools on you for the question of expertise in assessing expertise, it looks like you’re okay on ‘P’ and that we don’t have enough evidence on the rest.
Well-observed! Here’s my guess on where I rank on the various conditions above:
P—Process: Medium. I think my explicit process is still fairly decent, but my implicit processes still need work. E.g., I might perform well at identifying an expert if you gave me a decent amount of time to check markers with my framework, but I’m not fluent enough in my explicit models to do expertise assessments on the fly very well, Sherlock Holmes-style.
I—Interaction: Medium. I’ve spent dozens of hours interacting with expertise assessment tasks, as mentioned in the article. However, for much of this interaction with the data, I did not have strong explicit models (I only developed the expert assessment framework last month.) Since my interaction with the data was not very model-guided for the majority of the time, it’s likely that I often didn’t pay attention to the right features of the data. So I may have been rather like Bob above:
F—Feedback: Low. Since I’ve only had well-developed explicit models for about a month, I still have only gotten minor feedback on my predictive power. I have run a few predictive exercises—they went well by the n is still small. My primary feedback method has been to generate lots of examples of people I am confident have expertise and check whether each marker can be found in all the examples. I also did the opposite: generate lots of examples of people I am confident lack expertise, and check whether each marker is absent from all the examples. I also used normal proxy methods that one can apply to check the robustness of theories without knowing much about them. (E.g., are there logical contradictions?) I used a couple other methods (e.g., running simulations and checking whether my system 1 yielded error signals), but I’d need to write a full-length article about them for these to make sense. For now, I will just say that they were weak feedback processes, but useful ones. Overall, I looked for correlation between the various feedback methods.
T—Time: Low-medium. I have probably spent more time training in specifically domain-general expertise assessment relative to most people in the world. But this is not saying much, since domain-general expertise assessment is not a thriving or even recognized field, as far as I can tell. Also, I have been only a small amount of time on the skill relative to the amount of training required to become skilled in domains falling into a similar reference class. (e.g., I think expertise assessment could be it’s own scientific discipline, and people spend years in order to gain sufficient expertise in scientific disciplines.)