Hey Peter—it’s partly for you, but also many other people who have the same questions.
I can comment on the choice of question in the survey as I’m the one who wrote it.
The reason we went for an ex post assessment rather than an ex ante one was that we thought people would be able to more reliably assess how they feel about a previous hire today, than remember how they felt about a previous hire in the past.
Asking people to remember what they thought before runs the risk that they will substitute a hard question (what they thought months ago before they knew how someone would work out) for an easy question (what do they think given what they know now). Then we’d get a similar answer, but it would look ex ante when it actually isn’t.
It also seemed quite difficult for organisations to forecast how much they’ll value a typical hire in the future because, among other reasons, it’s difficult to anticipate how successful future searches will be.
In retrospect I think the framing we chose was probably a mistake, because the two assessments are much more different than most readers understand them to be. I agree with your suggestions for improvements and, indeed, we concluded our blog post with our plans to ask new questions next year or else interview a smaller number of people in more depth.
Hopefully a different approach next year will help us avoid this confusion going forward.
As for the article being misleading, we’ve:
i) Commented that these roles are hard to fill at the point when these figures are first mentioned.
ii) Explained the ex post, ex ante distinction in the relevant section, and now added a link to this post.
iii) Noted we don’t have much confidence in the answers to that question and would not recommend that people update very much based on it.
Hey Peter—it’s partly for you, but also many other people who have the same questions.
I can comment on the choice of question in the survey as I’m the one who wrote it.
The reason we went for an ex post assessment rather than an ex ante one was that we thought people would be able to more reliably assess how they feel about a previous hire today, than remember how they felt about a previous hire in the past.
Asking people to remember what they thought before runs the risk that they will substitute a hard question (what they thought months ago before they knew how someone would work out) for an easy question (what do they think given what they know now). Then we’d get a similar answer, but it would look ex ante when it actually isn’t.
It also seemed quite difficult for organisations to forecast how much they’ll value a typical hire in the future because, among other reasons, it’s difficult to anticipate how successful future searches will be.
In retrospect I think the framing we chose was probably a mistake, because the two assessments are much more different than most readers understand them to be. I agree with your suggestions for improvements and, indeed, we concluded our blog post with our plans to ask new questions next year or else interview a smaller number of people in more depth.
Hopefully a different approach next year will help us avoid this confusion going forward.
As for the article being misleading, we’ve:
i) Commented that these roles are hard to fill at the point when these figures are first mentioned.
ii) Explained the ex post, ex ante distinction in the relevant section, and now added a link to this post.
iii) Noted we don’t have much confidence in the answers to that question and would not recommend that people update very much based on it.