There’s a decent amount in this post that I agree with, but also sufficiently poor epistemics/reasoning transparency[1] (in my view) and enough advice that seems inaccurate[2] that I would recommend people either (a) don’t read this post or (b) read it only with substantial skepticism and alongside other sources of advice (and not just Manager Tools, though maybe use that as well, just with skepticism). One place other advice can be found is in this quick collection I made: Quick notes/links on hiring/vetting
I do think this post has some advice many people would benefit from, but the post also confidently asserts some things that I think are probably inaccurate or bad advice.[2] And (in my view) the post doesn’t make it easy for people to discern the good from bad advice unless they already have good knowledge here (in which case they’ll also gain little from the post), since the post provides very little sources or arguments and has the same confident tone for all its advice (rather than presenting some things as more tentative).
Overall, I think that this post is below the epistemic standards we should want for the Forum. (I’m aware this is somewhat rude/aggressive to write, and I promise I come across as friendlier in person! Also, in part [1] of this comment I provide some concrete recommendations / constructive advice that may be helpful for future posts. And to be clear, I do expect that the author was well-intentioned and trying to be helpful when writing this.)
[1] Regarding poor epistemics / reasoning transparency in my view: It sounds like the author deliberately chose to release a short-ish and maybe quickly-ish written post rather than no post at all (“I won’t be arguing every point fully. [...] That’s because doing otherwise would have made the article ten times longer”). That’s a reasonable choice. But in this case I think it turns out too much of this advice is probably bad for the resulting quick post to be useful to many people. I think the ideal version of this post would involve maybe a bit more word count and a bit longer time spent (but doesn’t have to be much more) and would:
Indicate differing levels of confidence in different claims
Probably be substantially less confident on average
I’m not saying that all posts should be low-confidence. But I think that should probably be the case when:
the author has limited experience
I’m basing this on the author saying “Hiring is a wide field and I’ve only tilled a small patch of it myself”. To be clear, I’m not saying people should only write posts about things they have lots of experience on, just that limited experience plus the following things should warrant lower confidence.
The author didn’t have time to properly find sources or explain the arguments
This is relevant because in the process of doing so one often realises oneself that the claims rest on shakier ground than one thought.
The main source seems to be a podcast/site (Manager Tools) which (I think) itself has poor epistemics
I’ve engaged a fair bit with Manager Tools, and think it’s useful overall, but in my experience they also usually provide no evidence or arguments for their claims except their personal experience, and I think many of their claims/advice are bad. So when I’m engaging with it I’m doing so with skepticism and sifting for a few takeaways that are in hindsight obviously true or that I can test out easily myself.
Provide more of the reasoning/arguments for various claims, even if that just means saying more of what one already believes or thinks rather than thinking things through in more detail or finding more sources
(I haven’t checked any of the other claims/sources properly, so this may not be the only such issue)
Probably defer less to sources which don’t themselves have strong evidence bases or reasoning transparency (Manager Tools)
Maybe find more high-quality sources (though it’d also be reasonable to not bother with that)
[2] Some things in this post that I think are inaccurate or bad advice (though unfortunately don’t have time to explain my reasoning for most of these):
The claim that personality tests aren’t predictive of job performance
My prior belief from reading some papers a while ago and from what I think would make sense (given how well I think Big Five personality tests can predict some other things) is that they are predictive. And the paper cited in this post for the claim that they’re not predictive actually seems to be showing they are one of the most predictive things. [Edited to add: Whoops, I misread the source myself. Though still the source seems to be saying they’re weakly/moderately predictive, and have unusually high incremental validity when added to other specific assessments, rather than saying “not predictive”.]
Saying it’s bad to outsource things you don’t know because you can’t assess performance well in those cases, without flagging that this might not apply anywhere near as much in cases where an outsourcee comes highly recommended by people you trust and whose situation is similar to yours, and when the outsourcee has credible signals of sharing a lot of your values
(E.g., if they’re a credibly EA-aligned hiring service/advisor who comes recommended from relevant other people based on performance there.)
Suggesting filtering only via resumes rather than instead/also via screening questions that are basically a very short work test
Suggesting giving many interviews and relying heavily on them, and suggesting candidates answering the same questions from each interviewee
I think that latter advice could only make sense if interviewees don’t have a pre-set rubric and are just assessing answers subjectively, which seems unwise to me, or if the interviews are close to unstructured rather than structured, which also seems unwise to me. Also, relative to work tests, this takes up a lot of candidate and esp. hirer time.
Making final decisions based on a meeting where people verbally share their assessments one at a time (which could then be anchored by whoever speaks first or groupthink or whatever), rather than collecting scores along the way and sticking to them by default (though with room to manoeuvre) to minimise noise (see also the book Noise)
Those last three things basically seem consistent with the ways traditional hiring is bad and seems based on prizing subjective judgement and not really quantitatively assessing things or checking what research shows works best, in contrast to what’s common in EA orgs + leading tech firms (as far as I’m aware). And that seems consistent with my experience of the kind of advice Manager Tools tends to give.
Also, these are just most of the things that stood out to me when casually listening to the post (in audio form), and that’s enough to make me think that there may also be some other issues.
It would be nice if you moved your last paragraph first. Recommending (in bold) based on a casual listening not to read a post feels unfair. (Speaking about fairness, there are other posts about hiring (eg. https://forum.effectivealtruism.org/posts/XpnJKvr5BKEKcgvdD/perhaps-the-highest-leverage-meta-skill-an-ea-guide-to#comments) that are argued more thinly than my post and have gotten much less criticism.) I agree with your alternative recommendation to read the post with skepticism. That’s the case with any post.
I agree with some of your points. Especially the one about ways to outsource without knowing about the subject. I might work it into the article. (Two sentences added 2022-10-19.) (2022-10-22: I’ve worked it into the article.) Overall I think there are misunderstandings, which I should have worked harder to avoid.
The main point is not hiring advice. (2022-10-20: Added clarifying comments to this end.) The main point is that people who hire need to learn more about hiring. I’m merely demonstrating this by countering Hirely, which I do based on hiring guidance that I’m fairly confident is better than theirs. How this came about I describe in the section about the article’s history.
I agree that it would be nice to have more epistemic nuance and diligence. But this article wouldn’t exist had I tried for that, because I don’t have more time. Given the choice between not publishing the article and publishing it with some epistemic hubris, I thought the latter is more beneficial.
The main resource for this article is Manager Tools because I think their hiring guidance is an excellent starting point. This is based on wider reading and listening and thinking and experiencing, which I have been doing for years. I agree that their arguments are often sloppy and there are areas other than hiring in which they’re weaker than they admit. Your claim that ‘they also usually provide no evidence or arguments for their claims except their personal experience’ is wrong as it stands. They provide evidence for some of their claims, which I’ve linked in my section on why Manager Tools is a decent authority. (Note that I wrote ‘a decent’, not ‘the ultimate’ – epistemic humility.) I wish they would provide evidence for other claims, too. And the bulk of each podcast/whitepaper is made up of arguments for their claims, spotty though they might be at times. Personal experience can be a valid support for arguments, too, if it’s believable that you have a lot of it. It is believable that MT has a lot of personal experience (and data) about hiring, which I also point out in my section about them.
By the way:
I do think I’m fairly transparent and indicate levels of confidence to some degree, perhaps in a different way than you’re used to. I don’t have time to read and digest the full Reasoning Transparency article. But if I look at the top recommendations:
‘Open with a linked summary of key takeaways.’ – That’s the title in my article.
‘Throughout a document, indicate which considerations are most important to your key takeaways.’ – With some squinting that’s the table of contents in my article.
‘Throughout a document, indicate how confident you are in major claims, and what support you have for them.’ – I do indicate my support (Manager Tools) and why I think it’s a support. I am quite confident in my main claim and in the main supporting claims. Note also that I don’t write that outsourcing hiring is terrible, but that you need to be careful. And I don’t write ‘Hirely will cause you to make a bad hire’, but that they put you at risk of making a bad hire. Among others.
I think I’ve refuted part of your claim about personality tests in other comments. Added 2022-10-22: Even if Big Five assessments can predict some things well, I would expect that to only be the case when they’re not gamed. The one I know can easily be gamed. (Eg. just answer ‘very accurate’/‘very inaccurate’ to all the question that sound like pushing up/down the conscientiousness score.) And an important part of hiring is keeping out people that make themselves look better than they are.
Expanded 2022-10-23: Some of the arguments against the ‘bad’ hiring advice had already been addressed in my article. Eg.: I point out that the résumé screen is a flawed filter and needs to be supplemented or replaced, depending on the situation. I briefly argue why it’s okay to take a lot of time interviewing. There is also an easy fix for anchoring/groupthink in the interview results capture meeting (as well as other, more subtle, reasons to do it): Ask people to send you their recommendations and justifications before the meeting.
Added 2022-10-19, edited 2022-10-23: In general, the items about ‘inaccurate or bad advice’ have common-sense arguments for and against them. (If anyone wants me to write any of them out, let me know.) It would come down to analysing Manager Tools’ (unfortunately non-public) data as well as the research you’re referring to. I would expect ‘traditional hiring’ when done right to work roughly equally well to whatever good EA orgs and leading tech firms do. My default expectation when people say that the ‘traditional way to do X is bad’ is that it actually works well when done right, but that it has a bad image because it got corrupted over time/is usually done mediocrely.
There’s a decent amount in this post that I agree with, but also sufficiently poor epistemics/reasoning transparency[1] (in my view) and enough advice that seems inaccurate[2] that I would recommend people either (a) don’t read this post or (b) read it only with substantial skepticism and alongside other sources of advice (and not just Manager Tools, though maybe use that as well, just with skepticism). One place other advice can be found is in this quick collection I made: Quick notes/links on hiring/vetting
I do think this post has some advice many people would benefit from, but the post also confidently asserts some things that I think are probably inaccurate or bad advice.[2] And (in my view) the post doesn’t make it easy for people to discern the good from bad advice unless they already have good knowledge here (in which case they’ll also gain little from the post), since the post provides very little sources or arguments and has the same confident tone for all its advice (rather than presenting some things as more tentative).
Overall, I think that this post is below the epistemic standards we should want for the Forum. (I’m aware this is somewhat rude/aggressive to write, and I promise I come across as friendlier in person! Also, in part [1] of this comment I provide some concrete recommendations / constructive advice that may be helpful for future posts. And to be clear, I do expect that the author was well-intentioned and trying to be helpful when writing this.)
[1] Regarding poor epistemics / reasoning transparency in my view: It sounds like the author deliberately chose to release a short-ish and maybe quickly-ish written post rather than no post at all (“I won’t be arguing every point fully. [...] That’s because doing otherwise would have made the article ten times longer”). That’s a reasonable choice. But in this case I think it turns out too much of this advice is probably bad for the resulting quick post to be useful to many people. I think the ideal version of this post would involve maybe a bit more word count and a bit longer time spent (but doesn’t have to be much more) and would:
Indicate differing levels of confidence in different claims
Probably be substantially less confident on average
I’m not saying that all posts should be low-confidence. But I think that should probably be the case when:
the author has limited experience
I’m basing this on the author saying “Hiring is a wide field and I’ve only tilled a small patch of it myself”. To be clear, I’m not saying people should only write posts about things they have lots of experience on, just that limited experience plus the following things should warrant lower confidence.
The author didn’t have time to properly find sources or explain the arguments
This is relevant because in the process of doing so one often realises oneself that the claims rest on shakier ground than one thought.
The main source seems to be a podcast/site (Manager Tools) which (I think) itself has poor epistemics
I’ve engaged a fair bit with Manager Tools, and think it’s useful overall, but in my experience they also usually provide no evidence or arguments for their claims except their personal experience, and I think many of their claims/advice are bad. So when I’m engaging with it I’m doing so with skepticism and sifting for a few takeaways that are in hindsight obviously true or that I can test out easily myself.
Provide more of the reasoning/arguments for various claims, even if that just means saying more of what one already believes or thinks rather than thinking things through in more detail or finding more sources
Be based on more careful reading of sources, to avoid drawing from one source a conclusion that seems exactly opposite to what the source finds
(I haven’t checked any of the other claims/sources properly, so this may not be the only such issue)
Probably defer less to sources which don’t themselves have strong evidence bases or reasoning transparency (Manager Tools)
Maybe find more high-quality sources (though it’d also be reasonable to not bother with that)
[2] Some things in this post that I think are inaccurate or bad advice (though unfortunately don’t have time to explain my reasoning for most of these):
The claim that personality tests aren’t predictive of job performance
My prior belief from reading some papers a while ago and from what I think would make sense (given how well I think Big Five personality tests can predict some other things) is that they are predictive. And the paper cited in this post for the claim that they’re not predictive actually seems to be showing they are
one of the most predictive things.[Edited to add: Whoops, I misread the source myself. Though still the source seems to be saying they’re weakly/moderately predictive, and have unusually high incremental validity when added to other specific assessments, rather than saying “not predictive”.]Saying it’s bad to outsource things you don’t know because you can’t assess performance well in those cases, without flagging that this might not apply anywhere near as much in cases where an outsourcee comes highly recommended by people you trust and whose situation is similar to yours, and when the outsourcee has credible signals of sharing a lot of your values
(E.g., if they’re a credibly EA-aligned hiring service/advisor who comes recommended from relevant other people based on performance there.)
Suggesting filtering only via resumes rather than instead/also via screening questions that are basically a very short work test
Suggesting giving many interviews and relying heavily on them, and suggesting candidates answering the same questions from each interviewee
I think that latter advice could only make sense if interviewees don’t have a pre-set rubric and are just assessing answers subjectively, which seems unwise to me, or if the interviews are close to unstructured rather than structured, which also seems unwise to me. Also, relative to work tests, this takes up a lot of candidate and esp. hirer time.
Making final decisions based on a meeting where people verbally share their assessments one at a time (which could then be anchored by whoever speaks first or groupthink or whatever), rather than collecting scores along the way and sticking to them by default (though with room to manoeuvre) to minimise noise (see also the book Noise)
Those last three things basically seem consistent with the ways traditional hiring is bad and seems based on prizing subjective judgement and not really quantitatively assessing things or checking what research shows works best, in contrast to what’s common in EA orgs + leading tech firms (as far as I’m aware). And that seems consistent with my experience of the kind of advice Manager Tools tends to give.
Also, these are just most of the things that stood out to me when casually listening to the post (in audio form), and that’s enough to make me think that there may also be some other issues.
It would be nice if you moved your last paragraph first. Recommending (in bold) based on a casual listening not to read a post feels unfair. (Speaking about fairness, there are other posts about hiring (eg. https://forum.effectivealtruism.org/posts/XpnJKvr5BKEKcgvdD/perhaps-the-highest-leverage-meta-skill-an-ea-guide-to#comments) that are argued more thinly than my post and have gotten much less criticism.) I agree with your alternative recommendation to read the post with skepticism. That’s the case with any post.
I agree with some of your points. Especially the one about ways to outsource without knowing about the subject. I might work it into the article. (Two sentences added 2022-10-19.) (2022-10-22: I’ve worked it into the article.) Overall I think there are misunderstandings, which I should have worked harder to avoid.
The main point is not hiring advice. (2022-10-20: Added clarifying comments to this end.) The main point is that people who hire need to learn more about hiring. I’m merely demonstrating this by countering Hirely, which I do based on hiring guidance that I’m fairly confident is better than theirs. How this came about I describe in the section about the article’s history.
I agree that it would be nice to have more epistemic nuance and diligence. But this article wouldn’t exist had I tried for that, because I don’t have more time. Given the choice between not publishing the article and publishing it with some epistemic hubris, I thought the latter is more beneficial.
The main resource for this article is Manager Tools because I think their hiring guidance is an excellent starting point. This is based on wider reading and listening and thinking and experiencing, which I have been doing for years. I agree that their arguments are often sloppy and there are areas other than hiring in which they’re weaker than they admit. Your claim that ‘they also usually provide no evidence or arguments for their claims except their personal experience’ is wrong as it stands. They provide evidence for some of their claims, which I’ve linked in my section on why Manager Tools is a decent authority. (Note that I wrote ‘a decent’, not ‘the ultimate’ – epistemic humility.) I wish they would provide evidence for other claims, too. And the bulk of each podcast/whitepaper is made up of arguments for their claims, spotty though they might be at times. Personal experience can be a valid support for arguments, too, if it’s believable that you have a lot of it. It is believable that MT has a lot of personal experience (and data) about hiring, which I also point out in my section about them.
By the way:
I do think I’m fairly transparent and indicate levels of confidence to some degree, perhaps in a different way than you’re used to. I don’t have time to read and digest the full Reasoning Transparency article. But if I look at the top recommendations:
‘Open with a linked summary of key takeaways.’ – That’s the title in my article.
‘Throughout a document, indicate which considerations are most important to your key takeaways.’ – With some squinting that’s the table of contents in my article.
‘Throughout a document, indicate how confident you are in major claims, and what support you have for them.’ – I do indicate my support (Manager Tools) and why I think it’s a support. I am quite confident in my main claim and in the main supporting claims. Note also that I don’t write that outsourcing hiring is terrible, but that you need to be careful. And I don’t write ‘Hirely will cause you to make a bad hire’, but that they put you at risk of making a bad hire. Among others.
I think I’ve refuted part of your claim about personality tests in other comments. Added 2022-10-22: Even if Big Five assessments can predict some things well, I would expect that to only be the case when they’re not gamed. The one I know can easily be gamed. (Eg. just answer ‘very accurate’/‘very inaccurate’ to all the question that sound like pushing up/down the conscientiousness score.) And an important part of hiring is keeping out people that make themselves look better than they are.
Expanded 2022-10-23: Some of the arguments against the ‘bad’ hiring advice had already been addressed in my article. Eg.: I point out that the résumé screen is a flawed filter and needs to be supplemented or replaced, depending on the situation. I briefly argue why it’s okay to take a lot of time interviewing. There is also an easy fix for anchoring/groupthink in the interview results capture meeting (as well as other, more subtle, reasons to do it): Ask people to send you their recommendations and justifications before the meeting.
Added 2022-10-19, edited 2022-10-23: In general, the items about ‘inaccurate or bad advice’ have common-sense arguments for and against them. (If anyone wants me to write any of them out, let me know.) It would come down to analysing Manager Tools’ (unfortunately non-public) data as well as the research you’re referring to. I would expect ‘traditional hiring’ when done right to work roughly equally well to whatever good EA orgs and leading tech firms do. My default expectation when people say that the ‘traditional way to do X is bad’ is that it actually works well when done right, but that it has a bad image because it got corrupted over time/is usually done mediocrely.