It would be nice if you moved your last paragraph first. Recommending (in bold) based on a casual listening not to read a post feels unfair. (Speaking about fairness, there are other posts about hiring (eg. https://forum.effectivealtruism.org/posts/XpnJKvr5BKEKcgvdD/perhaps-the-highest-leverage-meta-skill-an-ea-guide-to#comments) that are argued more thinly than my post and have gotten much less criticism.) I agree with your alternative recommendation to read the post with skepticism. That’s the case with any post.
I agree with some of your points. Especially the one about ways to outsource without knowing about the subject. I might work it into the article. (Two sentences added 2022-10-19.) (2022-10-22: I’ve worked it into the article.) Overall I think there are misunderstandings, which I should have worked harder to avoid.
The main point is not hiring advice. (2022-10-20: Added clarifying comments to this end.) The main point is that people who hire need to learn more about hiring. I’m merely demonstrating this by countering Hirely, which I do based on hiring guidance that I’m fairly confident is better than theirs. How this came about I describe in the section about the article’s history.
I agree that it would be nice to have more epistemic nuance and diligence. But this article wouldn’t exist had I tried for that, because I don’t have more time. Given the choice between not publishing the article and publishing it with some epistemic hubris, I thought the latter is more beneficial.
The main resource for this article is Manager Tools because I think their hiring guidance is an excellent starting point. This is based on wider reading and listening and thinking and experiencing, which I have been doing for years. I agree that their arguments are often sloppy and there are areas other than hiring in which they’re weaker than they admit. Your claim that ‘they also usually provide no evidence or arguments for their claims except their personal experience’ is wrong as it stands. They provide evidence for some of their claims, which I’ve linked in my section on why Manager Tools is a decent authority. (Note that I wrote ‘a decent’, not ‘the ultimate’ – epistemic humility.) I wish they would provide evidence for other claims, too. And the bulk of each podcast/whitepaper is made up of arguments for their claims, spotty though they might be at times. Personal experience can be a valid support for arguments, too, if it’s believable that you have a lot of it. It is believable that MT has a lot of personal experience (and data) about hiring, which I also point out in my section about them.
By the way:
I do think I’m fairly transparent and indicate levels of confidence to some degree, perhaps in a different way than you’re used to. I don’t have time to read and digest the full Reasoning Transparency article. But if I look at the top recommendations:
‘Open with a linked summary of key takeaways.’ – That’s the title in my article.
‘Throughout a document, indicate which considerations are most important to your key takeaways.’ – With some squinting that’s the table of contents in my article.
‘Throughout a document, indicate how confident you are in major claims, and what support you have for them.’ – I do indicate my support (Manager Tools) and why I think it’s a support. I am quite confident in my main claim and in the main supporting claims. Note also that I don’t write that outsourcing hiring is terrible, but that you need to be careful. And I don’t write ‘Hirely will cause you to make a bad hire’, but that they put you at risk of making a bad hire. Among others.
I think I’ve refuted part of your claim about personality tests in other comments. Added 2022-10-22: Even if Big Five assessments can predict some things well, I would expect that to only be the case when they’re not gamed. The one I know can easily be gamed. (Eg. just answer ‘very accurate’/‘very inaccurate’ to all the question that sound like pushing up/down the conscientiousness score.) And an important part of hiring is keeping out people that make themselves look better than they are.
Expanded 2022-10-23: Some of the arguments against the ‘bad’ hiring advice had already been addressed in my article. Eg.: I point out that the résumé screen is a flawed filter and needs to be supplemented or replaced, depending on the situation. I briefly argue why it’s okay to take a lot of time interviewing. There is also an easy fix for anchoring/groupthink in the interview results capture meeting (as well as other, more subtle, reasons to do it): Ask people to send you their recommendations and justifications before the meeting.
Added 2022-10-19, edited 2022-10-23: In general, the items about ‘inaccurate or bad advice’ have common-sense arguments for and against them. (If anyone wants me to write any of them out, let me know.) It would come down to analysing Manager Tools’ (unfortunately non-public) data as well as the research you’re referring to. I would expect ‘traditional hiring’ when done right to work roughly equally well to whatever good EA orgs and leading tech firms do. My default expectation when people say that the ‘traditional way to do X is bad’ is that it actually works well when done right, but that it has a bad image because it got corrupted over time/is usually done mediocrely.
It would be nice if you moved your last paragraph first. Recommending (in bold) based on a casual listening not to read a post feels unfair. (Speaking about fairness, there are other posts about hiring (eg. https://forum.effectivealtruism.org/posts/XpnJKvr5BKEKcgvdD/perhaps-the-highest-leverage-meta-skill-an-ea-guide-to#comments) that are argued more thinly than my post and have gotten much less criticism.) I agree with your alternative recommendation to read the post with skepticism. That’s the case with any post.
I agree with some of your points. Especially the one about ways to outsource without knowing about the subject. I might work it into the article. (Two sentences added 2022-10-19.) (2022-10-22: I’ve worked it into the article.) Overall I think there are misunderstandings, which I should have worked harder to avoid.
The main point is not hiring advice. (2022-10-20: Added clarifying comments to this end.) The main point is that people who hire need to learn more about hiring. I’m merely demonstrating this by countering Hirely, which I do based on hiring guidance that I’m fairly confident is better than theirs. How this came about I describe in the section about the article’s history.
I agree that it would be nice to have more epistemic nuance and diligence. But this article wouldn’t exist had I tried for that, because I don’t have more time. Given the choice between not publishing the article and publishing it with some epistemic hubris, I thought the latter is more beneficial.
The main resource for this article is Manager Tools because I think their hiring guidance is an excellent starting point. This is based on wider reading and listening and thinking and experiencing, which I have been doing for years. I agree that their arguments are often sloppy and there are areas other than hiring in which they’re weaker than they admit. Your claim that ‘they also usually provide no evidence or arguments for their claims except their personal experience’ is wrong as it stands. They provide evidence for some of their claims, which I’ve linked in my section on why Manager Tools is a decent authority. (Note that I wrote ‘a decent’, not ‘the ultimate’ – epistemic humility.) I wish they would provide evidence for other claims, too. And the bulk of each podcast/whitepaper is made up of arguments for their claims, spotty though they might be at times. Personal experience can be a valid support for arguments, too, if it’s believable that you have a lot of it. It is believable that MT has a lot of personal experience (and data) about hiring, which I also point out in my section about them.
By the way:
I do think I’m fairly transparent and indicate levels of confidence to some degree, perhaps in a different way than you’re used to. I don’t have time to read and digest the full Reasoning Transparency article. But if I look at the top recommendations:
‘Open with a linked summary of key takeaways.’ – That’s the title in my article.
‘Throughout a document, indicate which considerations are most important to your key takeaways.’ – With some squinting that’s the table of contents in my article.
‘Throughout a document, indicate how confident you are in major claims, and what support you have for them.’ – I do indicate my support (Manager Tools) and why I think it’s a support. I am quite confident in my main claim and in the main supporting claims. Note also that I don’t write that outsourcing hiring is terrible, but that you need to be careful. And I don’t write ‘Hirely will cause you to make a bad hire’, but that they put you at risk of making a bad hire. Among others.
I think I’ve refuted part of your claim about personality tests in other comments. Added 2022-10-22: Even if Big Five assessments can predict some things well, I would expect that to only be the case when they’re not gamed. The one I know can easily be gamed. (Eg. just answer ‘very accurate’/‘very inaccurate’ to all the question that sound like pushing up/down the conscientiousness score.) And an important part of hiring is keeping out people that make themselves look better than they are.
Expanded 2022-10-23: Some of the arguments against the ‘bad’ hiring advice had already been addressed in my article. Eg.: I point out that the résumé screen is a flawed filter and needs to be supplemented or replaced, depending on the situation. I briefly argue why it’s okay to take a lot of time interviewing. There is also an easy fix for anchoring/groupthink in the interview results capture meeting (as well as other, more subtle, reasons to do it): Ask people to send you their recommendations and justifications before the meeting.
Added 2022-10-19, edited 2022-10-23: In general, the items about ‘inaccurate or bad advice’ have common-sense arguments for and against them. (If anyone wants me to write any of them out, let me know.) It would come down to analysing Manager Tools’ (unfortunately non-public) data as well as the research you’re referring to. I would expect ‘traditional hiring’ when done right to work roughly equally well to whatever good EA orgs and leading tech firms do. My default expectation when people say that the ‘traditional way to do X is bad’ is that it actually works well when done right, but that it has a bad image because it got corrupted over time/is usually done mediocrely.