Thanks for your comment on my article! I appreciate your thoughts and have left a lengthy answer.
Richard Möhn
Karma: 268
Who would get to see this policy statement? If I imagine myself in the shoes of a more conservative potential donor, who is checking the fine print as part of their due diligence, I would be put off by phrases like ‘metamour’, ‘consumption of drugs’ and the repeated mentioning of sexual relationships.
Some suggestions:
Go over this with a lawyer and let them formulate it right.
Replace ‘romantic and/or sexual’ with ‘intimate’.
Right now those terms are at the top level of the bullets. One could make them stand out less by turning the hierarchy around. Example:
Sufficient causes for recusal of a fund member: Applicant is a close family member, or is/was a romantic/sexual partner within the last year.
Not sufficient for recusal, but should be made public: Intimate relationship that lasted longer than two weeks and ended more than a year ago. / Current intimate relationship with a third person who is in an intimate relationship with the applicant.
Pull things together to make them more innocuous:
The fund member has shared particularly intense experiences with the applicant on the level of, eg. a week-long silent meditation retreat, a CFAR workshop, or a shared psychedelic experience.
Regarding the lawyer: Perhaps I have a wrong idea both about the legal nature of the document and about what a lawyer does. But yes, in my imagination a lawyer is able to advise you on how to formulate a policy clearly and remove vagueness. So I might go to the lawyer and ask: ‘If this were to be legally binding, how would we have to write it?’
My imagination might be wrong, though – I haven’t dealt with lawyers much.
It’s better if potential donors are persuaded by content, not by form, ie. choice of words.
https://www.lesswrong.com/posts/HDXLTFnSndhpLj2XZ/i-m-leaving-ai-alignment-you-better-stay is relevant to how independent research attempts could be improved. I describe my attempt at independent AI alignment research and how I could have done better. It applies to other fields, too.
Manager Tools has great guidance for writing résumés: https://www.manager-tools.com/products/resume-workbook And for applying and interviewing in general: https://www.manager-tools.com/products/interview-series
(I’m not affiliated with Manager Tools.)
<First sentence redacted.>
The above article, too, contains many questionable pieces of advice. Just one example: ‘Craft your questions so that it’s not embarrassing or difficult for candidates to admit that they don’t have the personal qualities that you are looking for.’ So you’re supposed to rely on candidates admitting that they are not right for the job? Take the article’s example of risk-taking – why wouldn’t you write in the job ad that risk-taking is required? Then the honest non-risk-takers can sort themselves out before going through the trouble of applying.
What about those who don’t sort themselves out? According to the article, you ask them a sort of trick question: ‘are you somebody who likes to jump head first into a problem or do you hold more of the philosophy of looking before you leap?’ This can work if you’ve successfully left the candidate in the dark on whether the job requires risk-taking. If the candidate knows that risk-taking is required and really wants the job (and who doesn’t?), guess which of the two alternatives they will pick.
Talking about going through the trouble of applying – the above article doesn’t address the feedback about hiring that made a splash on this forum a while ago: https://forum.effectivealtruism.org/posts/jmbP9rwXncfa32seH/after-one-year-of-applying-for-ea-jobs-it-is-really-really I don’t agree with the attitude of the latter article, but it does contain important points that ‘an EA Guide to Hiring’ would do well to address.
Addendum – a better question to find out about a candidate’s risk-taking. Say: ‘It’s hard to have a big impact without taking risks. Tell me about a time when you had to choose between a safe and a risky course of action. What were your considerations, how did you implement your choice and how did it turn out?’ After they’ve given an answer, probe. Ask follow-up questions. Always about the past. If all you dig up is more details about successful risk-taking in the past, there’s a good chance the candidate fulfills your risk-taking requirement. On the other hand, if they don’t have done much risk-taking, you will find out quickly using this course of questioning. This is not to say that you should turn your desk lamp on them and play a Stalin-era interrogator. All the points about being extremely friendly and putting the candidate at ease still apply.
Note to those who are downvoting this: Please describe your disagreements in a comment. Otherwise nobody can learn from them.
I would read over it and not draw any conclusions about the organization. Although I don’t think any job would match my dream job ever. So if I took all ‘dream job’ ads literally, I would have to stop reading them after the first sentence.
Thanks for attempting an explanation! I’ve now added a bit of clarification at the end of the introduction. Ie. I did write a closed criticism and gave them a chance to respond. They didn’t want me to publish it as-is, so I ‘rewrote’ it to be more general. As is obvious now, I should have been less lazy and rewritten the whole thing rather than only searching and replacing the org name and rewriting the introduction.
To reduce guessability further, I’ve now rewritten the quote as much as I could while preserving its flavour.
Thank you, too, for explaining your reservations! And sorry about the word ‘stud’ – I didn’t know the connotations. I’ve now replaced it with ‘ace’.
I’ll try to explain the general assertions and my disdain for scenario questions. This is not arguing against the points you made. It’s just a clarification, which I should work into the article.
-
General assertions – I wasn’t aware of this as a problem of my article. Thanks for pointing it out! – There are so many claims in the article that I don’t have space and time to argue them all properly. That’s why I have lots of Manager Tools links throughout, since they have whole podcasts/whitepapers on each topic, where they do argue things (mostly) properly.
-
Scenario questions – a better way to structure the argument: Assume you need someone who can manage projects well and you have limited time. Which is going to give you more (and more reliable) information about a candidate’s ability to manage projects? Asking about how they’ve managed past projects or about how they would manage project X? Which is more predictive? Now, an experienced person would probably justify scenario answers with examples from their experience. And someone inexperienced could only cite from the course they just visited. So I see how scenario questions can work. Behavioural questions are just a more direct way of getting at what I want to know. (This is still only a plausibility argument. Ultimately it comes down to data, which I haven’t looked at. But I mostly trust MT to have looked at the data.)
-
Good point, thanks! Manager Tools usually explain their guidance in detail, which makes it adaptable to all kinds of organizations. And since MT itself is a small company with, I guess, an unusual amount of trust among staff, I don’t think they would put out material that fails to apply to them.
But I do agree that wider reading is necessary. Paul Graham’s essays, for example, are a good counterpoint to MT’s corporate emphasis, too.
Makes sense. I’m a bit worried that people reading this will take away: ‘We’re a small shop, therefore MT doesn’t apply at all.’ This is not the case and I think Howie would agree. I’ve never worked at a big organization and MT still has helped me a lot. I’ve also read and listened to a ton of non-MT material on leadership, doing work, business, processes etc. So I could well be putting MT guidance in its proper context without being aware of it.
Addendum – from https://www.manager-tools.com/2019/01/manager-tools-data-one-ones-part-1-hall-fame-guidance:
Going back to company size, we’ve done 3 studies comparing the effect of WO3s in small, medium, and large organizations. We have never been able to find any significant difference in R&R improvements based on organization size. We have measured statistically similar improvements in companies of less than 50, and companies greater than 100,000 employees.
WO3s … weekly one-on-ones, R&R … results and retention
This is not directly relevant to the article above, but it’s about one-on-ones, which are a core MT thing and which chapter 4 of the Effective Manager book is about.
Another excerpt, talking about their data in general:
We’ve measured and followed tens of thousands of managers in academia, and government, and non-profits. Hospitals, and charities, and religious organizations, and medical practices, and retail firms like grocery stores and mall-based clothing chains. We’ve measured over a quarter of a million managers in the Fortune 500 alone.
They often say that their guidance is for 90 % of people 90 % of the time. And their goal is: ‘Every manager effective, every professional productive.’
(Since I realize that I sound like a shill for MT, I’ll say again that I’m not affiliated nor have any hidden agenda. It’s just that my article refers to a lot of MT material and I’m trying to add evidence for their authority.)
I agree. Thanks for taking the time to hash this out with me!
Thanks for pointing that out. I had only looked at the validity of each method on its own and not at the validity gain numbers. Don’t the results indicate, though, that you would have to also subject the candidate to a GMA test if you want to get validity from conscientiousness and integrity tests? And GMA tests are rarely performed in hiring processes.
Pulling up an addendum from below (added 2022-10-19):
I would explain the high incremental validity by the fact that a GMA test barely measures conscientiousness and integrity. In fact, footnote ‘c,d’ mentions that ‘the correlation between integrity and ability is zero’. But conscientiousness and integrity are important for job performance (depending on the job [and the hiring manager]). I would expect much lower incremental validity over structured interviews or work samples. Because these, when done well, tell a lot about conscientiousness and integrity by themselves.
Thanks! I’m glad you liked it!
They are the third and fourth row in the table, but the rows aren’t ordered by the validity column. When you order by the validity column, integrity tests are 8th and conscientiousness tests are 12th unless I’ve miscounted.
I admit that I cherry-picked this article, basically only looked at the validity numbers in the table, and don’t know anything else from that literature. This post has a wider view for those interested: https://forum.effectivealtruism.org/posts/j9JnmEZtgKcpPj8kz/hiring-how-to-do-it-better On the other hand, the validity table was excerpted on an 80,000 Hours page for years, which gives me some confidence by proxy. Also, the points of my post rely only lightly on this article.
Added 2022-10-19: It would be nice if you removed the bolding on the text that you found to be wrong. Otherwise the impatient reader is apt to miss the bracketed text above.
It would be nice if you moved your last paragraph first. Recommending (in bold) based on a casual listening not to read a post feels unfair. (Speaking about fairness, there are other posts about hiring (eg. https://forum.effectivealtruism.org/posts/XpnJKvr5BKEKcgvdD/perhaps-the-highest-leverage-meta-skill-an-ea-guide-to#comments) that are argued more thinly than my post and have gotten much less criticism.) I agree with your alternative recommendation to read the post with skepticism. That’s the case with any post.
I agree with some of your points. Especially the one about ways to outsource without knowing about the subject. I might work it into the article. (Two sentences added 2022-10-19.) (2022-10-22: I’ve worked it into the article.) Overall I think there are misunderstandings, which I should have worked harder to avoid.
The main point is not hiring advice. (2022-10-20: Added clarifying comments to this end.) The main point is that people who hire need to learn more about hiring. I’m merely demonstrating this by countering Hirely, which I do based on hiring guidance that I’m fairly confident is better than theirs. How this came about I describe in the section about the article’s history.
I agree that it would be nice to have more epistemic nuance and diligence. But this article wouldn’t exist had I tried for that, because I don’t have more time. Given the choice between not publishing the article and publishing it with some epistemic hubris, I thought the latter is more beneficial.
The main resource for this article is Manager Tools because I think their hiring guidance is an excellent starting point. This is based on wider reading and listening and thinking and experiencing, which I have been doing for years. I agree that their arguments are often sloppy and there are areas other than hiring in which they’re weaker than they admit. Your claim that ‘they also usually provide no evidence or arguments for their claims except their personal experience’ is wrong as it stands. They provide evidence for some of their claims, which I’ve linked in my section on why Manager Tools is a decent authority. (Note that I wrote ‘a decent’, not ‘the ultimate’ – epistemic humility.) I wish they would provide evidence for other claims, too. And the bulk of each podcast/whitepaper is made up of arguments for their claims, spotty though they might be at times. Personal experience can be a valid support for arguments, too, if it’s believable that you have a lot of it. It is believable that MT has a lot of personal experience (and data) about hiring, which I also point out in my section about them.
By the way:
I do think I’m fairly transparent and indicate levels of confidence to some degree, perhaps in a different way than you’re used to. I don’t have time to read and digest the full Reasoning Transparency article. But if I look at the top recommendations:
‘Open with a linked summary of key takeaways.’ – That’s the title in my article.
‘Throughout a document, indicate which considerations are most important to your key takeaways.’ – With some squinting that’s the table of contents in my article.
‘Throughout a document, indicate how confident you are in major claims, and what support you have for them.’ – I do indicate my support (Manager Tools) and why I think it’s a support. I am quite confident in my main claim and in the main supporting claims. Note also that I don’t write that outsourcing hiring is terrible, but that you need to be careful. And I don’t write ‘Hirely will cause you to make a bad hire’, but that they put you at risk of making a bad hire. Among others.
I think I’ve refuted part of your claim about personality tests in other comments. Added 2022-10-22: Even if Big Five assessments can predict some things well, I would expect that to only be the case when they’re not gamed. The one I know can easily be gamed. (Eg. just answer ‘very accurate’/‘very inaccurate’ to all the question that sound like pushing up/down the conscientiousness score.) And an important part of hiring is keeping out people that make themselves look better than they are.
Expanded 2022-10-23: Some of the arguments against the ‘bad’ hiring advice had already been addressed in my article. Eg.: I point out that the résumé screen is a flawed filter and needs to be supplemented or replaced, depending on the situation. I briefly argue why it’s okay to take a lot of time interviewing. There is also an easy fix for anchoring/groupthink in the interview results capture meeting (as well as other, more subtle, reasons to do it): Ask people to send you their recommendations and justifications before the meeting.
Added 2022-10-19, edited 2022-10-23: In general, the items about ‘inaccurate or bad advice’ have common-sense arguments for and against them. (If anyone wants me to write any of them out, let me know.) It would come down to analysing Manager Tools’ (unfortunately non-public) data as well as the research you’re referring to. I would expect ‘traditional hiring’ when done right to work roughly equally well to whatever good EA orgs and leading tech firms do. My default expectation when people say that the ‘traditional way to do X is bad’ is that it actually works well when done right, but that it has a bad image because it got corrupted over time/is usually done mediocrely.
I’ve written on LessWrong about a fairly airtight approach to psychosomatic wrist pain: A cognitive intervention for wrist pain
And I think it’s important that every article writing about physical causes and interventions also contain a section about psychosomatics. Because the people prone to psychosomatic wrist pain might read the warnings of permanent physical damage and disability, and enter a vicious circle of worrying leading to pain leading to more worrying. This is how it was for me, as I describe in the article.