Some surprising hiring practices I follow (as a hiring manager and grantmaker in EA)
The best approach to take to hiring differs by industry.
This means that best practice differs across startups, think tanks, video production, non-profits, academia, and news outlets will all have different practices. The practices that work best in the EA ecosystem will be different again – but unfortunately the effective altruism organisation landscape is much smaller than all of those, and so people haven’t written books on how to do it.
I have been working as a hiring manager and sometimes grant maker in this space for over ten years, and I have developed some views on which practices work well in this industry. And I’ve noticed that others have (semi-)independently converged on these.
Below I’ve written out a few of the ones which I follow. I’ve especially tried to list those which might seem most surprising to people coming from industries like the ones listed above.
I’m trying to do a pretty quick job of this so that it actually gets out there. [Having written that sentence, I proceeded not to publish the post for 2 years. So if you’re wondering why there are errors in it, it’s because this time round I’m actually going to post it rather than finesse it. Really.] I’m not trying to give a well rounded picture of how I do all hiring, partly because I just want to write about a few things that will be interesting to people and partly because hiring varies a lot by role and organisation.
Two resources I’ve found particularly useful for thinking about how I should hire are Sam Altman’s post on How to Hire and Who: The A Method of Hiring.
Run unstructured interviews
I periodically hear people say that everyone should run ‘structured’ interviews: ones where the questions are set in advance and the same is asked of everyone. That does seem like an improvement on some interviews. It means that the interview difficulty is uniform between different candidates. It also forces you to think in advance about what you most want to learn from the interviews.
On the other hand, there’s a limit to how much you can learn in a structured interview, because you can’t adapt your questioning on the fly if you notice some particular strength or weakness of a candidate. It can also be difficult to really get a good sense of how they reason and how good their judgement is.
A case I think of as being at the other extreme is university philosophy admissions interviews. When I’ve done these, the setup aimed to be as close as possible to getting a sense of how well people ‘do philosophy’ by probing in depth the argument a candidate makes. This style of interview is a pretty useful tool to have your toolkit.
Variety between candidates amongst interviews can also be really important. For high skill jobs, you actually want to know a whole host of things about a person before deciding to hire them. It’s likely you’ll have different uncertainties about different candidates. In some cases that will occur as the interview goes, so you’ll want the flexibility to change the questions as you. Sometimes it will be based on their performance in earlier rounds of the hiring process, so you might want to have interviews about different things with different candidates.
Work tests can be feasible and useful even for high-skill roles
Work tests seem to be the kind of thing which is usually thought of as primarily used for either hard skills like programming or lower skill, rote tasks. But I’ve found work tests pretty useful even for roles requiring high levels of soft skills like being an 80,000 Hours Advisor.
Designing work tests that can assess skills like high level management is indeed often harder than designing work tests for hard skills and rote tasks. But it’s very much not impossible.
It’s possible that one thing going on here is really a confusion of terms. The best ways of testing soft skills often include conversation, so might normally be thought of as interviews. For example, 80,000 Hours’ work test for its advising team involves a ~2 hour work test where the bulk of the time is spent on a call with a staff member, discussing possible advisees. It might be that this would be considered ‘best practice’, but referred to as an interview rather than a work test. We distinguish ‘work tests’ from ‘interviews’ in part by there being reading/writing involved in addition, and in part by the fact that it’s intended to mimic the work the person would do on the job.
Perhaps the weirdest ‘work test’ I’ve administered was a grant interview for a therapist. I hadn’t previously directly evaluated therapists for grants, and felt at sea on how to do it. I realised that typically when hiring I got most info from seeing a person having a go at something that was as close as possible to the actual job, and that in this case since I’ve seen a few different therapists in the past I had some sense of what ‘doing a good job at it’ looked like. It worked surprisingly well to talk through a few representative seeming issues with her to get a sense of how she would handle them.
My guess is that these types of work tests are underinvested in because they’re quite a lot of work to prepare for. They often vary role by role rather than being able to reuse them for lots of vacancies, and are tricky to design so that they provide useful information without being prohibitively time consuming for both parties.
Informal references are useful
The types of references people expect vary markedly across sectors:
In large companies and local governments, it’s often accepted practice to provide only written references, and for those to only say ‘the person worked in this role from x date to y date’. This avoids people taking issue with bad references, and can help avoid both abuse (threatening someone with a bad reference) and nepotism.
In academia, long reference letters are common for early stage academics who don’t have much of a publication record. There’s wide variety in them, but they try to get reasonably specific, including making comparative claims like ‘this is the best student I’ve taught this year’.
The book ‘Who’ recommends asking specifically for verbal references so the referee is more willing to be candid and so that you can ask specific questions.
Who to ask for references from also seems to vary by sector. The easy default is to ask the applicant to provide suggested referees and only reach out to them. ‘Who’ suggests specifically telling candidates you want to talk to their boss and that you’ll do that after interviewing them, in order that you get more accurate answers from them. Start-up founders sometimes recommend talking to essentially as many people as you can about a potential hire, to really get a picture of them.
Given these differences in advice, it seems useful to know what causes the differences to get a sense of which advice is likely most applicable for your situation.
One thing going on is likely to be the stakes involved in particular hiring decisions. If a large company or local council hires one mediocre candidate, the overall system can deal with it. By contrast, a start-up having one mediocre early employee could make the difference between success and failure.
Another difference is likely to be the overall size of the ecosystem an organisation is in. A recruiter for high level legal executives is working with a reasonably small, specialist pool. They would know people who have worked with particular candidates well enough to ask for a take on the person. They’ll be able to trust people to tell the truth because they’ll work together in future so need to keep trust.
I’ve found getting a broad range of references very useful in the past. Ideally, I can find people who I trust to give a reasonably honest take and have worked with both me and the applicant. The reason is that if I’ve worked with someone, I likely have a goodish sense of how good they are at the skills I’m evaluating, and also some sense of how effusive they typically are. That makes it easier to tell how good a reference actually is.
People in the EA ecosystem are often touchingly generous with their time and willingness to provide references.
Get information from people with conflicts of interest
Another type of reference I think sometimes gets an unwarranted blanket ban is from people with a conflict of interest. As a stark example, in one organisation I worked for, if a person was quitting or going on parental leave, they were not allowed to be involved in hiring for their replacement due to potential conflict of interest. (For example, they might want to ensure that their replacement is ‘like them’ rather than focusing on whether the person would do a good job). That seemed kind of crazy to me. After all, the person who had been in the role had the most information about what it was like to do the job and what was needed to do it well.
Unfortunately, correlation between people having information on a candidate and having a conflict of interest is pretty frequent. Someone who has worked extensively and closely with a candidate can provide detailed information on their strengths and weaknesses. But they also likely get on well and feel positively disposed to them. After all, they know each other well and often will have deliberately chosen to continue working together. Someone who works in the same lab as a grant applicant likely has a good sense of how good they are at their job. They also likely have a vested interest in their lab getting more grants since it will have spillover prestige effects.
Avoiding getting references or other information from people with a conflict of interest is plausibly better than totally ignoring conflicts of interest. But those usually aren’t the only options—you can typically get some sense of how serious a person’s conflict of interest is, and take their information into account appropriately downweighted.
You can put numbers on things (but don’t trust them too much)
Putting numbers on the qualities people have feels pretty gross, which is probably why using quantification in hiring is rather polarising. On the one hand, there’s some line of thinking that the different ways in which people are well and ill suited to particular roles isn’t quantifiable and if you try to quantify it you’ll just be introducing bias. On the other hand, people in favour of quantification tend to strongly recommend that you stick exactly to the ranking your weightings produced.
I think I probably use quantification of this kind a little less than I endorse, because it feels so difficult to decide what the weightings should be and then to actually figure out for each person how to assign them. I do think it’s often a useful exercise for getting me to be explicit about the types of traits I care more and less about, and what evidence I have that different candidates have those traits.
On the other hand, I think that it’s mostly impossible to get formulae you fully endorse and that actually capture everything you want and nothing else. For that reason, I would use the quantification as a tool for guiding my thinking, rather than as the only input.
Bonus suggestion for applicants: Ask about the people you’ll be working with
My impression is that it’s not very common for applicants to try to talk to people who might be able to give candid views of their potential future colleagues (including their manager). In many cases, that’s too difficult to do. In others, it’s reasonably straightforward. For example a PhD applicant could ask their prospective supervisor’s current grad students what it’s like to work with the supervisor. Yet, at least when I was applying to grad school, this was not very common.
Colleagues, particularly managers, make a huge difference to your work experience. The extent to which you have compatible styles, and the extent to which you trust the judgement and integrity of those you’ll be working with, is incredibly important. So it can be really useful to get a sense of those things before accepting a job offer.
You should make these enquiries as sensitively as possible. You might want to ask people who used to work at the company rather than (or as well as) those who currently do, since the former will feel more free to speak their mind.
Getting advice
I think it’s very important to get advice from people with significant hiring experience. That includes both consuming content about it and talking to people individually.
A significant risk of not following ‘best practice’ is trusting your own intuitions too much. Worse still would be just not putting much thought into hiring. I think we should in fact be putting more rather than less time into hiring. We should try to get a sense of what practices are used in a range of contexts, design a process that draws on the elements most sensible for a particular situation, then get feedback on it from people with relevant experience.
Of course, doing things entirely in a bespoke way is not always possible—in cases where you have limited time, borrowing an existing template is likely to work better. For those cases, it seems useful to have a sense of whose templates you want to borrow—likely depending on what context your situation is most similar to.
Personally, I’ve typically found start-up advice the most useful, since it’s aimed at small, young organisations with large ambitions.
I’ve also found it useful to reach out to people who seem to have relevant expertise when I have a specific quandary, to talk it through it with them. To find people it might be useful to talk to and would likely be happy to, you could go to an EA Global or reach out to people on the EA UK directory.
I’m grateful for comments from a number of people, but Brenton Mayer particularly helped shape the post and Will MacAskill’s haranguing caused it to actually get published.
I agree with a lot in this post! Especially doing work tests for high-skilled roles, and getting information from people with conflicts of interest. Thanks for writing this!
I’m not sure I agree with the straightforward reading of this (though maybe that’s slightly different than what you mean). I think the case for running semi-structured interviews is better than running structured ones. But overall, I don’t take “structured interviews” to mean “only ask the same set of questions”. I take at least some of the literature on it to refer to asking roughly the same questions, then using structured follow ups until you’ve gotten a lot of information on each question from a candidate. The important part of the “structure” to me is trying to get comparability on their skills on specific attributes, not comparability on their response to the question.
Some general reflections I’ve had about hiring that feel related to this, but also make me skeptical of deviating from best practice (e.g. develop a job analysis, test people directly on the items in the job analysis in a structured way).
The more hiring I do, the less confidence I feel about ~any practice. This pushes me more and more toward “see how they do at the actual job or the skills that seem needed for the job.”
I think that often what I want to do in hiring is basically very biased by what’s pleasant for me. Structured interviews are very boring, especially when you do a lot of them. It feels like I’m not getting to know someone deeply. But when I look back at my most successful hires, it’s almost always me following a well designed process. The only cases I see where I deviated heavily from the process and it went well was when I had worked extensively with the person before and knew them quite well (though this could just be evidence that I’m not very good at casually assessing people without structure).
Relatedly, this is anecdotal, but I notice organizations who are less formal in their hiring seem also less happy with their hires. I don’t feel as confident in this claim though.
Hiring is also hard if you’re progressively eliminating candidates across rounds, because you never can measure the candidates you rejected. The candidate pool is always biased by who you chose to advance already. This makes me feel like I’m never collecting particularly useful data on hiring in hiring rounds. I don’t ever learn how good the people I rejected were!
I agree that these are very useful, but I’ve also noticed them increasing in frequency in the ecosystem, and that concerns me. I think these can be really biasing. I’ve started turning down most requests for these for a few reasons, and now usually ask the hiring manager if they can get the candidate’s consent for my reference.
If someone left a workplace in a termination, sometimes, especially with employers of record, there might be a non-disparagement clause. My choice to take or turndown an informal reference might be primarily informed by my legal obligations, but because I can’t say that, the person collecting the reference doesn’t have any info on why I’m saying I won’t give a reference.
I often feel like informal references don’t pass a smell test of “if the candidate knew the hiring manager was collecting this, would they be okay with it.” That feels like I’m breaking a candidate’s trust.
I’ve been asked for informal references for my direct reports. I’m thankful that often these direct reports have shared that they were job searching with me, but if they hadn’t, they probably would rightfully consider this a pretty big violation of trust on the hiring manager’s end.
There are cases where the above don’t apply, but I do think candidates deserve to know that there is some level of reference collecting happening from references they didn’t share.
Nice comment. One follow-up.
Do you think this basically supports “hiring people you know to be good” and using your network and previous interactions with people not in an interview setting to seek out good candidates?
Yeah, I think it provides some evidence in favor of it, but there are lots of downsides to that too, like:
Obviously, there is risk a bias, etc. (e.g. I know a small subset of possible people!)
Lots of times, I don’t actually know someone who would be a particularly good fit.
I think that doing this is lower downside risk, but probably somewhat lower upside potential in expectation, and probably just varies case-to-case in how those shake out overall.
Isn’t this avoidable? I could imagine a system where you allow a small percentage of randomized “rejected” candidates to the next hiring round and, if they properly succeed in the next, allow them into the third round. I have essentially no experience with how hiring works, but it seems to me that this could i) increase the effort that goes into hiring only moderately, ii) still sounds kind of fair to the candidates, iii) and would give you some information on what your selection process actually selects for.
Yeah definitely, I think that would be a really reasonable thing to do, and is the kind of experimentation I want to see in hiring in the space that I talk about here!
Definitely agree this is a challenge and limits learning from hiring rounds. I don’t quite agree with the strength of this statement though. A great thing about the agency and tenacity of people in our community is that people often apply to a bunch of roles at similar organisations, and so you end up seeing what people you turned down work on from afar, and how well that goes. It seems reasonably informative to me to track which people end up doing great work at other orgs, and thinking through how much of that seems down to the role being different vs your hiring process having been a false negative, and thinking through what the process would have looked like to have avoided the false negative.
I agree. Very often I think that semi-structured interviews (which has a more or less closely planned structure, with the capacity to deviate), will be the best compromise between fully structured and fully unstructured interviews. I think it’s relatively rare that the benefits of being completely structured outweigh the benefits of at least potentially asking a relevant followup question, and rare that the benefits of being completely unstructured outweigh the benefits of having at least a fairly well developed plan, with key questions to ask going on.
Firstly, great post thanks for writing it!
Secondly, with regards to the quantification section:
I just wanted to mention something that I’ve been experimenting a bit with lately that I think has worked reasonably well when it comes to this? One of the problems here is the overindexing on the numbers that you assign to people and taking the numbers too seriously. A way to go around taking things to seriously is play and we did an experiment where we took this seriously.
When we took mentees into our latest research program we divided people up into different D&D Classes such as “wizard”, “paladin” and “engineer” based on their profiles. You’re not going to be able to make a decision fully based on the experience level someone has as a “paladin”, yet you’re not going to feel bad using the information.
I imagine it can be a bit hard to implement in an existing organisation but I do think this degree of playfulness opens up a safety in talking about hiring decisions that wasn’t there before. So I’ll likely continue to use this system.
I’ll post the list of classes below as well as how to evaluate their level from 1-10 if anyone is interested (you can also multi-class and experience is within a class):
Tank - Can take a bunch of work and get things done
Healer - Helps keep the team on track with excellent people management
Paladin—A leader that can heal but also take on a bunch of the operational work—generalist
Sorcerer—Communicator & creative that can magic things out into the real world intuitively
Bard—A communicator that has experience with talking with external stakeholders & writing beautiful prose about the work
Engineer—Technical person who can make all the technical stuff happen
Wizard—Organised researcher with deep knowledge in fields that can create foundational work
Diplomat—Understanding institutional design and governance structures and crafting policies and frameworks that enable coordination
Levels:
1 - Hasn’t slain rats yet—no experience
3 - Finished the sewer level—Finished undergrad + initial project in AI Safety
5 - Can fight wolves relatively well—Done with PhD + initial knowledge in AI Safety
7 - When you’re slaying an epic monster you want this person in your team—Experience with taking responsibility in difficult domains
9 - Could probably slay a dragon if they try—Wooow, this person is like so cool, god damn.
10 - Legendary expert—possible one of the best people in their field
Thanks for writing this! I strongly agree re work trials, unstructured interviews, informal references, and info being useful despite conflicts of interest
I often advise doing this, albeit slightly differently—talk to their recently graduated former PhD students, who have a better perspective on what the process led to and how valuable it was in retrospect. I think similar advice plausibly applies in corresponding cases—talk to people who used to work somewhere, instead of current employees.
Nice post, thanks for writing it Michelle. Speaking to multiple referees is a good tip.
I especially appreciate the advice to speak to learn about the org you will be joining and the people you will be working with. Very underrated advice. Usually people are so happy to accept the offer, while they should really try to jump on calls with some future collaborators and speak to former employees (which as you said is doubly effective as these people usually have nothing to hold back).
Executive summary: Drawing on over a decade of experience as a hiring manager and grantmaker in the EA ecosystem, the author outlines unconventional but effective hiring practices—such as unstructured interviews, flexible work tests, and informal or conflicted references—arguing that context-sensitive, information-rich, and adaptable approaches outperform standardized “best practices” borrowed from other industries.
Key points:
Unstructured interviews can yield deeper insight than rigid formats, allowing hiring managers to adapt questions to each candidate’s strengths, weaknesses, and reasoning style—especially valuable for high-skill roles.
Work tests are underused for roles requiring soft skills but can be powerful when tailored to mimic real work; they provide richer signals than traditional interviews despite being harder to design.
Informal and wide-ranging references—including from peers or acquaintances outside formal referee lists—often offer the most candid and useful information, especially in small ecosystems like EA.
Conflicted sources shouldn’t be ignored entirely; instead, evaluators should weigh their input proportionally, since people with conflicts often also have the best information.
Quantification in hiring can help clarify priorities and trade-offs but should guide rather than dictate decisions, as numerical models rarely capture all relevant qualities.
For applicants, the author advises discreetly seeking candid feedback from potential colleagues or former employees, since manager fit strongly influences job satisfaction.
Overall hiring philosophy: adapt best practices to EA’s small, high-stakes environment; borrow selectively from startup culture; and consult experienced hirers to refine bespoke approaches.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
Super useful! Thanks for writing!