Hiring Process and Takeaways from Fish Welfare Initiative

haven6 Apr 2020 5:30 UTC
114 points
12 comments11 min readEA link

Who should read this: This post will likely only be useful to those employers who will be directly involved in a hiring process, although the Recommendations for Applicants section should be useful for most job applicants. Job applicants may also find it interesting to learn about the employer side of the process.


We (the co-founders of Fish Welfare Initiative) recently completed our hiring process for our first new full-time employees: a Research Analyst and an Animal Welfare Specialist (more on that distinction later).

As neither of us had previous hiring experience, we set out to build a process based on the best available evidence on how to hire effectively, objectively, and kindly. The following is what we found and learned.

We hope that this process and the linked templates will be useful to your organization and will save you some of the large time investment required to create a new process. If you have any questions or comments, feel free to comment below or contact us.

Big Takeaways

  1. Probably the best hiring advice we received came from the CEO of a GiveWell-recommended charity. He looks for candidates who are “smart, nice, and really want the job.”

  2. Your hiring process is a reflection of your organization. To reflect FWI, we aimed to make our hiring process evidence-based, compassionate, unconventional/​innovative, and requiring some dedication.

  3. If you’re not already, you should use Calendly or another scheduling software to schedule all interviews.

  4. We found EA Facebook pages, our website, and personal recommendations to be the best places to find talented applicants.

  5. Score everything with a template, where applicant materials and questions are all scored quantitatively. This will help you increase objectivity. You should input these scores for each round into one master spreadsheet.

  6. With interviews, we updated away from asking the same somewhat shallow questions. Rather, asking fewer and more probing semi-structured questions provided more valuable information.

  7. Don’t be afraid to gather more information about a candidate: additional calls, emails, and interviews can all be helpful.

  8. Don’t be a jerk to your applicants. Too many employers are. Your applicants will appreciate you for how you treat them and leave with a good impression of your organization.

Resources We Used

We relied heavily on the following resources and highly recommend looking them over. We agree with most of the recommendations they make, and have tried to restrict this post to primarily our own original takeaways so as not to duplicate work.

Additionally, although it was published towards the end of our hiring process, Notes on hiring a copyeditor for CEA is also a good resource.

We are very grateful to the organizations and individuals who created these resources.

The Role We Hired For

We were originally looking for a researcher who had prior knowledge and (ideally) credentials with fish and animal welfare. As this was the first hire FWI was going to make, we also wanted someone who would be able to take a leadership role in shaping the organization.

We ended up advertising for two separate roles: a Research Analyst and an Animal Welfare Specialist.

Advertising for Two Separate Jobs

Initially, we were unsure whether we wanted someone who was an early-career generalist (flexible and value-aligned), or someone who was later-career and had more domain knowledge and credentials (although possibly less flexibility and value alignment). In order to attract both types of applicants (even though we only intended to hire one person), we advertised for two separate roles: a Research Analyst and an Animal Welfare Specialist. We believe this allowed us to attract a much broader pool of applicants than we otherwise would, and we were sufficiently pleased with the finalist candidates that we did end up hiring one for each role after all.

Takeaway: A role’s name and qualification requirements significantly impact who applies for it.

Recommendations for Applicants

For more in-depth recommendations, see posts by the Effective Altruism Foundation and Charity Entrepreneurship.

Here are two more recommendations we had for applicants that weren’t listed by EAF or CE:

  1. Check your spam folder. Our emails going to spam was a problem for several applicants.

  2. Show interest early and often to the organization. The candidates we were most excited about were the ones who reached out early to us, and who regularly responded to our emails and asked us questions throughout the process.

Application Process Overview

Our application process consisted of six broad stages:

  1. Initial application (includes filter questions, resume, cover letter, optional writing sample)

  2. 15-minute short interview

  3. Test task (up to 8 hours)

  4. 45-minute long interview

  5. Reference check

  6. Calls with finalists and answering remaining questions

Within 4.5 weeks, we received 82 applications. Two months following the close of our initial application we made two full-time offers, which were both accepted. The following table provides further detail on our application stages (the time estimates were estimated by us):

We did not use any preset cutoff score to determine the number of candidates who would proceed to each round. Rather, we created rough cutoff scores after scoring at least some of the applicants. We may think more about using preset scores and criteria in the future (we expect using these could make the process more objective), although it is difficult to know what score makes a good cutoff before having calibrated yourself by reviewing several applicants.

Over the course of the process, 5 candidates withdrew their applications. We expect that we could have slightly lowered this number with a speedier application process. Although we felt there was sufficient value in each of our sections to justify their time cost, we will think more in future hiring rounds about how we can shorten the application length (perhaps by running stages concurrently, instead of discretely, for different candidates).

Internally, we spent roughly 150 hours on the application process, divided between our then two-person team.

We estimate that our application process caused all of our applicants to spend a combined total of roughly 616 hours working. We take this as a good reminder to be cautious of how much time we ask for from applicants.

Application Process in Detail

Stage 0: Spreading the Word

The following table lists where our applicants heard about the job:

What We Learned:

  1. Posting on EA Facebook pages (e.g. Effective Altruism Job Postings) is very helpful for finding talented EA applicants.

  2. Asking particular people who know many potential candidates is also very helpful, and will produce a disproportionate number of qualified applicants. One of our 8 final candidates learned of the job via a tweet from a famous fish researcher.

  3. Veganjobs.com seems to be a very common way for people in the animal protection movement to find new jobs.

Stage 1: Initial Application

In the job ad, we asked applicants to complete a Google Form. This required them to upload their resume/​CV, cover letter, and an optional writing sample. It also included several filter questions, the answers to which we could easily read and use to rule out some of the candidates.

What We Learned:

  1. Filter questions were useful, especially logistical questions. In the future, we will also likely ask:

    1. “Are you willing to relocate to <insert city name>?”

    2. “When can you begin? We will only be considering candidates who can begin by <date>.”

    3. A question getting at whether they were involved with animal rights (this told us a lot about an applicant’s motivations and helped our shape future questions; we wanted someone who was fiercely passionate about animals but still able to work collaboratively with industry).

  2. Candidates from certain countries (especially in Europe) often include a headshot in their CVs. In order to avoid potential bias, in the future we will explicitly ask candidates on the form not to do this.

  3. We asked for specific salary expectations, and found this to be useful in immediately filtering out some people whom we could never afford. However, if you are recruiting internationally make sure to specify a currency for the salary.

  4. It is probably not a good idea to have optional materials in the first stage: it just made scoring more complicated. In the future, we will likely request additional materials such as a writing sample at a later stage.

Stage 2: Short Interview

The purpose of the short interview was to rule out candidates by screening them on knowledge of fish, communication skills, past history, tolerance for instability, and value alignment. Interviews lasted 15-30 minutes, and were largely structured.

We did several practice interviews before interviewing real applicants, and we found these helpful.

What We Learned:

  1. Interviews are hard to get right. Nearly everyone will say that they are good at the thing you ask about, and will be able to provide some vague examples. The trick is being able to endure some social discomfort by pressing candidates on their examples in order to assess how impressive the examples (and the candidate) actually are.

  2. We initially graded the applicant on each question. In the subsequent interview we later changed to having a list of general traits (e.g. value alignment, domain expertise, communication skills), and then adding a +1 or −1 each time the applicant said something relevant to that trait (For instance, for “value alignment” mentioning that altruism was an underlying motivation in their lives would get them a +1, while showing little passion when discussing their past work with fish welfare would get them a −1). We think this latter approach worked better.

  3. We were not very pleased with our interview questions, as we had so many (about 15) that we lacked time to ask follow-up questions. In the future, we will likely stick with just 3-6 questions in the first interview, and press candidates for further detail on each.

  4. See Hire with Your Head for helpful advice on interview questions that we wish we would have started following sooner.

Stage 3: Test Task

For the test task, we asked candidates to do a very similar task to the work they would do on the job.

What We Learned:

  1. The test task proved to be a good filter because several people just didn’t complete it (in our case 3 people, or 18% of the remaining applicants). Although this could be due to the more talented candidates with other offers not being willing to spend more time on our application, we suspect (and hope) that the more likely explanation is that these candidates were less serious about the position.

  2. We instructed candidates not to spend over 8 hours on the task. However, in the future we will make explicit exactly what the consequences are for surpassing this time cap (i.e. a point deduction vs disqualification of application). Leaving it ambiguous meant that some candidates went well over the limit and produced a more polished report, while others kept the time limit and produced a less finished report. This made evaluation somewhat harder. For shorter test tasks, requiring that they be done in a single, timed sitting would resolve this ambiguity.

  3. Our test task involved contacting certain people in the real world (we’re being intentionally vague here so we can use the task again) in order to test applicants’ communication abilities. This has the drawbacks that 1) some of it is beyond the applicant’s control, and 2) some applicants will find this very stressful. Retrospectively, we are glad we included such a task as such communication skills are an important part of the job and are difficult to test for in interviews.

  4. Up to 8 hours is a long time for a test task, and is more than most people will be accustomed to. While two applicants gave us negative feedback about this, we think the insight we gained into the applicant’s output ability and desire for the job well outweighs this time cost.

Stage 4: Long Interview

The purpose of the long interview was to gain deeper insight into the applicant’s experience, ethical alignment, and fit for the job. It is also an opportunity to get to know the applicant much better. We had our other co-founder conduct the long interview in order to ensure that both of us felt comfortable with the candidates.

What We Learned:

One of the most helpful questions we asked was: “We are most concerned with X part of your application. Do you think this is a valid concern?” For instance, we asked about lack of prior experience with fish, and comfort with being managed by a younger manager. In addition to gaining further evidence about a potential weakness, this also helped us assess a candidate’s ability to handle critical feedback.

Stage 5: Reference Check

We were initially unsure about the use of references, but retrospectively we are glad we used them. Even if nothing negative comes up, we found references to be useful to distinguish people who produce overall good work from people who produce overall exceptional work.

The best questions were the ones to which it was hardest to give vague positive answers. For instance, one of our favorite questions was: “Compared to all the students/​employees you worked with, what percentage would you place the applicant in?” You might follow it up with: “What would it take for them to go up five percentage points?”

Stage 6: Call with Finalists

One of the best pieces of advice we received was to take our time with the process, and not make a decision until we were extremely confident in it. We thus had informal calls with all of our finalists in order to get to know them better and confront any remaining doubts.

What We Learned:

Especially on small teams, someone’s likeability is, to some extent, both a valid and invalid reason to hire them. We must be cautious of bias, but on the other hand an organization will run more effectively if its employees generally like each other. Such phone calls (as with any informal discussions) will leave additional room for biases, but overall we think they were useful as they allowed us to get a fuller picture of what it would be like to work with someone.

Candidate Evaluation

For most stages, we graded candidates on a rubric. We think this was useful as it increased objectivity, and allowed us to go back and see exactly what we thought of one candidate compared to another. We also used weighted criteria, then took the sum of the products of each criteria and its weight for the final score of an applicant in a round.

See our Candidate Evaluation spreadsheet.

Calibrations

With most of our graded stages, we had some form of internal calibration to ensure that 1) both reviewers were interpreting the scoring criteria in the same way, and 2) neither reviewer had any bias at play (of course, this doesn’t help if both reviewers share the same bias). Our calibration involved selecting a random sample of candidates scored by one reviewer, and then having the other reviewer score them to see how the scores compared. This did not take much time and we are glad we did it.

See our Calibration spreadsheet.

Diversity and Inclusion

We took diversity, inclusion, and objectivity seriously in our application process. The best resource we found on the topic was Sentience Institute’s blog post Effective Strategies for Equity and Inclusion.

Here are two of the things we did that you may not already be doing. Both apply to the initial job ad:

  1. It can be tempting to put a laundry list of all the qualifications that a perfect applicant would have. However, you should only list the qualifications that are genuinely required, as a massive list of requirements may unduly lead to fewer female applicants (see article).

  2. Paste the ad into a gender decoder to ensure a roughly equal amount of masculine- and feminine-coded words. We used this one.

We decided against anonymizing parts of the application process, as the research appears to be mixed as to whether anonymization is actually helpful. However, we are unsure about this and may do some anonymization in future hiring rounds.

Not Being a Jerk

It’s easy to see candidates as just another data point on your spreadsheet, but behind each of those data points is a person who is likely stressed out about getting a job and spending a lot of time on your application. We tried several things to make the process as straightforward and kind as possible:

  1. We sent out a hiring process schedule to all those who applied (although retrospectively, we wished we would have just sent it to those who made it past the initial application, so as to not make it as obvious to the people who were eliminated in the first round).

  2. We notified everyone, as soon as we reasonably could, whether they would be moving on to the next round or not. It’s concerning how many employers don’t do this.

  3. We started the interviews with easier questions (e.g. “What are your biggest strengths?”) to help people overcome their initial anxiety.

  4. We sent personalized rejection emails to our finalist candidates.

We received positive feedback from the applicants, even after they had been rejected, on all of these things. Aside from the inherent good in being kind to people, we expect that it is instrumentally valuable to leave people with a good impression of your organization.

We wish we would have:

  1. Sent people a confirmation email after having received their test task. Not receiving a confirmation email likely left some people stressed about whether we received their task.

  2. Not (initially) made the test task due right after the holidays. However, we also didn’t want to lengthen the application process further.

  3. Thought more about how to shorten the overall application process length.

Learning How to Reject People

For many reviewers (including ourselves), having to reject a large number of talented, nice people is emotionally draining. The best piece of advice we can give regarding rejecting people is the following:

Rejection only means that the applicant is not the right fit for this position at this time. It does not mean that the applicant isn’t an awesome person or someone who will never be successful. By turning someone down for this job, you may in part be enabling them to get a future job where they would be even happier.

Templates

See this folder. It includes the following materials, which are openly available for download, modification, and use:

These are mostly the same as the questions and documents we used in our actual hiring process. Therefore, before using them we recommend re-reading the relevant section(s) of this post and making the changes that we suggest.

Conclusion

We hope this has been helpful to your organization. If you have further questions, feel free to comment below or contact us. Good luck!

haven6 Apr 2020 5:30 UTC
114 points
12 comments11 min readEA link