Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
I hadn’t seen this until now. I still hope you’ll do a follow up on the most recent round, since as I’ve said (repeatedly) elsewhere, I think you guys are the gold standard in the EA movement about how to do this well :)
One not necessarily very helpful thought:
is a noble goal, but somewhat in tension with this goal:
It’s really hard to make a strictly timed test, especially a sub-one-day one unstressful/intense.
This isn’t to say you shouldn’t do the latter, just to recognise that there’s a natural tradeoff between two imperatives here.
Another problem with timing is that you don’t get to equalise across all axes, so you can trade one bias for another. For example, you’re going to bias towards people who have access to an extra monitor or two at the time of taking the test, whose internet is faster or who are just in a less distracting location.
I don’t know that that’s really a solvable problem, and if not, the timed test seems probably the least of all evils, but again it seems like a tradeoff worth being aware of.
The dream is maybe some kind of self-contained challenge where you ask them to showcase some relevant way of thinking in a way in time isn’t super important, but I can’t think of any good version of that.
This looks great! Thanks for sharing the process that your team used. I really like reading about how different orgs design and use hiring systems. I also want to say that I strongly approve of grading responses rather than candidates, and of anonymizing the information for the graders.
A clarification question: applicants didn’t speak to anyone (or have any type of interview/conversation) until the work trail stage?
We did correspond via email, but yes that’s right—we didn’t have a video call with any candidates until the work trial.
I think there’s a case to have had a call before then, as suggested by one of the candidates that gave us feedback:
The reason it’s non-obvious to me whether that would have been worthwhile is that it would have lengthened the process (in our case, due to the timing of leave commitments, the delay would have been considerable).
Feel free to ignore this if you think it is prying a bit too much.
I’m hoping that you had clearly defined criteria for each of these different methods (for each of the questions on the application form, for the work tests and trials, and for the interview questions), rather than just using a general/gestalt “how much do I like this” evaluation. Would you be able to share a bit about what criteria you used and how they were chosen?
We did!
Our team put a lot of thought into the job description which highlights the essential and desirable skills we were looking for. Each test was written with these criteria in mind, and we also used them to help reviewers score responses.[1] This helped reviewers provide scores more consistently and purposefully. Just to avoid overstating things though, I’d add that we weren’t just trying to legalistically make sure every question had a neat correspondence to previously written criteria, but instead were thinking “is this representative of the type of work the role involves?”
This is probably a bit more in the weeds than necessary, but though the initial application questions were written with clear reference to essential/desirable skills in the job description, I didn’t convert that into a clear grading rubric for reviewers to use. This was just an oversight.
Thanks for taking the time to answer my questions. :)