I hadn’t seen this until now. I still hope you’ll do a follow up on the most recent round, since as I’ve said (repeatedly) elsewhere, I think you guys are the gold standard in the EA movement about how to do this well :)
One not necessarily very helpful thought:
Our work trial was overly intense and stressful, and unrepresentative of working at GWWC.
is a noble goal, but somewhat in tension with this goal:
In retrospect, we could have ensured this was done on a time-limited basis, or provided a more reasonable estimate.
It’s really hard to make a strictly timed test, especially a sub-one-day one unstressful/intense.
This isn’t to say you shouldn’t do the latter, just to recognise that there’s a natural tradeoff between two imperatives here.
Another problem with timing is that you don’t get to equalise across all axes, so you can trade one bias for another. For example, you’re going to bias towards people who have access to an extra monitor or two at the time of taking the test, whose internet is faster or who are just in a less distracting location.
I don’t know that that’s really a solvable problem, and if not, the timed test seems probably the least of all evils, but again it seems like a tradeoff worth being aware of.
The dream is maybe some kind of self-contained challenge where you ask them to showcase some relevant way of thinking in a way in time isn’t super important, but I can’t think of any good version of that.
I hadn’t seen this until now. I still hope you’ll do a follow up on the most recent round, since as I’ve said (repeatedly) elsewhere, I think you guys are the gold standard in the EA movement about how to do this well :)
One not necessarily very helpful thought:
is a noble goal, but somewhat in tension with this goal:
It’s really hard to make a strictly timed test, especially a sub-one-day one unstressful/intense.
This isn’t to say you shouldn’t do the latter, just to recognise that there’s a natural tradeoff between two imperatives here.
Another problem with timing is that you don’t get to equalise across all axes, so you can trade one bias for another. For example, you’re going to bias towards people who have access to an extra monitor or two at the time of taking the test, whose internet is faster or who are just in a less distracting location.
I don’t know that that’s really a solvable problem, and if not, the timed test seems probably the least of all evils, but again it seems like a tradeoff worth being aware of.
The dream is maybe some kind of self-contained challenge where you ask them to showcase some relevant way of thinking in a way in time isn’t super important, but I can’t think of any good version of that.