Huh. I’m really surprised that they find this useful. One of the main ways that Wave employees’ productivity has varied is in how quickly they can accomplish a task at a given level of quality, which varies by an order of magnitude between our best and worst candidates. (Or equivalently, how good of a job they can do in a fixed amount of time.) It seems like not time-boxing the work sample would make it much, much harder to make an apples-to-apples quality comparison between applicants, because slower applicants can spend more time to reach the same level of quality.
It seems easier to increase the efficiency of your work than the quality. All else equal, I’m tentatively more interested in people who can do very high-quality work inefficiently than people doing mediocre work quickly – because I expect that the the former are more likely to eventually do high-quality, highly efficient work.
Some people tend to get very nervous with timed tests and mess up badly; it seems good to give them the opportunity to prove themselves in a less stressful environment.
My current view is to ask for both timed and untimed tests, and make the untimed tests very simple/short (such that you could complete it in 20 minutes if you had to and there’s very little benefit to spending >2h on it).
It seems easier to increase the efficiency of your work than the quality.
In software engineering, I’ve found the exact opposite. It’s relatively easy for me to train people to identify and correct flaws in their own code–I point out the problems in code review and try to explain the underlying heuristics/models I’m using, and eventually other people learn the same heuristics/models. On the other hand, I have no idea how to train people to work more quickly.
(Of course there are many reasons why other types of work might be different from software eng!)
I expect that good software engineers are more likely to figure out for themselves how to be more efficient than they are to figure out how to increase their work quality. So it’s not obvious what to infer from “it’s harder for an employer to train people to work faster”—does it just mean that the employer has less need to train the slow, high quality worker?
I hadn’t noticed the discrepancy before between the conversation notes test and their other tests, which generally read something like this:
“This test should require somewhere between X and Y hours of work; please send us your work, even if it’s incomplete, after Y hours.”
Adjusting the notes test seems like a good step, or at least asking applicants how much time they spent, so that there’s a clear tradeoff between speed and thoroughness (maybe it’s the case that a slightly messy four-hour test gets as good a score as a better eight-hour test, and Open Phil would be happy to consider both, or something like that).
Huh. I’m really surprised that they find this useful. One of the main ways that Wave employees’ productivity has varied is in how quickly they can accomplish a task at a given level of quality, which varies by an order of magnitude between our best and worst candidates. (Or equivalently, how good of a job they can do in a fixed amount of time.) It seems like not time-boxing the work sample would make it much, much harder to make an apples-to-apples quality comparison between applicants, because slower applicants can spend more time to reach the same level of quality.
Two points that speak against this view a bit:
It seems easier to increase the efficiency of your work than the quality. All else equal, I’m tentatively more interested in people who can do very high-quality work inefficiently than people doing mediocre work quickly – because I expect that the the former are more likely to eventually do high-quality, highly efficient work.
Some people tend to get very nervous with timed tests and mess up badly; it seems good to give them the opportunity to prove themselves in a less stressful environment.
My current view is to ask for both timed and untimed tests, and make the untimed tests very simple/short (such that you could complete it in 20 minutes if you had to and there’s very little benefit to spending >2h on it).
In software engineering, I’ve found the exact opposite. It’s relatively easy for me to train people to identify and correct flaws in their own code–I point out the problems in code review and try to explain the underlying heuristics/models I’m using, and eventually other people learn the same heuristics/models. On the other hand, I have no idea how to train people to work more quickly.
(Of course there are many reasons why other types of work might be different from software eng!)
I expect that good software engineers are more likely to figure out for themselves how to be more efficient than they are to figure out how to increase their work quality. So it’s not obvious what to infer from “it’s harder for an employer to train people to work faster”—does it just mean that the employer has less need to train the slow, high quality worker?
Good point, agree it depends on the type of work.
I hadn’t noticed the discrepancy before between the conversation notes test and their other tests, which generally read something like this:
“This test should require somewhere between X and Y hours of work; please send us your work, even if it’s incomplete, after Y hours.”
Adjusting the notes test seems like a good step, or at least asking applicants how much time they spent, so that there’s a clear tradeoff between speed and thoroughness (maybe it’s the case that a slightly messy four-hour test gets as good a score as a better eight-hour test, and Open Phil would be happy to consider both, or something like that).