Thanks for mentioning the thing about the conversation notes test. It was simply an oversight to not explicitly say “Please don’t spend more than X hours on this work test,” and I’ve now added such a sentence to our latest draft of those work test instructions. We had explicit time limits for our other two tests.
I would advocate for controllably timed work tests whenever possible. Simply saying “please don’t spend more than X hours on this work test” gives the opportunity to cheat by spending more time. Incentives for cheating are strong, because:
The tasks usually have tight time limits, so spending additional time will improve your results.
Applicants know the application process is highly competitive.
Applicants know that EA organisations put a lot of value on work test performance.
If you have enough applicants, some will cheat, and they will get a significant advantage. In rare cases, this may even deter people from applying. There was one position were I was planning to apply but then didn’t because they had a non-controllably timed worktest (I don’t want to cheat, somebody probably will cheat, and I am not super-well qualified for the position anyway so I would really need to shine in the work test → not worth applying). (I admit that this deterrence probably doesn’t happen often)
Great tools online for doing controllably timed work tests exist.
(I realize that it is not always possible to control the time limit, e.g. when the task is too long to be done in one sitting. I have no recommendation for what to do then, other than that I think Jonas Vollmer’s comment in this thread seems reasonable).
My current view is to ask for both timed and untimed tests, and make the untimed tests very simple/short (such that you could complete it in 20 minutes if you had to and there’s very little benefit to spending >2h on it).
Thanks for mentioning the thing about the conversation notes test. It was simply an oversight to not explicitly say “Please don’t spend more than X hours on this work test,” and I’ve now added such a sentence to our latest draft of those work test instructions. We had explicit time limits for our other two tests.
I would advocate for controllably timed work tests whenever possible. Simply saying “please don’t spend more than X hours on this work test” gives the opportunity to cheat by spending more time. Incentives for cheating are strong, because:
The tasks usually have tight time limits, so spending additional time will improve your results.
Applicants know the application process is highly competitive.
Applicants know that EA organisations put a lot of value on work test performance.
If you have enough applicants, some will cheat, and they will get a significant advantage. In rare cases, this may even deter people from applying. There was one position were I was planning to apply but then didn’t because they had a non-controllably timed worktest (I don’t want to cheat, somebody probably will cheat, and I am not super-well qualified for the position anyway so I would really need to shine in the work test → not worth applying). (I admit that this deterrence probably doesn’t happen often)
Great tools online for doing controllably timed work tests exist.
(I realize that it is not always possible to control the time limit, e.g. when the task is too long to be done in one sitting. I have no recommendation for what to do then, other than that I think Jonas Vollmer’s comment in this thread seems reasonable).