I would advocate for controllably timed work tests whenever possible. Simply saying “please don’t spend more than X hours on this work test” gives the opportunity to cheat by spending more time. Incentives for cheating are strong, because:
The tasks usually have tight time limits, so spending additional time will improve your results.
Applicants know the application process is highly competitive.
Applicants know that EA organisations put a lot of value on work test performance.
If you have enough applicants, some will cheat, and they will get a significant advantage. In rare cases, this may even deter people from applying. There was one position were I was planning to apply but then didn’t because they had a non-controllably timed worktest (I don’t want to cheat, somebody probably will cheat, and I am not super-well qualified for the position anyway so I would really need to shine in the work test → not worth applying). (I admit that this deterrence probably doesn’t happen often)
Great tools online for doing controllably timed work tests exist.
(I realize that it is not always possible to control the time limit, e.g. when the task is too long to be done in one sitting. I have no recommendation for what to do then, other than that I think Jonas Vollmer’s comment in this thread seems reasonable).
My current view is to ask for both timed and untimed tests, and make the untimed tests very simple/short (such that you could complete it in 20 minutes if you had to and there’s very little benefit to spending >2h on it).
(My answers might be very different to the ones that anonymousthrowaway might give).
Even if not a direct answer to your question, maybe it helps to illuminate the dynamics that contributed towards me making that many applications before moving on to other EA things. I personally did NOT think that jobs at EA organisations had clearly higher expected value than my other top options (which were working in biosecurity basically anywhere, or upskilling in machine learning via a master’s degree). (There were very few exceptions, like Chief of Staff for Will or RA at OpenPhil, which I thought were outstandingly good).
Then why did I apply to so many positions?
1. I thought EA organisations were really talent-starved and needed me. I also thought that it would be easy to get a job. This is clearly my fault, and I think there would have been ways for me to reach a better knowledge of the situation. But I went to several EAGs, talked to dozens of people, read most of 80000 hours advice, and I think it was definitely quite easy to come away from these with the impression I had.
2. I got pretty far in a few application processes, so that encouraged me to continue applying.
3. Somehow, EA positions were the positions I heard about. These were the positions that 80000 hours emailed me about, that I got invitations to apply for, that I found on the websites I was visiting anyway, and so on. Of course, the bar for applying for such a position is lower than if you first have to find positions yourself. 80000 hours never emailed me about a position in biosecurity.
4. EA was the thing I knew most about. I knew a lot about the EA-sphere anyway. So it is easier to evaluate to which organisation to apply to, than say in biosecurity (which is a vast, scary field that I have very little knowledge about). If I apply to OpenPhil, that is definitely at least good. If I pick a random biosecurity organisation, I have to do my homework to figure out if it is promising or not.
5. Psychologically: My other top options (biosecurity anywhere, most likely not long-term-focused, or upskilling in ML) felt like “now I have to work really hard for some time, and maybe later I will be able to contribute to X-risk reduction”. So still like I haven’t quite made it. In contrast, working for a major EA org (in my imagination) felt like “Yes, I have made it now. I am doing super valuable, long-termism-relevant work”.
6. Working at an EA organisation was the only thing that would contribute towards the long-termist agenda that I could hope to do RIGHT NOW. For the other things, I would need upskilling. So if discount rates on work are high, that makes EA orgs more attractive. (However, I don’t believe that discount rates on “valuable work in general” are anywhere as high as the rates often cited in the EA-sphere, so that did not make me think that jobs at EA orgs are clearly better than my alternatives).
Finally, I think that the fact that I thought it would be easy to get hired by an EA organisation really is quite crucial. My points 3.,4.,5.,6. really mainly became relevant against this background. It’s the difference between
“I could apply for an EA org, or do this other thing where I have to look for options first, have less of a clue about the field, which is more inconvenient and psychologically more challenging.”
“I could apply for an EA org. It’s pretty unlikely that I will get hired. Might as well try it a few times, but in the meantime, let’s see what options there are in biosecurity”
Now, without clearly defining WHAT it even is that we want to improve, here are a few ways that I think this could be improved. (a rather lose connection of thoughts)
To improve 1):
Communication:
Write more posts like the OP :-)
Improve communication about talent constraints (already happening)
Sadly, I don’t have many very concrete suggestions here. But I do think this is crucial, and could make the difference between the two points of view in quotation marks above.
Several things that can be tweaked in application processes, mostly related to more transparency (there are some orgs who are doing this very well. I thought e.g. this job description was good):
Say how many applicants you had last time
If you send personalised invitations, include how many people you are sending personalised invitations to.
Clearly state upfront how involved the application processes will be. If you don’t know, then give your best guess and a longest-case scenario.
To improve 3):
Hard to say. There is probably currently really no capacity for that, but if 80000 hours would have emailed me about positions in biosecurity, I would have applied. Having presented one position and only needing to evaluate if it is good is much easier than having to find a position within a large, scary field. They probably really don’t have the capacity for that (which includes figuring out which positions in biosecurity are good), but maybe as a long-term vision.
To improve 4):
Good career guides would be very valuable. This doesn’t even need to be from 80000 hours, but might come from somebody who has researched something for their own career. Maybe we can have EA grants for people writing up their findings? A good career guide for biosecurity, especially one that acknowledges that countries other than the US exist ( :-) ), would have been so, so great.