Application forms for EA jobs often give an estimate for how long you should expect it to take; often these estimates are *wildly* too low ime. (And others I know have said this too). This is bad because it makes the estimates unhelpful for planning, and because it probably makes people feel bad about themselves, or worry that they’re unusually slow, when they take longer than the estimate.
Imo, if something involves any sort of writing from scratch, you should expect applicants to take at least an hour, and possibly more. (For context, I’ve seen application forms which say ‘this application should take 10 minutes’ and more commonly ones estimating 20 minutes or 30 minutes).
It doesn’t take long to type 300 words if you already know what you’re going to say and don’t particularly care about polish (I wrote this post in less than an hour probably). But job application questions —even ‘basic’ ones like ‘why do you want this job?’ and ‘why would you be a good fit?’—take more time. You may feel intuitively that you’d be a good fit for the job, but take a while to articulate why. You have to think about how your skills might help with the job, perhaps cross-referencing with the job description. And you have to express everything in appropriately-formal and clear language.
Job applications are also very high-stakes, and many people find them difficult or ‘ugh-y’, which means applicants are likely to take longer to do them than they “should”, due to being stuck or procrastinating.
Maybe hirers put these time estimates because they don’t want applicants to spend too long on the first-stage form (for most of them, it won’t pay off, after all!) This respect for people’s time is laudable. But if someone really wants the job, they *will* feel motivated to put effort into the application form.
There’s a kind of coordination problem here too. Let’s imagine there’s an application for a job that I really want, and on the form it says ‘this application should take you approximately 30 minutes’. If I knew that all the other applicants were going to set a timer for half an hour, write what came to mind, then send off the form without polishing it too much, I also might do that. But as far as I know, they are spending hours polishing their answers. I don’t want to incorrectly seem worse than other candidates and lose out on the job just because I took the time estimates more literally than other people!
‘Aren’t you just unusually slow and neurotic?’ -No; I’d guess that I write faster than average, and I’m really not perfectionist about job applications.
Suggestion: if you’re hiring, include a link at the end of the application form where people can anonymously report how long it actually took them.
As a former applicant for many EA org roles, I strongly agree! I recall spending on average 2-8 times longer on some initial applications than was estimated by many job ads.
As someone who just helped drive a hiring process for Giving What We Can (for a Research Communicator role) I feel a bit daft having experienced it on the other side, but not having learned from it. I/we did not do a good enough job here. We had a few initial questions that we estimated would take ~20-60 minutes, and in retrospect I now imagine many candidates would have spent much longer than this (I know I would have).
Over the coming month or so I’m hoping to draft a post with reflections on what we learned from this, and how we would do better next time (inspired by Aaron Gertler’s 2020 post on hiring a copyeditor for CEA). I’ll be sure to include this comment and its suggestion (having a link at the end of the application form where people can report how long it actually took to fill the form in) in that post.
Might there be a way to time submissions? I know some tests I have taken for prospective employers are timed. This means candidates e.g. only gets 1 hour both to see the questions asked and to answer them. This might also remove any bias in recruitment as someone with a full-time job and caretaker responsibilities might not have the luxury of spending 6 x the time on an application, while someone in a more privileged position can even spend longer than that.
In the hiring round I mentioned, we did time submissions for the work tests, and at least my impression is we found a way of doing so worked out fairly well. Having a timed component for the initial application is also possible, but might require more of an ‘honour code’ system as setting up a process that allows for verification of the time spent is a pretty a big investment for the first stage of an application.
Yes, there are ways to time submissions, and (from my perspective) they aren’t particularly difficult to find or to use. I suspect that any organization not using them doesn’t have can’t find a timing tool as a reason, and more likely has chose not to devote the resources to improving this process, or hasn’t thought of it or hasn’t bothered with it as a reason.
A second thought I had is also that timed responses might be beneficial for the hiring organization. This could be because of two reasons. First, at work, you do not have 4 hours to polish an email to a potential donor. You have 10 minutes because you have a mountain of other important things to do. As such, having a strictly timed assessment is likely to give a more realistic view of the expected performance on the job. Secondly, timed responses will also make for a more apples-to-apples comparison, where you are more likely to select the best candidates instead of the candidates with the most time and/or the largest network of educated family and friends willing to help out polish responses.
We had a few initial questions that we estimated would take ~20-60 minutes, and in retrospect I now imagine many candidates would have spent much longer than this (I know I would have).
Michael, I’m wondering if more transparency would have helped here? As a simplistic example, there is a big difference between these two questions:
Tell us about a time when you took initiative in a work context.
and
Tell us about a time when you took initiative in a work context. We are specifically looking for candidates that have done this in relation to people management, can describe the process and the results/impact, and can demonstrate taking initiative by doing something fairly innovative.
I’m not sure I follow what you mean by transparency in this context. Do you mean being more transparent about what exactly we were looking for? In our case we asked for <100 words on “Why are you interested in this role?” and “Briefly, what is your experience with effective giving and/or effective altruism?” and we were just interested in seeing if applicants’ interest/experienced aligned with the skills, traits and experience we listed in the job descriptions.
I mean transparency in the sense of how the answers are assessed/evaluated. This basically gives candidates a little bit more guidance and structure.
An analogy that I like to use is rather silly, but it works: I might ask a candidate to describe to me how physically fit he are, and he tells me about how many weights he can lift and how fast you can run. But it turns out that I’m actually interested in flexibility and endurance rather than power and speed, and I’ll reject this candidate since he didn’t demonstrate flexibility or endurance. So it is true that he described physical fitness and that I’m assessing based on your physical fitness, but it’s also true that the information offered and what I wanted to assess were very different.
I don’t have any particularly strong views, and would be interested in what others think.
Broadly, I feel like I agree that more specificity/transparency is helpful, though I don’t feel convinced that it’s not also worth asking at some stage in the application an open-ended question like “Why are you interested in the role?”. Not sure I can explain/defend my intuitions here much right now but I would like to think more on it when I get around to writing some reflections on the Research Communicator hiring process.
I just want to say that I love seeing this kind of thing on the EA Forum, and it is so different from most other parts of the internet: I have a proposal or a suggestion, and it doesn’t quite mesh with what you think/feel. Neither of us have great justifications or clear data, and rather than ad hominems or posturing or some type of ‘battle,’ there is simply a bit of exchange and some reflection.
I really like that your response was reflective/pensive, rather than aggressive or defensive. Thanks for being one of the people that makes the internet ever-so-slightly better than it otherwise would be. ☺
Two (barely) related thoughts that I’ve wanted to bring up. Sorry if it’s super off topic.
Rethink priorities application for a role I applied for two years ago told applicants it was timed application and not to take over two hours. However there was no actual verification of this; it was simply a Google form. The first round I “cheated” and took about 4 hours. I made it to the second round. I felt really guilty about this so made sure not to go over on the second round. I didn’t finish all the questions and did not get to the next round. I was left with the unsavory feeling that they were incentivizing dishonest behavior and it could have easily been solved by using something similar to tech companies where a timer starts when you open the task. I haven’t applied for other stuff since so maybe they fixed this.
Charity entrepreneurship made a post a couple months back extending their deadline for the incubator because they thought it was worth it to get good candidates. I decided to apply and made it a few rounds in. I would say I spent like 10 ish hours doing the tasks. I might be misremembering, but at the time of extension I’m pretty sure they already had 2000-4000 applicants. Considering the time it took me, and assuming other applicants were similar, and the amount of applicants they already had, I’m not sure it was actually positive ev extending the deadline.
Neither of these things are really that big of a deal but thought I’d share
Peter Wildeford from Rethink Priorities here. I think about this sort of thing a lot. I’m disappointed in your cheating but appreciate your honesty and feedback.
We’ve considered many times about using a time verification system and even tried it once. But it was a pretty stressful experience for applicants since the timer then required the entire task to be done in one sitting. The system we used also introduced some logistical difficulty on our end.
We’d like to try to make things as easy for our applicants as possible since it’s already such a stressful experience. At the same time, we don’t want to incentivize cheating or make people feel like they have to cheat to stay ahead. It’s a difficult trade-off. But so far I think it’s been working—we’ve been hiring a lot of honest and high integrity people that I trust greatly and don’t feel like I need a timer to micromanage them.
More recently, we’ve been experimenting with more explicit honor code statements. We’ve also done more to pre-test all our work tests to ensure the time limits are reasonable and practical. We’ll continue to think and experiment around this and I’m very open to feedback from you or others about how to do this better.
Hi Peter thanks for the response—I am/was disappointed in myself also.
I assumed RP had thought about this. and I hear what you are saying about the trade-off. I don’t have kids or anything like that and I can’t really relate to struggling to sit down for a few hours straight but I totally believe this is an issue for some applicants and I respect that.
What I am more familiar with is doing school during COVID. My experience left me with a strong impression that even relatively high-integrity people will cheat in this version of the prisoner’s dilemma. Moreover, it will cause them tons of stress and guilt, but they are way less likely to bring it up than someone who is caused issues from having to take the test in one sitting because no one wants to out themselves as a cheater or even thinking about cheating.
I will say in school there is something additionally frustrating or tantalizing about seeing your math tests that usually have a 60% average be in the 90%s and having that confirmation that everyone in your class is cheating but given the people applying are thoughtful and smart they probably would assign this a high probability anyway.
If I had to bet, I would guess a decent chunk of the current employees who took similar tests (>20%) at RP did go over time limits but ofc this is pure speculation on my part. I just do think a significant portion of people will cheat in this situation (10-50%) and given a random split between the cheaters and non-cheaters, the people who cheat are going to have better essays and you are more likely to select them.
(to be clear I’m not saying that even if the above is true that you should definitely time the tests, I could still understand it not being worth it)
I’d be very interested in information about the second claim: that the incubator round already had 2k applicants and thus the time from later applicants was a waste.
Did you end up accepting late applicants? Did they replace earlier applicants who would otherwise have been accepted, or increase the total class size? Do you have a guess for the effects of the new participants?
Or more generally: how do you think about the time unaccepted applicants spend on applications?
My guess is that evaluating applications is expensive so you wouldn’t invite more if it didn’t lead to a much higher quality class, but I’m curious for specifics. CE has mentioned before that the gap between top and median participant is huge, which I imagine plays into the math.
I think you might have replied on the wrong subthread but a few things.
This is the post I was referring to. At the time of extension, they claim they had ~3k applicants. They also infer that they had way fewer (in quantity or quality) applicants for the fish welfare and tobacco taxation projects but I’m not sure exactly how to interpret their claim.
Did you end up accepting late applicants? Did they replace earlier applicants who would otherwise have been accepted, or increase the total class size? Do you have a guess for the effects of the new participants?
using some pretty crude math + assuming both applicant pools are the same, each additional applicant has ~.7% chance of being one of the 20 best applicants (I think they take 10 or 20). so like 150 applicants to get one replaced. if they had to internalize the costs to the candidates, and lets be conservative and say 20 bucks a candidate, then that would be about 3k per extra candidate replaced.
and this doesn’t included the fact that the returns consistently diminish. and they also have to spend more time reviewing candidates, and even if a candidate is actually better, this doesn’t guarantee they will correctly pick them. you can probably add another couple thousands for these considerations so maybe we go with ~5k?
Then you get into issues of fit vs quality, grabbing better quality candidates might help CE counterfactual value but doesn’t help the EA movement much since your pulling from the talent pool. And lastly it’s sort of unfair to the people who applied on time but that’s hard to quantify.
and I think 20 bucks per candidate is really really conservative. I value my time closer to 50$ an hour than 2$ and I’d bet most people applying would probably say something above 15$.
So my very general and crude estimate IMO is they are implicitly saying they value replacing a candidate at 2k-100k, and most likely somewhere between 5-50k. I wonder if we asked them how much they would have to pay for one candidate getting replaced at the time they extended what they would say.
if anyone thinks I missed super obvious considerations or made a mistake lmk.
If we don’t find more potential founders we may not be able to launch charities in Tobacco Taxation and Fish Welfare
This is apparently a pattern
In recent years we have had more charity ideas than we have been able to find founders for.
Seems pretty plausible they value a marginal new charity at $100k, or even $1m, given the amount of staff time and seed funding that go into each participant.
I also suspect they’re more limited by applicant quality than number of spaces.
That post further says
it is true that we get a lot of applicants (~3 thousand). But, and it’s a big but, ~80% of the applications are speculative, from people outside the EA community and don’t even really understand what we do. Of the 300 relevant candidates we receive, maybe 20 or so will make it onto the program.
If you assume that the late applicants recruited by posting on EAF are in the “relevant” pool, those aren’t terrible odds.[1] And they provide feedback even to first round applicants, which is a real service to applicants and cost to CE.
I don’t know if they’re doing the ideal thing here, but they are doing way better than I imagined from your comment.
I don’t love treating relevant and “within EA” as synonyms, but my guess is this that the real point is “don’t even really understand what we do”, and EA is a shorthand for the group that does.
I represent Rethink Priorities but the incubator Charlie is referencing was/is run by Charity Entrepreneurship, which is a different and fully separate org. So you would have to ask them.
If there are any of your questions you’d want me to answer with reference to Rethink Priorities, let me know!
As I have spent more time interacting with job application processes,[1] I lean more and more toward the opinion that broad/vague questions (such as ‘why are you interested in this job?’ and ‘why would you be a good fit?’) shouldn’t be used. I’ll ramble a bit about reasons, but I think the TLDR would be “poor applicant experience, and not very predictive of job performance.”
On the organizational side, my observations are that there often isn’t clear criteria for assessing / evaluating these questions[2], which means that the unofficial criteria often ends up being “do I like this answer.” I’d prefer something ever-so-slightly more rigorous, such as reject unless both A) there aren’t grammar/spelling mistakes, and B) the answer demonstrates that this person has at least a basic understanding of what our organization does.[3]
On the applicant side, there is a lot of uncertainty regarding what a good answer looks like, which makes the application feel like very arbitrary guess what the right answer is game. We might label it as low procedural justice. For a question such as “How did you hear about ORGANIZATION, and what makes you interested in working here?” an honest answer will probably be penalized, and thus I suspect that most applicants who care about getting the job will spend a good deal of effort on impression management, shying away from saying/describing how the appeal is a combination of prestige, good salary, company culture, the professional network, and the feeling of making a positive impact.
These broad/vague questions are probably useful for eliminating particularly bad fit applications.[4] But I do not have confidence in the ability of these question to do any more than to eliminate the bottom ~15% of applications.
While also informing the applicant up front that “we don’t expect you to write an essay about how you’ve have a lifelong desire to work in an entry-level research positions. We are just looking to make sure you have at least a surface level understanding of our industry and our mission. We’d like for you to demonstrate that you have some knowledge or experience related to our field/industry.”
“Bad fit” is a pretty fuzzy concept, but I’m thinking roughly about people who give answers that don’t demonstrate a modicum of knowledge or experience in the relevant field. If I am applying to Open Philanthropy, these would probably be answers such as “Overall I want to give pursue goodness for people, present and future,” or “I can succeed in this role because of my experience as a JOB_TITLE. My organization and attention to detail enabled me to exceed expectations in that role.” If I am the hiring manager, I want to see that the applicant has read the job description and is able to demonstrate some familiarity with the area of work.
Thanks for saying this. This totally rhymes with my experience. I assume that if an application says it will take 15 minutes, I will probably need to spend at least an hour on it (assuming I actually care about getting the job).
Application forms for EA jobs often give an estimate for how long you should expect it to take; often these estimates are *wildly* too low ime. (And others I know have said this too). This is bad because it makes the estimates unhelpful for planning, and because it probably makes people feel bad about themselves, or worry that they’re unusually slow, when they take longer than the estimate.
Imo, if something involves any sort of writing from scratch, you should expect applicants to take at least an hour, and possibly more. (For context, I’ve seen application forms which say ‘this application should take 10 minutes’ and more commonly ones estimating 20 minutes or 30 minutes).
It doesn’t take long to type 300 words if you already know what you’re going to say and don’t particularly care about polish (I wrote this post in less than an hour probably). But job application questions —even ‘basic’ ones like ‘why do you want this job?’ and ‘why would you be a good fit?’—take more time. You may feel intuitively that you’d be a good fit for the job, but take a while to articulate why. You have to think about how your skills might help with the job, perhaps cross-referencing with the job description. And you have to express everything in appropriately-formal and clear language.
Job applications are also very high-stakes, and many people find them difficult or ‘ugh-y’, which means applicants are likely to take longer to do them than they “should”, due to being stuck or procrastinating.
Maybe hirers put these time estimates because they don’t want applicants to spend too long on the first-stage form (for most of them, it won’t pay off, after all!) This respect for people’s time is laudable. But if someone really wants the job, they *will* feel motivated to put effort into the application form.
There’s a kind of coordination problem here too. Let’s imagine there’s an application for a job that I really want, and on the form it says ‘this application should take you approximately 30 minutes’. If I knew that all the other applicants were going to set a timer for half an hour, write what came to mind, then send off the form without polishing it too much, I also might do that. But as far as I know, they are spending hours polishing their answers. I don’t want to incorrectly seem worse than other candidates and lose out on the job just because I took the time estimates more literally than other people!
‘Aren’t you just unusually slow and neurotic?’
-No; I’d guess that I write faster than average, and I’m really not perfectionist about job applications.
Suggestion: if you’re hiring, include a link at the end of the application form where people can anonymously report how long it actually took them.
As a former applicant for many EA org roles, I strongly agree! I recall spending on average 2-8 times longer on some initial applications than was estimated by many job ads.
As someone who just helped drive a hiring process for Giving What We Can (for a Research Communicator role) I feel a bit daft having experienced it on the other side, but not having learned from it. I/we did not do a good enough job here. We had a few initial questions that we estimated would take ~20-60 minutes, and in retrospect I now imagine many candidates would have spent much longer than this (I know I would have).
Over the coming month or so I’m hoping to draft a post with reflections on what we learned from this, and how we would do better next time (inspired by Aaron Gertler’s 2020 post on hiring a copyeditor for CEA). I’ll be sure to include this comment and its suggestion (having a link at the end of the application form where people can report how long it actually took to fill the form in) in that post.
Might there be a way to time submissions? I know some tests I have taken for prospective employers are timed. This means candidates e.g. only gets 1 hour both to see the questions asked and to answer them. This might also remove any bias in recruitment as someone with a full-time job and caretaker responsibilities might not have the luxury of spending 6 x the time on an application, while someone in a more privileged position can even spend longer than that.
In the hiring round I mentioned, we did time submissions for the work tests, and at least my impression is we found a way of doing so worked out fairly well. Having a timed component for the initial application is also possible, but might require more of an ‘honour code’ system as setting up a process that allows for verification of the time spent is a pretty a big investment for the first stage of an application.
Yes, there are ways to time submissions, and (from my perspective) they aren’t particularly difficult to find or to use. I suspect that any organization not using them doesn’t have can’t find a timing tool as a reason, and more likely has chose not to devote the resources to improving this process, or hasn’t thought of it or hasn’t bothered with it as a reason.
A second thought I had is also that timed responses might be beneficial for the hiring organization. This could be because of two reasons. First, at work, you do not have 4 hours to polish an email to a potential donor. You have 10 minutes because you have a mountain of other important things to do. As such, having a strictly timed assessment is likely to give a more realistic view of the expected performance on the job. Secondly, timed responses will also make for a more apples-to-apples comparison, where you are more likely to select the best candidates instead of the candidates with the most time and/or the largest network of educated family and friends willing to help out polish responses.
I’m looking forward to reading a post with reflections on lessons learned. :)
Michael, I’m wondering if more transparency would have helped here? As a simplistic example, there is a big difference between these two questions:
and
I’m not sure I follow what you mean by transparency in this context. Do you mean being more transparent about what exactly we were looking for? In our case we asked for <100 words on “Why are you interested in this role?” and “Briefly, what is your experience with effective giving and/or effective altruism?” and we were just interested in seeing if applicants’ interest/experienced aligned with the skills, traits and experience we listed in the job descriptions.
I mean transparency in the sense of how the answers are assessed/evaluated. This basically gives candidates a little bit more guidance and structure.
An analogy that I like to use is rather silly, but it works: I might ask a candidate to describe to me how physically fit he are, and he tells me about how many weights he can lift and how fast you can run. But it turns out that I’m actually interested in flexibility and endurance rather than power and speed, and I’ll reject this candidate since he didn’t demonstrate flexibility or endurance. So it is true that he described physical fitness and that I’m assessing based on your physical fitness, but it’s also true that the information offered and what I wanted to assess were very different.
What do you think about Joseph’s thoughts on those types of questions here: https://forum.effectivealtruism.org/posts/4towuFeBfbGn8hJGs/amber-dawn-s-shortform?commentId=2N7JqCYzyt7FHCti2
I don’t have any particularly strong views, and would be interested in what others think.
Broadly, I feel like I agree that more specificity/transparency is helpful, though I don’t feel convinced that it’s not also worth asking at some stage in the application an open-ended question like “Why are you interested in the role?”. Not sure I can explain/defend my intuitions here much right now but I would like to think more on it when I get around to writing some reflections on the Research Communicator hiring process.
I just want to say that I love seeing this kind of thing on the EA Forum, and it is so different from most other parts of the internet: I have a proposal or a suggestion, and it doesn’t quite mesh with what you think/feel. Neither of us have great justifications or clear data, and rather than ad hominems or posturing or some type of ‘battle,’ there is simply a bit of exchange and some reflection.
I really like that your response was reflective/pensive, rather than aggressive or defensive. Thanks for being one of the people that makes the internet ever-so-slightly better than it otherwise would be. ☺
Two (barely) related thoughts that I’ve wanted to bring up. Sorry if it’s super off topic.
Rethink priorities application for a role I applied for two years ago told applicants it was timed application and not to take over two hours. However there was no actual verification of this; it was simply a Google form. The first round I “cheated” and took about 4 hours. I made it to the second round. I felt really guilty about this so made sure not to go over on the second round. I didn’t finish all the questions and did not get to the next round. I was left with the unsavory feeling that they were incentivizing dishonest behavior and it could have easily been solved by using something similar to tech companies where a timer starts when you open the task. I haven’t applied for other stuff since so maybe they fixed this.
Charity entrepreneurship made a post a couple months back extending their deadline for the incubator because they thought it was worth it to get good candidates. I decided to apply and made it a few rounds in. I would say I spent like 10 ish hours doing the tasks. I might be misremembering, but at the time of extension I’m pretty sure they already had 2000-4000 applicants. Considering the time it took me, and assuming other applicants were similar, and the amount of applicants they already had, I’m not sure it was actually positive ev extending the deadline.
Neither of these things are really that big of a deal but thought I’d share
Hi Charlie,
Peter Wildeford from Rethink Priorities here. I think about this sort of thing a lot. I’m disappointed in your cheating but appreciate your honesty and feedback.
We’ve considered many times about using a time verification system and even tried it once. But it was a pretty stressful experience for applicants since the timer then required the entire task to be done in one sitting. The system we used also introduced some logistical difficulty on our end.
We’d like to try to make things as easy for our applicants as possible since it’s already such a stressful experience. At the same time, we don’t want to incentivize cheating or make people feel like they have to cheat to stay ahead. It’s a difficult trade-off. But so far I think it’s been working—we’ve been hiring a lot of honest and high integrity people that I trust greatly and don’t feel like I need a timer to micromanage them.
More recently, we’ve been experimenting with more explicit honor code statements. We’ve also done more to pre-test all our work tests to ensure the time limits are reasonable and practical. We’ll continue to think and experiment around this and I’m very open to feedback from you or others about how to do this better.
Hi Peter thanks for the response—I am/was disappointed in myself also.
I assumed RP had thought about this. and I hear what you are saying about the trade-off. I don’t have kids or anything like that and I can’t really relate to struggling to sit down for a few hours straight but I totally believe this is an issue for some applicants and I respect that.
What I am more familiar with is doing school during COVID. My experience left me with a strong impression that even relatively high-integrity people will cheat in this version of the prisoner’s dilemma. Moreover, it will cause them tons of stress and guilt, but they are way less likely to bring it up than someone who is caused issues from having to take the test in one sitting because no one wants to out themselves as a cheater or even thinking about cheating.
I will say in school there is something additionally frustrating or tantalizing about seeing your math tests that usually have a 60% average be in the 90%s and having that confirmation that everyone in your class is cheating but given the people applying are thoughtful and smart they probably would assign this a high probability anyway.
If I had to bet, I would guess a decent chunk of the current employees who took similar tests (>20%) at RP did go over time limits but ofc this is pure speculation on my part. I just do think a significant portion of people will cheat in this situation (10-50%) and given a random split between the cheaters and non-cheaters, the people who cheat are going to have better essays and you are more likely to select them.
(to be clear I’m not saying that even if the above is true that you should definitely time the tests, I could still understand it not being worth it)
I’d be very interested in information about the second claim: that the incubator round already had 2k applicants and thus the time from later applicants was a waste.
Did you end up accepting late applicants? Did they replace earlier applicants who would otherwise have been accepted, or increase the total class size? Do you have a guess for the effects of the new participants?
Or more generally: how do you think about the time unaccepted applicants spend on applications?
My guess is that evaluating applications is expensive so you wouldn’t invite more if it didn’t lead to a much higher quality class, but I’m curious for specifics. CE has mentioned before that the gap between top and median participant is huge, which I imagine plays into the math.
I think you might have replied on the wrong subthread but a few things.
This is the post I was referring to. At the time of extension, they claim they had ~3k applicants. They also infer that they had way fewer (in quantity or quality) applicants for the fish welfare and tobacco taxation projects but I’m not sure exactly how to interpret their claim.
using some pretty crude math + assuming both applicant pools are the same, each additional applicant has ~.7% chance of being one of the 20 best applicants (I think they take 10 or 20). so like 150 applicants to get one replaced. if they had to internalize the costs to the candidates, and lets be conservative and say 20 bucks a candidate, then that would be about 3k per extra candidate replaced.
and this doesn’t included the fact that the returns consistently diminish. and they also have to spend more time reviewing candidates, and even if a candidate is actually better, this doesn’t guarantee they will correctly pick them. you can probably add another couple thousands for these considerations so maybe we go with ~5k?
Then you get into issues of fit vs quality, grabbing better quality candidates might help CE counterfactual value but doesn’t help the EA movement much since your pulling from the talent pool. And lastly it’s sort of unfair to the people who applied on time but that’s hard to quantify.
and I think 20 bucks per candidate is really really conservative. I value my time closer to 50$ an hour than 2$ and I’d bet most people applying would probably say something above 15$.
So my very general and crude estimate IMO is they are implicitly saying they value replacing a candidate at 2k-100k, and most likely somewhere between 5-50k. I wonder if we asked them how much they would have to pay for one candidate getting replaced at the time they extended what they would say.
if anyone thinks I missed super obvious considerations or made a mistake lmk.
That post says opens with
This is apparently a pattern
Seems pretty plausible they value a marginal new charity at $100k, or even $1m, given the amount of staff time and seed funding that go into each participant.
I also suspect they’re more limited by applicant quality than number of spaces.
That post further says
If you assume that the late applicants recruited by posting on EAF are in the “relevant” pool, those aren’t terrible odds.[1] And they provide feedback even to first round applicants, which is a real service to applicants and cost to CE.
I don’t know if they’re doing the ideal thing here, but they are doing way better than I imagined from your comment.
I don’t love treating relevant and “within EA” as synonyms, but my guess is this that the real point is “don’t even really understand what we do”, and EA is a shorthand for the group that does.
Yep after walking through it in my head plus re- reading the post, doesn’t seem egregious to me.
Hi Elizabeth,
I represent Rethink Priorities but the incubator Charlie is referencing was/is run by Charity Entrepreneurship, which is a different and fully separate org. So you would have to ask them.
If there are any of your questions you’d want me to answer with reference to Rethink Priorities, let me know!
Oops, should have read more carefully, sorry about that.
As I have spent more time interacting with job application processes,[1] I lean more and more toward the opinion that broad/vague questions (such as ‘why are you interested in this job?’ and ‘why would you be a good fit?’) shouldn’t be used. I’ll ramble a bit about reasons, but I think the TLDR would be “poor applicant experience, and not very predictive of job performance.”
On the organizational side, my observations are that there often isn’t clear criteria for assessing / evaluating these questions[2], which means that the unofficial criteria often ends up being “do I like this answer.” I’d prefer something ever-so-slightly more rigorous, such as reject unless both A) there aren’t grammar/spelling mistakes, and B) the answer demonstrates that this person has at least a basic understanding of what our organization does.[3]
On the applicant side, there is a lot of uncertainty regarding what a good answer looks like, which makes the application feel like very arbitrary guess what the right answer is game. We might label it as low procedural justice. For a question such as “How did you hear about ORGANIZATION, and what makes you interested in working here?” an honest answer will probably be penalized, and thus I suspect that most applicants who care about getting the job will spend a good deal of effort on impression management, shying away from saying/describing how the appeal is a combination of prestige, good salary, company culture, the professional network, and the feeling of making a positive impact.
These broad/vague questions are probably useful for eliminating particularly bad fit applications.[4] But I do not have confidence in the ability of these question to do any more than to eliminate the bottom ~15% of applications.
Both from the company side of filtering/selecting applications, and from the applicant side of submitting applications.
But I have seen a minority of organizations that actually use a rubric and have clear and job-relevant criteria. Good for you guys!
While also informing the applicant up front that “we don’t expect you to write an essay about how you’ve have a lifelong desire to work in an entry-level research positions. We are just looking to make sure you have at least a surface level understanding of our industry and our mission. We’d like for you to demonstrate that you have some knowledge or experience related to our field/industry.”
“Bad fit” is a pretty fuzzy concept, but I’m thinking roughly about people who give answers that don’t demonstrate a modicum of knowledge or experience in the relevant field. If I am applying to Open Philanthropy, these would probably be answers such as “Overall I want to give pursue goodness for people, present and future,” or “I can succeed in this role because of my experience as a JOB_TITLE. My organization and attention to detail enabled me to exceed expectations in that role.” If I am the hiring manager, I want to see that the applicant has read the job description and is able to demonstrate some familiarity with the area of work.
I found this reflection interesting and in general really like hearing your thoughts on hiring, Joseph :)
Aww, thanks. That makes me smile and tear up a bit.
Thanks for saying this. This totally rhymes with my experience. I assume that if an application says it will take 15 minutes, I will probably need to spend at least an hour on it (assuming I actually care about getting the job).