Peter Wildeford from Rethink Priorities here. I think about this sort of thing a lot. I’m disappointed in your cheating but appreciate your honesty and feedback.
We’ve considered many times about using a time verification system and even tried it once. But it was a pretty stressful experience for applicants since the timer then required the entire task to be done in one sitting. The system we used also introduced some logistical difficulty on our end.
We’d like to try to make things as easy for our applicants as possible since it’s already such a stressful experience. At the same time, we don’t want to incentivize cheating or make people feel like they have to cheat to stay ahead. It’s a difficult trade-off. But so far I think it’s been working—we’ve been hiring a lot of honest and high integrity people that I trust greatly and don’t feel like I need a timer to micromanage them.
More recently, we’ve been experimenting with more explicit honor code statements. We’ve also done more to pre-test all our work tests to ensure the time limits are reasonable and practical. We’ll continue to think and experiment around this and I’m very open to feedback from you or others about how to do this better.
Hi Peter thanks for the response—I am/was disappointed in myself also.
I assumed RP had thought about this. and I hear what you are saying about the trade-off. I don’t have kids or anything like that and I can’t really relate to struggling to sit down for a few hours straight but I totally believe this is an issue for some applicants and I respect that.
What I am more familiar with is doing school during COVID. My experience left me with a strong impression that even relatively high-integrity people will cheat in this version of the prisoner’s dilemma. Moreover, it will cause them tons of stress and guilt, but they are way less likely to bring it up than someone who is caused issues from having to take the test in one sitting because no one wants to out themselves as a cheater or even thinking about cheating.
I will say in school there is something additionally frustrating or tantalizing about seeing your math tests that usually have a 60% average be in the 90%s and having that confirmation that everyone in your class is cheating but given the people applying are thoughtful and smart they probably would assign this a high probability anyway.
If I had to bet, I would guess a decent chunk of the current employees who took similar tests (>20%) at RP did go over time limits but ofc this is pure speculation on my part. I just do think a significant portion of people will cheat in this situation (10-50%) and given a random split between the cheaters and non-cheaters, the people who cheat are going to have better essays and you are more likely to select them.
(to be clear I’m not saying that even if the above is true that you should definitely time the tests, I could still understand it not being worth it)
I’d be very interested in information about the second claim: that the incubator round already had 2k applicants and thus the time from later applicants was a waste.
Did you end up accepting late applicants? Did they replace earlier applicants who would otherwise have been accepted, or increase the total class size? Do you have a guess for the effects of the new participants?
Or more generally: how do you think about the time unaccepted applicants spend on applications?
My guess is that evaluating applications is expensive so you wouldn’t invite more if it didn’t lead to a much higher quality class, but I’m curious for specifics. CE has mentioned before that the gap between top and median participant is huge, which I imagine plays into the math.
I think you might have replied on the wrong subthread but a few things.
This is the post I was referring to. At the time of extension, they claim they had ~3k applicants. They also infer that they had way fewer (in quantity or quality) applicants for the fish welfare and tobacco taxation projects but I’m not sure exactly how to interpret their claim.
Did you end up accepting late applicants? Did they replace earlier applicants who would otherwise have been accepted, or increase the total class size? Do you have a guess for the effects of the new participants?
using some pretty crude math + assuming both applicant pools are the same, each additional applicant has ~.7% chance of being one of the 20 best applicants (I think they take 10 or 20). so like 150 applicants to get one replaced. if they had to internalize the costs to the candidates, and lets be conservative and say 20 bucks a candidate, then that would be about 3k per extra candidate replaced.
and this doesn’t included the fact that the returns consistently diminish. and they also have to spend more time reviewing candidates, and even if a candidate is actually better, this doesn’t guarantee they will correctly pick them. you can probably add another couple thousands for these considerations so maybe we go with ~5k?
Then you get into issues of fit vs quality, grabbing better quality candidates might help CE counterfactual value but doesn’t help the EA movement much since your pulling from the talent pool. And lastly it’s sort of unfair to the people who applied on time but that’s hard to quantify.
and I think 20 bucks per candidate is really really conservative. I value my time closer to 50$ an hour than 2$ and I’d bet most people applying would probably say something above 15$.
So my very general and crude estimate IMO is they are implicitly saying they value replacing a candidate at 2k-100k, and most likely somewhere between 5-50k. I wonder if we asked them how much they would have to pay for one candidate getting replaced at the time they extended what they would say.
if anyone thinks I missed super obvious considerations or made a mistake lmk.
If we don’t find more potential founders we may not be able to launch charities in Tobacco Taxation and Fish Welfare
This is apparently a pattern
In recent years we have had more charity ideas than we have been able to find founders for.
Seems pretty plausible they value a marginal new charity at $100k, or even $1m, given the amount of staff time and seed funding that go into each participant.
I also suspect they’re more limited by applicant quality than number of spaces.
That post further says
it is true that we get a lot of applicants (~3 thousand). But, and it’s a big but, ~80% of the applications are speculative, from people outside the EA community and don’t even really understand what we do. Of the 300 relevant candidates we receive, maybe 20 or so will make it onto the program.
If you assume that the late applicants recruited by posting on EAF are in the “relevant” pool, those aren’t terrible odds.[1] And they provide feedback even to first round applicants, which is a real service to applicants and cost to CE.
I don’t know if they’re doing the ideal thing here, but they are doing way better than I imagined from your comment.
I don’t love treating relevant and “within EA” as synonyms, but my guess is this that the real point is “don’t even really understand what we do”, and EA is a shorthand for the group that does.
I represent Rethink Priorities but the incubator Charlie is referencing was/is run by Charity Entrepreneurship, which is a different and fully separate org. So you would have to ask them.
If there are any of your questions you’d want me to answer with reference to Rethink Priorities, let me know!
Hi Charlie,
Peter Wildeford from Rethink Priorities here. I think about this sort of thing a lot. I’m disappointed in your cheating but appreciate your honesty and feedback.
We’ve considered many times about using a time verification system and even tried it once. But it was a pretty stressful experience for applicants since the timer then required the entire task to be done in one sitting. The system we used also introduced some logistical difficulty on our end.
We’d like to try to make things as easy for our applicants as possible since it’s already such a stressful experience. At the same time, we don’t want to incentivize cheating or make people feel like they have to cheat to stay ahead. It’s a difficult trade-off. But so far I think it’s been working—we’ve been hiring a lot of honest and high integrity people that I trust greatly and don’t feel like I need a timer to micromanage them.
More recently, we’ve been experimenting with more explicit honor code statements. We’ve also done more to pre-test all our work tests to ensure the time limits are reasonable and practical. We’ll continue to think and experiment around this and I’m very open to feedback from you or others about how to do this better.
Hi Peter thanks for the response—I am/was disappointed in myself also.
I assumed RP had thought about this. and I hear what you are saying about the trade-off. I don’t have kids or anything like that and I can’t really relate to struggling to sit down for a few hours straight but I totally believe this is an issue for some applicants and I respect that.
What I am more familiar with is doing school during COVID. My experience left me with a strong impression that even relatively high-integrity people will cheat in this version of the prisoner’s dilemma. Moreover, it will cause them tons of stress and guilt, but they are way less likely to bring it up than someone who is caused issues from having to take the test in one sitting because no one wants to out themselves as a cheater or even thinking about cheating.
I will say in school there is something additionally frustrating or tantalizing about seeing your math tests that usually have a 60% average be in the 90%s and having that confirmation that everyone in your class is cheating but given the people applying are thoughtful and smart they probably would assign this a high probability anyway.
If I had to bet, I would guess a decent chunk of the current employees who took similar tests (>20%) at RP did go over time limits but ofc this is pure speculation on my part. I just do think a significant portion of people will cheat in this situation (10-50%) and given a random split between the cheaters and non-cheaters, the people who cheat are going to have better essays and you are more likely to select them.
(to be clear I’m not saying that even if the above is true that you should definitely time the tests, I could still understand it not being worth it)
I’d be very interested in information about the second claim: that the incubator round already had 2k applicants and thus the time from later applicants was a waste.
Did you end up accepting late applicants? Did they replace earlier applicants who would otherwise have been accepted, or increase the total class size? Do you have a guess for the effects of the new participants?
Or more generally: how do you think about the time unaccepted applicants spend on applications?
My guess is that evaluating applications is expensive so you wouldn’t invite more if it didn’t lead to a much higher quality class, but I’m curious for specifics. CE has mentioned before that the gap between top and median participant is huge, which I imagine plays into the math.
I think you might have replied on the wrong subthread but a few things.
This is the post I was referring to. At the time of extension, they claim they had ~3k applicants. They also infer that they had way fewer (in quantity or quality) applicants for the fish welfare and tobacco taxation projects but I’m not sure exactly how to interpret their claim.
using some pretty crude math + assuming both applicant pools are the same, each additional applicant has ~.7% chance of being one of the 20 best applicants (I think they take 10 or 20). so like 150 applicants to get one replaced. if they had to internalize the costs to the candidates, and lets be conservative and say 20 bucks a candidate, then that would be about 3k per extra candidate replaced.
and this doesn’t included the fact that the returns consistently diminish. and they also have to spend more time reviewing candidates, and even if a candidate is actually better, this doesn’t guarantee they will correctly pick them. you can probably add another couple thousands for these considerations so maybe we go with ~5k?
Then you get into issues of fit vs quality, grabbing better quality candidates might help CE counterfactual value but doesn’t help the EA movement much since your pulling from the talent pool. And lastly it’s sort of unfair to the people who applied on time but that’s hard to quantify.
and I think 20 bucks per candidate is really really conservative. I value my time closer to 50$ an hour than 2$ and I’d bet most people applying would probably say something above 15$.
So my very general and crude estimate IMO is they are implicitly saying they value replacing a candidate at 2k-100k, and most likely somewhere between 5-50k. I wonder if we asked them how much they would have to pay for one candidate getting replaced at the time they extended what they would say.
if anyone thinks I missed super obvious considerations or made a mistake lmk.
That post says opens with
This is apparently a pattern
Seems pretty plausible they value a marginal new charity at $100k, or even $1m, given the amount of staff time and seed funding that go into each participant.
I also suspect they’re more limited by applicant quality than number of spaces.
That post further says
If you assume that the late applicants recruited by posting on EAF are in the “relevant” pool, those aren’t terrible odds.[1] And they provide feedback even to first round applicants, which is a real service to applicants and cost to CE.
I don’t know if they’re doing the ideal thing here, but they are doing way better than I imagined from your comment.
I don’t love treating relevant and “within EA” as synonyms, but my guess is this that the real point is “don’t even really understand what we do”, and EA is a shorthand for the group that does.
Yep after walking through it in my head plus re- reading the post, doesn’t seem egregious to me.
Hi Elizabeth,
I represent Rethink Priorities but the incubator Charlie is referencing was/is run by Charity Entrepreneurship, which is a different and fully separate org. So you would have to ask them.
If there are any of your questions you’d want me to answer with reference to Rethink Priorities, let me know!
Oops, should have read more carefully, sorry about that.