Thanks for your comment! I wanted to try to clarify a few things regarding the two claims you see us as making. I agree there are major benefits to providing feedback to applicants. But there are also significant costs, too, and I want to explain why it’s at least a non-obvious decision what the right choice is here.
On (1), I agree with Sam that it wouldn’t be the right prioritization for our team right now to give detailed feedback to >1600 applications we rejected, and would cut into our total output for the year significantly. I think it could be done if need be, but it would be really hard and require an innovative approach. So I don’t think we should be doing this now, but I’m not saying that we won’t try to find ways to give more feedback in the future (see below).
On (2), although we want to effectively allocate at least $100M this year, we don’t plan to do 100% of this using this particular process without growing our team. In our announcement post, we said we would try four different processes and see what works best. We could continue all, some, or none of them. We have given out considerably less than $100M via the open call (more in our progress update in a month or so); and, as I mentioned in another comment, for larger and/or more complex grants the investigation process often takes longer than two weeks.
On hiring someone to do this: I think there are good reasons for us not to hire an extra person whose job is to give feedback to everyone. Most importantly: there are lots of things we could hire for, I take early hiring decisions very seriously because they affect the culture and long-term trajectory of the organization, and we want to take those decisions slowly and deliberately. I also think it’s important to maintain a certain quality bar for this kind of feedback, and this would likely require significant oversight from the existing team.
Will we provide feedback to rejected applicants in the future? Possibly, but I think this involves complex tradeoffs and isn’t a no-brainer. I’ll try to explain some of the reasons I see it this way, even at scale. A simple and unfortunate reason is that there are a lot of opportunities for angry rejected applicants—most of whom we do not know at all and aren’t part of the effective altruism community—to play “gotcha” on Twitter (or with lawsuit threats) in response to badly worded feedback, and even if the chances of this happening are small for any single rejected application, the cumulative chances of this happening once are substantial if you’re giving feedback to thousands of people. (I think this may be why even many public-spirited employers and major funders don’t provide such feedback.) I could imagine a semi-standardized process that gave more feedback to people who wanted it and very nearly got funded. (A model that I heard TripleByte used sounds interesting to me.) We’ll have to revisit these questions the next time we have an open call, and we’ll take the conversation here into account—we really appreciate your feedback!
A model that I heard TripleByte used sounds interesting to me.
I wrote a comment about TripleByte’s feedback process here; this blog post is great too. In our experience, the fear of lawsuits and PR disasters from giving feedback to rejected candidates was much overblown, even at a massive scale. (We gave every candidate feedback regardless of how well they performed on our interview.)
Something I didn’t mention in my comment is that much of TripleByte’s feedback email was composed of prewritten text blocks carefully optimized to be helpful and non-offensive. While interviewing a candidate, I would check boxes for things like “this candidate used their debugger poorly”, and then their feedback email would automatically include a prewritten spiel with links on how to use a debugger well (or whatever). I think this model could make a lot of sense for the fund:
It makes giving feedback way more scalable. There’s a one-time setup cost of prewriting some text blocks, and probably a minor ongoing cost of gradually improving your blocks over time, but the marginal cost of giving a candidate feedback is just 30 seconds of checking some boxes. (IIRC our approach was to tell candidates “here are some things we think it might be helpful for you to read” and then when in doubt, err on the side of checking more boxes. For funding, I’d probably take it a step further, and rank or score the text blocks according to their importance to your decision. At TripleByte, we would score the candidate on different facets of their interview performance and send them their scores—if you’re already scoring applications according to different facets, this could be a cheap way to provide feedback.)
Minimize lawsuit risk. It’s not that costly to have a lawyer vet a few pages of prewritten text that will get reused over and over. (We didn’t have a lawyer look over our feedback emails, and it turned out fine, so this is a conservative recommendation.)
Minimize PR risk. Someone who posts their email to Twitter can expect bored replies like “yeah, they wrote the exact same thing in my email.” (Again, PR risk didn’t seem to be an issue in practice despite giving lots of freeform feedback along with the prewritten blocks, so this seems like a conservative approach to me.)
If I were you, I think I’d experiment with hiring one of the writers of the TripleByte feedback emails as a contractor or consultant. Happy to make an intro.
A few final thoughts:
Without feedback, a rejectee is likely to come up with their own theory of why they were rejected. You have no way to observe this theory or vet its quality. So I think it’s a mistake to hold yourself to a high bar. You just have to beat the rejectee’s theory. (BTW, most of the EA rejectee theories I’ve heard have been very cynical.)
You might look into liability insurance if you don’t have it already; it probably makes sense to get it for other reasons anyway. I’d be curious how the cost of insurance changes depending on the feedback you’re giving.
Will we provide feedback to rejected applicants in the future? Possibly, but I think this involves complex tradeoffs and isn’t a no-brainer
So I don’t think we should be doing this now, but I’m not saying that we won’t try to find ways to give more feedback in the future (see below).
Very much appreciate the considerate engagement with this. Wanted to flag that my primary response to your initial comment can be found here.
All this makes a lot of sense to me. I suspect some people got value out of the presentation of this reasoning. My goal here was to bring this set of consideration to yours and Sam’s attention and upvote its importance, hopefully it’s factored into what is definitely non-obvious and complex to decide moving forward. Great to see how thoughtful you all have been and thanks again!
Thanks for the response, and thanks for being open to improving your process, and I agree with many of your points about the importance of scaling teams cautiously.
Thanks for your comment! I wanted to try to clarify a few things regarding the two claims you see us as making. I agree there are major benefits to providing feedback to applicants. But there are also significant costs, too, and I want to explain why it’s at least a non-obvious decision what the right choice is here.
On (1), I agree with Sam that it wouldn’t be the right prioritization for our team right now to give detailed feedback to >1600 applications we rejected, and would cut into our total output for the year significantly. I think it could be done if need be, but it would be really hard and require an innovative approach. So I don’t think we should be doing this now, but I’m not saying that we won’t try to find ways to give more feedback in the future (see below).
On (2), although we want to effectively allocate at least $100M this year, we don’t plan to do 100% of this using this particular process without growing our team. In our announcement post, we said we would try four different processes and see what works best. We could continue all, some, or none of them. We have given out considerably less than $100M via the open call (more in our progress update in a month or so); and, as I mentioned in another comment, for larger and/or more complex grants the investigation process often takes longer than two weeks.
On hiring someone to do this: I think there are good reasons for us not to hire an extra person whose job is to give feedback to everyone. Most importantly: there are lots of things we could hire for, I take early hiring decisions very seriously because they affect the culture and long-term trajectory of the organization, and we want to take those decisions slowly and deliberately. I also think it’s important to maintain a certain quality bar for this kind of feedback, and this would likely require significant oversight from the existing team.
Will we provide feedback to rejected applicants in the future? Possibly, but I think this involves complex tradeoffs and isn’t a no-brainer. I’ll try to explain some of the reasons I see it this way, even at scale. A simple and unfortunate reason is that there are a lot of opportunities for angry rejected applicants—most of whom we do not know at all and aren’t part of the effective altruism community—to play “gotcha” on Twitter (or with lawsuit threats) in response to badly worded feedback, and even if the chances of this happening are small for any single rejected application, the cumulative chances of this happening once are substantial if you’re giving feedback to thousands of people. (I think this may be why even many public-spirited employers and major funders don’t provide such feedback.) I could imagine a semi-standardized process that gave more feedback to people who wanted it and very nearly got funded. (A model that I heard TripleByte used sounds interesting to me.) We’ll have to revisit these questions the next time we have an open call, and we’ll take the conversation here into account—we really appreciate your feedback!
I wrote a comment about TripleByte’s feedback process here; this blog post is great too. In our experience, the fear of lawsuits and PR disasters from giving feedback to rejected candidates was much overblown, even at a massive scale. (We gave every candidate feedback regardless of how well they performed on our interview.)
Something I didn’t mention in my comment is that much of TripleByte’s feedback email was composed of prewritten text blocks carefully optimized to be helpful and non-offensive. While interviewing a candidate, I would check boxes for things like “this candidate used their debugger poorly”, and then their feedback email would automatically include a prewritten spiel with links on how to use a debugger well (or whatever). I think this model could make a lot of sense for the fund:
It makes giving feedback way more scalable. There’s a one-time setup cost of prewriting some text blocks, and probably a minor ongoing cost of gradually improving your blocks over time, but the marginal cost of giving a candidate feedback is just 30 seconds of checking some boxes. (IIRC our approach was to tell candidates “here are some things we think it might be helpful for you to read” and then when in doubt, err on the side of checking more boxes. For funding, I’d probably take it a step further, and rank or score the text blocks according to their importance to your decision. At TripleByte, we would score the candidate on different facets of their interview performance and send them their scores—if you’re already scoring applications according to different facets, this could be a cheap way to provide feedback.)
Minimize lawsuit risk. It’s not that costly to have a lawyer vet a few pages of prewritten text that will get reused over and over. (We didn’t have a lawyer look over our feedback emails, and it turned out fine, so this is a conservative recommendation.)
Minimize PR risk. Someone who posts their email to Twitter can expect bored replies like “yeah, they wrote the exact same thing in my email.” (Again, PR risk didn’t seem to be an issue in practice despite giving lots of freeform feedback along with the prewritten blocks, so this seems like a conservative approach to me.)
If I were you, I think I’d experiment with hiring one of the writers of the TripleByte feedback emails as a contractor or consultant. Happy to make an intro.
A few final thoughts:
Without feedback, a rejectee is likely to come up with their own theory of why they were rejected. You have no way to observe this theory or vet its quality. So I think it’s a mistake to hold yourself to a high bar. You just have to beat the rejectee’s theory. (BTW, most of the EA rejectee theories I’ve heard have been very cynical.)
You might look into liability insurance if you don’t have it already; it probably makes sense to get it for other reasons anyway. I’d be curious how the cost of insurance changes depending on the feedback you’re giving.
Very much appreciate the considerate engagement with this. Wanted to flag that my primary response to your initial comment can be found here.
All this makes a lot of sense to me. I suspect some people got value out of the presentation of this reasoning. My goal here was to bring this set of consideration to yours and Sam’s attention and upvote its importance, hopefully it’s factored into what is definitely non-obvious and complex to decide moving forward. Great to see how thoughtful you all have been and thanks again!
Thanks for the response, and thanks for being open to improving your process, and I agree with many of your points about the importance of scaling teams cautiously.