I’d like to challenge this. There are simultaneous claims that:
it’s impossible to give constructive feedback on thousands of applications
It is possible to effectively (in an expected value sense) allocate $100m - $1b a year using this process which evaluates thousands of applications from a broad range of applicants related to a broad spectrum of ideas over just a two week period
I don’t think both can be true in the long run. Like others in the comments suggested both may be a question of further investment in and improvement of the process. There is a lot of room for improvement: any feedback is better than no feedback, it doesn’t have to be super constructive -just knowing if anyone even spent more than a minute looking at your application is useful info that applicants currently don’t have.
Wanting to be constructive: would there be arguments against hiring an extra person whose job is to observe the decision making process (I assume there is a kind of internal log of decisions/opinions), and formulate non-zero feedback on applications?
It would be very surprising if there weren’t an opportunity cost to providing feedback. Those might include:
Senior management time to oversee the project, bottlenecking other plans
PR firefighting and morale counselling when 1 in ~100 people get angry at what you say and cause you grief (this will absolutely happen)
Any hires capable of thinking up and communicating helpful feedback (this is difficult!) could otherwise use that time to read and make decisions on more grant proposals in more areas — or just improve the decision-making among the same pool of applicants.
That there’s an opportunity cost doesn’t show it’s not worth it but my guess is right now it would be huge mistake for Future Fund to provide substantial feedback except in rare cases.
That could change in future if their other streams of successful applicants dry up and improving the projects of people who were previously rejected becomes the best way to find new things they want to fund.
huge mistake for Future Fund to provide substantial feedback except in rare cases.
Yep, I’d imagine what makes sense is between ‘highly involved and coordinated attempt to provide feedback at scale’ and ‘zero’. I think it’s tempting to look away from how harmful ‘zero’ can be at scale
> That could change in future if their other streams of successful applicants dry up and improving the projects of people who were previously rejected becomes the best way to find new things they want to fund.
Agreed – this seems like a way to pick up easy wins and should be a good go-to for grant makers to circle back. However, banking on this as handling the concerns that were raised doesn’t account for all the things that come with unqualified rejection and people deciding to do other things, leave EA, incur critical stakeholder instability etc. as a result.
In other words, for the consequentialist-driven among us, I don’t think that community health is a nice-to-have if we’re serious about having a community of highly effective people working urgently on hard/complex things
“However, banking on this as handling the concerns that were raised doesn’t account for all the things that come with unqualified rejection and people deciding to do other things, leave EA, incur critical stakeholder instability etc. as a result. ”
I mean I think people are radically underestimating the opportunity cost of doing feedback properly at the moment. If I’m right then getting feedback might reduce people’s chances of getting funded by say, 30%, or 50%, because the throughput for grants will be much reduced.
I would probably rather have a 20% chance of getting funding for my project without feedback than a 10% chance with feedback, though people’s preferences may vary.
(Alternatively all the time spent explaining and writing and corresponding will mean worse projects get funded as there’s not much time left to actually think through which projects are most impactful.)
Rob, I think you’re consistently arguing against a point few people are making. You talk about ongoing correspondence with projects, or writing (potentially paragraphs of) feedback. Several people in this thread have suggested that pre-written categories of feedback would be a huge improvement from the status quo, and I can’t see anything you’ve said that actually argues against that.
Also, as someone who semi-regularly gives feedback to 80+ people, I’ve never found it to make my thinking worse, but I’ve sometimes found it makes my thinking better.
I’m not saying there’s no cost to feedback. Of course there’s a cost! But these exaggerations are really frustrating to read, because I actually do this kind of work and the cost of what I’m proposing is a lot lower than you keep suggesting.
I’ve got a similar feeling to Khorton. Happy to have been pre-empted there.
It could be helpful to consider what it is that legibility in the grant application process (for which post-application feedback is only one sort) is meant to achieve. Depending on the grant maker’s aims, this can non-exhaustively include developing and nurturing talent, helping future applicants self-select, orienting projects on whether they are doing a good job, being a beacon and marketing instrument, clarifying and staking out an epistemic position, serving an orientation function for the community etc.
And depending on the basket of things the grant maker is trying to achieve, different pieces of legibility affect ‘efficiency’ in the process. For example, case studies and transparent reasoning about accepted and rejected projects, published evaluations, criteria for projects to consider before applying, hazard disclaimers, risk profile declarations, published work on the grant makers theory of change, etc. can give grant makers ‘published’ content to invoke during the post-application process that allows for the scaling of feedback. (e.g. our website states that we don’t invest in projects that rapidly accelerate ‘x’). There are other forms of pro-active communication and stratifying applicant journeys that would make things even more efficient.
FTX did what they did, and there is definitely a strong case for why they did it that way. In moving forward , I’d be curious to see if they acknowledge and make adjustments in light of the fact that different forms and degrees of legibility can affect the community.
Okay, upon review, that was a little bit too much of a rhetorical flourish at the end. Basically, I think there’s something seriously important to consider here about how process can negatively affect community health and alignment, which I believe to be important for this community in achieving the plurality of ambitious goals we’re shooting for. I believe FTX could definitely affect in a very positive way if they wanted to
Thanks for your comment! I wanted to try to clarify a few things regarding the two claims you see us as making. I agree there are major benefits to providing feedback to applicants. But there are also significant costs, too, and I want to explain why it’s at least a non-obvious decision what the right choice is here.
On (1), I agree with Sam that it wouldn’t be the right prioritization for our team right now to give detailed feedback to >1600 applications we rejected, and would cut into our total output for the year significantly. I think it could be done if need be, but it would be really hard and require an innovative approach. So I don’t think we should be doing this now, but I’m not saying that we won’t try to find ways to give more feedback in the future (see below).
On (2), although we want to effectively allocate at least $100M this year, we don’t plan to do 100% of this using this particular process without growing our team. In our announcement post, we said we would try four different processes and see what works best. We could continue all, some, or none of them. We have given out considerably less than $100M via the open call (more in our progress update in a month or so); and, as I mentioned in another comment, for larger and/or more complex grants the investigation process often takes longer than two weeks.
On hiring someone to do this: I think there are good reasons for us not to hire an extra person whose job is to give feedback to everyone. Most importantly: there are lots of things we could hire for, I take early hiring decisions very seriously because they affect the culture and long-term trajectory of the organization, and we want to take those decisions slowly and deliberately. I also think it’s important to maintain a certain quality bar for this kind of feedback, and this would likely require significant oversight from the existing team.
Will we provide feedback to rejected applicants in the future? Possibly, but I think this involves complex tradeoffs and isn’t a no-brainer. I’ll try to explain some of the reasons I see it this way, even at scale. A simple and unfortunate reason is that there are a lot of opportunities for angry rejected applicants—most of whom we do not know at all and aren’t part of the effective altruism community—to play “gotcha” on Twitter (or with lawsuit threats) in response to badly worded feedback, and even if the chances of this happening are small for any single rejected application, the cumulative chances of this happening once are substantial if you’re giving feedback to thousands of people. (I think this may be why even many public-spirited employers and major funders don’t provide such feedback.) I could imagine a semi-standardized process that gave more feedback to people who wanted it and very nearly got funded. (A model that I heard TripleByte used sounds interesting to me.) We’ll have to revisit these questions the next time we have an open call, and we’ll take the conversation here into account—we really appreciate your feedback!
A model that I heard TripleByte used sounds interesting to me.
I wrote a comment about TripleByte’s feedback process here; this blog post is great too. In our experience, the fear of lawsuits and PR disasters from giving feedback to rejected candidates was much overblown, even at a massive scale. (We gave every candidate feedback regardless of how well they performed on our interview.)
Something I didn’t mention in my comment is that much of TripleByte’s feedback email was composed of prewritten text blocks carefully optimized to be helpful and non-offensive. While interviewing a candidate, I would check boxes for things like “this candidate used their debugger poorly”, and then their feedback email would automatically include a prewritten spiel with links on how to use a debugger well (or whatever). I think this model could make a lot of sense for the fund:
It makes giving feedback way more scalable. There’s a one-time setup cost of prewriting some text blocks, and probably a minor ongoing cost of gradually improving your blocks over time, but the marginal cost of giving a candidate feedback is just 30 seconds of checking some boxes. (IIRC our approach was to tell candidates “here are some things we think it might be helpful for you to read” and then when in doubt, err on the side of checking more boxes. For funding, I’d probably take it a step further, and rank or score the text blocks according to their importance to your decision. At TripleByte, we would score the candidate on different facets of their interview performance and send them their scores—if you’re already scoring applications according to different facets, this could be a cheap way to provide feedback.)
Minimize lawsuit risk. It’s not that costly to have a lawyer vet a few pages of prewritten text that will get reused over and over. (We didn’t have a lawyer look over our feedback emails, and it turned out fine, so this is a conservative recommendation.)
Minimize PR risk. Someone who posts their email to Twitter can expect bored replies like “yeah, they wrote the exact same thing in my email.” (Again, PR risk didn’t seem to be an issue in practice despite giving lots of freeform feedback along with the prewritten blocks, so this seems like a conservative approach to me.)
If I were you, I think I’d experiment with hiring one of the writers of the TripleByte feedback emails as a contractor or consultant. Happy to make an intro.
A few final thoughts:
Without feedback, a rejectee is likely to come up with their own theory of why they were rejected. You have no way to observe this theory or vet its quality. So I think it’s a mistake to hold yourself to a high bar. You just have to beat the rejectee’s theory. (BTW, most of the EA rejectee theories I’ve heard have been very cynical.)
You might look into liability insurance if you don’t have it already; it probably makes sense to get it for other reasons anyway. I’d be curious how the cost of insurance changes depending on the feedback you’re giving.
Will we provide feedback to rejected applicants in the future? Possibly, but I think this involves complex tradeoffs and isn’t a no-brainer
So I don’t think we should be doing this now, but I’m not saying that we won’t try to find ways to give more feedback in the future (see below).
Very much appreciate the considerate engagement with this. Wanted to flag that my primary response to your initial comment can be found here.
All this makes a lot of sense to me. I suspect some people got value out of the presentation of this reasoning. My goal here was to bring this set of consideration to yours and Sam’s attention and upvote its importance, hopefully it’s factored into what is definitely non-obvious and complex to decide moving forward. Great to see how thoughtful you all have been and thanks again!
Thanks for the response, and thanks for being open to improving your process, and I agree with many of your points about the importance of scaling teams cautiously.
I’d like to challenge this. There are simultaneous claims that:
it’s impossible to give constructive feedback on thousands of applications
It is possible to effectively (in an expected value sense) allocate $100m - $1b a year using this process which evaluates thousands of applications from a broad range of applicants related to a broad spectrum of ideas over just a two week period
I don’t think both can be true in the long run. Like others in the comments suggested both may be a question of further investment in and improvement of the process. There is a lot of room for improvement: any feedback is better than no feedback, it doesn’t have to be super constructive -just knowing if anyone even spent more than a minute looking at your application is useful info that applicants currently don’t have.
Wanting to be constructive: would there be arguments against hiring an extra person whose job is to observe the decision making process (I assume there is a kind of internal log of decisions/opinions), and formulate non-zero feedback on applications?
It would be very surprising if there weren’t an opportunity cost to providing feedback. Those might include:
Senior management time to oversee the project, bottlenecking other plans
PR firefighting and morale counselling when 1 in ~100 people get angry at what you say and cause you grief (this will absolutely happen)
Any hires capable of thinking up and communicating helpful feedback (this is difficult!) could otherwise use that time to read and make decisions on more grant proposals in more areas — or just improve the decision-making among the same pool of applicants.
That there’s an opportunity cost doesn’t show it’s not worth it but my guess is right now it would be huge mistake for Future Fund to provide substantial feedback except in rare cases.
That could change in future if their other streams of successful applicants dry up and improving the projects of people who were previously rejected becomes the best way to find new things they want to fund.
Yep, I’d imagine what makes sense is between ‘highly involved and coordinated attempt to provide feedback at scale’ and ‘zero’. I think it’s tempting to look away from how harmful ‘zero’ can be at scale
> That could change in future if their other streams of successful applicants dry up and improving the projects of people who were previously rejected becomes the best way to find new things they want to fund.
Agreed – this seems like a way to pick up easy wins and should be a good go-to for grant makers to circle back. However, banking on this as handling the concerns that were raised doesn’t account for all the things that come with unqualified rejection and people deciding to do other things, leave EA, incur critical stakeholder instability etc. as a result.
In other words, for the consequentialist-driven among us, I don’t think that community health is a nice-to-have if we’re serious about having a community of highly effective people working urgently on hard/complex things
I mean I think people are radically underestimating the opportunity cost of doing feedback properly at the moment. If I’m right then getting feedback might reduce people’s chances of getting funded by say, 30%, or 50%, because the throughput for grants will be much reduced.
I would probably rather have a 20% chance of getting funding for my project without feedback than a 10% chance with feedback, though people’s preferences may vary.
(Alternatively all the time spent explaining and writing and corresponding will mean worse projects get funded as there’s not much time left to actually think through which projects are most impactful.)
Rob, I think you’re consistently arguing against a point few people are making. You talk about ongoing correspondence with projects, or writing (potentially paragraphs of) feedback. Several people in this thread have suggested that pre-written categories of feedback would be a huge improvement from the status quo, and I can’t see anything you’ve said that actually argues against that.
Also, as someone who semi-regularly gives feedback to 80+ people, I’ve never found it to make my thinking worse, but I’ve sometimes found it makes my thinking better.
I’m not saying there’s no cost to feedback. Of course there’s a cost! But these exaggerations are really frustrating to read, because I actually do this kind of work and the cost of what I’m proposing is a lot lower than you keep suggesting.
If it’s just a form where the main reason for rejection is chosen from a list then that’s probably fine/good.
I’ve seen people try to do written feedback before and find it a nightmare so I guess people’s mileage varies a fair bit.
I’ve got a similar feeling to Khorton. Happy to have been pre-empted there.
It could be helpful to consider what it is that legibility in the grant application process (for which post-application feedback is only one sort) is meant to achieve. Depending on the grant maker’s aims, this can non-exhaustively include developing and nurturing talent, helping future applicants self-select, orienting projects on whether they are doing a good job, being a beacon and marketing instrument, clarifying and staking out an epistemic position, serving an orientation function for the community etc.
And depending on the basket of things the grant maker is trying to achieve, different pieces of legibility affect ‘efficiency’ in the process. For example, case studies and transparent reasoning about accepted and rejected projects, published evaluations, criteria for projects to consider before applying, hazard disclaimers, risk profile declarations, published work on the grant makers theory of change, etc. can give grant makers ‘published’ content to invoke during the post-application process that allows for the scaling of feedback. (e.g. our website states that we don’t invest in projects that rapidly accelerate ‘x’). There are other forms of pro-active communication and stratifying applicant journeys that would make things even more efficient.
FTX did what they did, and there is definitely a strong case for why they did it that way. In moving forward , I’d be curious to see if they acknowledge and make adjustments in light of the fact that different forms and degrees of legibility can affect the community.
Okay, upon review, that was a little bit too much of a rhetorical flourish at the end. Basically, I think there’s something seriously important to consider here about how process can negatively affect community health and alignment, which I believe to be important for this community in achieving the plurality of ambitious goals we’re shooting for. I believe FTX could definitely affect in a very positive way if they wanted to
Thanks for your comment! I wanted to try to clarify a few things regarding the two claims you see us as making. I agree there are major benefits to providing feedback to applicants. But there are also significant costs, too, and I want to explain why it’s at least a non-obvious decision what the right choice is here.
On (1), I agree with Sam that it wouldn’t be the right prioritization for our team right now to give detailed feedback to >1600 applications we rejected, and would cut into our total output for the year significantly. I think it could be done if need be, but it would be really hard and require an innovative approach. So I don’t think we should be doing this now, but I’m not saying that we won’t try to find ways to give more feedback in the future (see below).
On (2), although we want to effectively allocate at least $100M this year, we don’t plan to do 100% of this using this particular process without growing our team. In our announcement post, we said we would try four different processes and see what works best. We could continue all, some, or none of them. We have given out considerably less than $100M via the open call (more in our progress update in a month or so); and, as I mentioned in another comment, for larger and/or more complex grants the investigation process often takes longer than two weeks.
On hiring someone to do this: I think there are good reasons for us not to hire an extra person whose job is to give feedback to everyone. Most importantly: there are lots of things we could hire for, I take early hiring decisions very seriously because they affect the culture and long-term trajectory of the organization, and we want to take those decisions slowly and deliberately. I also think it’s important to maintain a certain quality bar for this kind of feedback, and this would likely require significant oversight from the existing team.
Will we provide feedback to rejected applicants in the future? Possibly, but I think this involves complex tradeoffs and isn’t a no-brainer. I’ll try to explain some of the reasons I see it this way, even at scale. A simple and unfortunate reason is that there are a lot of opportunities for angry rejected applicants—most of whom we do not know at all and aren’t part of the effective altruism community—to play “gotcha” on Twitter (or with lawsuit threats) in response to badly worded feedback, and even if the chances of this happening are small for any single rejected application, the cumulative chances of this happening once are substantial if you’re giving feedback to thousands of people. (I think this may be why even many public-spirited employers and major funders don’t provide such feedback.) I could imagine a semi-standardized process that gave more feedback to people who wanted it and very nearly got funded. (A model that I heard TripleByte used sounds interesting to me.) We’ll have to revisit these questions the next time we have an open call, and we’ll take the conversation here into account—we really appreciate your feedback!
I wrote a comment about TripleByte’s feedback process here; this blog post is great too. In our experience, the fear of lawsuits and PR disasters from giving feedback to rejected candidates was much overblown, even at a massive scale. (We gave every candidate feedback regardless of how well they performed on our interview.)
Something I didn’t mention in my comment is that much of TripleByte’s feedback email was composed of prewritten text blocks carefully optimized to be helpful and non-offensive. While interviewing a candidate, I would check boxes for things like “this candidate used their debugger poorly”, and then their feedback email would automatically include a prewritten spiel with links on how to use a debugger well (or whatever). I think this model could make a lot of sense for the fund:
It makes giving feedback way more scalable. There’s a one-time setup cost of prewriting some text blocks, and probably a minor ongoing cost of gradually improving your blocks over time, but the marginal cost of giving a candidate feedback is just 30 seconds of checking some boxes. (IIRC our approach was to tell candidates “here are some things we think it might be helpful for you to read” and then when in doubt, err on the side of checking more boxes. For funding, I’d probably take it a step further, and rank or score the text blocks according to their importance to your decision. At TripleByte, we would score the candidate on different facets of their interview performance and send them their scores—if you’re already scoring applications according to different facets, this could be a cheap way to provide feedback.)
Minimize lawsuit risk. It’s not that costly to have a lawyer vet a few pages of prewritten text that will get reused over and over. (We didn’t have a lawyer look over our feedback emails, and it turned out fine, so this is a conservative recommendation.)
Minimize PR risk. Someone who posts their email to Twitter can expect bored replies like “yeah, they wrote the exact same thing in my email.” (Again, PR risk didn’t seem to be an issue in practice despite giving lots of freeform feedback along with the prewritten blocks, so this seems like a conservative approach to me.)
If I were you, I think I’d experiment with hiring one of the writers of the TripleByte feedback emails as a contractor or consultant. Happy to make an intro.
A few final thoughts:
Without feedback, a rejectee is likely to come up with their own theory of why they were rejected. You have no way to observe this theory or vet its quality. So I think it’s a mistake to hold yourself to a high bar. You just have to beat the rejectee’s theory. (BTW, most of the EA rejectee theories I’ve heard have been very cynical.)
You might look into liability insurance if you don’t have it already; it probably makes sense to get it for other reasons anyway. I’d be curious how the cost of insurance changes depending on the feedback you’re giving.
Very much appreciate the considerate engagement with this. Wanted to flag that my primary response to your initial comment can be found here.
All this makes a lot of sense to me. I suspect some people got value out of the presentation of this reasoning. My goal here was to bring this set of consideration to yours and Sam’s attention and upvote its importance, hopefully it’s factored into what is definitely non-obvious and complex to decide moving forward. Great to see how thoughtful you all have been and thanks again!
Thanks for the response, and thanks for being open to improving your process, and I agree with many of your points about the importance of scaling teams cautiously.