I think this post could have been better researched, as it relates to how EA funders already work, and how they attempt to address problems.
Note that many funders do already consult multiple outside experts, especially for large grants, and do something like gather light-weight input. I don’t know if they have uniform systems for this, or have this written up publicly, but I have been asked for this type of input by 2 different funders, and know a third, Survival and Flourishing Fund, has said quite a bit about their more complex model.
Also, a critical problem with peer review funding, as has been widely discussed, is that it doesn’t promote long-shot bets or allow unpopular ideas to ever get explored, since you need a set of people to all rate the idea highly, instead of just one reviewer. This is a key reason that Survival and Flourishing—which as I mentioned, does use a group of expert reviewers—is structured to use their complex S-process, rather than ratings.
Thank you, David! From what you’ve said here it seems clear my post was missing critical information.
I’m not sure this post literally could have been much better researched, conditional on me writing it. I don’t feel entitled to contact funders to ask them about their process (perhaps I should feel free to? I’m not sure). EA Funds website mentions briefly they “engage expert-led teams of subject matter experts” in their decision-making, and that’s something I should have researched first and mentioned, but also, I think that gives away so little information that I learned more from your reply here than I would have from reading that.
Perhaps other funders describe their process in more detail, I don’t know, and if so, I concede that’s something I could have identified before writing the post.
So the only other way I can see this post could have had more information is that I could have asked more widely with people more familiar with the process than I am. But I’m not personally acquainted with anyone who I knew would know more about the process.
Or, finally, I could have left it to someone else to write, but then, if they didn’t, I wouldn’t have learned from you that grant-makers already engage in expert consultation.
It probably is obvious to you, considering your experiences. But the way grant making was described on the Doing Good Better post last week—something to the effect of “it helps to move to the bay and make friends with grant-makers”—suggests to me the process is pretty opaque to a lot of other people, not just to me, and so I suppose I’m glad I opened a conversation even if I don’t have a lot of insight to share on the process.
Edit: There is a point I’m trying to make, other than defending my own process, which is that the process in general is fairly opaque, and if the information you’re talking about is publicly available, I’m not aware of it! And that validates something of the transparency critiques from the DGB post last week.
I don’t feel entitled to contact funders to ask them about their process (perhaps I should feel free to? I’m not sure)
I do think you should feel free to do this! Open Phil, EA Funds, GiveWell, and ACE all have contact pages, and “I’m thinking about how EA funding could be better and I wanted to understand more how you work,” followed by specific questions that aren’t too much work to answer, is something I’d expect to be well received.
On the other hand, I don’t think you have to do this. Your post, as is, still is helpful in describing how the NIH and NSF make funding decisions. Personally, though, I would want to at talk to existing funders and learn a bit about their current process (or learn that they don’t want to talk about it) before including proposals about what I think they should do differently ;)
I agree that this is tricky to do, because the processes aren’t so well publicly documented. (Not that they should be—funders providing information about their processes make them more gameable, as most government funding is!)
I do think that you could have asked more people with knowledge of the process to review the post, and also think that the Survival and Flourishing Fund documents what they do pretty clearly, including both their writeup, and at least one forum post by a reviewer documenting it pretty extensively.
I think this post could have been better researched, as it relates to how EA funders already work, and how they attempt to address problems.
Note that many funders do already consult multiple outside experts, especially for large grants, and do something like gather light-weight input. I don’t know if they have uniform systems for this, or have this written up publicly, but I have been asked for this type of input by 2 different funders, and know a third, Survival and Flourishing Fund, has said quite a bit about their more complex model.
Also, a critical problem with peer review funding, as has been widely discussed, is that it doesn’t promote long-shot bets or allow unpopular ideas to ever get explored, since you need a set of people to all rate the idea highly, instead of just one reviewer. This is a key reason that Survival and Flourishing—which as I mentioned, does use a group of expert reviewers—is structured to use their complex S-process, rather than ratings.
Thank you, David! From what you’ve said here it seems clear my post was missing critical information.
I’m not sure this post literally could have been much better researched, conditional on me writing it. I don’t feel entitled to contact funders to ask them about their process (perhaps I should feel free to? I’m not sure). EA Funds website mentions briefly they “engage expert-led teams of subject matter experts” in their decision-making, and that’s something I should have researched first and mentioned, but also, I think that gives away so little information that I learned more from your reply here than I would have from reading that.
Perhaps other funders describe their process in more detail, I don’t know, and if so, I concede that’s something I could have identified before writing the post.
So the only other way I can see this post could have had more information is that I could have asked more widely with people more familiar with the process than I am. But I’m not personally acquainted with anyone who I knew would know more about the process.
Or, finally, I could have left it to someone else to write, but then, if they didn’t, I wouldn’t have learned from you that grant-makers already engage in expert consultation.
It probably is obvious to you, considering your experiences. But the way grant making was described on the Doing Good Better post last week—something to the effect of “it helps to move to the bay and make friends with grant-makers”—suggests to me the process is pretty opaque to a lot of other people, not just to me, and so I suppose I’m glad I opened a conversation even if I don’t have a lot of insight to share on the process.
Edit: There is a point I’m trying to make, other than defending my own process, which is that the process in general is fairly opaque, and if the information you’re talking about is publicly available, I’m not aware of it! And that validates something of the transparency critiques from the DGB post last week.
I do think you should feel free to do this! Open Phil, EA Funds, GiveWell, and ACE all have contact pages, and “I’m thinking about how EA funding could be better and I wanted to understand more how you work,” followed by specific questions that aren’t too much work to answer, is something I’d expect to be well received.
On the other hand, I don’t think you have to do this. Your post, as is, still is helpful in describing how the NIH and NSF make funding decisions. Personally, though, I would want to at talk to existing funders and learn a bit about their current process (or learn that they don’t want to talk about it) before including proposals about what I think they should do differently ;)
I agree that this is tricky to do, because the processes aren’t so well publicly documented. (Not that they should be—funders providing information about their processes make them more gameable, as most government funding is!)
I do think that you could have asked more people with knowledge of the process to review the post, and also think that the Survival and Flourishing Fund documents what they do pretty clearly, including both their writeup, and at least one forum post by a reviewer documenting it pretty extensively.