Thanks for laying this out. My quick take is that I’m uncertain how useful such evaluations will be in practice. I could see my reaction to them being anything between “while I won’t buy the bottom-line number as best-guess estimate or only input into my decision, this evaluation overall was super useful, if only to identify the most important uncertainties regarding this grant’s/project’s impact” and “this all feels so brittle and speculative that I don’t feel comfortable using any of the numbers, and all the qualitative insights I can clean from them were sort of obvious”.
But I think it’d be very worth trying.
My best guess is to agree with you that we should be doing more evaluations of roughly the kind you suggest on the margin. (At least, say, the EAIF, which is something I can speak to from my own experience. I also have a fuzzy more general sense that it would be good for many EA orgs/people to do so, but there I have a much poorer sense of the opportunity cost and the extent to which such things might happen internally.)
FWIW if you or someone else with a track record at making the required kind of quantitative estimates wanted to do some retrospective evaluations of EAIF grants in the spirit proposed here, I’d be very likely in favor of funding this.
Thanks for laying this out. My quick take is that I’m uncertain how useful such evaluations will be in practice. I could see my reaction to them being anything between “while I won’t buy the bottom-line number as best-guess estimate or only input into my decision, this evaluation overall was super useful, if only to identify the most important uncertainties regarding this grant’s/project’s impact” and “this all feels so brittle and speculative that I don’t feel comfortable using any of the numbers, and all the qualitative insights I can clean from them were sort of obvious”.
But I think it’d be very worth trying.
My best guess is to agree with you that we should be doing more evaluations of roughly the kind you suggest on the margin. (At least, say, the EAIF, which is something I can speak to from my own experience. I also have a fuzzy more general sense that it would be good for many EA orgs/people to do so, but there I have a much poorer sense of the opportunity cost and the extent to which such things might happen internally.)
FWIW if you or someone else with a track record at making the required kind of quantitative estimates wanted to do some retrospective evaluations of EAIF grants in the spirit proposed here, I’d be very likely in favor of funding this.