I’m late to the discussion, but I’m curious how much of the potential value would be unlocked—at least for modest size / many grants orgs like EA Funds—if we got a better writeup for a random ~10 percent of grants (with the selection of the ten percent happening after the grant decisions were made).
If the idea is to see the quality of the median grant, not assess individual grants, then a random sample should work ~as well as writing and polishing for dozens and dozens of grants a year.
I’m late to the discussion, but I’m curious how much of the potential value would be unlocked—at least for modest size / many grants orgs like EA Funds—if we got a better writeup for a random ~10 percent of grants (with the selection of the ten percent happening after the grant decisions were made).
Great suggestion, Jason! I think that would be over 50 % as valuable as detailed write-ups for all grants.
Actually, the grants which were described in this post on the Long-Term Future Fund (LTFF) and this on the Effective Altruism Infrastructure Fund (EAIF) were randomly selected after being divided into multiple tiers according to their cost-effectiveness[1]. I think this procedure was great. I would just make the probability of a grant being selected proportional to its size. The 5th and 95th percentile amount granted are 2.00 k$ and 234 k$, which is a wide range, so it is specially important to make larger grants being more likely to be picked if one is just analysing a few grants as opposed to dozens of grants (as it was the case for the posts). Otherwise there is a risk of picking small grants which are not representative of the mean grant.
There is still the question about how detailed the write-ups of the selected grants should be. They are just a few paragraphs in the posts I linked above, which in my mind is not enough to make a case for the value of the grants without many unstated background assumptions.
If the idea is to see the quality of the median grant, not assess individual grants, then a random sample should work ~as well as writing and polishing for dozens and dozens of grants a year.
Nitpick. I think we should care about the quality of the mean (not median) grant weighted by grant size, which justifies picking each grant with a probability proportional to its size.
On the nitpick: After reflection, I’d go with a mixed approach (somewhere between even odds and weighted odds of selection). If the point is donor oversight/evaluation/accountability, then I am hesitant to give the grantmakers too much information ex ante on which grants are very likely/unlikely to get the public writeup treatment. You could do some sort of weighted stratified sampling, though.
I think grant size also comes into play on the detail level of the writeup. I don’t think most people want more than a paragraph, maximum, on a $2K grant. I’d hope for considerably more on $234K. So the overweighting of small grants relative to their percentage of the dollar-amount pie would be at least somewhat counterbalanced by them getting briefer writeups if selected. So the expected-words-per-dollar figures might be somewhat similar throughout the range of grant sizes.
If the point is donor oversight/evaluation/accountability, then I am hesitant to give the grantmakers too much information ex ante on which grants are very likely/unlikely to get the public writeup treatment.
Great point! I had not thought about that. On the other hand, I assume grantmakers are already spending more time on assessing larger grants. So I wonder whether the distribution of the granted amount is sufficiently heavy-tailed for grantmakers to be influenced to spend too much time on them due to their higher chance of being selected for having longer write-ups.
I think grant size also comes into play on the detail level of the writeup.
Another nice point. I agree the level of detail of the write-up should be proportional to the granted amount.
I think I have an older discussion about managing conflicts of interest in grantmaking the back of my mind. I think that’s part of why I would want to see a representative sample of small-to-midsize grant writeups.
I’m late to the discussion, but I’m curious how much of the potential value would be unlocked—at least for modest size / many grants orgs like EA Funds—if we got a better writeup for a random ~10 percent of grants (with the selection of the ten percent happening after the grant decisions were made).
If the idea is to see the quality of the median grant, not assess individual grants, then a random sample should work ~as well as writing and polishing for dozens and dozens of grants a year.
Great suggestion, Jason! I think that would be over 50 % as valuable as detailed write-ups for all grants.
Actually, the grants which were described in this post on the Long-Term Future Fund (LTFF) and this on the Effective Altruism Infrastructure Fund (EAIF) were randomly selected after being divided into multiple tiers according to their cost-effectiveness[1]. I think this procedure was great. I would just make the probability of a grant being selected proportional to its size. The 5th and 95th percentile amount granted are 2.00 k$ and 234 k$, which is a wide range, so it is specially important to make larger grants being more likely to be picked if one is just analysing a few grants as opposed to dozens of grants (as it was the case for the posts). Otherwise there is a risk of picking small grants which are not representative of the mean grant.
There is still the question about how detailed the write-ups of the selected grants should be. They are just a few paragraphs in the posts I linked above, which in my mind is not enough to make a case for the value of the grants without many unstated background assumptions.
Nitpick. I think we should care about the quality of the mean (not median) grant weighted by grant size, which justifies picking each grant with a probability proportional to its size.
I know you are aware of this, since you commented on the post on LTFF, but I am writing this here for readers who did not know about the posts.
On the nitpick: After reflection, I’d go with a mixed approach (somewhere between even odds and weighted odds of selection). If the point is donor oversight/evaluation/accountability, then I am hesitant to give the grantmakers too much information ex ante on which grants are very likely/unlikely to get the public writeup treatment. You could do some sort of weighted stratified sampling, though.
I think grant size also comes into play on the detail level of the writeup. I don’t think most people want more than a paragraph, maximum, on a $2K grant. I’d hope for considerably more on $234K. So the overweighting of small grants relative to their percentage of the dollar-amount pie would be at least somewhat counterbalanced by them getting briefer writeups if selected. So the expected-words-per-dollar figures might be somewhat similar throughout the range of grant sizes.
Great point! I had not thought about that. On the other hand, I assume grantmakers are already spending more time on assessing larger grants. So I wonder whether the distribution of the granted amount is sufficiently heavy-tailed for grantmakers to be influenced to spend too much time on them due to their higher chance of being selected for having longer write-ups.
Another nice point. I agree the level of detail of the write-up should be proportional to the granted amount.
I think I have an older discussion about managing conflicts of interest in grantmaking the back of my mind. I think that’s part of why I would want to see a representative sample of small-to-midsize grant writeups.