I’m late to the discussion, but I’m curious how much of the potential value would be unlocked—at least for modest size /​ many grants orgs like EA Funds—if we got a better writeup for a random ~10 percent of grants (with the selection of the ten percent happening after the grant decisions were made).
Great suggestion, Jason! I think that would be over 50 % as valuable as detailed write-ups for all grants.
Actually, the grants which were described in this post on the Long-Term Future Fund (LTFF) and this on the Effective Altruism Infrastructure Fund (EAIF) were randomly selected after being divided into multiple tiers according to their cost-effectiveness[1]. I think this procedure was great. I would just make the probability of a grant being selected proportional to its size. The 5th and 95th percentile amount granted are 2.00 k$ and 234 k$, which is a wide range, so it is specially important to make larger grants being more likely to be picked if one is just analysing a few grants as opposed to dozens of grants (as it was the case for the posts). Otherwise there is a risk of picking small grants which are not representative of the mean grant.
There is still the question about how detailed the write-ups of the selected grants should be. They are just a few paragraphs in the posts I linked above, which in my mind is not enough to make a case for the value of the grants without many unstated background assumptions.
If the idea is to see the quality of the median grant, not assess individual grants, then a random sample should work ~as well as writing and polishing for dozens and dozens of grants a year.
Nitpick. I think we should care about the quality of the mean (not median) grant weighted by grant size, which justifies picking each grant with a probability proportional to its size.
On the nitpick: After reflection, I’d go with a mixed approach (somewhere between even odds and weighted odds of selection). If the point is donor oversight/​evaluation/​accountability, then I am hesitant to give the grantmakers too much information ex ante on which grants are very likely/​unlikely to get the public writeup treatment. You could do some sort of weighted stratified sampling, though.
I think grant size also comes into play on the detail level of the writeup. I don’t think most people want more than a paragraph, maximum, on a $2K grant. I’d hope for considerably more on $234K. So the overweighting of small grants relative to their percentage of the dollar-amount pie would be at least somewhat counterbalanced by them getting briefer writeups if selected. So the expected-words-per-dollar figures might be somewhat similar throughout the range of grant sizes.
If the point is donor oversight/​evaluation/​accountability, then I am hesitant to give the grantmakers too much information ex ante on which grants are very likely/​unlikely to get the public writeup treatment.
Great point! I had not thought about that. On the other hand, I assume grantmakers are already spending more time on assessing larger grants. So I wonder whether the distribution of the granted amount is sufficiently heavy-tailed for grantmakers to be influenced to spend too much time on them due to their higher chance of being selected for having longer write-ups.
I think grant size also comes into play on the detail level of the writeup.
Another nice point. I agree the level of detail of the write-up should be proportional to the granted amount.
I think I have an older discussion about managing conflicts of interest in grantmaking the back of my mind. I think that’s part of why I would want to see a representative sample of small-to-midsize grant writeups.
Great suggestion, Jason! I think that would be over 50 % as valuable as detailed write-ups for all grants.
Actually, the grants which were described in this post on the Long-Term Future Fund (LTFF) and this on the Effective Altruism Infrastructure Fund (EAIF) were randomly selected after being divided into multiple tiers according to their cost-effectiveness[1]. I think this procedure was great. I would just make the probability of a grant being selected proportional to its size. The 5th and 95th percentile amount granted are 2.00 k$ and 234 k$, which is a wide range, so it is specially important to make larger grants being more likely to be picked if one is just analysing a few grants as opposed to dozens of grants (as it was the case for the posts). Otherwise there is a risk of picking small grants which are not representative of the mean grant.
There is still the question about how detailed the write-ups of the selected grants should be. They are just a few paragraphs in the posts I linked above, which in my mind is not enough to make a case for the value of the grants without many unstated background assumptions.
Nitpick. I think we should care about the quality of the mean (not median) grant weighted by grant size, which justifies picking each grant with a probability proportional to its size.
I know you are aware of this, since you commented on the post on LTFF, but I am writing this here for readers who did not know about the posts.
On the nitpick: After reflection, I’d go with a mixed approach (somewhere between even odds and weighted odds of selection). If the point is donor oversight/​evaluation/​accountability, then I am hesitant to give the grantmakers too much information ex ante on which grants are very likely/​unlikely to get the public writeup treatment. You could do some sort of weighted stratified sampling, though.
I think grant size also comes into play on the detail level of the writeup. I don’t think most people want more than a paragraph, maximum, on a $2K grant. I’d hope for considerably more on $234K. So the overweighting of small grants relative to their percentage of the dollar-amount pie would be at least somewhat counterbalanced by them getting briefer writeups if selected. So the expected-words-per-dollar figures might be somewhat similar throughout the range of grant sizes.
Great point! I had not thought about that. On the other hand, I assume grantmakers are already spending more time on assessing larger grants. So I wonder whether the distribution of the granted amount is sufficiently heavy-tailed for grantmakers to be influenced to spend too much time on them due to their higher chance of being selected for having longer write-ups.
Another nice point. I agree the level of detail of the write-up should be proportional to the granted amount.
I think I have an older discussion about managing conflicts of interest in grantmaking the back of my mind. I think that’s part of why I would want to see a representative sample of small-to-midsize grant writeups.