Question: Does anyone know how big the pool of farm-animal charities ACE has considered actually is? And do we know if the best ones are a lot better than the mean?
The reason I’m asking is that the amount of regression to the mean/ winner’s curse we should expect depends (mostly) on two things: How big is the sample out of which we’ve selected the best charities and what’s the actual mean effectiveness in the appropriate reference class? (Variance would also be a factor)
To make this more concrete, suppose you let 4 people do a run and pick the fastest. If you let them run again, the previous winner will probably be roughly as fast as the first time. But if you did the same experiment with 1,000,000 people, the first winner would probably be significantly slower the next time because they just had a really good day the first time.
My guess would be that the number of charities we could call part of the sample is lower for farm animal advocacy than for global poverty, which would mean that we should expect less regression to the mean.
The second point is that in case the best charities in the farm animal advocacy sector aren’t actually that far above the mean, we shouldn’t expect our estimate of their effectiveness to decrease as much as we get better information.
Looking at the ACE website, they recommend 3 and examined 100 organizations. The cost-effectiveness estimates for the top charities seem to be heavily based on the same studies (with related methodological issues) suggesting that veg*n ads/flyers have incredibly high persuasion rates. The analysis of media impressions also uses those same studies to estimate how much meat reduction media coverage will give. There is also some discussion of policy changes and legal activity, as well as RCTs, but they place less of a role in the cost-effectiveness estimates.
So it looks like the reason the estimates are so high is because of the claims that 2% of people reached with ads/flyers will give up meat, combined with selection of organizations whose activities are evaluated in terms of ads, flyers and videos. I would worry about regression on the studies more than on the selection of charities conditional on the studies being right.
Question: Does anyone know how big the pool of farm-animal charities ACE has considered actually is? And do we know if the best ones are a lot better than the mean?
The reason I’m asking is that the amount of regression to the mean/ winner’s curse we should expect depends (mostly) on two things: How big is the sample out of which we’ve selected the best charities and what’s the actual mean effectiveness in the appropriate reference class? (Variance would also be a factor)
To make this more concrete, suppose you let 4 people do a run and pick the fastest. If you let them run again, the previous winner will probably be roughly as fast as the first time. But if you did the same experiment with 1,000,000 people, the first winner would probably be significantly slower the next time because they just had a really good day the first time.
My guess would be that the number of charities we could call part of the sample is lower for farm animal advocacy than for global poverty, which would mean that we should expect less regression to the mean.
The second point is that in case the best charities in the farm animal advocacy sector aren’t actually that far above the mean, we shouldn’t expect our estimate of their effectiveness to decrease as much as we get better information.
Looking at the ACE website, they recommend 3 and examined 100 organizations. The cost-effectiveness estimates for the top charities seem to be heavily based on the same studies (with related methodological issues) suggesting that veg*n ads/flyers have incredibly high persuasion rates. The analysis of media impressions also uses those same studies to estimate how much meat reduction media coverage will give. There is also some discussion of policy changes and legal activity, as well as RCTs, but they place less of a role in the cost-effectiveness estimates.
So it looks like the reason the estimates are so high is because of the claims that 2% of people reached with ads/flyers will give up meat, combined with selection of organizations whose activities are evaluated in terms of ads, flyers and videos. I would worry about regression on the studies more than on the selection of charities conditional on the studies being right.