As noted in my introduction, EA organizations often have to make assumptions about how long a policy intervention matters in calculating cost-effectiveness. Typically people assume that passing a policy is equivalent to having it in place for around five years more or moving the start date of the policy forward by around five years. These results suggest that this is off by more than an order of magnitude.
I think this overstates the effects of policy interventions. In cost-effectiveness analyses of policy interventions, the five years you are referring to usually respect a period over which a certain amount of annual counterfactual benefits apply. In this type of models, the choice of the length of the period and annual counterfactual benefits are not independent[1]. The longer the period, the lower the annual counterfactual benefits. I suspect EA organisations may often be choosing a shorter period to account for the caveats you commented about:
Some things to keep in mind:
Impacts might change over time (e.g., a policy stops mattering in 50 years even if it’s in place). If you think, e.g., transformative AI will upend everything, that might be what you need to think about here in terms of how long this policy change matters.
I’m looking at whether this policy or a version of it will be in place. It’s possible policies will be substituted for in some way in ways that make things wash out somewhat. (For instance, we don’t pass one animal welfare policy, but we pass some policy to shrink the farming sector.) I think this effect is small given the lack of differences by policy topic—this should be much more of an issue for some topics than others—but see the next point.
There are some hints of less persistence for policies where there’s more room for negotiation/more ways to dial it up and down. See my reply to Erich Grunewald lower down—for taxes and Congressional legislation, it seems like the effect on whether some possibly weaker version of the policy eventually passes might wash out.
Yes, it’s a good point that benefits and length of the period are not independent, and I agree with the footnote too.
I would note that the factors I mentioned there don’t seem like they should change things that much for most issues. I could see using 50-100 years rather than, e.g., 150 years as my results would seem to suggest, but I do think 5-10 years is an order of magnitude off.
I would note that the factors I mentioned there don’t seem like they should change things that much for most issues. I could see using 50-100 years rather than, e.g., 150 years as my results would seem to suggest, but I do think 5-10 years is an order of magnitude off.
Could you elaborate on why you think multiplying your results by a factor of 0.5 would be enough? Do you think it would be possible to study the question with empirical data, by looking not only into how much time the policy changes persisted counterfactually, but also into the target outcomes (e.g. number of caged hens for policy around animal welfare standards)? I am guessing this would be much harder, but that there are some questions in this vicinity one could try to answer more empirically to get a sense of how much the persistence estimates you got have to be adjusted downwards.
Great post!
I think this overstates the effects of policy interventions. In cost-effectiveness analyses of policy interventions, the five years you are referring to usually respect a period over which a certain amount of annual counterfactual benefits apply. In this type of models, the choice of the length of the period and annual counterfactual benefits are not independent[1]. The longer the period, the lower the annual counterfactual benefits. I suspect EA organisations may often be choosing a shorter period to account for the caveats you commented about:
Ideally, one would model the effect decaying over time, instead of being constant for a certain period, and then dropping to 0.
Yes, it’s a good point that benefits and length of the period are not independent, and I agree with the footnote too.
I would note that the factors I mentioned there don’t seem like they should change things that much for most issues. I could see using 50-100 years rather than, e.g., 150 years as my results would seem to suggest, but I do think 5-10 years is an order of magnitude off.
Could you elaborate on why you think multiplying your results by a factor of 0.5 would be enough? Do you think it would be possible to study the question with empirical data, by looking not only into how much time the policy changes persisted counterfactually, but also into the target outcomes (e.g. number of caged hens for policy around animal welfare standards)? I am guessing this would be much harder, but that there are some questions in this vicinity one could try to answer more empirically to get a sense of how much the persistence estimates you got have to be adjusted downwards.