Thanks for this! I appreciate the transparency, and all the work that has gone into it from so many people! I was curious about one part of the AMR pull funding CEA (cells C22 and C30), which says: “Based on limited data, it seems like advocacy can increase chance success by 10-15%”
I was wondering what data that was based on? In the sheet the 10-15% is used as percentage points, such that the chance of the PASTEUR Act passing goes from 7.5% to 20% (alternate estimate 10% to 17.5%). The implied 2.6X (alternate 1.75X) multiplier as a result of advocacy seems quite high, but I’m unsure. My intuition’s just based on the size and complexity of the U.S. legislative machinery, but for professionallobbying this 2020 post suggests a central estimate of 2.5% change (95%CI 0% to 5%), and this LSE study suggested an average impact of 0.05 percentage points. I’d weakly expect a new non-profit to be less effective than the average of professional lobbyists.
On its own it doesn’t look like this concern changes the bottom line though—even with just a 0.03 percentage point impact, it is still $35/DALY!
Hi Ben, happy to justify this. I was responsible for the alternate estimate of 10%-17.5%
– –
These numbers here are consistent with our other estimates of policy change. Other estimates were easier (but not easy) to justify as they were in better evidenced areas and tended to range between 5% and 40% (see 2022 ideas here). Areas with the best data were road safety policy, where we looked at looked at 84 case studies finding a 48% chance of policy success, and food fortification policy, where we looked at 62 case studies (in Annex) with a 47% chance of success – we scaled these numbers downwards but they are not low.
Other EA estimates are also in this ballpark – e.g. Nuno’s estimates on OpenPhil criminal justice reform are in the same ball park and give 7-50% and 1-10% chance of policy change success. Also personal experience suggests reasonably high numbers too – CE has one long-running policy charity (LEEP) and it seems to have been pretty successful, driving policy change within 6 months. My own experience was also fairly successful. I think this allows us to put relatively high priors on policy change of 5 to 20 percentage points increase.
That said my 7.5 percentage points increase here was guesswork. I wish I had more time to look into it but it was driven by intuitions based of case studies and experience, that is mostly not US based. I would be open to hearing from experienced US policy advocates that it is too high (or too low).
– –
On the articles you link to:
The informational lobbying post estimates that lobbying raises the chance of policy change from “very low” to 2.5%. I think this is consistent with a raise from 10% to 17.5%.
My view on policy (based on working in the field) is that the most impact per $ comes when advocating for a policy issue that might just happen anyway – e.g. advocating for something where the political door is ajar and it just needs a nudge. I think moving the needle from 15%-40% is about as easy as moving the needle from 1%-5%.
I am unfamiliar with the LSE study and will have to have a look at it. Maybe it will lead me to be more pessimistic.
– –
Note 1: I work for CE. Note 2: I think policy change in the animals welfare space is a bit different so assume I am not talking about animal welfare work in any of the above.
Just a note here as the author of that lobbying post you cite: the CEA including the 2.5% change in chance of success is intended to be illustrative — well, conservative, but it’s based on nothing more than a rough sense of effect magnitude from having read all those studies for the lit review. The specific figures included in the CEA are very rough. As Stephen Clare pointed out in the comments, it’s also probably not realistic to have modeled that is normal on the [0,5] 95% CI.
Thanks for this! I appreciate the transparency, and all the work that has gone into it from so many people! I was curious about one part of the AMR pull funding CEA (cells C22 and C30), which says: “Based on limited data, it seems like advocacy can increase chance success by 10-15%”
I was wondering what data that was based on? In the sheet the 10-15% is used as percentage points, such that the chance of the PASTEUR Act passing goes from 7.5% to 20% (alternate estimate 10% to 17.5%). The implied 2.6X (alternate 1.75X) multiplier as a result of advocacy seems quite high, but I’m unsure. My intuition’s just based on the size and complexity of the U.S. legislative machinery, but for professional lobbying this 2020 post suggests a central estimate of 2.5% change (95%CI 0% to 5%), and this LSE study suggested an average impact of 0.05 percentage points. I’d weakly expect a new non-profit to be less effective than the average of professional lobbyists.
On its own it doesn’t look like this concern changes the bottom line though—even with just a 0.03 percentage point impact, it is still $35/DALY!
Hi Ben, happy to justify this. I was responsible for the alternate estimate of 10%-17.5%
– –
These numbers here are consistent with our other estimates of policy change. Other estimates were easier (but not easy) to justify as they were in better evidenced areas and tended to range between 5% and 40% (see 2022 ideas here). Areas with the best data were road safety policy, where we looked at looked at 84 case studies finding a 48% chance of policy success, and food fortification policy, where we looked at 62 case studies (in Annex) with a 47% chance of success – we scaled these numbers downwards but they are not low.
Other EA estimates are also in this ballpark – e.g. Nuno’s estimates on OpenPhil criminal justice reform are in the same ball park and give 7-50% and 1-10% chance of policy change success. Also personal experience suggests reasonably high numbers too – CE has one long-running policy charity (LEEP) and it seems to have been pretty successful, driving policy change within 6 months. My own experience was also fairly successful. I think this allows us to put relatively high priors on policy change of 5 to 20 percentage points increase.
That said my 7.5 percentage points increase here was guesswork. I wish I had more time to look into it but it was driven by intuitions based of case studies and experience, that is mostly not US based. I would be open to hearing from experienced US policy advocates that it is too high (or too low).
– –
On the articles you link to:
The informational lobbying post estimates that lobbying raises the chance of policy change from “very low” to 2.5%. I think this is consistent with a raise from 10% to 17.5%.
My view on policy (based on working in the field) is that the most impact per $ comes when advocating for a policy issue that might just happen anyway – e.g. advocating for something where the political door is ajar and it just needs a nudge. I think moving the needle from 15%-40% is about as easy as moving the needle from 1%-5%.
I am unfamiliar with the LSE study and will have to have a look at it. Maybe it will lead me to be more pessimistic.
– –
Note 1: I work for CE. Note 2: I think policy change in the animals welfare space is a bit different so assume I am not talking about animal welfare work in any of the above.
That all makes sense and seems thorough, thanks!
Just a note here as the author of that lobbying post you cite: the CEA including the 2.5% change in chance of success is intended to be illustrative — well, conservative, but it’s based on nothing more than a rough sense of effect magnitude from having read all those studies for the lit review. The specific figures included in the CEA are very rough. As Stephen Clare pointed out in the comments, it’s also probably not realistic to have modeled that is normal on the [0,5] 95% CI.
Thanks for the clarification!