Very impressive and interesting piece, thanks for this! I am a colleague of smclare at Founders Pledge and work a lot on modeling policy advocacy across charities. Would be great to have a chat.
It’s a great and very useful summary of the literature, hugely valuable.
That being said, I am less convinced by the approach taken in the cost-effectiveness model which seems to somewhat contradict the prior analysis which stresses (a) contextuality and (b) strategic choice. As far as I can tell, the modeling tries to be entirely general (contradicting a)) and assumes (b) independence between variables that are very likely related (such as spending and success probability). Please let me know if I completely misunderstand what you are doing!
My sense from doing this kind of work in charity evaluation is that we would want to move towards a “suite” of models for different situations—e.g. (i) pushing genuinely new ideas, (ii) opportunistically exploiting policy windows (such as stimulus season), (iii) pushing high-impact low probability policies, (iv) averting policy rollbacks etc. and that the cost effectiveness for these kinds of things will be very different and essentially unrelated to a generic estimate such as the one featured at the end of your piece.
To me it seems that this kind of classifying into the sub-models will be the only way to realistically bound parameters of interest and also that the dynamics underlying are sufficiently different that they should probably have their own models.
Some minor points/comments: Something I found missing is a bit are standard explanations for why there is so little money in politics, namely (1)because dominant firms in districts have lots of indirect power via employees voting so that they do not need to spend money. (2) It also seems that the report somewhat under-emphasizes the idea of lobbying equilibria where marginal increases by one side would be quickly countered, which would make it look like additional money could be effective when in effect it is not.
I also think that the conclusion which, I believe, mostly draws from Baumgaertner ” (80%) Well-resourced interest groups are no more or less likely to achieve policy success, in general, than their less well-resourced opponents.” is quite surprising and I would be curious to find out why you think that / in how far you trust that conclusion.
Your criticism of the cost-effectiveness model is fair. Thematically, I guess it does contradict the spirit of my prior analysis in that it avoids the concerns of strategic choice. I was actively trying to be as general as possible, and actively trying to err on the side of greater uncertainty by not including any assumptions about correlatedness, though it occurs to me now that making such an assumption (e.g. a correlation between expenditure and likelihood of success) would actually have increased the variance of the final estimate, which would have been more in line with my goals. When I have time, I may comment here with an updated CEA.
I also agree that the only useful way to do this analysis is, as you’ve described, with a suite of models for different scenarios. I don’t have a defense for not having done this beyond my own capacity constraints, though I hope it’s more useful to have included the flawed model than not to have one at all (what do you think?).
I also think that the conclusion which, I believe, mostly draws from Baumgaertner ” (80%) Well-resourced interest groups are no more or less likely to achieve policy success, in general, than their less well-resourced opponents.” is quite surprising and I would be curious to find out why you think that / in how far you trust that conclusion.
Thanks for this, in particular. I think your surprise stems from a lack of clarity on my part. The reason I have high confidence in this conclusion is that it’s a much weaker claim than it might seem. It does stem primarily from Baumgartner et al and from Burstein and Linton (2002). The claim here is that resource-rich groups are no more or less likely to get what they want—holding all else equal, including absolute expenditure and the spending differential between groups and their opponents.
There are three types of claim that are closely related: 1) Groups that spend more relative to their opposition on a given policy are likelier to win 2) Groups that spend more in absolute terms are likelier to win 3) Groups that have more money to spend are likelier to win
So I found fairly consistent evidence for (1), some evidence for (2), and no real evidence for (3). It’s not obvious to me that (3) should be the case irrespective of (1): why would resource-rich groups succeed in lobbying if they deploy those resources poorly? It seems like the success of resource-rich groups is dependent upon (1), and that (3) should not be true when in isolation, unmediated by (1). Although Baumgartner et al conduct an observational study, the size of their (to me, convincingly representative) sample to me suggests that if such an effect exists, it should be observable as a correlation in their analysis. The association they observe is pretty small.
I have to say, though, that in writing this comment, my confidence in this conclusion has eased up a bit, so I’m curious to hear your response. I also think that since Baumgartner et al do find a small effect, I probably overstate the case here.
Baumgartner et al offer a theoretical take on this: ”...organizations rarely lobby alone. Citizen groups, like others, typically participate in policy debates alongside other actors of many types who share the same goals. For every citizen group opposing an action by a given industrial group, for example, there may also be an ally coming from a competing industry with which the group can join forces” (p.12). So it’s important to recognize that the finding here is about individual parties, not “sides” or coalitions advocating a given policy.
Finally, I’m curious to hear your take on the two potential money-in-politics explanations you mentioned. I’ve never found (1) particularly convincing—it’s not clear to me that firms and their employees have the same interests, or that (if they do) the marginal value of regulatory capture isn’t still high. But I agree that I underemphasized (2) and think it would be useful to have in this thread the “inside view” on lobbying equilibria from someone who works in the field.
Groups that spend more relative to their opposition on a given policy are likelier to win… Although Baumgartner et al conduct an observational study, the size of their (to me, convincingly representative) sample to me suggests that if such an effect exists, it should be observable as a correlation in their analysis. The association they observe is pretty small.
I just skimmed this thread, so apologies if I missed a comment on this. But Baumgartner et al. don’t argue that money doesn’t matter. They believe that the reason there’s little evidence in their study that money affects outcomes is because “the status quo already reflects the distribution of power in previous rounds of the policy process.” I.e. they don’t see large changes during their 4 year study period, because the situation at the start of their study period tended to favour the resource-rich groups.
After your comments and @jackva’s, I actually struck this conclusion. I was trying to make a more modest statement that upon reflection (thanks to you) is (1) not such a valuable claim and (2) not well-supported enough to have >50% confidence in. It’s true Baumgartner don’t find that money doesn’t matter; my initial (now disavowed) read was that if resources mattered independent of deployment strategy, then we’d expect to see a much stronger correlation even in the observational context. I sort of think that this observation holds true even given the passage you’ve cited, but it’s definitely not a top-level extract from the lit review and definitely needs a considerably more robust defense than I am prepared to muster.
Thanks for your reply and sorry for the slight delay! Would you like to present your work in a session at Founders Pledge (we have a Journal Club where we discuss relevant research and having you as a guest speaker there sounds like a good idea)?
On the cost effectiveness model On the model generality, I think my view is that even a general and simple model should not be unrealistic / systematically biased; because we trust the models more than our intuitions and generally fail—I think—to do intuitive adjustments on the model (e.g. we probably underestimate intuitively how much independence/dependence assumptions matter).
On the substance of that question, I am not sure I understand your reasoning (but see point above :)). To me it seems that when expenditure is positively correlated with success probability—what seems to be implied by a view where actors are strategic and at least mildly successful at being so—would that not (a) increase the cost effectiveness and (b) reduce the overall uncertainty?
Because we often trust models more than we should, I weakly lean towards having less models—personally, for me the conclusion “this is a really fascinating piece and now we need to think about building models for these different situations and considerations” would be fine whereas with the very rough model I see the risk of incorrect updates, e.g. people not looking into it more because they think that the model fairly represents the uncertainty and under the given range it does not look attractive for some interventions (where the benefit is less large).
But these are just personal philosophical views on modeling, weakly held.
On Baumgartner
Thanks for this, this is really clarifying.
The rephrasing makes this a lot clearer, I had originally read this as “giving more money might be useless” which would undermine the whole case for investigating advocacy charities from an EA perspective, so I am glad I misunderstood that and that this is now clearer.
I would agree that one should not update too much from this correlational evidence. Mostly, because what seems to be the dependent variable here—success probability—is itself confounded by a strategic choice to engage with issues so it is not clear that the same success probability across different levels of resources expresses no difference in strength, rather than being confounded by smaller groups not trying harder things.
On money in politics
(1) On employees as explanation of influence, I do think this is a pretty strong explanation for a couple of reasons.
(i) While employees and employers might not see eye to eye with regards to such as issues as labor policy, they have essentially the same interests with regards to the companies they own/work for—and this is where the lack of money paradox is focused, the lack of money in pork barrel political settings.
(ii) The argument does not rest on explicit voting intentions of employees, but can simply work with references to employment levels in districts, etc., it hands a very powerful argument to local business leaders vis-a-vis their political representatives.
(iii) Campaign finance as well as general influence of business leaders can lead to a situation where the political representatives are already “captured”, where there is no need for additional spending.
(iii) Empirically, it seems well-supported (or so I remember from my political science days, but I cannot find the paper so I might recall that wrongly).
(2) On lobbying equilibria, I am unsure who the relevant experts would be—but I would trust the political science / political economy literature there more than people with very local “inside view” expertise (as it is a dynamic system-level feature, something that seems more accurately to observe with data than based on individual experience).
And just to conclude on this, I think there are many cases where lobbying is probably very good, I just think that the introduction overstates this in not fully considering explanations that make this less surprising and give reasons to think that there can also be many situations where additional money will not lead to additional influence, just higher spending.
Points all well-taken. I’d love to share with FP’s journal club, though I hasten to add that I’m still making edits and modifications based on your feedback, @smclare’s, and others.
With respect to uncertainty in the CE calculation, my thinking was (am I making a dumb mistake here?) that because
Var(XY)=E(X2Y2)−E(XY)2 and Cov(X2Y2)=E(X2Y2)−E(X2)E(Y2) , then Var(XY)=Cov(X2,Y2)+E(X2)E(Y2)−E(XY)2. So if covariance is nonzero, then (I think?) the variance of the product of two correlated random variables should be bigger than in the uncorrelated counterfactual.
To me, the main value of the CE model was in the sensitivity analysis—working through it really helped me think about what “effective lobbying” would have to be able to do, and where the utility would lie in doing so. I think if it doesn’t serve this purpose for the reader, then I agree this document would have been better off without the model altogether.
Thanks for your thoughts on money in politics. Vis (1) I have to think more about this, but I do definitely view the topic a little differently. For instance, it’s not obvious to me that economic arguments and political representation do the necessary work of regulatory capture. Boeing is in Washington and Northrop Grumman is in Virginia. It seems clear that the representatives of the relevant districts are prepared to argue for earmarks that will benefit their constituents… but these companies are still in direct competition, and it seems like there’s still strategic benefit to each in getting the rest of Congress on their side. I might misunderstand- maybe we’re reaching the limits of asynchronous discussion on this topic.
Vis (2), the “inside view” I was talking about was actually yours, as someone who thinks about this professionally- so thank you for your thoughts!
1) I don’t see incompleteness as an issue—what is good for Journal Club is bringing in lots of interesting ideas which your post certainly does, updates you made and are working on are fine. So if that would work for you, I would suggest you as a speaker for Journal Club and we could see when it would fit over the next month or so?
2) My reading of your model—which might be wrong—was that you assumed independence between variables related to cost/effort and variables of success probability. It seems to me that when they are positively correlated rather than independent, cost efficiency would increase and become more narrow, because what this says is that worlds of high spending and success will be more likely to co-occur and worlds of high spending and no success less likely to occur than under independence. Does this make sense?
3) I think on money in politics my understanding is that a couple of intensely motivated politicians—e.g. the representatives where headquarters of companies are—can be quite sufficient for pork barrel style politics because they tend to fill committee positions important for their respective economic interests and they can easily bargain with other legislators.
1) Sounds good to me! We can connect about it over DM.
2) Your reading is right. A priori, a positive correlation means lower cost-effectiveness in expectation. However, I’m not sure if it means anything generally for the median cost-effectiveness (which I tried to work with in my existing CEA), irrespective of the other model parameters. And in my existing setup, if worlds of high spending and high success are more likely co-occur, and worlds with low spending and low success are more likely to co-occur, then I believe the distribution of their product would have been more dispersed, since there would be more values at the extremes (high/high and low/low) then there would be if they were independent. But I’m pretty convinced now that a better approach would have been, as you’ve suggested, to do separate CEAs conditional on various assumed interventions. Rather than change the parameters of independent distributions as I did in the posted analysis, the true next step is probably to re-model under varying assumptions about the covariance of the different variables.
3) I have a different sense of this, but not an overwhelmingly different sense, and I’m going to think about it some more.
It also seems that the report somewhat under-emphasizes the idea of lobbying equilibria where marginal increases by one side would be quickly countered, which would make it look like additional money could be effective when in effect it is not.
Thanks for sharing this. I’d be interested if you’re aware of empirical evidence of this effect/reaction happening? Or is this largely a theoretical concern?
Very impressive and interesting piece, thanks for this! I am a colleague of smclare at Founders Pledge and work a lot on modeling policy advocacy across charities. Would be great to have a chat.
It’s a great and very useful summary of the literature, hugely valuable.
That being said, I am less convinced by the approach taken in the cost-effectiveness model which seems to somewhat contradict the prior analysis which stresses (a) contextuality and (b) strategic choice. As far as I can tell, the modeling tries to be entirely general (contradicting a)) and assumes (b) independence between variables that are very likely related (such as spending and success probability). Please let me know if I completely misunderstand what you are doing!
My sense from doing this kind of work in charity evaluation is that we would want to move towards a “suite” of models for different situations—e.g. (i) pushing genuinely new ideas, (ii) opportunistically exploiting policy windows (such as stimulus season), (iii) pushing high-impact low probability policies, (iv) averting policy rollbacks etc. and that the cost effectiveness for these kinds of things will be very different and essentially unrelated to a generic estimate such as the one featured at the end of your piece.
To me it seems that this kind of classifying into the sub-models will be the only way to realistically bound parameters of interest and also that the dynamics underlying are sufficiently different that they should probably have their own models.
Some minor points/comments: Something I found missing is a bit are standard explanations for why there is so little money in politics, namely (1)because dominant firms in districts have lots of indirect power via employees voting so that they do not need to spend money. (2) It also seems that the report somewhat under-emphasizes the idea of lobbying equilibria where marginal increases by one side would be quickly countered, which would make it look like additional money could be effective when in effect it is not.
I also think that the conclusion which, I believe, mostly draws from Baumgaertner ” (80%) Well-resourced interest groups are no more or less likely to achieve policy success, in general, than their less well-resourced opponents.” is quite surprising and I would be curious to find out why you think that / in how far you trust that conclusion.
Hello and thank you for your response!
Your criticism of the cost-effectiveness model is fair. Thematically, I guess it does contradict the spirit of my prior analysis in that it avoids the concerns of strategic choice. I was actively trying to be as general as possible, and actively trying to err on the side of greater uncertainty by not including any assumptions about correlatedness, though it occurs to me now that making such an assumption (e.g. a correlation between expenditure and likelihood of success) would actually have increased the variance of the final estimate, which would have been more in line with my goals. When I have time, I may comment here with an updated CEA.
I also agree that the only useful way to do this analysis is, as you’ve described, with a suite of models for different scenarios. I don’t have a defense for not having done this beyond my own capacity constraints, though I hope it’s more useful to have included the flawed model than not to have one at all (what do you think?).
Thanks for this, in particular. I think your surprise stems from a lack of clarity on my part. The reason I have high confidence in this conclusion is that it’s a much weaker claim than it might seem. It does stem primarily from Baumgartner et al and from Burstein and Linton (2002). The claim here is that resource-rich groups are no more or less likely to get what they want—holding all else equal, including absolute expenditure and the spending differential between groups and their opponents.
There are three types of claim that are closely related:
1) Groups that spend more relative to their opposition on a given policy are likelier to win
2) Groups that spend more in absolute terms are likelier to win
3) Groups that have more money to spend are likelier to win
So I found fairly consistent evidence for (1), some evidence for (2), and no real evidence for (3). It’s not obvious to me that (3) should be the case irrespective of (1): why would resource-rich groups succeed in lobbying if they deploy those resources poorly? It seems like the success of resource-rich groups is dependent upon (1), and that (3) should not be true when in isolation, unmediated by (1). Although Baumgartner et al conduct an observational study, the size of their (to me, convincingly representative) sample to me suggests that if such an effect exists, it should be observable as a correlation in their analysis. The association they observe is pretty small.
I have to say, though, that in writing this comment, my confidence in this conclusion has eased up a bit, so I’m curious to hear your response. I also think that since Baumgartner et al do find a small effect, I probably overstate the case here.
Baumgartner et al offer a theoretical take on this: ”...organizations rarely lobby alone. Citizen groups, like others, typically participate in policy debates alongside other actors of many types who share the same goals. For every citizen group opposing an action by a given industrial group, for example, there may also be an ally coming from a competing industry with which the group can join forces” (p.12). So it’s important to recognize that the finding here is about individual parties, not “sides” or coalitions advocating a given policy.
Finally, I’m curious to hear your take on the two potential money-in-politics explanations you mentioned. I’ve never found (1) particularly convincing—it’s not clear to me that firms and their employees have the same interests, or that (if they do) the marginal value of regulatory capture isn’t still high. But I agree that I underemphasized (2) and think it would be useful to have in this thread the “inside view” on lobbying equilibria from someone who works in the field.
I just skimmed this thread, so apologies if I missed a comment on this. But Baumgartner et al. don’t argue that money doesn’t matter. They believe that the reason there’s little evidence in their study that money affects outcomes is because “the status quo already reflects the distribution of power in previous rounds of the policy process.” I.e. they don’t see large changes during their 4 year study period, because the situation at the start of their study period tended to favour the resource-rich groups.
After your comments and @jackva’s, I actually struck this conclusion. I was trying to make a more modest statement that upon reflection (thanks to you) is (1) not such a valuable claim and (2) not well-supported enough to have >50% confidence in. It’s true Baumgartner don’t find that money doesn’t matter; my initial (now disavowed) read was that if resources mattered independent of deployment strategy, then we’d expect to see a much stronger correlation even in the observational context. I sort of think that this observation holds true even given the passage you’ve cited, but it’s definitely not a top-level extract from the lit review and definitely needs a considerably more robust defense than I am prepared to muster.
Hi Matt,
Thanks for your reply and sorry for the slight delay!
Would you like to present your work in a session at Founders Pledge (we have a Journal Club where we discuss relevant research and having you as a guest speaker there sounds like a good idea)?
On the cost effectiveness model
On the model generality, I think my view is that even a general and simple model should not be unrealistic / systematically biased; because we trust the models more than our intuitions and generally fail—I think—to do intuitive adjustments on the model (e.g. we probably underestimate intuitively how much independence/dependence assumptions matter).
On the substance of that question, I am not sure I understand your reasoning (but see point above :)). To me it seems that when expenditure is positively correlated with success probability—what seems to be implied by a view where actors are strategic and at least mildly successful at being so—would that not (a) increase the cost effectiveness and (b) reduce the overall uncertainty?
Because we often trust models more than we should, I weakly lean towards having less models—personally, for me the conclusion “this is a really fascinating piece and now we need to think about building models for these different situations and considerations” would be fine whereas with the very rough model I see the risk of incorrect updates, e.g. people not looking into it more because they think that the model fairly represents the uncertainty and under the given range it does not look attractive for some interventions (where the benefit is less large).
But these are just personal philosophical views on modeling, weakly held.
On Baumgartner
Thanks for this, this is really clarifying.
The rephrasing makes this a lot clearer, I had originally read this as “giving more money might be useless” which would undermine the whole case for investigating advocacy charities from an EA perspective, so I am glad I misunderstood that and that this is now clearer.
I would agree that one should not update too much from this correlational evidence. Mostly, because what seems to be the dependent variable here—success probability—is itself confounded by a strategic choice to engage with issues so it is not clear that the same success probability across different levels of resources expresses no difference in strength, rather than being confounded by smaller groups not trying harder things.
On money in politics
(1) On employees as explanation of influence, I do think this is a pretty strong explanation for a couple of reasons.
(i) While employees and employers might not see eye to eye with regards to such as issues as labor policy, they have essentially the same interests with regards to the companies they own/work for—and this is where the lack of money paradox is focused, the lack of money in pork barrel political settings.
(ii) The argument does not rest on explicit voting intentions of employees, but can simply work with references to employment levels in districts, etc., it hands a very powerful argument to local business leaders vis-a-vis their political representatives.
(iii) Campaign finance as well as general influence of business leaders can lead to a situation where the political representatives are already “captured”, where there is no need for additional spending.
(iii) Empirically, it seems well-supported (or so I remember from my political science days, but I cannot find the paper so I might recall that wrongly).
(2) On lobbying equilibria, I am unsure who the relevant experts would be—but I would trust the political science / political economy literature there more than people with very local “inside view” expertise (as it is a dynamic system-level feature, something that seems more accurately to observe with data than based on individual experience).
And just to conclude on this, I think there are many cases where lobbying is probably very good, I just think that the introduction overstates this in not fully considering explanations that make this less surprising and give reasons to think that there can also be many situations where additional money will not lead to additional influence, just higher spending.
Points all well-taken. I’d love to share with FP’s journal club, though I hasten to add that I’m still making edits and modifications based on your feedback, @smclare’s, and others.
With respect to uncertainty in the CE calculation, my thinking was (am I making a dumb mistake here?) that because
Var(XY)=E(X2Y2)−E(XY)2 and Cov(X2Y2)=E(X2Y2)−E(X2)E(Y2) , then Var(XY)=Cov(X2,Y2)+E(X2)E(Y2)−E(XY)2. So if covariance is nonzero, then (I think?) the variance of the product of two correlated random variables should be bigger than in the uncorrelated counterfactual.
To me, the main value of the CE model was in the sensitivity analysis—working through it really helped me think about what “effective lobbying” would have to be able to do, and where the utility would lie in doing so. I think if it doesn’t serve this purpose for the reader, then I agree this document would have been better off without the model altogether.
Thanks for your thoughts on money in politics. Vis (1) I have to think more about this, but I do definitely view the topic a little differently. For instance, it’s not obvious to me that economic arguments and political representation do the necessary work of regulatory capture. Boeing is in Washington and Northrop Grumman is in Virginia. It seems clear that the representatives of the relevant districts are prepared to argue for earmarks that will benefit their constituents… but these companies are still in direct competition, and it seems like there’s still strategic benefit to each in getting the rest of Congress on their side. I might misunderstand- maybe we’re reaching the limits of asynchronous discussion on this topic.
Vis (2), the “inside view” I was talking about was actually yours, as someone who thinks about this professionally- so thank you for your thoughts!
Hi Matt,
1) I don’t see incompleteness as an issue—what is good for Journal Club is bringing in lots of interesting ideas which your post certainly does, updates you made and are working on are fine. So if that would work for you, I would suggest you as a speaker for Journal Club and we could see when it would fit over the next month or so?
2) My reading of your model—which might be wrong—was that you assumed independence between variables related to cost/effort and variables of success probability. It seems to me that when they are positively correlated rather than independent, cost efficiency would increase and become more narrow, because what this says is that worlds of high spending and success will be more likely to co-occur and worlds of high spending and no success less likely to occur than under independence. Does this make sense?
3) I think on money in politics my understanding is that a couple of intensely motivated politicians—e.g. the representatives where headquarters of companies are—can be quite sufficient for pork barrel style politics because they tend to fill committee positions important for their respective economic interests and they can easily bargain with other legislators.
1) Sounds good to me! We can connect about it over DM.
2) Your reading is right. A priori, a positive correlation means lower cost-effectiveness in expectation. However, I’m not sure if it means anything generally for the median cost-effectiveness (which I tried to work with in my existing CEA), irrespective of the other model parameters. And in my existing setup, if worlds of high spending and high success are more likely co-occur, and worlds with low spending and low success are more likely to co-occur, then I believe the distribution of their product would have been more dispersed, since there would be more values at the extremes (high/high and low/low) then there would be if they were independent. But I’m pretty convinced now that a better approach would have been, as you’ve suggested, to do separate CEAs conditional on various assumed interventions. Rather than change the parameters of independent distributions as I did in the posted analysis, the true next step is probably to re-model under varying assumptions about the covariance of the different variables.
3) I have a different sense of this, but not an overwhelmingly different sense, and I’m going to think about it some more.
Thanks for sharing this. I’d be interested if you’re aware of empirical evidence of this effect/reaction happening? Or is this largely a theoretical concern?