New intervention: paying farmers to not burn crops

Karthik Tadepalli22 Dec 2022 17:18 UTC
88 points
6 comments12 min readEA link

  • I summarize a recent paper evaluating an intervention to reduce air pollution: paying farmers not to burn their crop residue.

  • A pure conditional contract is ineffective, but when farmers are paid some fraction upfront, they reduce crop burning significantly.

  • The authors calculate that this contract saves a life for $5,000, comparable to GiveWell’s top charities; however, these averted deaths may be among the elderly and thus not reflect many years of life saved.

  • I also calculate promising climate co-benefits; this intervention reduces GHG emissions for roughly $36 a ton, which costs less than the social cost of carbon, but more than the best climate mitigation strategies.

  • I strongly recommend further research and piloting as a way to build on this single study, especially given the lack of scalable air pollution interventions in EA.

Air pollution causes at least 7 million premature deaths each year. Despite this, it has only recently surfaced as a top cause for effective altruists, with Open Philanthropy announcing South Asian air quality as its newest focus area last year. Even with this recent focus, grants have focused mostly on research rather than on interventions to actually improve air quality. The problem is that we don’t yet have shovel-ready air pollution interventions, demonstrably cost-effective interventions that are feasible and scalable for charities to implement themselves (rather than relying on uncertain advocacy). I think one recent paper offers a cost-effective, feasible and scalable intervention to reduce air pollution, and we should investigate it much more closely.

Overview

One pernicious source of air pollution in developing countries[1] is crop residue burning, where farmers burn the remnants of their summer crop to quickly clear the fields for their winter crop. The pollution from this burning causes 66,000 premature deaths a year in India alone. Both bans on crop burning and subsidizing alternatives have failed, leaving crop burning in desperate need of a solution.

A new working paper by Kelsey Jack, Seema Jayachandran, Namrata Kala and Rohini Pande proposes one solution: paying farmers directly not to burn crop residue, with a partial payment upfront and the remainder conditional on not burning. They show this contract reduces crop burning with an RCT, while purely conditional payments have no effect. Importantly, they also calculate that this approach saves a life for around $5,000, which is competitive with GiveWell’s top charities.

I think this study poses an important line of research for EA organizations to pursue, and I conclude with some ways for us to use this research.

The paper

One of the authors has summarized the paper already.

In brief: We use a randomized trial to evaluate a program that financially rewarded farmers if they avoided burning their rice stubble. We tried out a standard incentive contract that paid the farmer after we verified that he’d complied with the contract terms. That approach had no impact. However, when the contract was tweaked so that some of the payment was made upfront, the financial rewards program became a very cost-effective way to reduce burning, saving a life for <$5000. [author’s emphasis]

I recommend reading her full summary, but here is a self-contained overview. The study took place in Punjab, India, where crop burning is an especially harmful practice. A breathtaking 10 out of the 11 most polluted cities in the world are in northern India, and much of their pollution comes from crop burning from Punjab and other northern states.

The study intervention is a form of payment for ecosystem services (PES) contract. PES contracts have been used and evaluated for other environmental goals (e.g. reducing deforestation by paying landowners and community members for forest land preserved), but are quite new as an intervention to reduce crop burning. In this context, a standard PES contract is made with a farmer at the beginning of the growing season and pays them a certain amount ($12 per acre) at the end if they do not burn their crop residue. In contrast, an upfront PES contract pays some fraction of this (either 25% or 50%) as an explicitly unconditional transfer up front, and the remainder at the end if the farmer does not burn their crop residue. The authors randomized farmers into three experimental groups:[2] a standard PES group, an upfront PES group, and a control group which got no contract.

To measure the impact of these two contracts, the authors train a machine learning model on satellite images of fields to classify whether the field has been burnt. They use this model to detect crop burning in each group. They find that standard PES has no effect, while upfront PES makes a farmer 12 percentage points less likely to burn any field. For comparison, 90% of control farmers burnt a field and 78% of upfront PES farmers burnt a field, so the vast majority of farmers are still engaged in crop burning but the change is meaningful.

The authors explore a few potential explanations for why paying some money upfront makes such a difference. They have no conclusive evidence for any story, but two explanations have some support in their data:

  1. Cash constraints − 70% of farmers said that cash constraints were important in their decisions of how to manage crop residue, so it’s plausible that paying people upfront allowed them to pursue more expensive ways of disposing of crop residue. However, this didn’t differ by treatment group.

  2. Trust—upfront PES farmers were 7 percentage points more likely to trust that they would get paid. However, the control mean was 85% of farmers trusting they would be paid, so this can’t explain that much of the effect.

Finally, some farmers argue that crop burning is necessary to prevent delays to winter cropping. However, the authors detect no significant effect of each contract on either winter yields or winter crop times.[3] This is likely because farmers adopt other ways of removing crop residue; upfront PES farmers were more likely to buy a baler, which is used to remove residue from the field.

Cost-effectiveness

One unique feature of this paper is that it explicitly considers the cost-effectiveness of its proposed intervention, with a simple approach:

  1. The authors use the treatment effect of upfront PES, and the cost of farmer payments, to calculate the cost per reduced acre of burning as being $37 per acre with the upfront PES contracts.[4]

  2. They assume 50% of rice acreage in three Indian states (Punjab, Haryana, Uttar Pradesh) is burned, based on this paper, translating to 8.8 million acres of burning.

  3. They take the Global Burden of Disease estimate that crop burning caused 66,000 premature deaths in India in 2015, and thus each acre of burning averted reduces death by on average.

  4. $37 to reduce deaths by 0.0075 implies that one death is averted for 37⁄0.0075 = $4,933.

Thus, they conclude that they save a life for around $5,000. For comparison, GiveWell calculates that the Against Malaria Foundation saves a life for $3,800 to $8,500 across countries. So this estimate makes PES for crop burning competitive with GiveWell’s top charities.[5]

A very important caveat with these results is that averting premature deaths is not the same as saving lives. Averting the death of an elderly person saves fewer years of life than saving a child. The mortality burden of air pollution is largely experienced by elderly people. One way to solve this issue is with the DALY framework (disability-adjusted life years, which count the years of life lost to both death and disability). This post previously included a cost-effectiveness analysis based on the DALY framework, but I’ve removed it.[6]

Furthermore, two other caveats would increase the cost per life saved:

  1. The authors have two methods of estimating reduced crop burning from satellite data[7] and the estimate of $37 per acre comes from their preferred method. The other method gives a smaller effect and a 1.5x higher cost per acre burned, i.e. $55 per acre of reduced burning. This adjustment increases the cost per life saved to $7,333 using the authors’ specification or $2,260 by my specification.

  2. This cost-effectiveness calculation only uses the cost of payments as an input, rather than any other costs of delivery (e.g. surveyors to verify non-burning, fees associated with writing and enforcing contracts). The authors do not report the costs of delivery which would definitely raise the cost per life saved.[8]

Climate co-benefits

While air pollution is likely the biggest source of harms from crop burning, crop burning also has a clear carbon footprint that the upfront PES contract helps reduce. One study estimates that crop burning in India is responsible for 9.1 million tons of CO2-equivalent GHGs[9] annually. With 8.8 millions of acres burned, this means each acre unburned was responsible for 1.034 tons of GHGs averted. At a cost of $37 per unburned acre, upfront PES contracts offset 1 ton of GHGs for $36. This is a lower cost than all reasonable estimates of the social cost of carbon (the damage to society from 1 additional ton of GHGs). Thus, even if we ignore the air pollution benefits, the social benefits from PES contracts likely exceed their costs.

However, they are not the cheapest way to offset GHGs, and this estimate is still within an order of magnitude of reasonable social cost of carbon estimates, so it’s possible that the pure climate benefits are less than the intervention’s costs. As a result, the climate motivation for PES contracts on crop burning is weaker than the air pollution motivation.

Nonetheless, even with these caveats, the climate impacts are a clear co-benefit; they might not be enough to make PES contracts for crop burning a cost-effective intervention, but they make PES contracts look even better than the already-impressive picture from air pollution benefits.

Warning: wild speculation. These co-benefits could provide an appealing on-ramp for organizations to use the huge pool of carbon offset money to fund PES contracts for crop burning. Carbon offsets have faced serious scrutiny about their impact. If there is a lot of “dumb money” that organizations want to move towards more effective solutions, then PES contracts for crop burning could offer one such solution.

How would this scale?

Programs that work in an RCT need not succeed when scaled up by a government or charity. The policy implications of this study depend on how well its results would scale. The study can’t answer this scaling question, but my best guess is that scaling would increase the efficacy of upfront PES:

  1. Legal enforcement capacity: currently, crop burning is illegal, but this ban is unenforceable because of how widespread the practice is. However, if PES contracts were scaled up and more farmers adopted them, it would become much more feasible to punish farmers who do burn crops. This in turn increases the deterrent effects of the ban, creating a virtuous cycle that reduces crop burning.

  2. Demonstration of feasible alternatives: farmers are aware of the harms of crop burning, but do it because they see no other way to preserve their winter crop. However, the study results suggest that some upfront PES farmers adopted alternative crop residue management methods without reducing winter crop yields. This implies farmers could learn more about these methods from implementing PES at scale, and that learning could reduce their crop burning organically.

  3. Increased trust: the authors have suggestive evidence that an important failure mode for standard PES contracts is that farmers don’t trust they will actually get paid at the end of the contract period. This is a natural concern for a new policy, but if an organization built an established reputation for abiding by its contracts, that could make farmers trust them more and thus be more willing to enter and stick to PES contracts.

However, one countervailing force reduces the likely cost-effectiveness of this policy at scale: the study districts were selected to have more crop burning than average. The study districts were chosen explicitly for their high rates of crop burning and low rates of alternative crop residue management methods. Thus, implementing PES contracts beyond their 300 study villages might have smaller effects if crop burning is less of a problem elsewhere.

Nonetheless, I don’t think this selection issue is critical. Millions of acres of farmland are burned each year, and the farmers in this study collectively own a measly 8,000 acres. I think a PES program would have to be huge before it reached a scale that would seriously diminish the study’s effects.

What do we do with this?

Let’s say you accept this study’s results for now. What can we do with this information?

The first takeaway is that PES is uniquely feasible for nonprofits to implement since it involves no regulation but rather a voluntary contract. Compare this to air pollution interventions that are only implementable by governments, e.g. vehicular efficiency standards—a nonprofit can only make progress by lobbying the government to enact standards, and success in that lobbying is uncertain. Thus, PES contracts are not just an academic proposal but a feasible and shovel-ready intervention that charities could scale up.

Second, we should do more groundwork to test this study’s results. We shouldn’t make important policy or allocate large sums of money based one study. Results can be p-hacked, false positives by random chance, fail to generalize to other places, or suffer from scaling issues. But the enormous cost-effectiveness of this intervention from a shallow investigation makes it a strong candidate for further research. Some very useful research to follow up on this would be:

  1. Pure replication. This would not be in any researcher’s incentives, but any organization that might consider implementing upfront PES as an intervention should do a pure replication of this study in their context. A neat feature of this study is that its measurement strategy uses commercially available satellite data, rather than confidential administrative data of any kind, so it can actually be replicated in any new context.

  2. A replication with more power to detect mechanisms. The study sample is too small to really dig into why upfront PES works so much better than standard PES, or why standard PES fails. A larger study could show more heterogeneous treatment effects: for example, if farmers with more assets were less responsive to upfront PES than farmers with fewer assets, we could conclude that upfront PES helps by alleviating cash constraints for farmers to invest in non-burning approaches. The authors are unable to detect such an effect, but we can’t say if that’s because it doesn’t exist or because of sample size issues.

Moreover, the climate co-benefits of PES contracts to reduce crop burning are a clear source of additional value. They make the case for PES contracts more robust to multiple priorities or value judgments. I also think they can provide a useful platform to access the large pool of carbon offset money: for climate-oriented funders, PES contracts for crop burning could be a generally effective way to reduce emissions while offering fantastic health co-benefits. From a practical perspective, this is quite important to keep in mind for funding sources if PES contracts for crop burning are found to be worth scaling up.

In short, PES contracts are an exciting and shovel-ready intervention to reduce air pollution, an area with few demonstrably cost-effective interventions for charities to implement. At first glance, upfront PES contracts for crop burning beat GiveWell’s top charities in their cost to save a life. I’m sure that upon further scrutiny, they will lose some of that luster, but they merit a lot of attention. We should fund more research into upfront PES, and a real charity should pilot it, so that we can test one of the most promising air pollution interventions yet.

Thanks to Oliver Kim and Tejas Subramaniam for helpful feedback.

  1. ↩︎

    Crop burning is most commonly associated with India, and this paper studies crop burning in India, but it is also a serious problem in southeast Asia, China and Africa.

  2. ↩︎

    The study actually split the groups further. The standard PES arm was split into two contract sizes ($12 per acre and $24 per acre), and the upfront PES arm was split into two upfront shares (25% and 50%). The authors end up pooling these together when reporting results because they had the same effects.

  3. ↩︎

    The point estimates are negative, so a more careful conclusion is that we can rule out negative effects larger than 10% of the control mean yield.

  4. ↩︎

    Since payments to farmers are at most $12 per acre, why is the cost to reduce burning $37 per acre? This is because upfront PES pays many farmers who end up reneging on the contract, so the payments to those farmers have to be built into the cost of reducing burning.

  5. ↩︎

    Of course, cost-effectiveness is not just based on the cost of saving a life. GiveWell assigns different moral weights to averting deaths for under-5 children and people over 5, so a proper comparison in GiveWell’s units would require a full accounting of health benefits based on age. I can’t do that, so I’m settling for showing the cost to save a life as a strong starting point for cost-effectiveness comparisons.

  6. ↩︎

    One study estimates that eliminating crop burning in Punjab, Haryana and Delhi could avert the loss of 149,000 DALYs. This means each unburned acre saves on average DALYs. If it costs $37 to save 0.017 DALYs, then the cost per DALY saved is $2,200 and the cost for a full life saved (70 DALYs) is $153,000. I previously underestimated this by 100x, because the published study estimates the burden as 14.9 million DALYs, which Marshall pointed out was corrected later. This 100x error led me to conclude that a full life could be saved for $1,500 rather than $153,000. This estimate has the advantage of incorporating years of life saved, but it suffers from lack of comparability to other charities. As I investigated further, I discovered that GiveWell used to report cost-effectiveness in terms of DALYs, but stopped because of their lack of confidence in DALY estimates. Thus, I’ve removed it from the main text and left only the caveat that deaths averted may not reflect the moral value of air pollution interventions.

  7. ↩︎

    The authors detect crop burning by training a random forest model on satellite images of fields. One approach is to equalize the false positive and false negative classifications by the model (balanced accuracy) and the other is to minimize false classifications overall (max accuracy). Since burning is more frequent than not burning, the max accuracy model over-classifies fields as being burned, which increases the detected rate of burning in the control group. This reduces the estimated effect of the PES contract (which is the treatment burning minus the control burning) and thus reduces the cost-effectiveness of the PES contract. The authors prefer to use balanced accuracy, which they argue has only a small loss in accuracy.

  8. ↩︎

    One threat to cost-effectiveness that I am not concerned about is the sensitivity to agricultural impacts. The study finds no impact on agricultural yields for winter crops, but it’s possible that this result is not true and PES contracts have negative impacts on yields. However, the health effects of pollution are so large that the economic effects would just be swamped by mortality effects for any plausible value of life. Moreover, the economic effects are bounded by farmers’ revealed preference: it’s hard to believe that farmers would take up a PES contract if it crippled their income.

  9. ↩︎

    Different GHGs have different warming effects, so “CO2-equivalent GHGs” weights each GHG by its warming effect relative to CO2 and sums up the emissions.

Karthik Tadepalli22 Dec 2022 17:18 UTC
88 points
6 comments12 min readEA link