On (1), I tend to agree and don’t think a lot of the CEA on health even takes into account the cost of treatment purchases/copays and clinic transport (something I’m sure you have great data on!) which is not insignificant if your annual cash income is <$200 and your kid potentially gets infected multiple times per annum. Some CEA doesn’t even include morbidity. But I don’t think there’s much scope for comparable medium term multiplier effects on a neighbourhood from typical health programs.
I think (2) is an important point. I’ve read studies before (not the Egger one which appears to have been the biggest factor here) which claim the remarkable finding that cash transfers had positive impact on things like domestic violence in neighbouring households that didn’t receive them, and thought to myself have you not considered the possibility that people have noticed the outsiders with clipboards asking personal questions seem to be associated in some way with their neighbours getting unexpected windfalls, and started to speculate about what sort of answers the NGOs are looking for...
It’s an interestingly large upward revision by GiveWell though, especially since they seem to have heavily discounted some of the study results
A quick look at Egger suggests their main novelty was that they considered positive spillover effects on other villages up to 2km away (whereas other studies didn’t or may even have used those neighbouring villages as controls with increases in their income reducing the estimated cash transfer effect size). This seems plausible, though it also seems plausible they’re bundling in other local effects with that. They seem to have plausible data that non-recipients are actually getting higher earned incomes by being paid to do more labour by recipients though, which is the sort of thing these programmes hope to achieve
”have you not considered the possibility that people have noticed the outsiders with clipboards asking personal questions seem to be associated in some way with their neighbours getting unexpected windfalls, and started to speculate about what sort of answers the NGOs are looking for...”
I wrote this piece and wanted to offer my $0.02 on Hawthorne effects driving these consumption spillover results as it’s not covered in the report. I don’t think this is likely to be a key driver of the large spillovers reported, for two reasons:
To measure consumption spillovers, Egger et al. is essentially comparing consumption in nearby non-recipient households (e.g. <2km away) to consumption in further away non-recipient households (e.g. 10km). For this to produce biased results, you’d have to think the nearer non-recipients are gaming their answers in a way that the further away non-recipients aren’t. That seems plausible to me – but it also seems plausible that the further away non-recipients will still be aware of the program (so might have similar, counterbalancing incentives)
Even if you didn’t buy this, I’m not convinced the bias would be in the direction you’re implying. The program studied in Egger et al. was means-tested – cash transfers were only given to households with thatched roofs. If you think nearby non-recipients are more likely to be gaming the system, it seems plausible to me that they’d infer poorer households are more likely to get cash, so it makes sense for them to understate their consumption. This would downward bias the results
Hawthorne effects for recipient consumption gains seem more intuitively concerning to me, and I’ve been wondering whether this could be part of the story behind these large recipient consumption gains at 5-7 years we’ve been sent. We’re not putting much weight on these results at the moment as they’ve not been externally scrutinized, but it’s something I plan to think more about if/when we revisit these.
On (1), I tend to agree and don’t think a lot of the CEA on health even takes into account the cost of treatment purchases/copays and clinic transport (something I’m sure you have great data on!) which is not insignificant if your annual cash income is <$200 and your kid potentially gets infected multiple times per annum. Some CEA doesn’t even include morbidity. But I don’t think there’s much scope for comparable medium term multiplier effects on a neighbourhood from typical health programs.
I think (2) is an important point. I’ve read studies before (not the Egger one which appears to have been the biggest factor here) which claim the remarkable finding that cash transfers had positive impact on things like domestic violence in neighbouring households that didn’t receive them, and thought to myself have you not considered the possibility that people have noticed the outsiders with clipboards asking personal questions seem to be associated in some way with their neighbours getting unexpected windfalls, and started to speculate about what sort of answers the NGOs are looking for...
It’s an interestingly large upward revision by GiveWell though, especially since they seem to have heavily discounted some of the study results
A quick look at Egger suggests their main novelty was that they considered positive spillover effects on other villages up to 2km away (whereas other studies didn’t or may even have used those neighbouring villages as controls with increases in their income reducing the estimated cash transfer effect size). This seems plausible, though it also seems plausible they’re bundling in other local effects with that. They seem to have plausible data that non-recipients are actually getting higher earned incomes by being paid to do more labour by recipients though, which is the sort of thing these programmes hope to achieve
I love the way you put this
”have you not considered the possibility that people have noticed the outsiders with clipboards asking personal questions seem to be associated in some way with their neighbours getting unexpected windfalls, and started to speculate about what sort of answers the NGOs are looking for...”
Hi Nick & David,
I wrote this piece and wanted to offer my $0.02 on Hawthorne effects driving these consumption spillover results as it’s not covered in the report. I don’t think this is likely to be a key driver of the large spillovers reported, for two reasons:
To measure consumption spillovers, Egger et al. is essentially comparing consumption in nearby non-recipient households (e.g. <2km away) to consumption in further away non-recipient households (e.g. 10km). For this to produce biased results, you’d have to think the nearer non-recipients are gaming their answers in a way that the further away non-recipients aren’t. That seems plausible to me – but it also seems plausible that the further away non-recipients will still be aware of the program (so might have similar, counterbalancing incentives)
Even if you didn’t buy this, I’m not convinced the bias would be in the direction you’re implying. The program studied in Egger et al. was means-tested – cash transfers were only given to households with thatched roofs. If you think nearby non-recipients are more likely to be gaming the system, it seems plausible to me that they’d infer poorer households are more likely to get cash, so it makes sense for them to understate their consumption. This would downward bias the results
Hawthorne effects for recipient consumption gains seem more intuitively concerning to me, and I’ve been wondering whether this could be part of the story behind these large recipient consumption gains at 5-7 years we’ve been sent. We’re not putting much weight on these results at the moment as they’ve not been externally scrutinized, but it’s something I plan to think more about if/when we revisit these.