[Epistemic status: Writing quickly about a complicated literature—I think the direction of this critique is right but would love to discuss further in the comments!]
I know authors don’t choose their own headlines, but this one really is a ridiculous overstatement. We can’t crown something the “best” way to help extremely poor people after only comparing two interventions (graduation vs. cash). And even in a head-to-head comparison, the available evidence suggests that the characteristics and effects of graduation-style programs vary so much across contexts that we should be extremely cautious about generalizing from a study of one particular program in one particular place.
At the risk of further beating an already dead horse, I also think it’s worth re-contextualizing this article in the context of the growth vs. randomista debate (which, IMO, has not advanced much at all since Hauke and John’s post in Jan. 2020. Here again is Pritchett’s damning figure comparing the gains from a representative $1,000,000,000 investment in Graduation-type programs to the value of various national growth accelerations:
Overall I feel pretty disappointed at how un-quantitative the linked Vox article is. Sigal summarizes the results of the latest BRAC study as “study subjects enjoying higher income and consumption even a full decade later.” But this is just saying the program has AN effect. This is trivial. What matters is the size of the effect. Unfortunately, Sci-Hub hasn’t indexed the 2021 study the Vox article focuses on. But I’ve looked at the results of previous graduation studies and haven’t been blown away. Banerjee et al. (2016) report of BRAC that “seven years after the asset were first distributed, the monthly consumption of those assigned to treatment is 16 dollars–or 25%–higher [than those in the control group]”.
Of course, 16 extra dollars per month works out to $0.50 per day. I believe in diminishing returns to consumption such that an extra 50 cents per day is very meaningful if it boosts your consumption by 25%. But any claims that Graduation helps people “graduate from” or “escape” poverty assumes a low poverty line, such as the standard $1.25 per day. That is, the average very poor Graduation program participant remains very poor after participating in the program. I think it’s misleading to talk about them escaping poverty as Vox and Graduation proponents do: e.g. from the article “[BRAC] aims to “graduate” recipients out of extreme poverty.”
I’m impressed by the thoroughness of Sigal’s literature review here but still think it understates the extent to which these programs are controversial. In fact, I basically think we shouldn’t generalize at all from studies of one particular Graduation program. Some of these issues are discussed in what I think is a pretty good critical article, Kidd & Athias 2019, which I don’t see discussed in the Vox piece.
I agree with your disappointment about the lack of any quantitative aspect, and I’m adding the paper you linked to my reading list.
I’ve also been planning on reading selected books and papers from Further reading/References in the Growth and the case against randomista development for a while, but if you have other recommendations, I’d love to hear them.
Thanks for the link. I want to emphasize that I think this is a very good paper. The intro especially is well worth reading for its description of the program and poverty trap model.
Here’s a relevant quote; the results aren’t much of an update as the absolute treatment effect in terms of per-capita consumption didn’t change between years 7 and 10.
Average household per capita consumption was $1.35 (2018 PPP) at baseline in the control group and $2.90 by year ten. [...] [The treatment group’s] per capita consumption is $0.60 per day (0.6 standard deviations) higher than the control group at both years seven and ten, and income is 0.3 standard deviations higher. This temporal pattern of growing effects followed by persistence is consistent with the alleviation of a poverty trap, what BRAC describes as the graduation of treated households. However, it is also consistent with there being persistent effects without actually getting out of a trap: the control households do become less poor over time, and the treatment households are still not very rich by the time the treatment effect stabilizes (although their average consumption is above the World Bank threshold for “moderate poverty”).
(p. 472, emphasis mine)
Note that the authors’ wording is more cautious and nuanced than the Vox article.
[Epistemic status: Writing quickly about a complicated literature—I think the direction of this critique is right but would love to discuss further in the comments!]
I know authors don’t choose their own headlines, but this one really is a ridiculous overstatement. We can’t crown something the “best” way to help extremely poor people after only comparing two interventions (graduation vs. cash). And even in a head-to-head comparison, the available evidence suggests that the characteristics and effects of graduation-style programs vary so much across contexts that we should be extremely cautious about generalizing from a study of one particular program in one particular place.
At the risk of further beating an already dead horse, I also think it’s worth re-contextualizing this article in the context of the growth vs. randomista debate (which, IMO, has not advanced much at all since Hauke and John’s post in Jan. 2020. Here again is Pritchett’s damning figure comparing the gains from a representative $1,000,000,000 investment in Graduation-type programs to the value of various national growth accelerations:
Overall I feel pretty disappointed at how un-quantitative the linked Vox article is. Sigal summarizes the results of the latest BRAC study as “study subjects enjoying higher income and consumption even a full decade later.” But this is just saying the program has AN effect. This is trivial. What matters is the size of the effect. Unfortunately, Sci-Hub hasn’t indexed the 2021 study the Vox article focuses on. But I’ve looked at the results of previous graduation studies and haven’t been blown away. Banerjee et al. (2016) report of BRAC that “seven years after the asset were first distributed, the monthly consumption of those assigned to treatment is 16 dollars–or 25%–higher [than those in the control group]”.
Of course, 16 extra dollars per month works out to $0.50 per day. I believe in diminishing returns to consumption such that an extra 50 cents per day is very meaningful if it boosts your consumption by 25%. But any claims that Graduation helps people “graduate from” or “escape” poverty assumes a low poverty line, such as the standard $1.25 per day. That is, the average very poor Graduation program participant remains very poor after participating in the program. I think it’s misleading to talk about them escaping poverty as Vox and Graduation proponents do: e.g. from the article “[BRAC] aims to “graduate” recipients out of extreme poverty.”
I’m impressed by the thoroughness of Sigal’s literature review here but still think it understates the extent to which these programs are controversial. In fact, I basically think we shouldn’t generalize at all from studies of one particular Graduation program. Some of these issues are discussed in what I think is a pretty good critical article, Kidd & Athias 2019, which I don’t see discussed in the Vox piece.
I really appreciate this comment, thank you!
I agree with your disappointment about the lack of any quantitative aspect, and I’m adding the paper you linked to my reading list.
I’ve also been planning on reading selected books and papers from Further reading/References in the Growth and the case against randomista development for a while, but if you have other recommendations, I’d love to hear them.
Here’s a link to the Banerjee paper for those without institutional access.
Thanks for the link. I want to emphasize that I think this is a very good paper. The intro especially is well worth reading for its description of the program and poverty trap model.
Here’s a relevant quote; the results aren’t much of an update as the absolute treatment effect in terms of per-capita consumption didn’t change between years 7 and 10.
(p. 472, emphasis mine)
Note that the authors’ wording is more cautious and nuanced than the Vox article.