Here’s a thread in which a World Bank economist critiques GiveWell on research/publication methods. (GiveWell responds here.)
cole_haus
Karma: 995
I just feel like it’s hard to come away with much of long-term value. I sort of nod along as I read thinking, “That’s plausible,” and that’s about it. (To be concrete: I make Anki cards for most nonfiction I read and I’ve only made around 1o or 12 across 200 pages which is way fewer than normal for me.) I think I generally want my non-fiction to have at least one of:
Solid empirical findings (i.e. widely and repeatedly attested within the field)
Falsifiable models with some explanatory depth (i.e. not just mindless curve fitting or a listing of all possible causal factors)
Insightful conceptual analysis (e.g. mutually exclusive and collectively exhaustive taxonomies)
Regarding 1, several empirical studies are mentioned but they don’t seem to add up to a coherent or even non-contradictory whole.
There’s basically none of 2.
The book is probably closest to achieving number 3, but still not great. I would have liked, for example, if they talked about why the classic agenda of “collective action frames”, “mobilizing structures”, and “political opportunities” is a better organizational scheme than the alternatives.
The book also focuses more on apportioning credit and on the history of the thinking in the field than I’d prefer.
All that said, I understand different readers are looking for different things.
I remain pretty confused by this line of argument. I basically parse it as: we should strive to make the actions of developing countries similar to the (best) actions of developed countries. But actions seem of merely instrumental interest and what we actually care about is states (conditions) that are conducive to development.
The recommendations from these two perspectives (actions vs states) converge only insofar as the best actions are invariant across states. But this is quite a big claim and contradicted by e.g. Rodrik who insists that “Institutional innovations do not travel well”.
It seems like the development interventions we commonly see can be readily justified by the state-based view. For example, no, we didn’t see widespread deployment of insecticidal nets in the US, but, yes, we did see deliberate effort to achieve and good returns from achieving a low burden of infectious disease in the US. No, we didn’t have women’s self-help groups, but, yes, we did achieve a state of increased gender equality and of increased integration of women into the formal economy.
TL;DR: Why would we expect the same actions to produce the same end state given different starting states?
Another book in this area is Handbook of Social Movements Across Disciplines. Unfortunately, I’m most of the way through and it’s a bit underwhelming.
Here’s a half-baked argument for natalism vis-à-vis climate change:
Carbon emissions in the highly developed countries most EAs live in are generally trending in the right direction (i.e. there seems to be at least relative decoupling between emissions and consumption). The bulk of emissions growth over the next several decades will be in other large, rapidly developing countries like India and China. Green technology transfer is a way that highly developed countries can positively influence emissions in the critical rapidly developing countries (see e.g. this). Economic models generally propose that a larger population generates more ideas and a higher rate of technological change (e.g. Population Growth and Technological Change: One Million B.C. to 1990). Therefore, the (smallish?) direct impact of increased emissions from greater population in highly developed countries might be outweighed by more green technology and technology transfer to the crucial rapidly developing countries like China and India.
Thanks for writing this up!
For those interested in more info:
-
Harsanyi had two different theorems like this (his aggregation theorem and his impartial observer theorem) which rely on slightly different assumptions.
-
The main arguments against Harsanyi’s theorems were made by prominent economist Amartya Sen in what has become known as the “Harsanyi-Sen debate” or “Harsanyi-Sen-Weymark debate” (searchable terms). The gist of the counterargument is that “while Harsanyi has perhaps shown that overall good is a linear sum of individuals’ von Neumann–Morgenstern utilities, he has done nothing to establish any connection between the notion of von Neumann–Morgenstern utility and that of well-being, and hence that utilitarianism does not follow.”.
-
Chloramphenicol is an approved drug, but not approved for this purpose. Approving Chloramphenicol as a coronary treatment requires human trials that will probably cost $25 million.
I am extremely far from an expert here so there may be some subtlety, but off-label uses are generally possible. From Wikipedia:
However, once a drug has been approved for sale for one purpose, physicians are free to prescribe it for any other purpose that in their professional judgment is both safe and effective, and are not limited to official, FDA-approved indications. This off-label prescribing is most commonly done with older, generic medications that have found new uses but have not had the formal (and often costly) applications and studies required by the FDA to formally approve the drug for these new indications.
Edit: The full post at the link acknowledges this:
As an approved drug (though for another purpose) any doctor can prescribe Chloramphenicol for any purpose. Of course, they don’t know to do this. And — perhaps more importantly — such bold action can get American doctors sued for malpractice.
I just finished reading Democracy for Realists recently which argues that:
They demonstrate that voters—even those who are well informed and politically engaged—mostly choose parties and candidates on the basis of social identities and partisan loyalties, not political issues. They also show that voters adjust their policy views and even their perceptions of basic matters of fact to match those loyalties. When parties are roughly evenly matched, elections often turn on irrelevant or misleading considerations such as economic spurts or downturns beyond the incumbents’ control; the outcomes are essentially random. [...]
Achen and Bartels argue that democratic theory needs to be founded on identity groups and political parties, not on the preferences of individual voters.
I’m not fully settled on how much weight to give to this perspective, but I think it’s important to remember the empirical facts of voting as it happens in the real world and not just the idealizations of social choice theory. Presumably this leads to a quite different notion of the optimal electoral system and the optimal series of electoral reforms.
(This isn’t meant to say that the social choice theory perspective and the points brought up in this post are unimportant. I just thought it was an interesting book and a good reminder to look at this whole other set of criteria.)
A few of the summary points in Safe Drinking Water for Low-Income Regions are interesting and may provide reason for a bit of pessimism:
Safe drinking water from “source to sip” consists of a series of interactions between technologies, their delivery models, their scales and costs of production, and consumer uptake and consistent use. Safe drinking water is a system, not a product or an intervention.
It seems unlikely that household treatment and safe storage systems—with the possible exception of boiling—can be transformative at scale under current prices, delivery models, and preferences, but they are effective and protective in specific contexts.
Cost analyses for “low-cost” systems are usually reported on a partial basis, with installation costs and some operational costs included. The enabling costs of social marketing, mobilization, education, reminders, and community- or household-based unpaid labor are mentioned but not explicitly accounted for.
Delivery models and business models significantly affect costs and uptake, at all three scales of service. Yet they are rarely made explicit.
Safe water systems can be highly effective, but consumers undervalue drinking water quality and have low willingness and/or ability to pay for safety. This is a particular challenge for arsenic mitigation or avoidance, as arsenicosis is only evident after several years of exposure.
Without an in-depth of understanding of either deworming or WASH, it seems like the implementation issues for WASH may be more severe?
This doesn’t directly engage with the point of this post, but Safe Drinking Water for Low-Income Regions is a pretty good introduction to WASH IMO:
Well into the 21st century, safe and affordable drinking water remains an unmet human need. At least 1.8 billion people are potentially exposed to microbial contamination, and close to 140 million people are potentially exposed to unsafe levels of arsenic. Many new technologies, water quality assessments, health impact assessments, cost studies, and user preference studies have emerged in the past 20 years to further the laudable goal of safe drinking water for all. This article reviews (a) the current literature on safe water approaches with respect to their effectiveness in improving water quality and protectiveness in improving human health, (b) new work on the uptake and use of safe water systems among low-income consumers, (c) new research on the cash and labor costs of safe water systems, and (d) research on user preferences and valuations for safe water. Our main recommendation is that safe water from “source to sip” should be seen as a system; this entire system, rather than a discrete intervention, should be the object of analysis for technical, economic, and health assessments.
(Providing Safe Water: Evidence from Randomized Evaluations too.)
This isn’t looking at it from exactly the same angle as this post, but Incomplete Contracting and AI Alignment also looks at the alignment problem through the principal-agent lens:
We suggest that the analysis of incomplete contracting developed by law and economics researchers can provide a useful framework for understanding the AI alignment problem and help to generate a systematic approach to finding solutions. We first provide an overview of the incomplete contracting literature and explore parallels between this work and the problem of AI alignment. As we emphasize, misalignment between principal and agent is a core focus of economic analysis. We highlight some technical results from the economics literature on incomplete contracts that may provide insights for AI alignment researchers. Our core contribution, however, is to bring to bear an insight that economists have been urged to absorb from legal scholars and other behavioral scientists: the fact that human contracting is supported by substantial amounts of external structure, such as generally available institutions (culture, law) that can supply implied terms to fill the gaps in incomplete contracts. We propose a research agenda for AI alignment work that focuses on the problem of how to build AI that can replicate the human cognitive processes that connect individual incomplete contracts with this supporting external structure.
I didn’t see it among your links, but GiveWell has an interim intervention report on this. Their summary is:
What is its evidence of effectiveness? Results from multiple randomized controlled trials (RCTs) suggest that distributions of clean cookstoves do not have clear evidence of effectiveness at reducing health problems attributable to air pollution. The evidence we have reviewed in our preliminary investigation finds limited impacts on women’s health and no clear impacts on children’s health under typical use. Distributions of clean cookstoves may have been less effective than expected due to implementation challenges, such as low compliance with using the replacement stoves and failure of the cleaner stoves to reduce air pollution sufficiently.
How cost-effective is it? We have not produced a cost-effectiveness model for clean cookstoves because we have not yet seen strong enough evidence to model a health benefit of the intervention.
I’m also pretty uninformed on the biomarkers aspect. Beyond what they say in the paper:
One reason why we do not find significant effects on biomarkers at conventional levels may be power issues combined with relatively noisy measures. Another, related reason may be the composition of our sample: high levels of pro-inflammatory cytokines have been found for major depression; respondents in our sample, however, report, on average, only mild depressive symptomatology, pre-treatment. In fact, we find that only eight out of 133 respondents (about 6%) report strong depressive symptomatology, as indicated by PHQ-9 scores of fifteen or higher. Moreover, even amongst these, only about a third show associated elevated inflammation (Wium-Andersen and Nielsen, 2013). For cortisol, individual differences and timing of measurement matter; it has been found to be a rather short-term measure for stress (Miller et al., 2007).
I found Table 5 here gives a bit more context on the correlation between these biomarkers and different outcomes.
trained facilitator
This is how they describe their facilitators:
The course is manualised and scalable: each course is led by two volunteers – screened by Action for Happiness for motivation and skills, and once approved, provided with structured resources – as facilitators on an unpaid basis in their local communities. Recruitment of course leaders follows a carefully documented, standardised process: each candidate completes a Leader Registration process sharing their motivation and skills and is given clear instructions on what is required. Once potential course leaders have a co-leader, venue, and dates in mind, they complete a Course Application process. The team at Action for Happiness reviews this application and, if all criteria are met, arranges a call to discuss next steps. Once a course is fully approved, course leaders receive on-going guidance and support. There is also a post- course follow-up process.
Not sure if that’s what you had understood and meant with ‘trained facilitator’ (just wanted to make it clear that it doesn’t mean licensed behavioral therapist or something).
As far as comparisons, they say:
Impacts on subjective wellbeing, mental health, and pro-sociality are large: the course increases life satisfaction on a zero-to-ten scale by about one point, more than being partnered as opposed to being single (+0.6) or being employed as opposed to being unemployed (+0.7) (Clark et al., 2018). It is more than double the effect of ENHANCE, a 12-week course focusing primarily on positive habits, skills, and attitudes, which is probably the most comparable intervention (Kushlev et al., 2017). 28 However, the authors are able to track outcomes over a longer period of time, up to six months post-treatment. Finally, the effect on life satisfaction is somewhat larger than effects found in trials by the UK Big Lottery Fund, which funded a wide range of wellbeing programmes (fourteen portfolios, each consisting of three to 34 actual trials) from 2008 to 2015 at a volume of £200 million. Trials typically included community-based activities such as horticultural activities, cooking lessons, or sports events. As a conservative estimate, they increased life satisfaction on a zero-to-ten scale by, on average, 0.5 points for six months post-treatment (New Economics Foundation-Centre for Local Economic Strategies, 2013). Different from our intervention, however, these trials all targeted specific groups with mental health needs, including overweight adults, families with young children, or people with substance use disorders.
Thanks for your thoughts!
Yes, regarding persistence they also note:
To look at treatment effect persistence, we exploit data points at follow-up in an extended sample. As all respondents have been treated at follow-up, we cannot estimate causal effects, so that results are exploratory.
A Local Community Course That Raises Mental Wellbeing and Pro-Sociality
Also just came across this claim in this paper:
In the nationally representative UK Household Longitudinal Survey (“Understanding Society”), for example, average life satisfaction, measured on a scale from one to seven whereby higher values denote higher wellbeing, was not significantly higher in 2016 than in 1996 (5.3 vs. 5.2), despite large rises in real incomes.
Couldn’t quickly chase down source data up through 2016--best I could find was this through 2008.
How Poverty Ends: The Many Paths to Progress—and Why They Might Not Continue is from Abhijit V. Banerjee and Esther Duflo (the recent econ Nobel Laureates who won for their RCT work) and I think can reasonably be read as a response to the criticisms of Lant Pritchett (probably the most vocal advocate of the line of thinking this post represents).
Key excerpts:
Economists, ourselves included, have spent entire careers studying development and poverty, and the uncomfortable truth is that the field still doesn’t have a good sense of why some economies expand and others don’t. There is no clear formula for growth.
¶
What, then, are policymakers left with? There are some things clearly worth avoiding: hyperinflation; extremely overvalued fixed exchange rates; communism in its Soviet, Maoist, or North Korean varieties; the kind of total government chokehold on private enterprise that India had in the 1970s, with state ownership of everything from shipyards to shoe factories. But this is not particularly helpful advice today, given that hardly anyone is reaching for such extreme options anymore.
¶
There simply is no accepted recipe for how to make poor countries achieve permanently high growth. Even the experts seem to have accepted this. In 2006, the World Bank asked the economist Michael Spence to lead a commission on economic growth. In its final report, the group recognized that there are no general principles for growth and that no two instances of economic expansion are quite alike. Easterly described their efforts in less charitable terms: “After two years of work by the commission of 21 world leaders and experts, an 11-member working group, 300 academic experts, 12 workshops, 13 consultations, and a budget of $4m, the experts’ answer to the question of how to attain high growth was roughly: we do not know, but trust experts to figure it out.”
¶
If there is a common thread, it is that the fastest growth appears to come from reallocating poorly allocated resources—that is, putting capital and labor toward their most productive use. But eventually, the returns from that process diminish, at which point countries need to find a new strategy for combating poverty.
¶
One very real danger is that in trying to hold on to fast growth, countries facing sharply slowing growth will veer toward policies that hurt the poor now in the name of future growth. In a bid to preserve growth, many countries have interpreted the prescription to be business friendly as a license to enact all kinds of anti-poor, pro-rich policies, such as tax cuts for the rich and bailouts for corporations.
¶
Looking back, it is clear that many of the important successes of the last few decades were the result not of economic growth but of a direct focus on improving particular outcomes, even in countries that were and have remained very poor. The under-five mortality rate, for example, has fallen drastically across the world, even in some very poor countries whose economies have not grown particularly fast. Credit goes mostly to policymakers’ focus on newborn care, vaccination, and malaria prevention.
¶
This is patient work: spending money by itself does not necessarily deliver real education or good health. But unlike with growth, experts actually know how to make progress. One big advantage of focusing on clearly defined interventions is that these policies have measurable objectives and therefore can be directly evaluated. Researchers can experiment with them, abandon the ones that don’t work, and improve the ones that do.
In addition to Will MacAskill’s critique of functional decision theory (MIRI-originated and intended to be relevant for AI alignment), there’s this write-up by someone that refereed FDT’s submission to a philosophy journal: