JPHoughton

Karma: 218

JPHoughton 13 Mar 2022 13:15 UTC
1 point
0 ∶ 0
in reply to: Charles Dillon ’s comment on: Concerns about AMF from GiveWell reading—Part 4
Agree a simple calculation as outlined wouldn’t be hard. That would effectively increase the cost-per-life-saved by 20%, say, which is noteworthy but not fundamentally changing things.
The real risk is the longer-term, hard-to-measure impacts which may hold back economic progress generally. These are by definition hard to fit in to a cost-per-life saved calculation but that doesn’t mean the impacts don’t exist. Knowing these risks exist and intervening anyway is a choice some donors will be comfortable with but others will not.

JPHoughton 13 Mar 2022 12:28 UTC
3 points
0 ∶ 0
in reply to: Aaron Gertler’s comment on: Concerns about AMF from GiveWell reading—Part 4
Thanks Aaron for this comment, and the longer one you made elsewhere.
On the excerpt, I think there’s anecdotal evidence for bullet points 1-3 in a few places. GiveWell mention this in an old blogpost about people not buying subsidised nets:
Two different people told us in off-the-record conversations that they thought that this occurred because the mothers offered subsidized bednets believed that they would be able to acquire free nets at some other point. There have been periodic free ITN distributions in many sub-Saharan African countries over the last decade, and the international consensus seems to be that governments should distribute ITNs free of charge in malaria-endemic areas. Accordingly, it should not be especially surprising that citizens may expect bednets to be provided free of charge, and may not move to purchase them even if they are available at subsidized prices in the marketplace. If we reasonably expect to be given something for free in a relatively short time window, why buy it now?
If this happens in a minority of locations, we may just accept that’s the cost of doing business. In order to do a large amount of good overall we accept we’re going to do a small amount of harm. If it happens in a large number of locations, the harm is increased.
Agree with the logic of your last bullet above, though I don’t know the difference in difference in impact between 20% coverage and 80% coverage. It isn’t necessarily linear—could be better or worse than that—and from memory the RCTs here all had universal coverage so no insight from there.
In terms of other stories where distributions may be net negative, I outlined a few in the comments recently. For citizens, workers, business owners and governments there may be wider impacts beyond the market-demand-for-nets idea you’ve described above. How important you think those ideas are probably depends on how you think the economy really works and what might promote or prevent future growth.

JPHoughton 11 Mar 2022 8:00 UTC
2 points
0 ∶ 0
in reply to: Lucas Lewit-Mendes’s comment on: Concerns about AMF from GiveWell reading—Part 4
Thanks Lucas. Agree that these may be reasons for 100% coverage to be a reasonable philanthropic target which would be unachievable through commercial means.
Your first point includes the idea that some people may not purchase ITNs because they expect to be given them for free. This reinforces the idea in the essay that there is an extra cost to such distributions as nobody can make a living from selling nets in areas where people think the price should be zero.

JPHoughton 11 Mar 2022 7:46 UTC
1 point
0 ∶ 0
in reply to: david_reinstein’s comment on: Concerns about AMF from GiveWell reading—Part 4
I don’t have a specific intervention/opportunity in mind for the scenario where health spending is broken.
I’m reminded of a survey of several poor countries which revealed many were not following best practice for treating complications in pregnancy and childbirth despite the treatments being cheap and well-known. Digging into it showed there wasn’t a single reason for this, so no single intervention would change things everywhere.
If the underlying reality is locals make bad choices, as normal individuals and health practitioners and policiticians, I don’t think distributing nets for free is going to make much difference. The moral argument for intervening seems to be a lot weaker too. In that set-up we have replaced Singer’s drowning child with an adult who refuses swimming lessons and ignores all advice to stay out of the water.

JPHoughton 10 Mar 2022 9:08 UTC
3 points
0 ∶ 0
in reply to: david_reinstein’s comment on: Concerns about AMF from GiveWell reading—Part 4
Agree with your outline of first-order impacts. My concern is with wider consequences not included in that view. The world is a complicated place and unintended side effects can be significant and negative. This is even possible under the assumption that such distributions are viewed as a good thing by locals. For example:
For citizens, the lesson may be “Random acts from people far away determine my circumstances, so there’s little I can do to improve my lot”. That would hold back all economic development in a way that outstripped the benefit of anti-malarial bednets.
For workers, the lesson may be “You can lose your job and your whole industry overnight, so don’t specialise”. That’s a bad outcome because specialisation is a huge driver of economic progress.
For business owners, the lesson may be “The risks of starting a new venture are higher than before”. Destroying incentives to start useful businesses is certainly a backwards step.
For governments, the lesson may be “It’s better to be appealing to foreign donors than to sort out our own problems”. Another backward step, because no nation in history became rich waiting for others to help them out.
I don’t think these the the most likely outcomes, just that they are possible. If somebody thought these things were not a risk, they would need to have an extremely high level of confidence in their model of international development and be able to explain why.

JPHoughton 10 Mar 2022 7:10 UTC
1 point
0 ∶ 0
in reply to: david_reinstein’s comment on: Concerns about AMF from GiveWell reading—Part 4
I like “steelmanning”, so thanks for sharing that.
Sin taxes & behavioural nudges seem to support my point rather than work against it. The US banned alcohol and discovered many people kept on drinking anyway, so now limits itself to talking a good game and collecting the extra tax income. Most health professionals are very clear that alcohol is bad, and many claim if it were invented today governments would ban it like so many other drugs. Yet no government I know says “We’ve looked at the scientific evidence, this is a clear example of people making bad choices, so we’re going to force them to make a good choice”.
I know almost nothing about the DRC government. My best guess is that it makes OK spending choices. More than a quarter of government spending is on health, about half as much again on education. (3.3% and 1.5% of GDP, out of a total spend of 11.9%.) That seems reasonable. High level statistics on health & education have been improving over the last 20 years despite there being a civil war for a lot of that time. Turning it around, if we think the DRC government, health administrators, doctors and nurses make bad spending decisions then that seems like a much better opportunity to improve things.

JPHoughton 7 Mar 2022 9:08 UTC
2 points
0 ∶ 0
in reply to: david_reinstein’s comment on: Concerns about AMF from GiveWell reading—Part 4
I agree that humans are generally bad at risk management for low-likelihood-high-impact events. I think this is not limited to black plague era peoples or those living with malaria today. I believe it’s a feature of human brains which scientific knowledge helps mitigate.
Despite this, we generally let people make their own decisions about things which affect their health. People smoke, drink, over-eat, fail to exercise, etc but societies rarely force interventions on their own residents to prevent this. If a benevolent foreign government gave every citizen in my country a free exercise bike worth $300 I’d think they missed an opportunity to do even more good by just giving everybody $300.
Moving from individuals to groups, it isn’t clear to me that the governments of bednet-receiving countries would make these choices either. The main recipient of AMF bednets this year is the DRC, whose govenment spends 12% of GDP each year, about a quarter of which is on healthcare. A bednet per person costs about 1% of GDP (since nets costs around $5.50 each and GDP is around $550 per person). If a govenment with access to scientific knowledge hasn’t spent its first 12% of GDP on this project, how confident are we that’s what it would spend the next 1% on?

JPHoughton 6 Mar 2022 15:41 UTC
2 points
0 ∶ 0
in reply to: Charles He’s comment on: Concerns about AMF from GiveWell reading—Part 4
I agree that in theory you can measure such economic impacts. In practice I don’t believe anybody is.
If a body of practical knowledge on this point exists, then it would be straightforward to quantify the economic downsides of bednet distibutions and include it in the GiveWell calculation. I am confident GiveWell are intellectually honest enough to do such a thing. I believe the reason they haven’t done this is that the information isn’t out there.
When the information isn’t out there, all you can do is make general/theoretical points and share anecdotes like the one I linked to above. There must be a risk of doing harm when we ignore such general points and anecdotes with the only justification being “the evidence must be out there”, even though nobody has pointed to where the evidence actually is.

JPHoughton 15 Jan 2022 9:20 UTC
1 point
0 ∶ 0
in reply to: Matt_Sharp’s comment on: Concerns about AMF from GiveWell reading—Part 3
I think we both agree that bednets give a 17% reduction in mortality. The question is what mortality rate to apply this 17% to.
GiveWell say 11.9.
I say 7.7.
Based on your points I thought you were either saying (a) 7.7, agreeing with me, (b) an adjusted version of 7.7, which I calculate to be 7.66. Either way we’re agreeing here.

JPHoughton 14 Jan 2022 18:28 UTC
1 point
0 ∶ 0
in reply to: Matt_Sharp’s comment on: Concerns about AMF from GiveWell reading—Part 3
I agree for those that get a net it’s a good thing and mortality is reduced, and also agree mortality would increase again if no further nets appeared. However this point isn’t material given the volumes involved. 0.4m children at a rate of 6.4 and 11.6m at a rate of 7.7 is an average rate of 7.66 for the 12m children in total.
Even this small effect is diluted further when you consider the 6.4 rate only applies for 1. 74 years after the 2014-2016 distributions, so much lower when you measure mortality in 2019.
Previous AMF distributions fall into the level of rounding error in this counterfactual, which is why I’m saying an uplift from 7.7 to 11.9 is unreasonable.

JPHoughton 12 Jan 2022 7:55 UTC
3 points
0 ∶ 0
in reply to: Matt_Sharp’s comment on: Concerns about AMF from GiveWell reading—Part 3
Thanks for your comments. Agree with your suggested edit—there’s now two references to the Democratic Republic of Congo. Note that for the 2021 v2 model this is AMF’s total distributions as no other countries were expected to get distributions but regardless it’s worth stating explicitly.
Net distributions cover the whole community, they are not targetted at just under-5s. Using GiveWell’s figures, 16% of the population is under 5. Scaled up 1.8 people per net this suggests coverage for 1.4m * 16% * 1.8 = 0.4m young children. That’s not going to materially change the mortality rate in a country with c.12m under 5s.
Also any protection from 2014-2016 distributions would be massively diluted by the time you measure the mortality rate in 2019, which is when the 7.7 number comes from. GiveWell estimates that DRC bednets effectively last 1.74 years on average.
I can confirm that the 7.7 mortality rate is an unadjusted country-wide mortality rate and 11.9 is the rate GiveWell estimates would occur with no distributions from AMF.

JPHoughton 26 Nov 2021 8:02 UTC
4 points
0 ∶ 0
in reply to: Lucas Lewit-Mendes’s comment on: Concerns about AMF from GiveWell reading—Part 2
Thanks Lucas. You are right that there is a version 3 of this spreadsheet and my post is based on version 2. I originally took a copy of the spreadsheet in late August and v3 appeared in September.
The only difference I can see between the two is the allocation of funding across countries (row 7), with the rest of the values agreeing once this is adjusted. Still, this is a vital difference for my essay, and agree v3 gets the average cost-per-life-saved pretty close to the stated range.
GiveWell don’t justify this change in their changelog and it seems a little finger-in-the-air to me. Some quick research shows AMF’s $108m of recently committed funding almost exclusively went to DRC. That said, it’s not clear where the next $80m will go as AMF do not publicly identify countries involved until an Agreement is signed.
Loreno, thanks again for your efforts here. I agree with what you said, and have tweaked the post to say “2021 v2” for clarity.

JPHoughton 26 Nov 2021 7:43 UTC
1 point
0 ∶ 0
in reply to: Charles He’s comment on: Concerns about AMF from GiveWell reading—Part 2
Charles
No need to apologise for a lack of response. Thanks for sharing your high-level view here even with the limitation that explaining that view is hard. It’s much better for me to have that comment than none at all, and I know the feeling of struggling to explain a view.
For info, I agree with the thrust of what you were saying about real-world constraints on distributions, though agree this is not what the post was actually about.

JPHoughton 26 Nov 2021 7:37 UTC
1 point
0 ∶ 0
in reply to: Lorenzo Buonanno’s comment on: Concerns about AMF from GiveWell reading—Part 2
Lorenzo, thanks for diving in here to help out. I can confirm this is indeed the source I used.
Charles, I didn’t expect anybody to bother checking my source values. I described them as “GiveWell’s 2021 cost effectiveness calculation” which I think satisfies your criteria on context, age & purpose of source. Googling that gives me Lorenzo’s first link above.
Inspired by your comment I’ve now figured out how to add in line links for sources so that will perhaps help address this type of comment in future.

JPHoughton 23 Oct 2021 13:49 UTC
1 point
0 ∶ 0
in reply to: Matt_Sharp’s comment on: Concerns about AMF from GiveWell reading—Part 1
Thanks for the comments Matt. I’ve adjusted and improved the post based on your input.
I was aware of this info and assumed everybody else would be too, so I just took it as read. However, I agree these points are not clear enough in the original post above.
I’ve now changed the heading to add the clarity that it only applies to non-RCT/”real world” distributions. I’ve also inserted a sentence in the final paragraph to make it clear the RCTs do show such evidence and this is the basis for GiveWell’s recommendation.

JPHoughton 23 Oct 2021 13:27 UTC
12 points
0 ∶ 0
in reply to: Charles He’s comment on: Concerns about AMF from GiveWell reading—Part 1
Thanks Charles for your detailed response.
I agree with your central point that it’s very hard to use statistics to prove anything. In particular, you need a huge amount of data and there is lots of noise as the real world is not a clean & tidy place.
For bednets, we do have a huge amount of data. The World Malaria Report 2011, used in GiveWell’s macro review, says 145 million bednets were distrubuted in sub-Saharan Africa in 2010 alone [1]. That’s theoretical coverage for around 30% of the population [2]. This is a massive level of intervention.
For malaria, we also have lots of noise. The same World Malaria Report puts annual deaths in the range 537,000-907,000. That’s a pretty wide confidence interval. The Lancet gives 929,000-1,685,000 deaths per year. That’s a wider range than the first and the two ranges don’t even overlap. [3]
I understand GiveWell’s position (and yours?) to be “There is so much noise, the real world observations don’t really tell you anything. You have to focus on the Randomised Control Trials as proving the concept & monitor AMF to ensure competent delivery”. This might well be right. However, it is then unclear what information could ever be supplied to change GiveWell’s mind. How many bednets would we have to distribute with no evidence of impact before we revisit the recommendation? A billion? 100 billion? Put another way, imagine 10 years from now we find out that bednet distributions had much less impact than we expected. What would be the evidence that demonstrates this, and where might we look now for clues that such evidence is emerging?
More generally, if an intervention can’t stand out from the statistical noise then I’m not sure it passes my personal threshold for a top intervention. As a minimum this means the scale of the problem, and so the scale of our impact, is not well understood. An intervention that can’t stand out from statistical noise has no way of providing feedback to providers on when it is going well or badly, and so has no way to avoid mistakes and no way to improve. Finally, there’s also a psychological element about certainty of impact that will be a big deal to some donors, but that’s a topic for another day.
[1] Source: https://www.who.int/malaria/world_malaria_report_2011/WMR2011_factsheet.pdf
[2] Based on GiveWell’s assumption of 1.8 people covered per net & a population of 869m, as per here: https://www.statista.com/statistics/805605/total-population-sub-saharan-africa/
[3] Source: https://blog.givewell.org/2013/01/23/guest-post-from-david-barry-about-deworming-cost-effectiveness/
What links here?
- Charles He's comment on Concerns about AMF from GiveWell reading—Part 2 by JPHoughton (21 Nov 2021 20:44 UTC; 5 points)

JPHoughton 23 Oct 2021 11:50 UTC
20 points
1 ∶ 0
in reply to: Linch’s comment on: Concerns about AMF from GiveWell reading—Part 1
Thanks Linch, interesting thoughts.
To clarify, my point is not just there’s no direct empirical evidence of AMF’s specific distributions saving lives. My point is that there is no direct evidence of any non-RCT/”real world” distributions saving lives.
Further, this is not because nobody is looking for such evidence. GiveWell’s macro review of the evidence suggests every time somebody has looked for evidence of non-RCT/”real world” distributions saving lives they’ve come up with nothing.
I agree with your summary of the GiveWell argument (strong RCT evidence + AMF as competent distributor). However, in order to turn these two facts into a prediction about future we need to add the assumption that the RCT evidence applies to future distributions. This is the weak link in the chain. As you say, differences in malarial load could distort things. Differences in the underlying health of the population, differences in net usage and increasing insecticide resistance are other contenders, along with many more I’m sure. If we can’t see any evidence of impact after distributing hundreds of millions of bednets then it seems reasonable to question if this key assumption is leading us astray.