The mention of the specific errors found in DCP2 estimates of de-worming efficacy, seem to be functioning here as guilt by association. I can’t see any reason they should be extrapolated to all other calculations in different chapters of a >1000 page document. The figure from DCP2 for trachoma treatment directly references the primary source, so it’s highly unlikely to be vulnerable to any spreadsheet errors.
The table Toby cites and you reference here (Table 50.1 from DCP2) says “trichiasis surgery”. This means surgical treatment for a late stage of trachoma. Trichiasis is not synonymous with trachoma, but a late and severe complication of trachoma infection, by which stage eyelashes are causing corneal friction. It doesn’t ‘sometimes’ lead to blindness, though that is true of trachoma infections when the whole spectrum is considered. Trichiasis frequently causes corneal damage leading to visual impairment and blindness. You are right to point out that not every person with trichiasis will develop blindness, and a “Number Needed to Treat” is needed to correct the estimate from $20 per case of blindness prevented. However we don’t have good epidemiological data to say whether that number is 1, 2, 10 or more. Looking at the literature it’s likely to be closer to 2 than 10. The uncertainty factor encoded in Peter Singer’s use of $100 per person would allow for a number needed to treat of 5.
In this case the term “cure” is appropriate—as trichiasis is the condition being treated by surgery. At one point Toby’s essay talks about curing blindness as well as curing trachoma. Strictly speaking trichiasis surgery is tertiary prevention (treatment of a condition which has already caused damage to prevent further damage.), but the error is not so egregious as to elicit the scorn of the hypothetical doctor you quote below. (Source: I am a medical doctor specialising in infectious diseases, I think the WHO fact sheet you link to is overly simplifying matters when it states “blindness caused by trachoma is irreversible”).
Thank you very much for writing this. Ironically, I did not do enough fact-checking before making public claims. Now I am not even sure I was right to say that everyone should frequently check facts in this manner because it takes a lot of time and it’s easy to make mistakes, especially when it’s not the field of expertise for most of us.
Trichiasis surgery then does seem to be absurdly effective in preventing blindness and pain. I am puzzled why GiveWell hasn’t looked into it more. Well, they explain it here. The same uncertainty about “Number Needed to Treat”.
I want to ask if you don’t mind:
When literature says that surgery costs ~$20-60 or $7.14, is that for both eyes?
Do you think that it’s fair to say that it costs say $100 to prevent trachoma-induced blindness? Or is there too much uncertainty to use such number when introducing EA?
I think it’s laudable to investigate the basis for claims as you’ve done. It’s fair to say evidence appraisal and communication really is a specialist area in its own right, and outside our ares of expertise it’s common to make errors in doing so. And while we all like evidence confirms what we think, other biases may be at play. I think some people in effective altruism also put a high value on identifying and admitting mistakes, so we might also be quick to jump on a contrary assessment even if it has some errors of its own.
I think your broader point about communicating the areas and extent of uncertainty is important, but the solution to how we do that when communicating in different domains is not simple. For example, you can look at how NICE investigates the efficacy of clinical interventions. They have to distill 1000′s of pages of evidence into a decision, and even the ‘summary’ of that can be 100s of pages long. At the front of that will be an ‘executive summary’ which can’t possibly capture all the ares of uncertainty and imperfect evidence, but usually represents their best assessment because ultimately they have to make concrete recommendations.
Another approach is that seen in the Cochrane Systematic Reviews. These take a very careful approach to criticising the methodology of all studies included in their analysis. A running joke though its that every Cochrane review reaches the same conclusion: “More Evidence is Needed”. This is precise and careful, but often lacks any practical conclusion.
My main area of uncertainty on that figure is around number needed to treat. I’ve spoken to a colleague who is an ophthalmologist and has treated trichiasis in Ghana. Her response was “trachoma with trichiasis always causes blindness”. But in the absence of solid epidemiology to back it up, I think it’s wise to allow for NNT being higher than 1. I would be comfortable with saying that for about $100 we can prevent trachoma-induced blindness, in order to contrast that with things that we consider a reasonable buy in other contexts. (I haven’t assessed any orgs to know if there are orgs who do it for that little: they may for instance do surgeries on a wider range of conditions with varying DALYs gained per dollar spent).
The mention of the specific errors found in DCP2 estimates of de-worming efficacy, seem to be functioning here as guilt by association. I can’t see any reason they should be extrapolated to all other calculations in different chapters of a >1000 page document. The figure from DCP2 for trachoma treatment directly references the primary source, so it’s highly unlikely to be vulnerable to any spreadsheet errors.
The table Toby cites and you reference here (Table 50.1 from DCP2) says “trichiasis surgery”. This means surgical treatment for a late stage of trachoma. Trichiasis is not synonymous with trachoma, but a late and severe complication of trachoma infection, by which stage eyelashes are causing corneal friction. It doesn’t ‘sometimes’ lead to blindness, though that is true of trachoma infections when the whole spectrum is considered. Trichiasis frequently causes corneal damage leading to visual impairment and blindness. You are right to point out that not every person with trichiasis will develop blindness, and a “Number Needed to Treat” is needed to correct the estimate from $20 per case of blindness prevented. However we don’t have good epidemiological data to say whether that number is 1, 2, 10 or more. Looking at the literature it’s likely to be closer to 2 than 10. The uncertainty factor encoded in Peter Singer’s use of $100 per person would allow for a number needed to treat of 5.
In this case the term “cure” is appropriate—as trichiasis is the condition being treated by surgery. At one point Toby’s essay talks about curing blindness as well as curing trachoma. Strictly speaking trichiasis surgery is tertiary prevention (treatment of a condition which has already caused damage to prevent further damage.), but the error is not so egregious as to elicit the scorn of the hypothetical doctor you quote below. (Source: I am a medical doctor specialising in infectious diseases, I think the WHO fact sheet you link to is overly simplifying matters when it states “blindness caused by trachoma is irreversible”).
[Edited to add DOI: I’m married to Toby Ord]
Thank you very much for writing this. Ironically, I did not do enough fact-checking before making public claims. Now I am not even sure I was right to say that everyone should frequently check facts in this manner because it takes a lot of time and it’s easy to make mistakes, especially when it’s not the field of expertise for most of us.
Trichiasis surgery then does seem to be absurdly effective in preventing blindness and pain. I am puzzled why GiveWell hasn’t looked into it more. Well, they explain it here. The same uncertainty about “Number Needed to Treat”.
I want to ask if you don’t mind:
When literature says that surgery costs ~$20-60 or $7.14, is that for both eyes?
Do you think that it’s fair to say that it costs say $100 to prevent trachoma-induced blindness? Or is there too much uncertainty to use such number when introducing EA?
Thanks for responding!
I think it’s laudable to investigate the basis for claims as you’ve done. It’s fair to say evidence appraisal and communication really is a specialist area in its own right, and outside our ares of expertise it’s common to make errors in doing so. And while we all like evidence confirms what we think, other biases may be at play. I think some people in effective altruism also put a high value on identifying and admitting mistakes, so we might also be quick to jump on a contrary assessment even if it has some errors of its own.
I think your broader point about communicating the areas and extent of uncertainty is important, but the solution to how we do that when communicating in different domains is not simple. For example, you can look at how NICE investigates the efficacy of clinical interventions. They have to distill 1000′s of pages of evidence into a decision, and even the ‘summary’ of that can be 100s of pages long. At the front of that will be an ‘executive summary’ which can’t possibly capture all the ares of uncertainty and imperfect evidence, but usually represents their best assessment because ultimately they have to make concrete recommendations.
Another approach is that seen in the Cochrane Systematic Reviews. These take a very careful approach to criticising the methodology of all studies included in their analysis. A running joke though its that every Cochrane review reaches the same conclusion: “More Evidence is Needed”. This is precise and careful, but often lacks any practical conclusion.
Re your 2 questions:
It’s $7.14 for 1 eye (in 2001) with 77% success, according to this source: https://www.ncbi.nlm.nih.gov/pubmed/11471088 In Toby Ord’s essay he uses this to derive the “less than $20 per person” figure (7.14 *2 /(0.77) = $18.5 ) https://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/moral_imperative.pdf So that’s both eyes (in 2001 terms).
My main area of uncertainty on that figure is around number needed to treat. I’ve spoken to a colleague who is an ophthalmologist and has treated trichiasis in Ghana. Her response was “trachoma with trichiasis always causes blindness”. But in the absence of solid epidemiology to back it up, I think it’s wise to allow for NNT being higher than 1. I would be comfortable with saying that for about $100 we can prevent trachoma-induced blindness, in order to contrast that with things that we consider a reasonable buy in other contexts. (I haven’t assessed any orgs to know if there are orgs who do it for that little: they may for instance do surgeries on a wider range of conditions with varying DALYs gained per dollar spent).