Regrettably, I misspoke in my TED talk when I referred to “curing” blindness from trachoma. I should have said “preventing.” (I used to talk about curing blindness by performing cataract surgery, and that may be the cause of the slip.) But there is a source for the figure I cited, and it is not GiveWell. I give the details in The Most Good You Can Do”, in an endnote on p. 194, but to save you all looking it up, here it is:
“I owe this comparison to Toby Ord, “The moral imperative towards cost-effectiveness,” http://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/moral_imperative.pdf. Ord suggests a figure of $20 for preventing blindness; I have been more conservative. Ord explains his estimate of the cost of providing a guide dog as follows: “Guide Dogs of America estimate $19,000 for the training of the dog. When the cost of training the recipient to use the dog is included, the cost doubles to $38,000. Other guide dog providers give similar estimates, for example Seeing Eye estimates a total of $50,000 per person/dog partnership, while Guiding Eyes for the Blind estimates a total of $40,000.” His figure for the cost of preventing blindness by treating trachoma comes from Joseph Cook et al., “Loss of vision and hearing,” in Dean Jamison et al., eds., Disease Control Priorities in Developing Countries, 2d ed. (Oxford: Oxford University Press, 2006), 954. The figure Cook et al. give is $7.14 per surgery, with a 77 percent cure rate. I thank Brian Doolan of the Fred Hollows Foundation for discussion of his organization’s claim that it can restore sight for $25. GiveWell suggests a figure of $100 for surgeries that prevent one to thirty years of blindness and another one to thirty years of low vision but cautions that the sources of these figures are not clear enough to justify a high level of confidence.”
Now, maybe there is some more recent research casting doubt on this figure, but note that the numbers I use allow that the figure may be $100 (typically, when I speak on this, I give a range, saying that for the cost of training one guide dog, we may be able to prevent somewhere between 400 − 1600 cases of blindness. Probably it isn’t necessary even to do that. The point would be just as strong if it were 400, or even 40.
To begin with, I want to say that my goal is not to put blame on anyone but to change how we speak and act in the future.
His figure for the cost of preventing blindness by treating trachoma comes from Joseph Cook et al., “Loss of vision and hearing,” in Dean Jamison et al., eds., Disease Control Priorities in Developing Countries, 2d ed. (Oxford: Oxford University Press, 2006), 954. The figure Cook et al. give is $7.14 per surgery, with a 77 percent cure rate.
I am looking at this table from the cited source (Loss of Vision and Hearing, DCP2). It’s 77% cure rate for trachoma that sometimes develops into blindness. Not 77% cure rate for blindness. At least that’s how I interpret it, I can’t be sure because the cited source of the figure in the DCP2’s table doesn’t even mention trachoma! From what I’ve read, sometimes recurrences happen so 77% cure rate from trachoma is much much more plausible. I’m afraid Toby Ord made the mistake of implying that curing trachoma = preventing blindness.
What is more, Toby Ord used the same DCP2 report that GiveWell used and GiveWell found major errors in it. To sum up very briefly:
Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43.
[...]
The estimates on deworming are the only DCP2 figures we’ve gotten enough information on to examine in-depth.
Regarding Fred Hollows Foundation, please see GiveWell’s page about them and this blog post. In my eyes these discredit organization’s claim that it restores sight for $25.
In conclusion, without further research we have no basis for the claim that trachoma surgeries can prevent 400, or even 40 cases of blindness for $40,000. We simply don’t know. I wish we did, I want to help those people in the video.
I also want to add that preventing 400 blindness cases for $40,000 (i.e. one case for $100) to me sounds much more effective than top GiveWell’s charities. GiveWell seem to agree, see citations from this page
Based on very rough guesses at major inputs, we estimate that cataract programs may cost $112-$1,250 per severe visual impairment reversed
[...]
Based on prior experience with cost-effectiveness analyses, we expect our estimate of cost per severe visual impairment reversed to increase with further evaluation.
[...]
Our rough estimate of the cost-effectiveness of cataract surgery suggests that it may be competitive with our priority programs; however, we retain a high degree of uncertainty.
We tell the trachoma example and then advertise GiveWell, showing that GiveWell’s top and standout charities are not even related to blindness and no one in EA ever talks about blindness. So people probably assume that GiveWell’s recommended charities are much more effective than surgery that cures blindness for $100 but they are not.
Because GiveWell’s estimates for cataract surgeries are based on guesses, I think we shouldn’t use those figures in introductory EA talks as well. We can tell the disclaimers but the person who hears the example might skip them when retelling the thought experiment (out of desire to sound more convincing). And then the same will happen.
These are good points and I’m suitably chastened for not being sufficiently thorough in checking Toby Ord’s claims, I’m pleased to see that GiveWell is again investigating treating blindness: http://blog.givewell.org/2017/05/11/update-on-our-views-on-cataract-surgery/. In this very recent post, they say:
“We believe there is evidence that cataract surgeries substantially improve vision. Very roughly, we estimate that the cost-effectiveness of cataract surgery is ~$1,000 per severe visual impairment reversed.[1]” The footnote reads:
“This estimate is on the higher end of the range we calculated, because it assumes additional costs due to demand generation activities, or identifying patients who would not otherwise have known about surgery. We use this figure because we expect that GiveWell is more likely to recommend an organization that can demonstrate, through its demand generation activities, that it is causing additional surgeries to happen. The $1,000 figure also reflects our sense that cost-effectiveness in general tends to worsen (become more expensive) as we spend more time building our model of any intervention. Finally, it is a round figure that communicates our uncertainty about this estimate overall.
But it’s reasonable to say that until they complete this investigation, which will be years rather than months, it may be better to avoid using the example of preventing or curing blindness.”
So the options seem to be either not using the example of blindness at all, or using this rough figure of $1000, with suitable disclaimers. It still leads to 40 cases of severe visual impairment reversed v. 1 case of providing a blind person with a guide dog.
The mention of the specific errors found in DCP2 estimates of de-worming efficacy, seem to be functioning here as guilt by association. I can’t see any reason they should be extrapolated to all other calculations in different chapters of a >1000 page document. The figure from DCP2 for trachoma treatment directly references the primary source, so it’s highly unlikely to be vulnerable to any spreadsheet errors.
The table Toby cites and you reference here (Table 50.1 from DCP2) says “trichiasis surgery”. This means surgical treatment for a late stage of trachoma. Trichiasis is not synonymous with trachoma, but a late and severe complication of trachoma infection, by which stage eyelashes are causing corneal friction. It doesn’t ‘sometimes’ lead to blindness, though that is true of trachoma infections when the whole spectrum is considered. Trichiasis frequently causes corneal damage leading to visual impairment and blindness. You are right to point out that not every person with trichiasis will develop blindness, and a “Number Needed to Treat” is needed to correct the estimate from $20 per case of blindness prevented. However we don’t have good epidemiological data to say whether that number is 1, 2, 10 or more. Looking at the literature it’s likely to be closer to 2 than 10. The uncertainty factor encoded in Peter Singer’s use of $100 per person would allow for a number needed to treat of 5.
In this case the term “cure” is appropriate—as trichiasis is the condition being treated by surgery. At one point Toby’s essay talks about curing blindness as well as curing trachoma. Strictly speaking trichiasis surgery is tertiary prevention (treatment of a condition which has already caused damage to prevent further damage.), but the error is not so egregious as to elicit the scorn of the hypothetical doctor you quote below. (Source: I am a medical doctor specialising in infectious diseases, I think the WHO fact sheet you link to is overly simplifying matters when it states “blindness caused by trachoma is irreversible”).
Thank you very much for writing this. Ironically, I did not do enough fact-checking before making public claims. Now I am not even sure I was right to say that everyone should frequently check facts in this manner because it takes a lot of time and it’s easy to make mistakes, especially when it’s not the field of expertise for most of us.
Trichiasis surgery then does seem to be absurdly effective in preventing blindness and pain. I am puzzled why GiveWell hasn’t looked into it more. Well, they explain it here. The same uncertainty about “Number Needed to Treat”.
I want to ask if you don’t mind:
When literature says that surgery costs ~$20-60 or $7.14, is that for both eyes?
Do you think that it’s fair to say that it costs say $100 to prevent trachoma-induced blindness? Or is there too much uncertainty to use such number when introducing EA?
I think it’s laudable to investigate the basis for claims as you’ve done. It’s fair to say evidence appraisal and communication really is a specialist area in its own right, and outside our ares of expertise it’s common to make errors in doing so. And while we all like evidence confirms what we think, other biases may be at play. I think some people in effective altruism also put a high value on identifying and admitting mistakes, so we might also be quick to jump on a contrary assessment even if it has some errors of its own.
I think your broader point about communicating the areas and extent of uncertainty is important, but the solution to how we do that when communicating in different domains is not simple. For example, you can look at how NICE investigates the efficacy of clinical interventions. They have to distill 1000′s of pages of evidence into a decision, and even the ‘summary’ of that can be 100s of pages long. At the front of that will be an ‘executive summary’ which can’t possibly capture all the ares of uncertainty and imperfect evidence, but usually represents their best assessment because ultimately they have to make concrete recommendations.
Another approach is that seen in the Cochrane Systematic Reviews. These take a very careful approach to criticising the methodology of all studies included in their analysis. A running joke though its that every Cochrane review reaches the same conclusion: “More Evidence is Needed”. This is precise and careful, but often lacks any practical conclusion.
My main area of uncertainty on that figure is around number needed to treat. I’ve spoken to a colleague who is an ophthalmologist and has treated trichiasis in Ghana. Her response was “trachoma with trichiasis always causes blindness”. But in the absence of solid epidemiology to back it up, I think it’s wise to allow for NNT being higher than 1. I would be comfortable with saying that for about $100 we can prevent trachoma-induced blindness, in order to contrast that with things that we consider a reasonable buy in other contexts. (I haven’t assessed any orgs to know if there are orgs who do it for that little: they may for instance do surgeries on a wider range of conditions with varying DALYs gained per dollar spent).
It’s pretty much like you said in this comment and I completely agree with you and am putting it here because of how well I think you’ve driven home the point:
...I myself once mocked a co-worker for taking an effort to recycle when the same effort could do so much more impact for people in Africa. That’s wrong in any case, but I was probably wrong in my reasoning too because of numbers.
Also, I’m afraid that some doctor will stand up during an EA presentation and say
You kids pretend to be visionaries, but in reality you don’t have the slightest idea what you are talking about. Firstly, it’s impossible to cure trachoma induced blindness. Secondly [...] You should go back to play in your sandboxes instead of preaching adults how to solve real world problems
Also, I’m afraid that the doctor might be partially right
Also, my experience has persistently been that the blindness vs trachoma example is quite off-putting in an “now this person who might have gotten into EA is going to avoid it” kind of way. So if we want more EAs, this example seems miserably inept at getting people into EA. I myself have stopped using the example in introductory EA talks altogether. I might be an outlier though and will start using it again if provided a good argument that it works well, but I suspect I’m not the only one that has seen better results introducing EAs by not bringing up this example at all. Now with all the uncertainty around it, it would seem that both emotions and numbers argue against the EA community using this example in introductory talks? Save it for the in-depth discussions that happen after an intro instead?
I strongly agree with both of the comments you’ve written in this thread so far, but the last paragraph here seems especially important. Regarding this bit, though:
I might be a bit of an outlier
This factor may push in the opposite way than you’d think, given the context. Specifically, if people who might have gotten into EA in the past ended up avoiding it because they were exposed to this example, then you’d expect the example to be more popular than it would be if everyone who once stood a reasonable chance of becoming an EA (or even a hardcore EA) had stuck around to give you their opinion on whether you should use that example. So, keep doing what you’re doing! I like your approach.
Regrettably, I misspoke in my TED talk when I referred to “curing” blindness from trachoma. I should have said “preventing.” (I used to talk about curing blindness by performing cataract surgery, and that may be the cause of the slip.) But there is a source for the figure I cited, and it is not GiveWell. I give the details in The Most Good You Can Do”, in an endnote on p. 194, but to save you all looking it up, here it is:
“I owe this comparison to Toby Ord, “The moral imperative towards cost-effectiveness,” http://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/moral_imperative.pdf. Ord suggests a figure of $20 for preventing blindness; I have been more conservative. Ord explains his estimate of the cost of providing a guide dog as follows: “Guide Dogs of America estimate $19,000 for the training of the dog. When the cost of training the recipient to use the dog is included, the cost doubles to $38,000. Other guide dog providers give similar estimates, for example Seeing Eye estimates a total of $50,000 per person/dog partnership, while Guiding Eyes for the Blind estimates a total of $40,000.” His figure for the cost of preventing blindness by treating trachoma comes from Joseph Cook et al., “Loss of vision and hearing,” in Dean Jamison et al., eds., Disease Control Priorities in Developing Countries, 2d ed. (Oxford: Oxford University Press, 2006), 954. The figure Cook et al. give is $7.14 per surgery, with a 77 percent cure rate. I thank Brian Doolan of the Fred Hollows Foundation for discussion of his organization’s claim that it can restore sight for $25. GiveWell suggests a figure of $100 for surgeries that prevent one to thirty years of blindness and another one to thirty years of low vision but cautions that the sources of these figures are not clear enough to justify a high level of confidence.”
Now, maybe there is some more recent research casting doubt on this figure, but note that the numbers I use allow that the figure may be $100 (typically, when I speak on this, I give a range, saying that for the cost of training one guide dog, we may be able to prevent somewhere between 400 − 1600 cases of blindness. Probably it isn’t necessary even to do that. The point would be just as strong if it were 400, or even 40.
EDIT: this comment contains some mistakes
To begin with, I want to say that my goal is not to put blame on anyone but to change how we speak and act in the future.
I am looking at this table from the cited source (Loss of Vision and Hearing, DCP2). It’s 77% cure rate for trachoma that sometimes develops into blindness. Not 77% cure rate for blindness. At least that’s how I interpret it, I can’t be sure because the cited source of the figure in the DCP2’s table doesn’t even mention trachoma! From what I’ve read, sometimes recurrences happen so 77% cure rate from trachoma is much much more plausible. I’m afraid Toby Ord made the mistake of implying that curing trachoma = preventing blindness.
What is more, Toby Ord used the same DCP2 report that GiveWell used and GiveWell found major errors in it. To sum up very briefly:
Regarding Fred Hollows Foundation, please see GiveWell’s page about them and this blog post. In my eyes these discredit organization’s claim that it restores sight for $25.
In conclusion, without further research we have no basis for the claim that trachoma surgeries can prevent 400, or even 40 cases of blindness for $40,000. We simply don’t know. I wish we did, I want to help those people in the video.
I think one thing that is happening is that we are too eager to believe any figures we find if they support an opinion we already hold. That severely worsens already existing problem of optimizer’s curse.
I also want to add that preventing 400 blindness cases for $40,000 (i.e. one case for $100) to me sounds much more effective than top GiveWell’s charities. GiveWell seem to agree, see citations from this page
We tell the trachoma example and then advertise GiveWell, showing that GiveWell’s top and standout charities are not even related to blindness and no one in EA ever talks about blindness. So people probably assume that GiveWell’s recommended charities are much more effective than surgery that cures blindness for $100 but they are not.
Because GiveWell’s estimates for cataract surgeries are based on guesses, I think we shouldn’t use those figures in introductory EA talks as well. We can tell the disclaimers but the person who hears the example might skip them when retelling the thought experiment (out of desire to sound more convincing). And then the same will happen.
These are good points and I’m suitably chastened for not being sufficiently thorough in checking Toby Ord’s claims,
I’m pleased to see that GiveWell is again investigating treating blindness: http://blog.givewell.org/2017/05/11/update-on-our-views-on-cataract-surgery/. In this very recent post, they say: “We believe there is evidence that cataract surgeries substantially improve vision. Very roughly, we estimate that the cost-effectiveness of cataract surgery is ~$1,000 per severe visual impairment reversed.[1]”
The footnote reads: “This estimate is on the higher end of the range we calculated, because it assumes additional costs due to demand generation activities, or identifying patients who would not otherwise have known about surgery. We use this figure because we expect that GiveWell is more likely to recommend an organization that can demonstrate, through its demand generation activities, that it is causing additional surgeries to happen. The $1,000 figure also reflects our sense that cost-effectiveness in general tends to worsen (become more expensive) as we spend more time building our model of any intervention. Finally, it is a round figure that communicates our uncertainty about this estimate overall. But it’s reasonable to say that until they complete this investigation, which will be years rather than months, it may be better to avoid using the example of preventing or curing blindness.” So the options seem to be either not using the example of blindness at all, or using this rough figure of $1000, with suitable disclaimers. It still leads to 40 cases of severe visual impairment reversed v. 1 case of providing a blind person with a guide dog.
agree :)
It looks like GiveWell put that project on hold in January 2018: https://www.givewell.org/charities/IDinsight/partnership-with-idinsight/cataract-surgery-project
The mention of the specific errors found in DCP2 estimates of de-worming efficacy, seem to be functioning here as guilt by association. I can’t see any reason they should be extrapolated to all other calculations in different chapters of a >1000 page document. The figure from DCP2 for trachoma treatment directly references the primary source, so it’s highly unlikely to be vulnerable to any spreadsheet errors.
The table Toby cites and you reference here (Table 50.1 from DCP2) says “trichiasis surgery”. This means surgical treatment for a late stage of trachoma. Trichiasis is not synonymous with trachoma, but a late and severe complication of trachoma infection, by which stage eyelashes are causing corneal friction. It doesn’t ‘sometimes’ lead to blindness, though that is true of trachoma infections when the whole spectrum is considered. Trichiasis frequently causes corneal damage leading to visual impairment and blindness. You are right to point out that not every person with trichiasis will develop blindness, and a “Number Needed to Treat” is needed to correct the estimate from $20 per case of blindness prevented. However we don’t have good epidemiological data to say whether that number is 1, 2, 10 or more. Looking at the literature it’s likely to be closer to 2 than 10. The uncertainty factor encoded in Peter Singer’s use of $100 per person would allow for a number needed to treat of 5.
In this case the term “cure” is appropriate—as trichiasis is the condition being treated by surgery. At one point Toby’s essay talks about curing blindness as well as curing trachoma. Strictly speaking trichiasis surgery is tertiary prevention (treatment of a condition which has already caused damage to prevent further damage.), but the error is not so egregious as to elicit the scorn of the hypothetical doctor you quote below. (Source: I am a medical doctor specialising in infectious diseases, I think the WHO fact sheet you link to is overly simplifying matters when it states “blindness caused by trachoma is irreversible”).
[Edited to add DOI: I’m married to Toby Ord]
Thank you very much for writing this. Ironically, I did not do enough fact-checking before making public claims. Now I am not even sure I was right to say that everyone should frequently check facts in this manner because it takes a lot of time and it’s easy to make mistakes, especially when it’s not the field of expertise for most of us.
Trichiasis surgery then does seem to be absurdly effective in preventing blindness and pain. I am puzzled why GiveWell hasn’t looked into it more. Well, they explain it here. The same uncertainty about “Number Needed to Treat”.
I want to ask if you don’t mind:
When literature says that surgery costs ~$20-60 or $7.14, is that for both eyes?
Do you think that it’s fair to say that it costs say $100 to prevent trachoma-induced blindness? Or is there too much uncertainty to use such number when introducing EA?
Thanks for responding!
I think it’s laudable to investigate the basis for claims as you’ve done. It’s fair to say evidence appraisal and communication really is a specialist area in its own right, and outside our ares of expertise it’s common to make errors in doing so. And while we all like evidence confirms what we think, other biases may be at play. I think some people in effective altruism also put a high value on identifying and admitting mistakes, so we might also be quick to jump on a contrary assessment even if it has some errors of its own.
I think your broader point about communicating the areas and extent of uncertainty is important, but the solution to how we do that when communicating in different domains is not simple. For example, you can look at how NICE investigates the efficacy of clinical interventions. They have to distill 1000′s of pages of evidence into a decision, and even the ‘summary’ of that can be 100s of pages long. At the front of that will be an ‘executive summary’ which can’t possibly capture all the ares of uncertainty and imperfect evidence, but usually represents their best assessment because ultimately they have to make concrete recommendations.
Another approach is that seen in the Cochrane Systematic Reviews. These take a very careful approach to criticising the methodology of all studies included in their analysis. A running joke though its that every Cochrane review reaches the same conclusion: “More Evidence is Needed”. This is precise and careful, but often lacks any practical conclusion.
Re your 2 questions:
It’s $7.14 for 1 eye (in 2001) with 77% success, according to this source: https://www.ncbi.nlm.nih.gov/pubmed/11471088 In Toby Ord’s essay he uses this to derive the “less than $20 per person” figure (7.14 *2 /(0.77) = $18.5 ) https://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/moral_imperative.pdf So that’s both eyes (in 2001 terms).
My main area of uncertainty on that figure is around number needed to treat. I’ve spoken to a colleague who is an ophthalmologist and has treated trichiasis in Ghana. Her response was “trachoma with trichiasis always causes blindness”. But in the absence of solid epidemiology to back it up, I think it’s wise to allow for NNT being higher than 1. I would be comfortable with saying that for about $100 we can prevent trachoma-induced blindness, in order to contrast that with things that we consider a reasonable buy in other contexts. (I haven’t assessed any orgs to know if there are orgs who do it for that little: they may for instance do surgeries on a wider range of conditions with varying DALYs gained per dollar spent).
It’s pretty much like you said in this comment and I completely agree with you and am putting it here because of how well I think you’ve driven home the point:
Also, my experience has persistently been that the blindness vs trachoma example is quite off-putting in an “now this person who might have gotten into EA is going to avoid it” kind of way. So if we want more EAs, this example seems miserably inept at getting people into EA. I myself have stopped using the example in introductory EA talks altogether. I might be an outlier though and will start using it again if provided a good argument that it works well, but I suspect I’m not the only one that has seen better results introducing EAs by not bringing up this example at all. Now with all the uncertainty around it, it would seem that both emotions and numbers argue against the EA community using this example in introductory talks? Save it for the in-depth discussions that happen after an intro instead?
I strongly agree with both of the comments you’ve written in this thread so far, but the last paragraph here seems especially important. Regarding this bit, though:
This factor may push in the opposite way than you’d think, given the context. Specifically, if people who might have gotten into EA in the past ended up avoiding it because they were exposed to this example, then you’d expect the example to be more popular than it would be if everyone who once stood a reasonable chance of becoming an EA (or even a hardcore EA) had stuck around to give you their opinion on whether you should use that example. So, keep doing what you’re doing! I like your approach.