I’m extremely skeptical of the finding that reading glasses dramatically increase income. After looking into this topic for the past year (having initially been excited by these studies), I would now guess that the significant findings are more a result of experimenter demand effects than of reading glasses.
For example, in the paper on tea pickers you mention, the research team made seven unannounced visits to assess “compliance with study glasses” over the course of the 11-week trial. But we know from past research (e.g. Zwane et al. 2011) that monitoring changes behavior (as evidenced by the uptick in glasses usage). In that case, the estimate from the trial will capture the effect of monitoring + reading glasses which is not an effect we’d ever observe in the real world. The productivity of the control group also increases by 20% between baseline and endline but I don’t believe the authors provide any potential explanations for this substantial increase.
Perhaps more tellingly, I’ve now visited tea growing regions in rural Kenya multiple times this past year. I’ve observed hospitals giving reading glasses to tea pluckers and I’ve asked people if they would use them when plucking tea and they all say no. Or if I go to an area where the hospital has distributed glasses in the past, I have never found workers wearing them in the field. That’s not to say they don’t appreciate the glasses. Many people love them and report using them daily, but they’re using the glasses to read the bible or look at their phone! Yes, this evidence is anecdotal, but it’s hard to reconcile the study findings with my observations.
Furthermore, the idea that someone in rural Kenya would have to travel to Nairobi to get a pair of reading glasses is not accurate in my experience (unless they live really close to Nairobi). The hospitals I went to in rural Kenya offered reading glasses and there are also multiple optometrists in larger towns. Yes, people won’t be able to buy reading glasses at the grocery store or at CVS like here in the US, but glasses aren’t crazily inaccessible.
On this point, if reading glasses really did generate an extra $140 in earnings a year, why isn’t everyone clamoring to get a pair? There would be huge incentives for businesses to expand and offer the product in rural areas at higher prices. And workers should expend substantial effort to obtain a pair. In practice, we don’t see this happening. In my view, it is more likely that THRIVE overestimates the return to reading glasses than that hundreds of millions of workers worldwide substantially underestimate the benefit.
Additionally, the 33.4% statistic you cite from THRIVE isn’t even statistically significant! The authors report the coefficient from a median regression, but the standard error for this estimate is not mentioned.
At the moment, I’m working on a formal comment aggregating these considerations + many others and I aim to wrap that up shortly.
Also a longer response: I do think the lack of demand is worrying and could be suggestive that these studies are not showing real world effects. I haven’t spent enough time in rural Kenya to know how hard it is to get glasses, but I am updating based on what you say!
I do think it is easy to underestimate how bad your vision has gotten and not use glasses you need. Personally, I have failed to notice that my prescription has gotten out of date and continued to use old glasses, and then finally get around to getting new ones and I do notice a productivity improvement. Since your brain can somewhat compensate, I think it’s easy to underestimate the returns to a correct Rx.
I am very excited to see your longer comment when it’s done—make sure to sent it to both OP and Erin Crossett at GiveWell as well; it sounds like we will all benefit from it!
I share a lot of Drew’s skepticism about the study, especially the experimenter demand effects. If monitoring alone is enough to increase productivity, I think it’s quite plausible that there is some further response (beyond a direct effect of the glasses on vision) to monitoring plus the provision of glasses. Even as a large proponent of quantile regression in many applications, I do think OLS is more appropriate for a cost effectiveness analysis. A median shift could be consistent both with a much larger or much smaller (even negative) impact on aggregate utility.
However, I do think the point about glasses as an experience good is a good one and could quite possibly be at play here. If getting glasses for work is not a normal activity, it could be easy to underestimate the benefits of doing so.
Connor makes great points as always, and I appreciate the detailed response and the openness Lauren!
I very much agree with you both that glasses are an experience good and people systematically underestimate the value as their eyesight slowly gets worse. I think it’d be very interesting to study different ways of stimulating demand and generating awareness. I’m especially interested in the free screening model some systems like LV Prasad have adopted.
And to be clear, I would guess that providing reading glasses is cost-effective based on the disability aversion alone. People were very happy to glasses and did benefit from them. I am just skeptical there are many further income benefits beyond that. On that topic, I have finished a first draft of the comment, so hopefully can circulate soon!
Thanks so much for this comment Drew. I love on-the-ground experience like this and the insights you can bring, especially paired with data analysis like you’ve done here. We don’t get this very much on the forum.
If Kenya is anything like UgAnda, reading glasses are available in most towns like you say but they are a bit absurdly expensive.
I’m interested that the study quoted there 33 precent increase in productivity rather than the delta between the intervention and control group (12 percent). That seems quite sloppy.
The demand things is fascinating, although I’ve got moved feelings on it there also isn’t much demand for buying mosquito nets either.
Weird off-the-cuff question but maybe intentionally inducing something like experimenter demand effects would be a worthwhile intervention? After figuring out a way of not making recipients feel cheated or patronized, of course.
I’m extremely skeptical of the finding that reading glasses dramatically increase income. After looking into this topic for the past year (having initially been excited by these studies), I would now guess that the significant findings are more a result of experimenter demand effects than of reading glasses.
For example, in the paper on tea pickers you mention, the research team made seven unannounced visits to assess “compliance with study glasses” over the course of the 11-week trial. But we know from past research (e.g. Zwane et al. 2011) that monitoring changes behavior (as evidenced by the uptick in glasses usage). In that case, the estimate from the trial will capture the effect of monitoring + reading glasses which is not an effect we’d ever observe in the real world. The productivity of the control group also increases by 20% between baseline and endline but I don’t believe the authors provide any potential explanations for this substantial increase.
Perhaps more tellingly, I’ve now visited tea growing regions in rural Kenya multiple times this past year. I’ve observed hospitals giving reading glasses to tea pluckers and I’ve asked people if they would use them when plucking tea and they all say no. Or if I go to an area where the hospital has distributed glasses in the past, I have never found workers wearing them in the field. That’s not to say they don’t appreciate the glasses. Many people love them and report using them daily, but they’re using the glasses to read the bible or look at their phone! Yes, this evidence is anecdotal, but it’s hard to reconcile the study findings with my observations.
Furthermore, the idea that someone in rural Kenya would have to travel to Nairobi to get a pair of reading glasses is not accurate in my experience (unless they live really close to Nairobi). The hospitals I went to in rural Kenya offered reading glasses and there are also multiple optometrists in larger towns. Yes, people won’t be able to buy reading glasses at the grocery store or at CVS like here in the US, but glasses aren’t crazily inaccessible.
On this point, if reading glasses really did generate an extra $140 in earnings a year, why isn’t everyone clamoring to get a pair? There would be huge incentives for businesses to expand and offer the product in rural areas at higher prices. And workers should expend substantial effort to obtain a pair. In practice, we don’t see this happening. In my view, it is more likely that THRIVE overestimates the return to reading glasses than that hundreds of millions of workers worldwide substantially underestimate the benefit.
Additionally, the 33.4% statistic you cite from THRIVE isn’t even statistically significant! The authors report the coefficient from a median regression, but the standard error for this estimate is not mentioned.
At the moment, I’m working on a formal comment aggregating these considerations + many others and I aim to wrap that up shortly.
Also a longer response: I do think the lack of demand is worrying and could be suggestive that these studies are not showing real world effects. I haven’t spent enough time in rural Kenya to know how hard it is to get glasses, but I am updating based on what you say!
I do think it is easy to underestimate how bad your vision has gotten and not use glasses you need. Personally, I have failed to notice that my prescription has gotten out of date and continued to use old glasses, and then finally get around to getting new ones and I do notice a productivity improvement. Since your brain can somewhat compensate, I think it’s easy to underestimate the returns to a correct Rx.
I am very excited to see your longer comment when it’s done—make sure to sent it to both OP and Erin Crossett at GiveWell as well; it sounds like we will all benefit from it!
I share a lot of Drew’s skepticism about the study, especially the experimenter demand effects. If monitoring alone is enough to increase productivity, I think it’s quite plausible that there is some further response (beyond a direct effect of the glasses on vision) to monitoring plus the provision of glasses. Even as a large proponent of quantile regression in many applications, I do think OLS is more appropriate for a cost effectiveness analysis. A median shift could be consistent both with a much larger or much smaller (even negative) impact on aggregate utility.
However, I do think the point about glasses as an experience good is a good one and could quite possibly be at play here. If getting glasses for work is not a normal activity, it could be easy to underestimate the benefits of doing so.
Connor makes great points as always, and I appreciate the detailed response and the openness Lauren!
I very much agree with you both that glasses are an experience good and people systematically underestimate the value as their eyesight slowly gets worse. I think it’d be very interesting to study different ways of stimulating demand and generating awareness. I’m especially interested in the free screening model some systems like LV Prasad have adopted.
And to be clear, I would guess that providing reading glasses is cost-effective based on the disability aversion alone. People were very happy to glasses and did benefit from them. I am just skeptical there are many further income benefits beyond that. On that topic, I have finished a first draft of the comment, so hopefully can circulate soon!
Thanks so much for this comment Drew. I love on-the-ground experience like this and the insights you can bring, especially paired with data analysis like you’ve done here. We don’t get this very much on the forum.
If Kenya is anything like UgAnda, reading glasses are available in most towns like you say but they are a bit absurdly expensive.
I’m interested that the study quoted there 33 precent increase in productivity rather than the delta between the intervention and control group (12 percent). That seems quite sloppy.
The demand things is fascinating, although I’ve got moved feelings on it there also isn’t much demand for buying mosquito nets either.
Please post and comment more!
Very interested to see it when it’s done! Always happy to update my thoughts based on new evidence.
Weird off-the-cuff question but maybe intentionally inducing something like experimenter demand effects would be a worthwhile intervention? After figuring out a way of not making recipients feel cheated or patronized, of course.