Hi carneades, in reply I just want to make 2 general points here:
Many things need to be done in the developing world, e.g.the ones you mentioned: protecting people against malaria, creating jobs, improving the quality of governments… The most effective intervention for one purpose could be not very useful for another, but that’s still okay because it would be better than trying to do something that serves multiple purposes but is ineffective in all of them. (e.g. for protecting people against malaria, the most effective intervention could be distributing bed nets for free; for job creating, it could be improving the infrastructure and legal system; for improving the quality of governments it could be educating voters—though for this to affect government provision of bed nets down the road would take many years and if we solely rely on this to protect people against malaria we would miss the opportunity to save many lives now compared to using the most effective intervention on this)
By saying the studies are good I mean the researchers take into account that respondents may not report the truth and seem to manage to find out the truth despite that. This is not just because I trust these famous economists to do a good job or I know some of them personally and know that they care about doing a good job, but I also find evidence from the papers. For instance in the Cohen and Dupas paper on free distribution of bed nets, it says (on p14, under “III.C Data”) “During the home visits, respondents were asked to show the net, whether they had started using it, and who was sleeping under it. Surveyors checked to see whether the net had been taken out of the packaging, whether it was hanging, and the condition of the net.” In the Haushofer and Shapiro paper on Give Directly, on p28 they talk about potential desirability bias on alcohol and tobacco and how they address it, and on p32 they mention assets including metal roofs and livestocks which surveyors could easily check (I don’t think they mentioned surveyors checking this but the study is done by IPA in the same region I work in Kenya so I imagine it should have the high standard of work done by this organization and surveyors should check things when they can). In general economists don’t like to rely on survey data precisely because people may lie, and in developing countries when this is often inevitable we try hard to get around the problem and mention how we address them in papers (otherwise you would get a lot of questioning from presentation audience, referees, etc. so there’s plenty of incentive in academia to do that; e.g. I can imagine these two studies having survived such scrutiny).
Also regarding your point of how things on the ground could appear to be completely different from the truth if you don’t know the community well enough, in my experience typically local surveyors understand the context well enough to be able to explain to us foreigners what is happening. I agree that this could be a problem for some donors/organizations/studies that aren’t very careful, so I can only say that this won’t be a big problem if things are done well (which should be the case for charities recommended by GiveWell but I don’t know about others).
(I spent 5 months in a town in western Kenya, not super integrated into the local community or fluent in the local language so take this with a grain of salt if you want, but I do know very well how this kind of studies and surveys work since I did one myself.)
Certainly there are many different kinds of interventions that we can implement. Some interventions are mutually exclusive, others are not. Toms shoes comes and donates shoes to a community, where another organization is attempting to improve small businesses, particularly tailors, some of the few professionals in very small communities. The donation puts many local tailors out of business. Therefore, we either need to develop interventions which are not mutually exclusive or choose between interventions. If we want to do the first, organizations like AMF should help local businesses instead of destroying them. If we want to do the second, we should choose between interventions by asking the people what they need, not deciding what we think is best without ever talking to the people on the ground. If they don’t want us to save their lives at the expense of job opportunities, we should not. People should have the right to choose their own interventions, especially when those interventions can do their communities harm. If you want to go in without the community’s consent, the least that you can do first is do no harm.
As for the studies, at the end of the day I’m a philosophical skeptic all the way down (which means that I have so serious concerns about the relation between truth and the scientific method, for legitimate philosophical reasons, such as the problem of induction , and the problem of underdetermination, check out the Stanford Encyclopedia of Philosophy for more on that) but we can put that aside for now. As for the trust, I trust the people that live here about what they need, and what they are doing more than I trust foreigners who don’t spend much time with the people, famous economists or not. Your quote about nets fails to address a number of concerns, such as communities that put up nets whenever they see the inspectors coming (you would be surprise how many people do this), and the fact that most people here hang out outside until well after peak mosquito biting time, so even if they sleep under a mosquito net, and are fully truthful in an interview, since they don’t actually understand or appreciate the importance of bed nets, and someone values something much less when they get it for free. Checking for a metal roof seems like a clear method, but there are so many more which could be lied about. Livestock wanders freely here, and is rarely kept in any kind of pen, and only the people that live in a village really know who owns what. I’m not saying that all of their metrics are necessarily off, I’m saying that I’m not convinced that they are correct. As for the academic criticism, I’m concerned that the people that provide the scrutiny are, unfortunately, not the people with an in-depth experiential understanding of the cultural practices of these communities. I am also concerned that there is a gap between something being able to survive academic criticism, and providing a fully accurate picture of the state of affairs here.
As for the final point, why do you think that the local surveyors are able to solicit truthful responses any more than foreigners? If those local surveyors did not grow up in the same tiny area of the village, the families will lie through their teeth. I have lived here for years, and I am close to fluent in a local language and I have seen families lie to every census worker that comes through, local or foreign. I have also seen translators blatantly lie to researchers about what respondents say. Truth is not valued here. Unless you do speak the language and have lived with a family for quite some time, there’s no way for you to know if they actually sleep under a bed net, if they actually exclusively breast feed, or if they really own that goat. As noted before, I cannot say that this happens everywhere as Africa is a big place, and cultures differ drastically. Perhaps in East Africa there are different cultural practices which make it easier to conduct research, but here even the Government cannot correctly take a census.
Thanks for taking the time to engage in in this discussion. I hope that you reconsider donating to charities which ignore what communities actually want in favor of providing cheap, but ultimately harmful interventions.
Hi carneades, in reply I just want to make 2 general points here:
Many things need to be done in the developing world, e.g.the ones you mentioned: protecting people against malaria, creating jobs, improving the quality of governments… The most effective intervention for one purpose could be not very useful for another, but that’s still okay because it would be better than trying to do something that serves multiple purposes but is ineffective in all of them. (e.g. for protecting people against malaria, the most effective intervention could be distributing bed nets for free; for job creating, it could be improving the infrastructure and legal system; for improving the quality of governments it could be educating voters—though for this to affect government provision of bed nets down the road would take many years and if we solely rely on this to protect people against malaria we would miss the opportunity to save many lives now compared to using the most effective intervention on this)
By saying the studies are good I mean the researchers take into account that respondents may not report the truth and seem to manage to find out the truth despite that. This is not just because I trust these famous economists to do a good job or I know some of them personally and know that they care about doing a good job, but I also find evidence from the papers. For instance in the Cohen and Dupas paper on free distribution of bed nets, it says (on p14, under “III.C Data”) “During the home visits, respondents were asked to show the net, whether they had started using it, and who was sleeping under it. Surveyors checked to see whether the net had been taken out of the packaging, whether it was hanging, and the condition of the net.” In the Haushofer and Shapiro paper on Give Directly, on p28 they talk about potential desirability bias on alcohol and tobacco and how they address it, and on p32 they mention assets including metal roofs and livestocks which surveyors could easily check (I don’t think they mentioned surveyors checking this but the study is done by IPA in the same region I work in Kenya so I imagine it should have the high standard of work done by this organization and surveyors should check things when they can). In general economists don’t like to rely on survey data precisely because people may lie, and in developing countries when this is often inevitable we try hard to get around the problem and mention how we address them in papers (otherwise you would get a lot of questioning from presentation audience, referees, etc. so there’s plenty of incentive in academia to do that; e.g. I can imagine these two studies having survived such scrutiny).
Also regarding your point of how things on the ground could appear to be completely different from the truth if you don’t know the community well enough, in my experience typically local surveyors understand the context well enough to be able to explain to us foreigners what is happening. I agree that this could be a problem for some donors/organizations/studies that aren’t very careful, so I can only say that this won’t be a big problem if things are done well (which should be the case for charities recommended by GiveWell but I don’t know about others).
(I spent 5 months in a town in western Kenya, not super integrated into the local community or fluent in the local language so take this with a grain of salt if you want, but I do know very well how this kind of studies and surveys work since I did one myself.)
Thanks again. Here are my thoughts.
Certainly there are many different kinds of interventions that we can implement. Some interventions are mutually exclusive, others are not. Toms shoes comes and donates shoes to a community, where another organization is attempting to improve small businesses, particularly tailors, some of the few professionals in very small communities. The donation puts many local tailors out of business. Therefore, we either need to develop interventions which are not mutually exclusive or choose between interventions. If we want to do the first, organizations like AMF should help local businesses instead of destroying them. If we want to do the second, we should choose between interventions by asking the people what they need, not deciding what we think is best without ever talking to the people on the ground. If they don’t want us to save their lives at the expense of job opportunities, we should not. People should have the right to choose their own interventions, especially when those interventions can do their communities harm. If you want to go in without the community’s consent, the least that you can do first is do no harm.
As for the studies, at the end of the day I’m a philosophical skeptic all the way down (which means that I have so serious concerns about the relation between truth and the scientific method, for legitimate philosophical reasons, such as the problem of induction , and the problem of underdetermination, check out the Stanford Encyclopedia of Philosophy for more on that) but we can put that aside for now. As for the trust, I trust the people that live here about what they need, and what they are doing more than I trust foreigners who don’t spend much time with the people, famous economists or not. Your quote about nets fails to address a number of concerns, such as communities that put up nets whenever they see the inspectors coming (you would be surprise how many people do this), and the fact that most people here hang out outside until well after peak mosquito biting time, so even if they sleep under a mosquito net, and are fully truthful in an interview, since they don’t actually understand or appreciate the importance of bed nets, and someone values something much less when they get it for free. Checking for a metal roof seems like a clear method, but there are so many more which could be lied about. Livestock wanders freely here, and is rarely kept in any kind of pen, and only the people that live in a village really know who owns what. I’m not saying that all of their metrics are necessarily off, I’m saying that I’m not convinced that they are correct. As for the academic criticism, I’m concerned that the people that provide the scrutiny are, unfortunately, not the people with an in-depth experiential understanding of the cultural practices of these communities. I am also concerned that there is a gap between something being able to survive academic criticism, and providing a fully accurate picture of the state of affairs here.
As for the final point, why do you think that the local surveyors are able to solicit truthful responses any more than foreigners? If those local surveyors did not grow up in the same tiny area of the village, the families will lie through their teeth. I have lived here for years, and I am close to fluent in a local language and I have seen families lie to every census worker that comes through, local or foreign. I have also seen translators blatantly lie to researchers about what respondents say. Truth is not valued here. Unless you do speak the language and have lived with a family for quite some time, there’s no way for you to know if they actually sleep under a bed net, if they actually exclusively breast feed, or if they really own that goat. As noted before, I cannot say that this happens everywhere as Africa is a big place, and cultures differ drastically. Perhaps in East Africa there are different cultural practices which make it easier to conduct research, but here even the Government cannot correctly take a census.
Thanks for taking the time to engage in in this discussion. I hope that you reconsider donating to charities which ignore what communities actually want in favor of providing cheap, but ultimately harmful interventions.