Second, we’ve taken a very shallow look at a recent trial that compares cash transfers with therapy head-to-head. This trial finds a much smaller effect of therapy vs. cash.
We account for this trial in our meta-analysis—if we hadn’t incorporated it, therapy would look even a bit more cost-effective. Of course, the point of meta-analyses is to look at the whole evidence base, rather than just selecting one or two pieces of evidence; one could discount any meta-analysis this way by pointing to the trial with the lowest effect.
We don’t think one study should overshadow the results of a meta-analysis, which aggregates a much wider set of data (“beware the man of only one study” etc). If there was one study finding no impact of bednets, I doubt GiveWell would conclude it would be reasonable to discount all the previous data on bednets.
We account for this trial in our meta-analysis—if we hadn’t incorporated it, therapy would look even a bit more cost-effective. Of course, the point of meta-analyses is to look at the whole evidence base, rather than just selecting one or two pieces of evidence; one could discount any meta-analysis this way by pointing to the trial with the lowest effect.
We don’t think one study should overshadow the results of a meta-analysis, which aggregates a much wider set of data (“beware the man of only one study” etc). If there was one study finding no impact of bednets, I doubt GiveWell would conclude it would be reasonable to discount all the previous data on bednets.