I think there’s reason to be cautious with the “highest marginal information comes from studying neglected interventions” line of reasoning, because of the danger of studies not replicating. If we only ever test new ideas, and then suggest funding the new ideas which appear from their first study to have the highest marginal impact, it’s very easy to end up with several false positives being funded even if they don’t work particularly well.
In fact, in some sense the opposite argument could be made; it is possible that the highest marginal information gain will come from research research into a topic which is already receiving lots of funding. Mass deworming is the first example that springs to mind, mostly because there’s such a lack of clarity at the moment, but the marginal impact of finding new evidence about an intervention there’s lots of money in could still be very large.
I guess the rather sad thing is that the biggest impact comes from bad news: if an intervention is currently receiving lots of funding because the research picture looks positive, and a large study fails to replicate, a promising intervention now looks less so. If funding moves towards more promising causes as a result, this is a big positive impact, but it feels like a loss. It certainly feels less like good news than a promising initial study on a new cause area, but I’m not sure it actually results in a smaller impact.
I agree that non-replication is a danger. But I don’t think that positive results are specifically high-value; instead, I care about studies that tell me whether or not an intervention is worth funding.
I’d expect most studies of neglected interventions to turn up “negative” results, in the sense that those interventions will seem less valuable than the best ones we already know about. But it still seems good to conduct these studies, for a couple of reasons:
If an intervention does look high-value, there’s a chance we’ve stumbled onto a way to substantially improve our impact (pending replication studies).
Studying an intervention that is different from those currently recommended may help us gain new kinds of knowledge we didn’t have from studying other interventions.
For example, if the community is already well-versed in public health research, we might learn less from research on deworming than research on Chinese tobacco taxation policy (which could teach about generally useful topics like Chinese tax policy, the Chinese legislative process, and how Chinese corporate lobbying works).
Of course, doing high-quality China research might be more difficult and expensive, which cuts against it (and other research into “neglected” areas), but I do like the idea of EA having access to strong data on many different important areas.
That said, you do make good points, and I continue to think that neglectedness is less important as a consideration than scale or tractability.
Related: I really like GiveWell’s habit of telling us not only which charities they like, but which ones they looked at and deprioritized. This is helpful for understanding the extent to which the “deliberate neglect by rational funders” model pans out for a given intervention/charity, and I wish it were done by more grantmaking organizations.
I think there’s reason to be cautious with the “highest marginal information comes from studying neglected interventions” line of reasoning, because of the danger of studies not replicating. If we only ever test new ideas, and then suggest funding the new ideas which appear from their first study to have the highest marginal impact, it’s very easy to end up with several false positives being funded even if they don’t work particularly well.
In fact, in some sense the opposite argument could be made; it is possible that the highest marginal information gain will come from research research into a topic which is already receiving lots of funding. Mass deworming is the first example that springs to mind, mostly because there’s such a lack of clarity at the moment, but the marginal impact of finding new evidence about an intervention there’s lots of money in could still be very large.
I guess the rather sad thing is that the biggest impact comes from bad news: if an intervention is currently receiving lots of funding because the research picture looks positive, and a large study fails to replicate, a promising intervention now looks less so. If funding moves towards more promising causes as a result, this is a big positive impact, but it feels like a loss. It certainly feels less like good news than a promising initial study on a new cause area, but I’m not sure it actually results in a smaller impact.
I agree that non-replication is a danger. But I don’t think that positive results are specifically high-value; instead, I care about studies that tell me whether or not an intervention is worth funding.
I’d expect most studies of neglected interventions to turn up “negative” results, in the sense that those interventions will seem less valuable than the best ones we already know about. But it still seems good to conduct these studies, for a couple of reasons:
If an intervention does look high-value, there’s a chance we’ve stumbled onto a way to substantially improve our impact (pending replication studies).
Studying an intervention that is different from those currently recommended may help us gain new kinds of knowledge we didn’t have from studying other interventions.
For example, if the community is already well-versed in public health research, we might learn less from research on deworming than research on Chinese tobacco taxation policy (which could teach about generally useful topics like Chinese tax policy, the Chinese legislative process, and how Chinese corporate lobbying works).
Of course, doing high-quality China research might be more difficult and expensive, which cuts against it (and other research into “neglected” areas), but I do like the idea of EA having access to strong data on many different important areas.
That said, you do make good points, and I continue to think that neglectedness is less important as a consideration than scale or tractability.
Related: I really like GiveWell’s habit of telling us not only which charities they like, but which ones they looked at and deprioritized. This is helpful for understanding the extent to which the “deliberate neglect by rational funders” model pans out for a given intervention/charity, and I wish it were done by more grantmaking organizations.