Finding before funding: Why EA should probably invest more in research

Falk Lieder17 Aug 2022 8:48 UTC
102 points
11 comments9 min readEA link

[Disclaimer: Joel McGuire contributed in a personal capacity unrelated to the HLI.]

EA grantmakers dramatically differ in how they split their funds between doing good with the tools and knowledge we already have versus improving them through scientific research.[1] This raises the question of which proportion of its funds the EA community should invest in scientific research. Are we investing the correct percentage of our funds in scientific research? Should it be lower? Or should it be higher than it currently is?

We recently found that funding scientific research that aims to enable or create improved interventions can be 2-5 times as cost-effective as investing in the interventions that already exist. This seems plausible given that scientific research gave us the tools and knowledge we can now use to improve the future of humanity. If previous generations had not funded scientific research and R&D projects, there would be no medicines for preventing malaria, no long-lasting insecticide treated bed nets, no supplements for preventing vitamin A deficiency, no deworming tablets, and no vaccines. Strategically funding relevant scientific research could generate even more cost-effective opportunities to do good. But the more money we invest in research the less money we will have left for applying our best interventions. To help funding agencies and the EA community navigate this dilemma, we investigated how the total investment into a cause should be split between scientific research that might enable us to do more good in the future and doing good with the tools and knowledge that happen to be already available today.

Assumptions

Our analysis makes the following three assumptions:

  1. On average, additional research and R&D projects produce interventions that are substantially less cost-effective than the best interventions that already exist.

  2. The cost-effectiveness of newly developed interventions is variable. Therefore, research occasionally succeeds in inventing superior interventions, and the cost-effectiveness of those interventions can be substantially higher.

  3. When a more cost-effective intervention is invented, then the funding that the best previous intervention would have received will be invested into the new intervention instead.

  4. We measure the cost-effectiveness of research in the same units as the cost-effectiveness of the existing interventions it might help us improve on. The expected increase in the amount of good we can do with the improved interventions is a lower bound on the value of research.

Approach

We performed more than 300 million computer simulations to determine which way of splitting a fixed budget between developing better interventions and deploying interventions would generate the most good in total in expectation across different scenarios. For each scenario, we estimated the total good that could be achieved by first funding 0, 1, 2, 3, …, or 1000 research projects and then optimally allocating the remaining funds between the best interventions that resulted from the research and the best intervention that were available prior to the research. We then derived the optimal size of the research budget from the number of funded research projects that maximized the total good that the funding agency accomplished on average across 10,000 simulations. To derive robust recommendations, we replicated the main findings of our simulations across a wide range of conservative assumptions about the effectiveness of scientific research, the variability in the cost-effectiveness of new interventions. Concretely, we varied the average cost-effectiveness of new interventions between 10% and 110% of the cost-effectiveness of the best existing interventions with a standard deviation between 1% and 100%. In addition, we varied the total amount of funding in the simulated cause area between 10 million dollars and 1 billion dollars. We assumed uniform priors over these different scenarios. Moreover, we considered that the cost-effectiveness of new interventions could either follow a heavy-tailed distribution or a Normal distribution.

Results

1. We should initially spend more than half of our resources on scientific research. According to our simulations, the optimal philanthropic strategy is to first strategically improve the repertoire of interventions that are available for a given cause by funding research projects and then optimally allocate the remaining funds between the best new interventions and the best previous interventions. Concretely, our simulations suggest that the optimal strategy for causes with a budget of 100 million dollars or more is to initially allocate at least 52% of their annual spending to scientific research, and later invest around 45% of the initial endowment into the most effective interventions that the research produced.[2] Moreover, as the budget increases to 1 billion dollars, the optimal size of the initial research budget increases to 64-72% (see Figure 1). This is because the larger the budget, the larger the total gain from improvements in cost-effectiveness of each dollar spent.

Figure 1. Relative size of the research budget in % as a function of the size of the total budget, depending on whether the cost-effectiveness of new interventions follows a heavy-tailed distribution or not.

2. We should fund many research projects. According to our simulations, we should fund at least one research project for every 10 million dollars that are available for a given cause (see Figure 2). As illustrated in Figure 2, the optimal number of research projects increases even more rapidly with the total amount of funding that is available for a given cause or cause area when the cost-effectiveness of new interventions follows a heavy-tailed distribution.[3] On average across all simulated scenarios, the optimal number of research projects on causes backed by at least 50 million dollars is around 48 to 111 depending on whether the cost-effectiveness of new interventions follows a Normal distribution or a heavy-tailed distribution, respectively.

Figure 2. Optimal number of research projects as a function of the total amount of funding available for a given cause or cause area. Please note that both the x-axis and the y-axis are on a log-scale.

3. Why is this a good idea? According to our simulations, the recommended level of research funding would likely produce interventions that are 78% to 252% more cost-effective at doing good than the best already available interventions. While research takes time, once new interventions have been developed, they will, on average, allow us to do about twice as much good with each of the remaining dollars as is currently possible (see Figure 3). The resulting increase in the total good produced by our annual donations and investments will add up over time. Thus, even if investing into research temporarily reduced donations to charities that deploy existing interventions, we would be able to quickly make up for that and then still have money left to do more good than we could have done without the research. Concretely, according to our simulations, this strategy achieves between 23% and 56% more good with the initial budget than investing everything into the application of interventions that already exist today. Moreover, having developed more cost-effective interventions would also amplify the benefit of additional future donations and could attract more funding to the cause.

Figure 3. The long-run value of investing into scientific research. The orange line shows the total amount of good that can be accomplished by investing 50% of the annual budget into research for the decade, and then investing everything into the best new interventions. The blue line shows how much good we would accomplish if we always invested everything into the interventions that already exist today. Comparing the two lines shows that investing into research pays off in the long run. This is true even if investing into research would temporarily reduce how much money we can invest into the deployment of interventions that already exist.

4. The EA community’s investments in scientific research are much lower. So far, major EA funding agencies, such as GiveWell, Open Philanthropy, have spent much less than 50% of their resources on scientific research (see Figure 4). Only 1.6% of the grant funding that GiveWell has provided since 2014 supported projects with a research component. Open Philanthropy has invested about 25% of its funds into scientific research and R&D projects, and the Gates Foundation has invested approximately 30%. The Global Health and Development Fund, the Animal Welfare Fund, and the EA Infrastructure Fund invested between 13% and 28% of their funds into scientific research and R&D. For the FTX Future Fund the proportion of the investments that have gone into research and R&D projects is close to optimal. Only the Long-Term Future Fund invested more than 50% of its resources into scientific research and R&D projects (68%). However, this does not bring the total investment in research anywhere near the recommended 50% because the Long-Term Fund’s budget was only 0.01% of all funds.[4] Crucially, the total research funding provided jointly by all those grantmakers was only about 30%. Without the Gates Foundation, it would have been only about 18%.

Figure 4. Comparison of the relative size of the research budget of two existing grant makers with the optimal size of the research budget. The bar labeled “All” refers to the grants of GiveWell, Open Philanthropy, EA Funds, and the FTX Future Fund combined. For the FTX Future Fund the error bar shows the proportion of the publicly declared investments into research out of the total spending. For the other grantmakers the error bars show the 95% confidence intervals of our empirical estimates we obtained by annotating a random sample of 200-600 of their grants. The red error bar shows the proportion of the publicly declared investments into research out of the publicly declared investments only.

5. The recommendations are robust to uncertainty about the effectiveness of research. As shown in Figure 5 and Figure 6 the recommendations of our analysis hold for a wide range of assumptions about the expected cost-effectiveness of new interventions and its variability across different research projects. Concretely, we found that, on average, investing at least 50% of the initial annual budget in scientific research is optimal even if the new interventions are only about half as cost-effective as the best existing intervention, on average (see Figure 5). This recommendation holds almost regardless of whether the distribution of the cost-effectiveness of new interventions is heavy-tailed or not.

Figure 5. Optimal size of the research budget as a function of the expected cost-effectiveness of new interventions relative to the cost-effectiveness of the best existing interventions.
Figure 6. Optimal size of the research budget as a function of the variability of the cost-effectiveness of new interventions relative to the cost-effectiveness of the best existing intervention.

Discussion

We introduced a general method for determining how the grand total of the funds of a given cause or cause area should be split between charities that apply existing knowledge and interventions versus strategically selected basic and applied projects that might lead to more cost-effective interventions. We hope that, in the long run, the prudential application of our method to grant-making decisions will enable the EA community to do even more good by investing its funds more wisely. In particular, our analysis raises the question whether the EA community should invest more funds into scientific research and R&D projects. This question is very timely because our simulations suggest that the proportion of resources invested into research should increase with the total amount of funding available to a given cause area.

Some might object that academic research is already well funded. However, the focus areas and priorities of academic funding agencies are often misaligned with the goals of EA. This can make it extremely difficult for academics in certain fields to receive funding for EA-aligned research. Consequently, many academics who would be thrilled to have a more significant positive impact become discouraged and resign themselves to working on conventional topics. On the bright side, this creates an easy opportunity for the EA community to increase the value of academic research. All we need to do is to rectify the lack of funding opportunities for scientific research on crucial EA topics. Suppose we do decide to fund more research. In that case, we will face the practical challenge of determining which topics and interventions additional research should focus on could become a practical obstacle to increasing research funding to its optimal level. We have recently begun to develop a method for identifying impactful research topics, but much more work remains to be done.

Because there are usually multiple funders in each cause area, achieving the optimal split does not require that every grantmaker allocates the same proportion of their budget to scientific research. Rather, some funders may specialize in funding projects applying existing interventions, whereas others may specialize in funding scientific research and R&D projects. From this perspective, the results of our analysis can be interpreted as a recommendation for how the total funds of a given cause area should be allocated between funding agencies specializing in research versus intervention. In practice, many funding agencies, such as Open Philanthropy, support both.

Grantmakers that follow this approach could use the results of our analysis to determine whether they could increase their positive impact by increasing or decreasing the proportion of funds they allocate to research on a cause-by-cause basis. For each area, the optimal adjustments will depend on how funds are currently allocated in that area and how much research has already been conducted. Usually, the budget of a single program or grantmaker is only a fraction of the total funding for a given area. Thus, the implications of our analysis for an individual program or grantmaker depend on how other funders split their funds between research and applications. In highly neglected cause areas, most of the funding might come from within the EA community. In other areas, such as global health and development, governmental, corporate, and private philanthropy outside the EA community should also be considered to the extent that its funds are used effectively. Thus, when the other funders in a given cause area spent much less than 50% of their resources on developing more cost-effective interventions, it might be warranted to invest an even more significant fraction of the budget into scientific research. Conversely, suppose previous investments in a given cause area have almost exclusively focussed on scientific research. In that case, it might be optimal for an EA grant maker to focus primarily or solely on the findings’ applications.

Because the implications of our analysis depend on how funds are currently allocated in a given cause area, researching how funds are currently allocated in different areas is an important direction for future work. This research should be accompanied by determining the optimal fund allocation for a given cause area based on the potential gains that additional research could achieve. Through this investigation, we might identify which cause areas should increase their investment in scientific research and R&D projects and which, if any, should decrease it.

The simulations reported in this post make a number of rather abstract, theoretical assumptions about the cost-effectiveness of new interventions that might result from additional research and development projects. We believe that those theoretical assumptions can and should be replaced by data-driven estimates that reflect the effectiveness of scientific research and R&D projects tackling crucial problems in the cause area(s) that specific funding agencies or programs are working on. In fact, for many cause areas, both the expected value and the standard deviation of the relative difference between the cost-effectiveness of new interventions and the cost-effectiveness of the best existing intervention can be estimated from the effect sizes reported in relevant meta-analyses. Moreover, whether the distribution of the cost-effectiveness is heavy-tailed can be gauged from histograms of the cost-effectiveness of newly developed interventions and the prior distribution on the available budget can be replaced by the actual budget of a specific funding agency or program.

If you would like to know more about the methods and results of our simulations, you can read our detailed technical report. If you would like to use our method, feel free to reach out to us. We are happy to share the code with you and help you adapt it to your questions.

Author Contributions

Falk Lieder conducted the simulations and analysis and wrote the first draft. Joel McGuire gave feedback on the draft and contributed the ideas for the Assumptions section and recommended comparing the optimal versus actual sizes of grant maker’s research budgets. Joel was only involved in a personal capacity unrelated to his work at HLI.

Acknowledgements

Matthias Stelter gave insightful feedback on an earlier draft and made concrete suggestions for how to improve it. This analysis is part of the research priorities project, which is an ongoing collaboration with Izzy Gainsburg, Philipp Schönegger, Emily Corwin-Renner, Abbigail Novick Hoskin, Louis Tay, Isabel Thielmann, Mike Prentice, John Wilcox, Will Fleeson, and many others. The model is partly inspired by Jack Malde’s earlier post on whether effective altruists should fund scientific research. We would also like to thank Michael Plant from the Happier Lives Institute, Emily Corwin-Renner, and Cecilia Tilli for helpful pointers and discussions.

  1. ^

    On the low end of the spectrum, only 1.6% of GiveWell’s investments supported scientific research or the development of new interventions. On the high end of the spectrum, the Long-Term Future Fund provided about 70% of its funds to research and R&D projects.

  2. ^

    This means that out of all funds that will ever be spent on a given cause about 28.6% should be invested into research.

  3. ^

    How many more research projects we should fund on a given topic also depends on how much research has already been conducted on that topic. We simulated the effect of prior research by adjusting the distribution of the difference between the cost-effectiveness of new interventions and the cost-effectiveness of existing interventions.

  4. ^

    Even if we don’t account for the grants made by the Gates Foundation, the budget of the Long-Term Future Fund is still only about 0.2% of the total funds invested according to the principles of EA.

Falk Lieder17 Aug 2022 8:48 UTC
102 points
11 comments9 min readEA link