I joined the psychology department at UCLA as an Assistant Professor in July 2023. Prior to that, I was an independent research group leader at the MPI for Intelligent Systems in Tübingen. I completed my Ph.D. in the Computational Cognitive Science Lab at UC Berkeley in 2013, obtained a master’s degree in Neural Systems and Computation from ETH Zurich, and completed two simultaneous bachelor’s degrees in Cognitive Science and Mathematics/Computer Science at the University of Osnabrück.
Falk Lieder
Finding before funding: Why EA should probably invest more in research
Forecasting the cost-effectiveness of trying something new
Estimating the cost-effectiveness of scientific research
Predicting the cost-effectiveness of deploying a new intervention
Estimating the cost-effectiveness of previous R&D projects
Developing interventions for promoting prosocial behavior might be more cost-effective at increasing well-being than providing psychotherapy
Predicting the cost-effectiveness of future R&D projects and academic research
I think the best-known study on the subject is
Oliner, S. P. (1992). Altruistic personality: rescuers of Jews in Nazi Europe. Simon and Schuster.
Two other good articles on this subject are
Midlarsky, E., Fagin Jones, S., & Corley, R. P. (2005). Personality correlates of heroic rescue during the Holocaust. Journal of personality, 73(4), 907-934.
How quickly should grantees impacted by recent events apply to this call? Is there a hard or soft deadline for these applications? I have to decide how much time I should invest in adapting, updating, and improving the previous application. I assume you want applicants to attach a proposal detailing the planned projects, the project’s pathway to impact, and evidence of its chances to succeed.
MetaScience Symposium on Identifying Impactful Research Topics
Will you consider applications for specific projects or only for the general operating expenses of the entire organization?
I have investigated the issues you highlighted, diagnosed the underlying errors, and revised the model accordingly. The root of the problem was that I had sourced some of the estimates of the frequency of prosocial behavior from studies on social behavior under special, unrepresentative conditions, such as infants interacting with adults for 10 min while being observed by researchers and prosocial behavior in TV series. I have removed those biased estimates of the frequency of prosocial behavior in the real world. As a consequence, the predicted lifetime increase in the number of kind acts per person reached by the intervention dropped from 1600 to 64. The predicted cost-effectiveness of the research dropped from 110 times the cost-effectiveness of StrongMinds to 7.5 times the cost-effectiveness of StrongMinds.
In producing this revised version, I also made a few additional improvements. The most consequential of those was to base the estimated cost of deploying the intervention on empirical data on the effectiveness of online advertising in $ per install.
I am currently using Squiggle to program a much more rigorous version of this analysis. That version will include additional improvements and rigorously document and justify each of the model’s assumptions.
Here is my post on the proof-of-concept that this approach can be applied to predict the cost-effectiveness of funding more research on a specific topic: https://forum.effectivealtruism.org/posts/aRFCrJaozrHearPAh/doing-research-on-promoting-prosocial-behavior-might-be-100
The submission form keeps telling me “Your response is too large. Try shortening some answers.” even though the total number of characters is significantly lower than 25,000. What should I do? Would you like me to share a link to a PDF with our answers to all questions or upload a single PDF with all of that information?
Thank you for your feedback, Stan!
I think the appropriateness of E[CE] as a prioritization criterion depends on the nature of the decision problem.
I think the expected value of the cost-effectiveness ratio is the appropriate prioritization criterion for the following scenario: i) a decision-maker is considering which organization should receive a given fixed amount of money (m), and ii) each organization (i) turns every dollar it receives into some uncertain amount of value (CE_i). In that case, the expected utility of giving the money to organization i is E[U_i]= m*E[CE_i]. Therefore, the way to maximize expected utility is to give the money to the organization with the highest expected cost-effectiveness. In this scenario, the consequences of contributing $1 to a project with an expected cost-effectiveness of 1 WELLBY/$ are almost identical in both scenarios. Most of the expected utility comes from the possibility that the project might be highly cost-effective. If the project is not highly cost-effective, then the $1 contribution accomplishes very little, regardless of whether the project costs $10,000, $100,000, or $1,000,000.
In my view, your example illustrates that the expected cost-effectiveness ratio is an inappropriate prioritization criterion if the funder has to decide whether to pay 100% of the project’s costs without knowing how much that will be. In that scenario, I think the appropriate prioritization criterion would be E[B]-E[CE_alt]*E[C], where E[CE_alt] is the expected cost-effectiveness of the most promising project that the funder could fund instead.
I think the second decision problem describes the situation of a researcher or funder who is committed to seeing their project through until the end. By contrast, the first decision problem corresponds to a researcher/funder intending to allocate a fixed amount of time/money to one project or another (e.g., 3 years of personal time or 1 million dollars) and then move on to another project after that.
Thank you for engaging with and critiquing the cost-effectiveness analysis, Michael! There seem to be a few misunderstandings I would like to correct.
The CEE in the linked Guesstimate looks optimistic to the point of being impossible. Given the quoted numbers of 32 acts of kindness per day with each act producing an average of 0.7 happy hours, that’s 22 happy hours produced per person-day of acts of kindness. If you said people’s acts of kindness increased overall happiness by 10%, I’d say that sounds too high. If you say it produces 22 happy hours, when the average person is only awake for 17 hours...well that’s not even possible.
The value you calculated is the sum of the additional happiness of all the people to whom the person was kind. This includes everyone they interacted with that day in any way. This includes everyone from the strangers they smiled at, to the friends they messaged, the colleagues they helped at work, the customers they served, their children, their partner, and their parents and other family members. If you consider that the benefit for the kindness might be benefited over more than a dozen people, then 22 hours of happiness, might be no more than 1-2 hours per person. Moreover, the estimates also take into account that a person who benefits from your kindness today might still be slightly more happy tomorrow.
I am also very skeptical of the reported claim that a one-time intervention of “watching an elevating video, enacting prosocial behaviors, and reflecting on how those behaviors relate to one’s value” (Baumsteiger 2019) can produce an average of 1600 additional acts of kindness per person. That number sounds about 1000x too high to me.
The intervention by Baumsteiger (2019) was a multi-session program that lasted 12 days and involved planning, performing, and documenting one’s prosocial behavior for 10 days in a row. The effect sizes distribution in the Guesstimate model is based on many different studies, some of which were even more intensive.
In general, psych studies are infamous for reporting impossibly massive effects and then failing to replicate.
Most of the estimates are based on meta-analyses of many studies. The results of meta-analyses are substantially more robust and more reliable than the result of a single study.
I think you are right that this first estimate was too optimistic. In particular, the probability distribution of the frequency of prosocial behavior is currently based on four estimates from different studies. One of those studies led to an estimate that appears to be far too high. This might be because they defined prosocial behavior more liberally because it involved interactions with children, or because participants knew that they were being observed. I will think about what the more general problem might be and how it can be addressed systematically.
Thank you, Vasco!
I agree that the optimal percentage of research funding is higher/lower for areas where less/more science and R&D have been done so far. We don’t really know yet how different areas correspond to which of the simulated scenarios. I think establishing this correspondence will be a crucial next step in our project. Moreover, some topics and potential interventions within a broad cause area, such as global health and development, might have been researched much less than others. Therefore, it probably makes most sense to apply our analysis at the level of specific research topics or interventions.Thank you for pointing out that the amount of available resources can change. Contrary to your intuition, I suspect that taking this into account would tilt the analysis in favor of even more research if the amount of funding an area receives depends on the cost-effectiveness and scalability of its best intervention. Successful research results in new interventions that are more cost-effective or more scalable than the best previous ones. This can significantly increase how much the cause area appeals to the EA community and how much finding it can absorb. Suppose, research in a cause area without any highly cost-effective interventions leads to the development of an intervention that is more cost-effective than the best interventions in any other area. That would probably increase the amount of money that will be donated to the cause. Or suppose that the research makes a highly effective intervention much more scalable. That would likely increase the amount of money that will be donated to the corresponding cause as well.
Thank you, Stuart! Your post strongly resonated with me. :) I agree that EA funding agencies should develop farsighted strategic agendas for the entire process of generating and utilizing knowledge and innovations that are crucial for humanity’s long-term survival and flourishing. I think this process should start with use-inspired basic research on crucial questions. The next step should be to translate the discoveries of that research into new interventions. Then those interventions should be tested on an increasingly larger scale and rigorously compared against the best existing interventions in terms of their cost-effectiveness. Once we have completed those steps to generate better interventions and knowledge about their effectiveness, we can exploit having a substantially improved set of opportunities to do good. A single program of a single funding agency could support all of these steps and thereby coordinate and guide the process of discovery, intervention development, and evaluation research towards what we most need to maximally improve the future of humanity. This farsighted strategic approach is not unheard of. The Development Innovation Ventures program and the Grand Challenges for Human Flourishing program are steps in that direction. I think EA Funding agencies could learn something from the design of those programs and combine their strategic approach with the latest insights and principles of Effective Altruism and good scientific practice. I think that would be extremely valuable.
Thank you for your insightful comments, Marshall!
The simulations do not distinguish between scientific research and R&D projects outside of academia. The relative usefulness of these two types of research is beyond the scope of the model. The main assumption of the simulations is that the research projects are selected strategically for their potential to enable or produce more cost-effective interventions.
I agree that the assumption about the cost-effectiveness of new interventions can and should be validated empirically. Estimating it from historical data is an important direction for future work, and I am planning to pursue it. I think the expected cost-effectiveness will be vastly different depending on the extent to which the research builds on established knowledge and techniques. In the extreme case of refining the best existing intervention, the expected cost-effectiveness of the new intervention would definitely be larger than 50%.
I am very shocked. What exactly happened? How could this happen? How could the CEA possibly let itself be infiltrated by a cult striving to take over the world? And how could an organization founded by academics fail to scrutinize Leverage’s pseudo-scientific and manipulative use of concepts and techniques related to psychotherapy and rationality? Did CEA ever consult an independent psychological scientist or psychotherapy researcher to assess the ethicality of what Leverage was doing, the accuracy of their claims, or the quality of their “research”? Didn’t it raise any red flags that the people inventing new methods of “psychotherapy” had no training in clinical psychology?