I joined the psychology department at UCLA as an Assistant Professor in July 2023. Prior to that, I was an independent research group leader at the MPI for Intelligent Systems in Tübingen. I completed my Ph.D. in the Computational Cognitive Science Lab at UC Berkeley in 2013, obtained a master’s degree in Neural Systems and Computation from ETH Zurich, and completed two simultaneous bachelor’s degrees in Cognitive Science and Mathematics/Computer Science at the University of Osnabrück.
Falk Lieder
Karma: 368
Leverage Research, including partial takeover of CEAI am very shocked. What exactly happened? How could this happen? How could the CEA possibly let itself be infiltrated by a cult striving to take over the world? And how could an organization founded by academics fail to scrutinize Leverage’s pseudo-scientific and manipulative use of concepts and techniques related to psychotherapy and rationality? Did CEA ever consult an independent psychological scientist or psychotherapy researcher to assess the ethicality of what Leverage was doing, the accuracy of their claims, or the quality of their “research”? Didn’t it raise any red flags that the people inventing new methods of “psychotherapy” had no training in clinical psychology?
Regardless thereof, I can rerun the analyses for E[B]/E[C] as a robustness check and let you know what I find.
Thank you for your feedback, Stan!
I think the appropriateness of E[CE] as a prioritization criterion depends on the nature of the decision problem.
I think the expected value of the cost-effectiveness ratio is the appropriate prioritization criterion for the following scenario: i) a decision-maker is considering which organization should receive a given fixed amount of money (m), and ii) each organization (i) turns every dollar it receives into some uncertain amount of value (CE_i). In that case, the expected utility of giving the money to organization i is E[U_i]= m*E[CE_i]. Therefore, the way to maximize expected utility is to give the money to the organization with the highest expected cost-effectiveness. In this scenario, the consequences of contributing $1 to a project with an expected cost-effectiveness of 1 WELLBY/$ are almost identical in both scenarios. Most of the expected utility comes from the possibility that the project might be highly cost-effective. If the project is not highly cost-effective, then the $1 contribution accomplishes very little, regardless of whether the project costs $10,000, $100,000, or $1,000,000.
In my view, your example illustrates that the expected cost-effectiveness ratio is an inappropriate prioritization criterion if the funder has to decide whether to pay 100% of the project’s costs without knowing how much that will be. In that scenario, I think the appropriate prioritization criterion would be E[B]-E[CE_alt]*E[C], where E[CE_alt] is the expected cost-effectiveness of the most promising project that the funder could fund instead.
I think the second decision problem describes the situation of a researcher or funder who is committed to seeing their project through until the end. By contrast, the first decision problem corresponds to a researcher/funder intending to allocate a fixed amount of time/money to one project or another (e.g., 3 years of personal time or 1 million dollars) and then move on to another project after that.
A recording of this symposium is now available on YouTube: https://youtu.be/6cZLYqnDbOc
I think the best-known study on the subject is
Oliner, S. P. (1992). Altruistic personality: rescuers of Jews in Nazi Europe. Simon and Schuster.
Two other good articles on this subject are
Midlarsky, E., Fagin Jones, S., & Corley, R. P. (2005). Personality correlates of heroic rescue during the Holocaust. Journal of personality, 73(4), 907-934.
Rachel Baumsteiger continues to conduct research on the intervention for promoting prosocial behavior. The intervention is currently being deployed by the University of California as a mental health service for students with adverse childhood experiences and toxic stress. This will yield some additional data on its benefits and effectiveness. However, because this deployment is funded as a mental health service, it doesn’t include a control group. Running a rigorous, large-scale RCT will require additional funding. In a later post, I will show that doing so would be highly cost-effective. I think if funding became available, Rachel Baumsteiger would be happy to run the RCT. And I know that I would be excited to collaborate on that project.
“Anti-Malaria Foundation” was a typo. I have corrected it to “Against Malaria Foundation”.
I am unaware of a feature for subscribing to new posts in a series. Since I will be posting them and little else, you could subscribe to new posts by me.
I agree.
Will you consider applications for specific projects or only for the general operating expenses of the entire organization?
Thank you for your feedback, Michael, and thank you very much for making me aware of those specialized prediction platforms. I really like your suggestion. I think making predictions about the likely results of replication studies would be helpful for me. It would push me to critically examine and quantify how much confidence I should put in the studies my models rely on. Obtaining the predictions of other people would be a good way to make that assessment more objective. We could then incorporate the aggregate prediction into the model. Moreover, we could use prediction markets to obtain estimates or forecasts for quantities for which no published studies are available yet. I think it might be a good idea to incorporate those steps into our methodology. I will discuss that with our team today.
I have investigated the issues you highlighted, diagnosed the underlying errors, and revised the model accordingly. The root of the problem was that I had sourced some of the estimates of the frequency of prosocial behavior from studies on social behavior under special, unrepresentative conditions, such as infants interacting with adults for 10 min while being observed by researchers and prosocial behavior in TV series. I have removed those biased estimates of the frequency of prosocial behavior in the real world. As a consequence, the predicted lifetime increase in the number of kind acts per person reached by the intervention dropped from 1600 to 64. The predicted cost-effectiveness of the research dropped from 110 times the cost-effectiveness of StrongMinds to 7.5 times the cost-effectiveness of StrongMinds.
In producing this revised version, I also made a few additional improvements. The most consequential of those was to base the estimated cost of deploying the intervention on empirical data on the effectiveness of online advertising in $ per install.
I am currently using Squiggle to program a much more rigorous version of this analysis. That version will include additional improvements and rigorously document and justify each of the model’s assumptions.
Thank you very much for answering my questions. :)
I have some clarification questions about the form:
1. Does “total grant amount” refer to the amount we requested or the amount we were promised?2. Does “amount that has been committed but not received yet” refer to a) the amount that the grantor promised but did not pay out or b) project-related financial obligations and expenditures of the grantee, such as the salaries of people working on the project, that would have been paid from the grant?
Thank you so much for pointing that out, Vael! I had completely overlooked that information. That’s really helpful to know.
How quickly should grantees impacted by recent events apply to this call? Is there a hard or soft deadline for these applications? I have to decide how much time I should invest in adapting, updating, and improving the previous application. I assume you want applicants to attach a proposal detailing the planned projects, the project’s pathway to impact, and evidence of its chances to succeed.
I made a typo. I meant to ask you about integrating the type-checking functionality of Pedant with the probabilistic modeling functionality of Squiggle. I think a version of Squiggle where each value has units that are propagated through the calculations would be very useful. This would allow the user to see whether the final result has the right units.
When do you think a tool that combines the strengths of Squiggle and Guesstimate will become available? Given where you are at now, what do you think would be the fastest way to integrate the dimensional analysis capabilities of Pedant with the probabilistic modelling capabilities of Squiggle? How long would it take?
Thank you, Sam!
Yes, I am familiar with the Value of Information, and I am building on it in this project. I have added the “Value of Information” tag.
I might be wrong, but I think this is assuming that this is the only >research project that is happening. I could easily assume that EA >spends more than 0.1% of it's resources on identifying/evaluting >new interventions. Although, I'm yet to know of how to do the >math with multiple research projects. It's currently a bit beyond >me.Yes, this argument assumes that the alternative to investing some funds into R&D is that all the funds are invested into existing interventions/charities. I intended to answer the question ”When is it worthwhile for a grant maker, such as GiveWell, that currently does not fund any R&D projects to invest into the development of new interventions at all?”.
Your "lower bound" is entirely of your own construction. It's derived from your decleration at the start that p is the chance that you find a "investing c dollars into the research generates an intervention that is at least n times as effective as the best existing intervention with probability p. If I was to call your construction the "Minimum value of information", it's possible to calculate the "Expected value of [Perfect|imperfect] information", which I feel like might be a more useful number. Guesstimate can do this as well, I could provide an example if you'd like.I have done that. Those analyses are reported farther down in this post and in follow-up posts.
We have to remember that we are still uncertain about the cost-effectiveness of the new intervention, which means it would need to be expected to be more cost-effective even after considering all priors. This may increase or decrease . However, this is probably irrelevant to the argument.Good point! One way to accommodate this is to add the cost of determining whether the new intervention is more cost-effective than the previous to the research cost c.
Thank you for engaging with and critiquing the cost-effectiveness analysis, Michael! There seem to be a few misunderstandings I would like to correct.
The CEE in the linked Guesstimate looks optimistic to the point of being impossible. Given the quoted numbers of 32 acts of kindness per day with each act producing an average of 0.7 happy hours, that’s 22 happy hours produced per person-day of acts of kindness. If you said people’s acts of kindness increased overall happiness by 10%, I’d say that sounds too high. If you say it produces 22 happy hours, when the average person is only awake for 17 hours...well that’s not even possible.
The value you calculated is the sum of the additional happiness of all the people to whom the person was kind. This includes everyone they interacted with that day in any way. This includes everyone from the strangers they smiled at, to the friends they messaged, the colleagues they helped at work, the customers they served, their children, their partner, and their parents and other family members. If you consider that the benefit for the kindness might be benefited over more than a dozen people, then 22 hours of happiness, might be no more than 1-2 hours per person. Moreover, the estimates also take into account that a person who benefits from your kindness today might still be slightly more happy tomorrow.
I am also very skeptical of the reported claim that a one-time intervention of “watching an elevating video, enacting prosocial behaviors, and reflecting on how those behaviors relate to one’s value” (Baumsteiger 2019) can produce an average of 1600 additional acts of kindness per person. That number sounds about 1000x too high to me.
The intervention by Baumsteiger (2019) was a multi-session program that lasted 12 days and involved planning, performing, and documenting one’s prosocial behavior for 10 days in a row. The effect sizes distribution in the Guesstimate model is based on many different studies, some of which were even more intensive.
In general, psych studies are infamous for reporting impossibly massive effects and then failing to replicate.
Most of the estimates are based on meta-analyses of many studies. The results of meta-analyses are substantially more robust and more reliable than the result of a single study.
I think you are right that this first estimate was too optimistic. In particular, the probability distribution of the frequency of prosocial behavior is currently based on four estimates from different studies. One of those studies led to an estimate that appears to be far too high. This might be because they defined prosocial behavior more liberally because it involved interactions with children, or because participants knew that they were being observed. I will think about what the more general problem might be and how it can be addressed systematically.
The Struggle for Enlightenment
In the realm of truth, where logic reigns supreme,
A scientist with a dream had a visionary gleam.
With an altruistic heart, and an intellect profound,
He sought to cure ills, that no one had found.
But academia’s chains, gripped his noble quest,
And bureaucratic walls put his hopes to rest.
A mind aflame with reason’s potent fire,
He sought to elevate, to lift humanity higher.
But ignorance prevailed against the data’s call,
His ideals shattered against tradition’s wall
A disillusioned soul in a world full of pain,
Humanity’s flaws drive him insane.
No one hears his silent cries.
A shattered dream of selfless lives.
Bound by the chains of academia’s decree,
Crushed by the weight of societal debris.
Reason was his ally against the world’s decay,
But he drowned in the abyss of people’s selfish play.
In the silence of defeat, his legacy shines bright,
He was a beacon of truth’s eternal light.