Expected value estimates we (cautiously) took literally—Oxford Prioritisation Project
By Tom Sittler
Cross-posted from the Oxford Prioritisation Project blog. We’re centralising all discussion on the Effective Altruism forum. To discuss this post, please comment here.
Summary: This post describes how we turned the cost-effectiveness estimates of our four shortlisted organisations into a final decision. In order to give adequately more weight to more robust estimates, we use the four model outputs to update a prior distribution over grantee organisation impacts.
Inspired by Michael Dickens’ example, we decided to formalise the intuition that more robust estimates should get higher weights. We did this by treating the outputs of our models as providing an evidence distribution, which we use to update a prior distribution over the cost-effectiveness of potential grantee organisations.
The code we used to compute the Bayesian posterior estimates is here.
The prior distribution
This is the unconditional (prior) probability distribution of the cost-effectiveness of potential grantees. We use a lognormal distribution. A theoretical justification for this is that we expect cost-effectiveness to be the result of a multiplicative rather than additive process. A possible empirical justification could be the distribution of cost-effectiveness of DCP-2 interventions. Again, this has been discussed at length elsewhere.
The parameters of a lognormal distribution are its scale and location. The scale is equal to the standard deviation of the natural logarithm of the values, and the location is equal to the mean of the natural logarithm of the values. The median of a lognormal distribution is the exponential of its location.
We choose the location parameter such that the median of the distribution is as cost-effective as highly effective global health interventions such a those recommended by GiveWell, which we estimate to provide a QALY for $50. Intuitively, this means that the set of organisations we were considering funding at the start of the project had a median cost-effectiveness of 0.02 QALYs/$.
We set the scale parameter as 0.5, which means that the standard deviation of the natural logarithm of our prior is 25 times the mean of the natural logarithm of our prior. This is a relatively poorly informed guess, which we arrived at mostly by looking at the choices of Michael Dickens and checking that they did not intuitively seem absurd to team members.
Had we chosen a scale parameter more than about 2.2 times as large, the Machine Intelligence Research Institute would have had the highest posterior cost-effectiveness estimate.
The evidence distribution
We fit the outputs of our models, which are lists of numbers, to a lognormal probability distribution. The fit is excellent, as you can see from the graphs below. On the log scale, the probability density function of our original data appears in black and the probability density function of data randomly generated from the lognormal distribution we fitted to the original data appears in red.
This is the graph for StrongMinds:
And the graph for MIRI:
The other graphs look very similar, so I’m not including them here. You can generate them using the code I provide.
What about negative values?
The models for Animal Charity Evaluators and Machine Intelligence Research institute contain negative values, so they cannot be fitted to a lognormal distribution.
Instead, we split the data into a positive and a negative lognormal, which we update separately on a positive and a negative prior.
Intuitively, we think that both interventions that do a large amount of good (in the tail of the positive prior) and interventions that do a large amount of hard (in the tail of the negative prior) are unlikely in priors.
Updating when distributions are lognormal
In my other post, I derive a closed-form solution to the problem of updating a lognormal prior using a lognormal evidence distribution.
A word on units: inside each of the four models, we convert all estimates to “Human-equivalent well-being-adjusted life-years” (HEWALYs). One HEWALY is a QALY, or a year of life as a fully healthy, modern-day human. If an action produces zero HEWALYs, we are indifferent between doing it and not doing it. Negative HEWALYs correspond to lives not worth living, and −1 HEWALY is as bad as 1 HEWALY is good. In other words, we are indifferent between causing 0 HEWALYs and causing both 1 HEWALY and −1 HEWALY.
A being can accrue more than 1 HEWALY per year, because life can be better than life as a fully healthy modern-day human. Symmetrically, a being can accrue less than −1 HEWALY per year.
You can view the results and code here. If you disagree with our prior parameters, we encourage you to try our own values and see what you come up with, in the style of GiveWell, who provide their parameters as estimated by each staff member. We also include commented-out code to visualise how the posterior estimates depend on the prior parameters.
Our prior strongly punishes MIRI. While the mean of its evidence distribution is 2,053,690,000 HEWALYs/$10,000, the posterior mean is only 180.8 HEWALYs/$10,000. If we set the prior scale parameter to larger than about 1.09, the posterior estimate for MIRI is greater than 1038 HEWALYs/$10,000, thus beating 80,000 Hours.
Our estimate of StrongMinds is lower than our prior. The StrongMinds evidence distribution had a mean 17.9 HEWALYs/$10,000 which is lower than the posterior of 18.5 HEWALYs/$10,000. We can interpret this in the following way: we found evidence that StrongMinds is has surprisingly (relative to our prior) low cost-effectiveness, so taking into account the prior leads us to increase our estimate of StrongMinds.