Error

Thanks for your entry!

This is really nice, major kudos for a) the result, b) pulling the research colaboration off. Also keen to see people build upon these, e.g., with the optimizer’s curse.

To get estimates comparable to each other, GiveWell CEAs use many variables that are the same across different charities. These include moral variables such as the discount rate and the moral weights, as well as a number or adjustment factors.

Because each of our CEAs was built in its separate notebook. Variables that should have more correlated uncertainty do not correlate.

Correlated uncertainty driven by using the same parameters across models: I’d love to see this, I expect it will effect the VOI calculations.

Also could be a good gateway towards the more difficult correlated uncertainty: the correlation between parameters within a model. I expect some of these to be highly correlated, and this might matter a lot. In fact, in the fistula model, some of these might largely be different takes on the same uncertain underlying variable? Here one might need to report on the VOI for something like a ‘cluster of variables’?

But this is more challenging; I guess it would probably start with some calibrated judgment/​educated guessing on the correlation parameter.

I think maybe the issue of correlated uncertainty is the biggest limitation to bringing uncertainty into the models. If the uncertainties are ‘super correlated in the same direction’ then the naive ‘worst case/​best case’ for everything is closer to being correct. If the uncertainties are ‘super negatively correlated’ then we may need to worry less about uncertainty, or perhaps worry about the largest of the uncertainties.

we use our uncertainty quantification to calculate the Expected Value of Perfect Information on all GiveWell Top Charities, and find that the cost would outweigh the benefits if GiveWell were to spend more than 11.25% of their budget on research for this information.

Small point: I would replace ‘budget’ with ‘funds they expect to influence’; considering the time horizon for this is difficult though, it probably requires a model of how the benefit of the research evolves over time, and some sort of explore/​exploit modeling;

Finger-exercise: If we think GW influences 500 million per year over the next 10 years, or 5 billion in total, 11% is 550 million, 1% 55 million. … or 5.5 million per year. (But as I said in the other comment, I don’t know where that ‘1/​10th rule’ came from).

I guess this assumes that all of the money they shift are allocated to the single top-EV charity.

https://​​www.givingwhatwecan.org/​​charities/​​givewell-unrestricted-fund suggests their operating expenses are about $11 million per year. But:

• that’s not all research obviously

• on the other hand, there are many others doing research in this space

• on the other hand, there are benefits outside GW

But of course this doesn’t count the VOI for the arrival of new charities, which could justify a greater expense.

Is there any basis for Hubbard’s ‘spend 1/​10th of the max’ rule of thumb? It seems arbitrary. Is there any evidence or information-theoretic justification for this?

On a related note, one could make a case that the ‘benefit of research for informing Givewell’s decisions’ might understate the global benefits, if the research informs other research and decisionmaking outside of GW.

The VOI simulation calculation discussion you give (“This expectation of value obtainable in the informed state) is super-interesting to me, not sure I’ve seen this laid out before, I love it!.

Maybe you could add 1 or 2 more lines or footnotes to make it a bit clearer; I had to think through ‘yeah these distributions represent our beliefs over the probability of each actual value for each charity … so sampling from these reflects what we can expect to learn.’ Maybe it just took me a moment because I didn’t start out as a Bayesian.

Also, I might state the VOI as

In the context of choosing among discrete alternatives, information is more valuable when:

It is more likely to change our mind about the best alternative
the value of the alternatives we are choosing among is likely to be larger The environment is high-stakes, implying a greater potential for producing value or harm, given the resources devoted to it.

I would want to disambiguate this so as to not be confused with a case where the context is high-stakes, but we are trying to choose between ‘purple bednets and pink bednets’ (nearly equivalent value)

What do you think?

I see a potential ‘buried headline’ here. It seems as though Helen Keller more or less ‘dominates’ the other charities!

• even its bottom 10th percentile is above the 50th percentile for all the others, and

• is above the 75th percentile of all but New Incentives (and the latter has extremely wide intervals).

Stepping outside the model itself, how confident are you in this result?