It took me a while to fully parse, but here are my thoughts, let me know if I misunderstood something.
I/ Re the 3000x example, I think I wasn’t particularly clear in the talk and this is a misunderstanding resulting from that. You’re right to point out that the expected uncertainty is not 3000x.
I meant this more to quickly demonstrate that if you put a couple of uncertainties together it quickly becomes quite hard to evaluate whether something meets a given bar, the range of outcomes is extremely large (if on regression to the mean the uncertainty in the example goes to ~40x, as you suggest, this is still not that helpful to evaluate whether something meets a bar).
And if we did this for a realistic case, e.g. something with 7+ uncertainties then this would be clear even taking into account regression to the mean. In our current trial runs, we definitely see differences that are significantly larger than 40x, i.e. the two options discussed in the talk do indeed look fairly similar in our model of the overall impact space.
II/ In general, we do not assume that all uncertainties are independent or that they all have the same distribution, right now we’ve enabled normal, lognormal and uniform distributions as well as correlations between all variables. For technological change, log normal distributions appear a good approximation of the data but this is not necessarily the same for other variables.
III/ “However, I would say we should conceptually care about E(“cost-effectiveness of A”)/E(“cost-effectiveness of B”), not E(“cost-effectiveness of A”/”cost-effectiveness of B”), where E is the expected value.”
Yes, in general we care about E(CE of A/CE of B). The different decomposition in the talk comes from the specific interest in that case, illustrating that even if we are quite uncertain in general about absolute values, we can make relatively confident statements about dominance relations, e.g. that in 91% of worlds a given org dominates even though the first intuitive reaction to the visualization would be “ah, this is all really uncertain, can we really know anything?”.
IV/ The less formalized versions of this overall framework have indeed already influenced a lot of other FP work in other cause areas, e.g. on bio and nuclear risk and also air pollution, and I do expect that some of the tools we are developing will indeed diffuse to other parts of the research team and potentially to other orgs (we aim to publish the code as well). This is very intentional, we try to push methodology forward in mid-to-high uncertainty contexts where little is published so far.
V/ Most of the donors to the Climate Fund are indeed not cause-neutral EAs and we mostly target non-EA audiences.
Yes, in general we care about E(CE of A/CE of B).
I meant we should in theory just care about r = E(“CE of A”)/E(“CE of B”)[1], and pick A over B if the expected cost-effectiveness of A is greater than that of B (i.e. if r > 1), even if A was worse than B in e.g. 90 % of the worlds. In practice, if A is better than B in 90 % of the worlds (in which case the 10th precentile of “CE of A”/”CE of B” would be 1), r will often be higher than 1, so focussing on r or E(“CE of A”/”CE of B”) will lead to the same decisions.
If r is what matters, to investigate whether one’s decision to pick A over B is robust, the aim of the sensitivity analysis would be ensuring that r > 1 under various plausible conditions. So, instead of checking whether the CE of A is often higher than the CE of B, one should be testing whether the expected CE of A if often higher than the expected CE of B.
In practice, it might be the case that:
If r > 1 and A is better than B in e.g. 90 % of the worlds, then the conclusion that r > 1 is robust, i.e. we can be confident that A will continue to be better than B upon further investigation.
If r > 1 and A is better than B in e.g. just 25 % of the worlds, then the conclusion that r > 1 is not robust, i.e. we cannot be confident that A will continue to be better than B upon further investigation.
In this piece, we tried to characterize the problem we face when making claims about expected impacts in a high-uncertainty environment such as climate philanthropy.
How do you think about adaptation (e.g. economic growth, adoption of air conditioning, and migration)? I forgot to finish this sentence in my last comment.
Hi Vasco,
Thanks for your thoughtful comment!
It took me a while to fully parse, but here are my thoughts, let me know if I misunderstood something.
I/ Re the 3000x example, I think I wasn’t particularly clear in the talk and this is a misunderstanding resulting from that. You’re right to point out that the expected uncertainty is not 3000x.
I meant this more to quickly demonstrate that if you put a couple of uncertainties together it quickly becomes quite hard to evaluate whether something meets a given bar, the range of outcomes is extremely large (if on regression to the mean the uncertainty in the example goes to ~40x, as you suggest, this is still not that helpful to evaluate whether something meets a bar).
And if we did this for a realistic case, e.g. something with 7+ uncertainties then this would be clear even taking into account regression to the mean. In our current trial runs, we definitely see differences that are significantly larger than 40x, i.e. the two options discussed in the talk do indeed look fairly similar in our model of the overall impact space.
II/ In general, we do not assume that all uncertainties are independent or that they all have the same distribution, right now we’ve enabled normal, lognormal and uniform distributions as well as correlations between all variables. For technological change, log normal distributions appear a good approximation of the data but this is not necessarily the same for other variables.
III/
“However, I would say we should conceptually care about E(“cost-effectiveness of A”)/E(“cost-effectiveness of B”), not E(“cost-effectiveness of A”/”cost-effectiveness of B”), where E is the expected value.”
Yes, in general we care about E(CE of A/CE of B). The different decomposition in the talk comes from the specific interest in that case, illustrating that even if we are quite uncertain in general about absolute values, we can make relatively confident statements about dominance relations, e.g. that in 91% of worlds a given org dominates even though the first intuitive reaction to the visualization would be “ah, this is all really uncertain, can we really know anything?”.
IV/ The less formalized versions of this overall framework have indeed already influenced a lot of other FP work in other cause areas, e.g. on bio and nuclear risk and also air pollution, and I do expect that some of the tools we are developing will indeed diffuse to other parts of the research team and potentially to other orgs (we aim to publish the code as well). This is very intentional, we try to push methodology forward in mid-to-high uncertainty contexts where little is published so far.
V/ Most of the donors to the Climate Fund are indeed not cause-neutral EAs and we mostly target non-EA audiences.
Thanks for the clarifications, Johannes!
I meant we should in theory just care about r = E(“CE of A”)/E(“CE of B”)[1], and pick A over B if the expected cost-effectiveness of A is greater than that of B (i.e. if r > 1), even if A was worse than B in e.g. 90 % of the worlds. In practice, if A is better than B in 90 % of the worlds (in which case the 10th precentile of “CE of A”/”CE of B” would be 1), r will often be higher than 1, so focussing on r or E(“CE of A”/”CE of B”) will lead to the same decisions.
If r is what matters, to investigate whether one’s decision to pick A over B is robust, the aim of the sensitivity analysis would be ensuring that r > 1 under various plausible conditions. So, instead of checking whether the CE of A is often higher than the CE of B, one should be testing whether the expected CE of A if often higher than the expected CE of B.
In practice, it might be the case that:
If r > 1 and A is better than B in e.g. 90 % of the worlds, then the conclusion that r > 1 is robust, i.e. we can be confident that A will continue to be better than B upon further investigation.
If r > 1 and A is better than B in e.g. just 25 % of the worlds, then the conclusion that r > 1 is not robust, i.e. we cannot be confident that A will continue to be better than B upon further investigation.
How do you think about adaptation (e.g. economic growth, adoption of air conditioning, and migration)? I forgot to finish this sentence in my last comment.
Note E(X/Y) is not equal to E(X)/E(Y).