Thanks for the analysis, Hauke! I strongly upvoted it.
The mean “CCEI’s effect of shifting deploy$ to RD&D$” of 5 % you used in UseCarlo is 12.5 (= 0.05/0.004) times the mean of 0.4 % respecting your Guesstimate model. Which one do you stand by? Since you say “CCEI is part of a much smaller coalition of only hundreds of key movers and shakers”, the smaller effect of 0.4 % (= 1⁄250) would be more appropriate assuming the same contribution for each member of such coalition.
I think you had better estimate the expected cost-effectiveness in t/$ instead of $/t:
The expected benefits in t are equal to the product between the cost and expected cost-effectiveness in t/$[1], not to the ratio between the cost and expected cost-effectiveness in $/t[2].
I appreciate the cost-effectiveness you present in your results table was correctly obtained with the 1st of the above methods. However, people could interpret it as referring to the mean cost per benefit, which would not be correct (since E(1/X) is not equal to 1/E(X)).
In your Guesstime model, you estimate the expected cost per benefit, which is not directly comparable to the expected benefit per cost that you calculated with UseCarlo.
The benefits can often be 0, thus resulting in numerical instabilities in the cost-effectiveness in $/t, although this does not apply to your case.
Regarding CCEI’s effect of shifting deploy$ to RD&D$:
Yes, in the Guesstimate model the confidence intervals went from 0.1% to 1% lognormally distributed, with a mean of ~0.4%
With UseCarlo I used a metalog distribution with parameters 0%, 0.1%, 2%, 10%, resulting in a mean of ~5%
So you’re right, there is indeed about an order of magnitude difference between the two estimates:
This is mostly driven by my assigning some credence to the possibility that CCEI might have had as much as a 10% influence, which I wouldn’t rule out entirely.
However, the confidence intervals of the two estimates are overlapping.
I agree this is the weakest part of the analysis. As I highlighted, it’s a guesstimate motivated by the qualitative analysis that CCEI is part of the coalition of key movers and shakers that shifted budget increases to energy RD&D.
I think both estimates are roughly valid given the information available. Without further analysis, I don’t have enough precision to zero in on the most likely value.
I lost access to UseCarlo during the writeup and the after the analysis was delayed for quite some time (I had initially pitched it to FTX as an Impact NFT).
I just wanted to get the post out rather than delay further. With more resources, one could certainly dig deeper and make the analysis more rigorous and detailed. But I hope it provides a useful starting point for discussion and further research.
One could further nuance this analysis e.g. by calculating marginal effect of our $1M on US climate policy philanthropy at the current ~$55M level vs. what it’s now.
Thanks also for the astute observation about estimating expected cost-effectiveness in t/$ vs $/t. You raise excellent points and I agree it would be more elegant to estimate it as t/$ for the reasons you outlined.
I really appreciate you taking the time to engage substantively with the post.
Thanks for the analysis, Hauke! I strongly upvoted it.
The mean “CCEI’s effect of shifting deploy$ to RD&D$” of 5 % you used in UseCarlo is 12.5 (= 0.05/0.004) times the mean of 0.4 % respecting your Guesstimate model. Which one do you stand by? Since you say “CCEI is part of a much smaller coalition of only hundreds of key movers and shakers”, the smaller effect of 0.4 % (= 1⁄250) would be more appropriate assuming the same contribution for each member of such coalition.
I think you had better estimate the expected cost-effectiveness in t/$ instead of $/t:
The expected benefits in t are equal to the product between the cost and expected cost-effectiveness in t/$[1], not to the ratio between the cost and expected cost-effectiveness in $/t[2].
I appreciate the cost-effectiveness you present in your results table was correctly obtained with the 1st of the above methods. However, people could interpret it as referring to the mean cost per benefit, which would not be correct (since E(1/X) is not equal to 1/E(X)).
In your Guesstime model, you estimate the expected cost per benefit, which is not directly comparable to the expected benefit per cost that you calculated with UseCarlo.
The benefits can often be 0, thus resulting in numerical instabilities in the cost-effectiveness in $/t, although this does not apply to your case.
E(“benefits (t)”) = E(“cost ($)”*”cost-effectiveness (t/$)”) = “cost ($)”*E(“cost-effectiveness (t/$)”).
E(“benefits (t)”) = E(“cost ($)”/”cost-effectiveness ($/t)”) = “cost ($)”*E(1/”cost-effectiveness ($/t)”) != “cost ($)”/E(“cost-effectiveness ($/t)”).
Great comment—thanks so much!
Regarding CCEI’s effect of shifting deploy$ to RD&D$:
Yes, in the Guesstimate model the confidence intervals went from 0.1% to 1% lognormally distributed, with a mean of ~0.4%
With UseCarlo I used a metalog distribution with parameters 0%, 0.1%, 2%, 10%, resulting in a mean of ~5%
So you’re right, there is indeed about an order of magnitude difference between the two estimates:
This is mostly driven by my assigning some credence to the possibility that CCEI might have had as much as a 10% influence, which I wouldn’t rule out entirely.
However, the confidence intervals of the two estimates are overlapping.
I agree this is the weakest part of the analysis. As I highlighted, it’s a guesstimate motivated by the qualitative analysis that CCEI is part of the coalition of key movers and shakers that shifted budget increases to energy RD&D.
I think both estimates are roughly valid given the information available. Without further analysis, I don’t have enough precision to zero in on the most likely value.
I lost access to UseCarlo during the writeup and the after the analysis was delayed for quite some time (I had initially pitched it to FTX as an Impact NFT).
I just wanted to get the post out rather than delay further. With more resources, one could certainly dig deeper and make the analysis more rigorous and detailed. But I hope it provides a useful starting point for discussion and further research.
One could further nuance this analysis e.g. by calculating marginal effect of our $1M on US climate policy philanthropy at the current ~$55M level vs. what it’s now.
Thanks also for the astute observation about estimating expected cost-effectiveness in t/$ vs $/t. You raise excellent points and I agree it would be more elegant to estimate it as t/$ for the reasons you outlined.
I really appreciate you taking the time to engage substantively with the post.