Error

This is my favorite in the series so far. I really enjoy the tacit knowledge flavor of it and that some of these lessons generalize beyond the CBA domain.

Google Sheets also has a criminally under appreciated tool—Google App Script

You can write custom code, as Javascript, and have it run on a schedule, or as a function in a spreadsheet cell, or as a clickable menu item

I do a lot of modelling in my job, and I have to say this is the best tacit knowledge piece I’ve read on modelling in a while (the MC gsheet template is a nice bonus too). Bookmarked for (I expect) frequent future reference. Thanks Richard.

This is fantastic thank you! Have already sent it to someone considering dong a CBA

“For any purpose other than an example calculation, never use a point estimate. Always do all math in terms of confidence intervals. All inputs should be ranges or probability distributions, and all outputs should be presented as confidence intervals.”

I weakly disagree with this “never” statement, as I think there is value in doing basic cost-benefit analysis without confidence intervals, especially for non mathsy indivuals or small orgs who want to look at potential cost effectiveness of their own or other’s interventions. I wouldn’t want to put some people off by setting this as a “minimum” bar. I also think that simple “lower and upper bound” ranges can sometimes be an easier way to do estimates, without strictly needing to calculate a confidence interval.

In saying that when, big organisations do CBA’s to actually make decisions or move large amounts of money, or for any academic purpose then yes I agree confindence intervals are what’s needed!

I would also say that for better or worse (probably for worse) the point estimate is by far the most practically discussed and used application of any CBA so I think its practially important to put more effort into getting your point estimate as accurate as possible, then it is to make sure you’re range is accurate.

Nice job again.

Agree with others. This is fantastic. Learned a lot.

Agree with others. This is fantastic. Learned a lot.

Executive summary: Effective cost-benefit analysis requires probabilistic thinking, transparency, iteration, and clear communication tailored to the audience, with an emphasis on real-world costs and investigative methods.

Key points:

1. Use Probabilistic Thinking – Always work with confidence intervals and probability distributions rather than point estimates.

2. Prioritize Transparency and Accessibility – Google Sheets enhances collaboration, reviewability, and comprehension for a broad audience.

3. Start Small and Iterate – Begin with a simple, minimal analysis and refine based on available data and stakeholder needs.

4. Investigate Real-World Costs – Use journalistic methods to gather practical cost estimates from industry experts rather than relying solely on academic sources.

5. Tailor Communication to Decision-Makers – Focus on clear, actionable insights rather than academic rigor, emphasizing practical impact over technical precision.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

Thanks for the post, Richard.

For any purpose other than an example calculation, never use a point estimate. Always do all math in terms of confidence intervals. All inputs should be ranges or probability distributions, and all outputs should be presented as confidence intervals.

I have run lots of Monte Carlo simulations, but have mostly moved away from them. I strongly endorse maximising expected welfare, so I think the final point estimate of the expected cost-effectiveness is all that matters in principle if it accounts for all the considerations. In practice, there are other inputs that matter because not all considerations will be modelled in that final estimate. However, I do not see this as an argument for modelling uncertainty per se. I see it as an argument for modelling the considerations which are currently not covered, at least informally (more implicitly), and ideally formally (more explicitly), such that the final point estimate of the expected cost-effectiveness becomes more accurate.

That being said, I believe modelling uncertainty is useful if it affects the estimation of the final expected cost-effectiveness. For example, one can estimate the expected effect size linked to a set of RCTs with inverse-variance weighting from w_1*e_1 + w_2*e_2 + … + w_n*e_n, where w_i and e_i are the weight and expected effect size of study i, and w_i = 1/​”variance of the effect size of study i”/​(1/​”variance of the effect size of study 1“ + 1/​”variance of the effect size of study 2” + … + 1/​”variance of the effect size of study n”). In this estimation, the uncertainty (variance) of the effect sizes of the studies matters because it directly affects the expected aggregated effect size.

Holden Karnofsky’s post Why we can’t take expected value estimates literally (even when they’re unbiased) is often mentioned to point out that unbiased point estimates do not capture all information. I agree, but the clear failures of point estimates described in the post can be mitigated by adequately weighting priors, as is illustrated in the post. Applying inverse-variance weighting, the final expected cost-effectiveness is “mean of the posterior cost-effectiveness” = “weight of the prior”*”mean of the prior cost-effectiveness” + “weight of the estimate”*”mean of the estimated cost-effectiveness” = (“mean of the prior cost-effectiveness”/​”variance of the prior cost-effectiveness” + “mean of the estimated cost-effectiveness”/​”variance of the estimated cost-effectiveness”)/​(1/​”variance of the prior cost-effectiveness” + 1/​”variance of the estimated cost-effectiveness”). If the estimated cost-effectiveness is way more uncertain than the prior cost-effectiveness, the prior cost-effectiveness will be weighted much more heavily, and therefore the final expected cost-effectiveness, which integrates information about the prior and estimated cost-effectiveness, will remain close to the prior cost-effectiveness.

It is still important to ensure that the final point estimate for the expected cost-effectiveness is unbiased. This requires some care in converting input distributions to point estimates, but Monte Carlo simulations requiring more than one distribution can very often be avoided. For example, if “cost-effectiveness” = (“probability of success”*”years of impact given success” + (1 - “probability of success”)*”years of impact given failure”)*”number of animals that can be affected”*”DALYs averted per animal-year improved”/​”cost”, and all these variables are independent (as usually assumed in Monte Carlo simulations for simplicity), the expected cost-effectiveness will be E(“cost-effectiveness”) = (“probability of success”*E(“years of impact given success”) + (1 - “probability of success”)*E(“years of impact given failure”))*E(“number of animals that can be affected”)*E(“DALYs averted per animal-year improved”)*E(1/​”cost”). This is because E(“constant a”*”distribution X” + “constant b”) = a*E(X) + b, and E(X*Y) = E(X)*E(Y) if X and Y are independent. Note:

• The input distributions should be converted to point estimates corresponding to their means.

  • You can make a copy of this sheet (presented here) to calculate the mean of uniform, normal, loguniform, lognormal, pareto and logistic distributions from 2 of their quantiles. For example, if “years of impact given success” follows a lognormal distribution with 5th and 95th percentiles of 3 and 30 years, one should set the cell B2 to 0.05, C2 to 0.95, B3 to 3, and C3 to 30, and then check E(“years of impact given success”) in cell C22, which is 12.1 years.

  • Replacing an input by its most likely value (its mode), or one which is as likely to be an underestimate as an overestimate (its median) may lead to a biased expected cost-effectiveness. For example, the median and mode of a lognormal distribution are always lower than its mean. So, if “years of impact given success” followed such distribution, replacing it with its most likely value, or one as likely to be too low as too high would result in underestimating the expected cost-effectiveness.

• The expected cost-effectiveness is proportional to E(1/​”cost”), which is only equal to 1/​E(“cost”) if “cost” is a constant, or practically equal if it is a fairly certain distribution compared to others influencing the cost-effectiveness. If “cost” is too uncertain to be considered constant, and there is not a closed formula to determine E(1/​”cost”) (there would be if “cost” followed a uniform distribution), one would have to run a Monte Carlo simulation to compute E(1/​”cost”), but it would only involve the distribution of the cost. For uniform, normal and lognormal distributions, Guesstimate would do. For other distributions, you can try Squiggle AI (I have not used it, but it seems quite useful).