For any purpose other than an example calculation, never use a point estimate. Always do all math in terms of confidence intervals. All inputs should be ranges or probability distributions, and all outputs should be presented as confidence intervals.
I have run lots of Monte Carlo simulations, but have mostly moved away from them. I strongly endorse maximising expected welfare, so I think the final point estimate of the expected cost-effectiveness is all that matters in principle if it accounts for all the considerations. In practice, there are other inputs that matter because not all considerations will be modelled in that final estimate. However, I do not see this as an argument for modelling uncertainty per se. I see it as an argument for modelling the considerations which are currently not covered, at least informally (more implicitly), and ideally formally (more explicitly), such that the final point estimate of the expected cost-effectiveness becomes more accurate.
That being said, I believe modelling uncertainty is useful if it affects the estimation of the final expected cost-effectiveness. For example, one can estimate the expected effect size linked to a set of RCTs with inverse-variance weighting from w_1*e_1 + w_2*e_2 + ⌠+ w_n*e_n, where w_i and e_i are the weight and expected effect size of study i, and w_i = 1/ââvariance of the effect size of study iâ/â(1/ââvariance of the effect size of study 1â + 1/ââvariance of the effect size of study 2â + ⌠+ 1/ââvariance of the effect size of study nâ). In this estimation, the uncertainty (variance) of the effect sizes of the studies matters because it directly affects the expected aggregated effect size.
Holden Karnofskyâs post Why we canât take expected value estimates literally (even when theyâre unbiased) is often mentioned to point out that unbiased point estimates do not capture all information. I agree, but the clear failures of point estimates described in the post can be mitigated by adequately weighting priors, as is illustrated in the post. Applying inverse-variance weighting, the final expected cost-effectiveness is âmean of the posterior cost-effectivenessâ = âweight of the priorâ*âmean of the prior cost-effectivenessâ + âweight of the estimateâ*âmean of the estimated cost-effectivenessâ = (âmean of the prior cost-effectivenessâ/ââvariance of the prior cost-effectivenessâ + âmean of the estimated cost-effectivenessâ/ââvariance of the estimated cost-effectivenessâ)/â(1/ââvariance of the prior cost-effectivenessâ + 1/ââvariance of the estimated cost-effectivenessâ). If the estimated cost-effectiveness is way more uncertain than the prior cost-effectiveness, the prior cost-effectiveness will be weighted much more heavily, and therefore the final expected cost-effectiveness, which integrates information about the prior and estimated cost-effectiveness, will remain close to the prior cost-effectiveness.
It is still important to ensure that the final point estimate for the expected cost-effectiveness is unbiased. This requires some care in converting input distributions to point estimates, but Monte Carlo simulations requiring more than one distribution can very often be avoided. For example, if âcost-effectivenessâ = (âprobability of successâ*âyears of impact given successâ + (1 - âprobability of successâ)*âyears of impact given failureâ)*ânumber of animals that can be affectedâ*âDALYs averted per animal-year improvedâ/ââcostâ, and all these variables are independent (as usually assumed in Monte Carlo simulations for simplicity), the expected cost-effectiveness will be E(âcost-effectivenessâ) = (âprobability of successâ*E(âyears of impact given successâ) + (1 - âprobability of successâ)*E(âyears of impact given failureâ))*E(ânumber of animals that can be affectedâ)*E(âDALYs averted per animal-year improvedâ)*E(1/ââcostâ). This is because E(âconstant aâ*âdistribution Xâ + âconstant bâ) = a*E(X) + b, and E(X*Y) = E(X)*E(Y) if X and Y are independent. Note:
The input distributions should be converted to point estimates corresponding to their means.
You can make a copy of this sheet (presented here) to calculate the mean of uniform, normal, loguniform, lognormal, pareto and logistic distributions from 2 of their quantiles. For example, if âyears of impact given successâ follows a lognormal distribution with 5th and 95th percentiles of 3 and 30 years, one should set the cell B2 to 0.05, C2 to 0.95, B3 to 3, and C3 to 30, and then check E(âyears of impact given successâ) in cell C22, which is 12.1 years.
Replacing an input by its most likely value (its mode), or one which is as likely to be an underestimate as an overestimate (its median) may lead to a biased expected cost-effectiveness. For example, the median and mode of a lognormal distribution are always lower than its mean. So, if âyears of impact given successâ followed such distribution, replacing it with its most likely value, or one as likely to be too low as too high would result in underestimating the expected cost-effectiveness.
The expected cost-effectiveness is proportional to E(1/ââcostâ), which is only equal to 1/âE(âcostâ) if âcostâ is a constant, or practically equal if it is a fairly certain distribution compared to others influencing the cost-effectiveness. If âcostâ is too uncertain to be considered constant, and there is not a closed formula to determine E(1/ââcostâ) (there would be if âcostâ followed a uniform distribution), one would have to run a Monte Carlo simulation to compute E(1/ââcostâ), but it would only involve the distribution of the cost. For uniform, normal and lognormal distributions, Guesstimate would do. For other distributions, you can try Squiggle AI (I have not used it, but it seems quite useful).
Doing the Monte Carlo using my sheet is easier than the method you presented for avoiding the Monte Carlo. It presents the mean, which is the expected value, and also the confidence interval.
There are some audiences that already understand uncertainty and have a SBF-style desire to only maximize expected utility. These audiences are rare. Most people need to be shown the uncertainty (even if they do not yet know they need it).
Some people will want or need to take the âsafe optionâ with a higher floor rather than try to maximize the expected value.
When done right, the confidence interval includes uncertainty in implementation. If it is done by an A-team that gets things right, you will get better results. Knowing the possible range is key to know how fragile the expected result is and how much care will be required to get things right.
Thanks for the post, Richard.
I have run lots of Monte Carlo simulations, but have mostly moved away from them. I strongly endorse maximising expected welfare, so I think the final point estimate of the expected cost-effectiveness is all that matters in principle if it accounts for all the considerations. In practice, there are other inputs that matter because not all considerations will be modelled in that final estimate. However, I do not see this as an argument for modelling uncertainty per se. I see it as an argument for modelling the considerations which are currently not covered, at least informally (more implicitly), and ideally formally (more explicitly), such that the final point estimate of the expected cost-effectiveness becomes more accurate.
That being said, I believe modelling uncertainty is useful if it affects the estimation of the final expected cost-effectiveness. For example, one can estimate the expected effect size linked to a set of RCTs with inverse-variance weighting from w_1*e_1 + w_2*e_2 + ⌠+ w_n*e_n, where w_i and e_i are the weight and expected effect size of study i, and w_i = 1/ââvariance of the effect size of study iâ/â(1/ââvariance of the effect size of study 1â + 1/ââvariance of the effect size of study 2â + ⌠+ 1/ââvariance of the effect size of study nâ). In this estimation, the uncertainty (variance) of the effect sizes of the studies matters because it directly affects the expected aggregated effect size.
Holden Karnofskyâs post Why we canât take expected value estimates literally (even when theyâre unbiased) is often mentioned to point out that unbiased point estimates do not capture all information. I agree, but the clear failures of point estimates described in the post can be mitigated by adequately weighting priors, as is illustrated in the post. Applying inverse-variance weighting, the final expected cost-effectiveness is âmean of the posterior cost-effectivenessâ = âweight of the priorâ*âmean of the prior cost-effectivenessâ + âweight of the estimateâ*âmean of the estimated cost-effectivenessâ = (âmean of the prior cost-effectivenessâ/ââvariance of the prior cost-effectivenessâ + âmean of the estimated cost-effectivenessâ/ââvariance of the estimated cost-effectivenessâ)/â(1/ââvariance of the prior cost-effectivenessâ + 1/ââvariance of the estimated cost-effectivenessâ). If the estimated cost-effectiveness is way more uncertain than the prior cost-effectiveness, the prior cost-effectiveness will be weighted much more heavily, and therefore the final expected cost-effectiveness, which integrates information about the prior and estimated cost-effectiveness, will remain close to the prior cost-effectiveness.
It is still important to ensure that the final point estimate for the expected cost-effectiveness is unbiased. This requires some care in converting input distributions to point estimates, but Monte Carlo simulations requiring more than one distribution can very often be avoided. For example, if âcost-effectivenessâ = (âprobability of successâ*âyears of impact given successâ + (1 - âprobability of successâ)*âyears of impact given failureâ)*ânumber of animals that can be affectedâ*âDALYs averted per animal-year improvedâ/ââcostâ, and all these variables are independent (as usually assumed in Monte Carlo simulations for simplicity), the expected cost-effectiveness will be E(âcost-effectivenessâ) = (âprobability of successâ*E(âyears of impact given successâ) + (1 - âprobability of successâ)*E(âyears of impact given failureâ))*E(ânumber of animals that can be affectedâ)*E(âDALYs averted per animal-year improvedâ)*E(1/ââcostâ). This is because E(âconstant aâ*âdistribution Xâ + âconstant bâ) = a*E(X) + b, and E(X*Y) = E(X)*E(Y) if X and Y are independent. Note:
The input distributions should be converted to point estimates corresponding to their means.
You can make a copy of this sheet (presented here) to calculate the mean of uniform, normal, loguniform, lognormal, pareto and logistic distributions from 2 of their quantiles. For example, if âyears of impact given successâ follows a lognormal distribution with 5th and 95th percentiles of 3 and 30 years, one should set the cell B2 to 0.05, C2 to 0.95, B3 to 3, and C3 to 30, and then check E(âyears of impact given successâ) in cell C22, which is 12.1 years.
Replacing an input by its most likely value (its mode), or one which is as likely to be an underestimate as an overestimate (its median) may lead to a biased expected cost-effectiveness. For example, the median and mode of a lognormal distribution are always lower than its mean. So, if âyears of impact given successâ followed such distribution, replacing it with its most likely value, or one as likely to be too low as too high would result in underestimating the expected cost-effectiveness.
The expected cost-effectiveness is proportional to E(1/ââcostâ), which is only equal to 1/âE(âcostâ) if âcostâ is a constant, or practically equal if it is a fairly certain distribution compared to others influencing the cost-effectiveness. If âcostâ is too uncertain to be considered constant, and there is not a closed formula to determine E(1/ââcostâ) (there would be if âcostâ followed a uniform distribution), one would have to run a Monte Carlo simulation to compute E(1/ââcostâ), but it would only involve the distribution of the cost. For uniform, normal and lognormal distributions, Guesstimate would do. For other distributions, you can try Squiggle AI (I have not used it, but it seems quite useful).
Several points:
Doing the Monte Carlo using my sheet is easier than the method you presented for avoiding the Monte Carlo. It presents the mean, which is the expected value, and also the confidence interval.
There are some audiences that already understand uncertainty and have a SBF-style desire to only maximize expected utility. These audiences are rare. Most people need to be shown the uncertainty (even if they do not yet know they need it).
Some people will want or need to take the âsafe optionâ with a higher floor rather than try to maximize the expected value.
When done right, the confidence interval includes uncertainty in implementation. If it is done by an A-team that gets things right, you will get better results. Knowing the possible range is key to know how fragile the expected result is and how much care will be required to get things right.