This is fantastic thank you! Have already sent it to someone considering dong a CBA
“For any purpose other than an example calculation, never use a point estimate. Always do all math in terms of confidence intervals. All inputs should be ranges or probability distributions, and all outputs should be presented as confidence intervals.”
I weakly disagree with this “never” statement, as I think there is value in doing basic cost-benefit analysis without confidence intervals, especially for non mathsy indivuals or small orgs who want to look at potential cost effectiveness of their own or other’s interventions. I wouldn’t want to put some people off by setting this as a “minimum” bar. I also think that simple “lower and upper bound” ranges can sometimes be an easier way to do estimates, without strictly needing to calculate a confidence interval.
In saying that when, big organisations do CBA’s to actually make decisions or move large amounts of money, or for any academic purpose then yes I agree confindence intervals are what’s needed!
I would also say that for better or worse (probably for worse) the point estimate is by far the most practically discussed and used application of any CBA so I think its practially important to put more effort into getting your point estimate as accurate as possible, then it is to make sure you’re range is accurate.
You are right, and I did walk it back a little in the “Deliver what the customer needs” section.
But you would have been more right before I posted the link to that Google Sheets Monte Carlo template. Before I had that, I would often default to a point estimate for very fast jobs. But this tool makes it a lot easier to start with the Monte Carlo to see if it works and is accepted.
I agree with him on inputs, but often the expected value is the most important output, in which case point estimates are still informative (sometimes more so than ranges). Also, CIs are often not the most informative indicator of uncertainty; a CEAC, CEAF, VOI, or p(error) given a known WTP threshold is often more useful, though perhaps less so in a CBA rather than a CEA/CUA.
If you have a sophisticated audience, sure. But outside a few niche subfields, very few people working in policy understand any of those things. My advice is for dealing with an audience of typical policy makers, the kind of people that you have to explain the concept of ‘value of statistical life’ to. If you give them an expected value, they will mentally collapse it into a point estimate with near-zero uncertainty, and/or treat it as a talking point that is barely connected to any real analysis.
This is fantastic thank you! Have already sent it to someone considering dong a CBA
“For any purpose other than an example calculation, never use a point estimate. Always do all math in terms of confidence intervals. All inputs should be ranges or probability distributions, and all outputs should be presented as confidence intervals.”
I weakly disagree with this “never” statement, as I think there is value in doing basic cost-benefit analysis without confidence intervals, especially for non mathsy indivuals or small orgs who want to look at potential cost effectiveness of their own or other’s interventions. I wouldn’t want to put some people off by setting this as a “minimum” bar. I also think that simple “lower and upper bound” ranges can sometimes be an easier way to do estimates, without strictly needing to calculate a confidence interval.
In saying that when, big organisations do CBA’s to actually make decisions or move large amounts of money, or for any academic purpose then yes I agree confindence intervals are what’s needed!
I would also say that for better or worse (probably for worse) the point estimate is by far the most practically discussed and used application of any CBA so I think its practially important to put more effort into getting your point estimate as accurate as possible, then it is to make sure you’re range is accurate.
Nice job again.
Thanks!
You are right, and I did walk it back a little in the “Deliver what the customer needs” section.
But you would have been more right before I posted the link to that Google Sheets Monte Carlo template. Before I had that, I would often default to a point estimate for very fast jobs. But this tool makes it a lot easier to start with the Monte Carlo to see if it works and is accepted.
Ha I love this I will definitely check that simulator out nice one!
I agree with him on inputs, but often the expected value is the most important output, in which case point estimates are still informative (sometimes more so than ranges). Also, CIs are often not the most informative indicator of uncertainty; a CEAC, CEAF, VOI, or p(error) given a known WTP threshold is often more useful, though perhaps less so in a CBA rather than a CEA/CUA.
If you have a sophisticated audience, sure. But outside a few niche subfields, very few people working in policy understand any of those things. My advice is for dealing with an audience of typical policy makers, the kind of people that you have to explain the concept of ‘value of statistical life’ to. If you give them an expected value, they will mentally collapse it into a point estimate with near-zero uncertainty, and/or treat it as a talking point that is barely connected to any real analysis.