I don’t know if this is useful to you, but we have a somewhat easier way of solving the same problem using what we call a simplified Monte Carlo estimation technique. This is described in the following post:
It is not quite as accurate as your method because it approximates the probability distributions as a three-value distribution, but it addresses the issue of CEA inputs being uncorrelated and can be done in any spreadsheet without a need of using any extra tools or web services. You just calculate for all combinations of inputs, and then use standard spreadsheet tools like cumulative probability plots or histogram calculators to get the probability distribution of results:
“In our CE estimation with uncertain inputs, we implement a highly simplified Monte Carlo method that we call a simplified Monte Carlo or “poor man’s” Monte Carlo calculation.
In our simplified Monte Carlo calculation, we initially estimate ranges for all or most of the input parameters, and represent these ranges by low, median, and high values. Given a probability distribution of what values a parameter may take, the low value represents the average value of the lowest 1⁄3 of possibilities, the median value represents the average of the middle 1⁄3 of probable values and the high value represents the average of the largest 1⁄3 of probably values. This approximates a probability distribution of possible input parameter values by three discrete values of equal probability.
Once all of the input parameters are represented by three values of equal probability, then the CE result is calculated for all combinations of input parameters. If each of the input parameters are independent and uncorrelated, then the set of CE values that result from all combinations of inputs all have equal probability. A histogram of the full set of CE results is then constructed to illustrate the full range of possible CE values and their respective approximate probabilities. ”
Thanks for your comment @Robert Van Buskirk, and great that you are also thinking of ways to incorporate uncertainty into CEAs. You note that your way also solves ‘the same problem’, but a key aspect of what we are arguing (and what others have suggested) is not just that a proper monte carlo approach avoids objective issues with an interval-based approach, but that it provides a wealth of more nuanced information and allows for much more discerning assessments to be made. This is approximated by the approach you suggest, but not fully realised. For example ‘A histogram of the full set of CE results is then constructed to illustrate the full range of possible CE values’ - but by definition the histogram won’t include the full range of possible values because values beyond the average of the top and bottom thirds are not included in the calculations. This might obscure or underestimate the possibility of extreme outcomes.
My second concern would just be that if one has the capacity to define what the average of the bottom third of the expected values are, and the average of the top third, this is in all likelihood enough information for defining an approximate distribution, or one could simply select a uniform distribution. In that case, why not just do a proper monte carlo? The main answer to this is often just that people don’t have the confidence to consider a possible probability distribution that might work, and that was the key thing I’m trying to help with in developing and sharing this application
Thanks Jamie: our method is most useful when one has a relatively small sample of field data. In that case it is easy to calculate the averages of the bottom third, middle third, and top third of values and this is good enough because the data sample is not sufficient to specify the distribution with any greater precision.
Our method can also be calculated in any spreadsheet extremely easily and quickly without using any plug-ins or tools.
But agreed, if someone has the time, data and capacity, your method is better.
I don’t know if this is useful to you, but we have a somewhat easier way of solving the same problem using what we call a simplified Monte Carlo estimation technique. This is described in the following post:
https://forum.effectivealtruism.org/posts/icxnuEHTXrPapHBQg/a-simplified-cost-effectiveness-estimation-methodology-for
It is not quite as accurate as your method because it approximates the probability distributions as a three-value distribution, but it addresses the issue of CEA inputs being uncorrelated and can be done in any spreadsheet without a need of using any extra tools or web services. You just calculate for all combinations of inputs, and then use standard spreadsheet tools like cumulative probability plots or histogram calculators to get the probability distribution of results:
Thanks for your comment @Robert Van Buskirk, and great that you are also thinking of ways to incorporate uncertainty into CEAs. You note that your way also solves ‘the same problem’, but a key aspect of what we are arguing (and what others have suggested) is not just that a proper monte carlo approach avoids objective issues with an interval-based approach, but that it provides a wealth of more nuanced information and allows for much more discerning assessments to be made. This is approximated by the approach you suggest, but not fully realised. For example ‘A histogram of the full set of CE results is then constructed to illustrate the full range of possible CE values’ - but by definition the histogram won’t include the full range of possible values because values beyond the average of the top and bottom thirds are not included in the calculations. This might obscure or underestimate the possibility of extreme outcomes.
My second concern would just be that if one has the capacity to define what the average of the bottom third of the expected values are, and the average of the top third, this is in all likelihood enough information for defining an approximate distribution, or one could simply select a uniform distribution. In that case, why not just do a proper monte carlo? The main answer to this is often just that people don’t have the confidence to consider a possible probability distribution that might work, and that was the key thing I’m trying to help with in developing and sharing this application
Thanks Jamie: our method is most useful when one has a relatively small sample of field data. In that case it is easy to calculate the averages of the bottom third, middle third, and top third of values and this is good enough because the data sample is not sufficient to specify the distribution with any greater precision.
Our method can also be calculated in any spreadsheet extremely easily and quickly without using any plug-ins or tools.
But agreed, if someone has the time, data and capacity, your method is better.