Because I had fun reading Circular A-4 and vehemently hated it I decided to write up my views. I have done this in the form of drafting two entries to this competition. If anyone wants to borrow some of these ideas in their entry then they would be welcome (I am not a US resident so not entering). Referencing me would be nice. See ideas below (or in a Google Doc here).
1. Going beyond quantitative analysis
Summary 1
Circular A-4 provides advice to regulatory analysts. However it focuses almost exclusively on quantitative Benefit-Cost Analysis (BCA). This risks missing the woods for all the trees. There are clear limits to BCAs and it is important to only use BCAs in conjunction with other methodologies. There are many other types of evidence out there that decision makers can be made aware of that analysis should look into. There is good precedent for this, for example advice for government analysts in the UK suggests this.
Suggested amendment 1
I suggest radically amending the kind of analysis that Circular A-4 guides analysts provide to decision makers along the following lines.
Amendments to the introduction should be made as follows:
A good regulatory analysis should include the following three six basic elements: (1) a statement of the need for the proposed action, (2) an examination of alternative approaches, and (3) an evaluation of the benefits and costs—quantitative and qualitative—of the proposed action and the main alternatives identified by the analysis [Add: (4) an evaluation of the past success, case studies, past precedent and quality of evidence for the proposed action (5) an evaluation of the strategic case and practical considerations for delivering the proposed action, and (6) a summary of expert views on the proposed action].
Furthermore additional sections (not drafted in full here) should be added to the guidance after Section G to cover:
H. Case study and historical analysis
I. Analysis of strategic and practical considerations
Quantitative analysis is prone to be affected by analysts and decision makers’ biases. Explicitly adjusting for these biases should be done as part of the analysis. There is good precedent for this, for example advice for government analysts in the UK suggests this. Adjustments can be made for both optimism bias (especially for any decisions that require government procurement or major infrastructure to be developed) and for presentism bias.
Suggested amendment 2
In section H the following should be added:
Adjustments to account for biases.
When conducting appraisal consideration should also be given to optimism bias – this is the proven tendency for appraisers to be optimistically biased about key project parameters, including capital costs and operating costs, project duration, and resulting benefits delivery. Optimistic rather than realistic projections result in undeliverable targets and if permitted across the board create institutional failure as all proposals fall consistently far short of promised results. For this reason, specific optimism bias adjustments must be applied as numbers are initially identified. Ideally adjustments should be based on an organisation’s own evidence base for historic levels of optimism bias. In the absence of robust organisation-specific estimates generic values are provided in the table below. There are currently no generic values available to be applied to benefits, only to costs, however an adjustment should be applied based on an organisation’s own evidence base.
[A table should then be inserted with figures for different kinds of cost inputs and how to adjust them for optimism bias, based on historical analysis of past BCA]
When conducting appraisal consideration should also be given to presentism bias – this is the tendency for decision makers to favour the short run benefits over the long run benefits and disfavour short-run costs over long-run costs. This leads to short term decision making. For this reason, specific presentism bias adjustments can be applied at the same time as a discount rate is applied. This should be in the form of a small positive amount that effectively lowers the discount rate. Ideally adjustments should be based on an organisation’s own evidence base for historic levels of presentism bias. In the absence of robust organisation-specific estimates a generic value of 1.5% per annum should be used. The resultant output should be provided to decision makers alongside the non-adjusted figure to support their decision making (not as a replacement to the unadjusted output)
(Note text borrowed from the UK government Green Book.)
Because I had fun reading Circular A-4 and vehemently hated it I decided to write up my views. I have done this in the form of drafting two entries to this competition. If anyone wants to borrow some of these ideas in their entry then they would be welcome (I am not a US resident so not entering). Referencing me would be nice. See ideas below (or in a Google Doc here).
1. Going beyond quantitative analysis
Summary 1
Circular A-4 provides advice to regulatory analysts. However it focuses almost exclusively on quantitative Benefit-Cost Analysis (BCA). This risks missing the woods for all the trees. There are clear limits to BCAs and it is important to only use BCAs in conjunction with other methodologies. There are many other types of evidence out there that decision makers can be made aware of that analysis should look into. There is good precedent for this, for example advice for government analysts in the UK suggests this.
Suggested amendment 1
I suggest radically amending the kind of analysis that Circular A-4 guides analysts provide to decision makers along the following lines.
Amendments to the introduction should be made as follows:
A good regulatory analysis should include the following
threesix basic elements: (1) a statement of the need for the proposed action, (2) an examination of alternative approaches, and (3) an evaluation of the benefits and costs—quantitative and qualitative—of the proposed action and the main alternatives identified by the analysis [Add: (4) an evaluation of the past success, case studies, past precedent and quality of evidence for the proposed action (5) an evaluation of the strategic case and practical considerations for delivering the proposed action, and (6) a summary of expert views on the proposed action].Furthermore additional sections (not drafted in full here) should be added to the guidance after Section G to cover:
H. Case study and historical analysis
I. Analysis of strategic and practical considerations
J. Summary of expert views
Essay 1
See the Google Doc here
2. Adjusting for presentism and optimism bias
Summary 2
Quantitative analysis is prone to be affected by analysts and decision makers’ biases. Explicitly adjusting for these biases should be done as part of the analysis. There is good precedent for this, for example advice for government analysts in the UK suggests this. Adjustments can be made for both optimism bias (especially for any decisions that require government procurement or major infrastructure to be developed) and for presentism bias.
Suggested amendment 2
In section H the following should be added:
Adjustments to account for biases.
When conducting appraisal consideration should also be given to optimism bias – this is the proven tendency for appraisers to be optimistically biased about key project parameters, including capital costs and operating costs, project duration, and resulting benefits delivery. Optimistic rather than realistic projections result in undeliverable targets and if permitted across the board create institutional failure as all proposals fall consistently far short of promised results. For this reason, specific optimism bias adjustments must be applied as numbers are initially identified. Ideally adjustments should be based on an organisation’s own evidence base for historic levels of optimism bias. In the absence of robust organisation-specific estimates generic values are provided in the table below. There are currently no generic values available to be applied to benefits, only to costs, however an adjustment should be applied based on an organisation’s own evidence base.
[A table should then be inserted with figures for different kinds of cost inputs and how to adjust them for optimism bias, based on historical analysis of past BCA]
When conducting appraisal consideration should also be given to presentism bias – this is the tendency for decision makers to favour the short run benefits over the long run benefits and disfavour short-run costs over long-run costs. This leads to short term decision making. For this reason, specific presentism bias adjustments can be applied at the same time as a discount rate is applied. This should be in the form of a small positive amount that effectively lowers the discount rate. Ideally adjustments should be based on an organisation’s own evidence base for historic levels of presentism bias. In the absence of robust organisation-specific estimates a generic value of 1.5% per annum should be used. The resultant output should be provided to decision makers alongside the non-adjusted figure to support their decision making (not as a replacement to the unadjusted output)
(Note text borrowed from the UK government Green Book.)