I want to highlight up front that I am a big supporter of any work that aims to improve institutional decision making. I believe it’s a highly impactful area with unparalleled potential given the decision power (in both terms of spending and benefit potential) of large institutions is immense. I personally feel there’s a big moral and EA argument in supporting solutions that could practically deliver benefits (even small returns given the scale).
In terms of cleaner questions upfront which get to the heart of my uncertainties:
How much will the elements combined improve the quality of decisions that are made? Tied to this—which elements could be cut if needed for time without undermining the benefits you’d expect?
Are there examples of previous decisions that have been made that have been run through this process, to show what different outcomes would have been generated?
Given the time investment needed to implement this process, why is it advantageous over existing solutions that have been shown to provide substantial improvements in decision making quality (under experimentation) but often face complaints over needing significant time and expertise investment (e.g. training on and aggregating Bayesian models)?
Further reflections if interested/useful Having read your paper, I have some concerns over how the solution can be implemented at a beneficial scale. I raise this particularly as a number of the problems you’ve mentioned in the White Paper (e.g. unstructured/limited consultations with experts or limited analysis of the problem space) are driven more by time constraints rather than a clear framework of how to do it. This is an important consideration as planning for catastrophic risks is only half of the problem—we can’t consistently (or at all) predict black swan events and thus decision making at speed in crises is incredibly (if not more) important, as Covid showed us.
Given decision science research, I query the heavy reliance on expert judgment as a key node to improve the predictive accuracy, as there’s a healthy body of evidence that suggests quality of reasoning as opposed to domain expertise is a better predictor for such accuracy. Your White Paper actually seems to account for this by proxy when it highlights specific reasoning methods to drive improved accuracy (e.g. IDEA framework).
In addition, I’m less sure how beneficial the democratic/deliberation process with citizens is for the risks you are targeting. The examples you note (such as abortion and LGBTQ+ issues) are primarily social issues which lend themselves well to citizen assemblies given they are moral in nature. On the other side, planning policy is quite heavily democratised in the UK and arguably has led to very bad outcomes given wider economic or societal benefits from construction are less tangible than personal concerns around changes to the local area. These externalities aren’t always accurately priced into people’s incentives and thus their judgements aren’t necessarily what’s best for society. Do you see a similar issue for catastrophic risks/how will you mitigate if so?
Thank you James for such a thorough response! Always pleased to see recognition of the still neglected potential of political technologies and existing best practices.
1. We aim to do aspects of the modelling ourselves, but by identifying in literature review existing open access models rather than building them from the data directly, time can be saved. An assembly can be conducted over a few months, as can the modelling be undertaken simultaneously with effective resourcing to parallelise phases of the Process, which is not unusual for a large policy initiative of any kind. I would emphasise that time is not the bottleneck when deciding how, from a meta- or strategic planning level, how to proceed. It is about building capacity so you are ready in future to act quickly. Badly and well prepared administration can both act quickly; the difference in what they chose to have ready is what we are aiming for with this work.
2. No one has combined all three components yet, however we want to test this because we see each success story and underlying logic of the components as mutually reinforcing. Some syntheses of parts of this exist: horizon scanning and public deliberation were advocated for in Kemp et al’s Climate Endgame: https://www.pnas.org/doi/10.1073/pnas.2108146119as well as numerous examples of DMDU (the Room for the River initiative for strategic planning of the Dutch Delta going over 40 years into the future) and participatory processes being reviewed at this year’s DMDU 2023 conference. For example, DMDU is a fundamentally different ontology for modelling; rather than trying consolidative modelling (which is legitimate, but is also often used to confirm biases or flatten a complex space), DMDU accepts that we need to explore many possible futures to increase the likelihood of being prepared for uncertainty. It also works when there are deeper disagreements on what theories or uncertainties to account for:
Citations useful for exploratory modelling include:
3. I would liken the time constraints as pertaining to the tactical level problem, which is still hindered or enabled by the quality of a pre-emptively prepared ‘strategic’ resilience plan. In short, due to these uncertainties, the point isn’t deciding from scratch as the risk is occurring—it is having done enough horizon scanning, scenario planning, and invested enough in resilient infrastructure, backups, failsafes, public education etc. before the time so that a faster dynamic response, as with DAPPs, can occur and be iterated (think nuclear war infomercials from the Cold War—if best practice wasn’t established for surviving a blast, and publicly communicated, there wouldn’t be time to figure this out rapidly on the day). If we don’t put in the time, work, and focus that something like our Process does before time, there will be no easy way to suddenly decide quickly because the ground wouldn’t have been prepared. Similarly, Bayesian inference is only as good as it’s priors, and falls apart with complex low probabilities collapse events where we had very little data to learn from Taleb N’s Statistical Consequences of Fat Tails: https://arxiv.org/ftp/arxiv/papers/2001/2001.10488.pdf
Collective intelligence, and broad and diverse sets of experts—not homogenous or narrow sets—have something to contribute to developing those models. As do the public for developing something that is viable for enforcement, not something likely to lead to social unrest if attempted. As such setting a values driven strategy that can be adjusted with different operational means when events hit is the goal, and as such time is not the only or primary bottleneck. The quality of preparations—which have also been sorely lacking as seen in Britain with the Covid-19 inquiry—is vital. Taiwan had prior experience with SARS, but also an effective test and trace system built up after that. Britain still does not—with better foresight, the need for resilience and robust decision making is seen more clearly, as we might now invest better for the next pandemic. The problem wasn’t only time, it was political partiality (which can be mitigated somewhat through a wider sample of the public, rather than a slim and vested interest beholden political party) and it was a lack of transparency that also allowed for abysmal prioritisation to stay hidden. Matt Boyd’s anticipatory governance paper here is solid on a range of wider recommendations for good existential and catastrophic risk policy making, including deliberation and transparency taken as essential (built from a wide sample of expert elicitation and examples of successful governance): https://ojs.victoria.ac.nz/pq/article/view/7313/6467
As to your last point, there is even more of a moral component to x-risks than perhaps issues of obvious social cleavage; it is a fundamental misstep to assume that how we prioritise, who we identify as more or less vulnerable, and what we want to invest in or pursue despite their risks, is not a moral question. Brushing them under the carpet and assuming we can technocratically address them is not viable. I would again refer here to a Matt Boyd paper on the need for deliberation on values, because even the ‘how’ questions carry deeper ‘why’ aspects to them: https://philarchive.org/rec/BOYERN
Rounds of iterated expert, modelling, and citizen deliberation can help price in exactly those externalities. This is why we don’t just say ‘let’s save time and do just a deliberation, or let’s make it more technical and do just a horizon scan’ - for these issues, by taking a similar amount of time as a longer policy consultation but with broader expert, generalist, complexity modelling, and finally public collective intelligence, we ensure there are phased opportunities to consider multiple levels of knowledge and values. For the greatest risks, we think this is a huge bargain for avoiding stumbling into them or failing to enforce their mitigation due to flawed, secretive, and self-sabotaging approaches to the problem spaces.
Thanks for posting this!
I want to highlight up front that I am a big supporter of any work that aims to improve institutional decision making. I believe it’s a highly impactful area with unparalleled potential given the decision power (in both terms of spending and benefit potential) of large institutions is immense. I personally feel there’s a big moral and EA argument in supporting solutions that could practically deliver benefits (even small returns given the scale).
In terms of cleaner questions upfront which get to the heart of my uncertainties:
How much will the elements combined improve the quality of decisions that are made? Tied to this—which elements could be cut if needed for time without undermining the benefits you’d expect?
Are there examples of previous decisions that have been made that have been run through this process, to show what different outcomes would have been generated?
Given the time investment needed to implement this process, why is it advantageous over existing solutions that have been shown to provide substantial improvements in decision making quality (under experimentation) but often face complaints over needing significant time and expertise investment (e.g. training on and aggregating Bayesian models)?
Further reflections if interested/useful
Having read your paper, I have some concerns over how the solution can be implemented at a beneficial scale. I raise this particularly as a number of the problems you’ve mentioned in the White Paper (e.g. unstructured/limited consultations with experts or limited analysis of the problem space) are driven more by time constraints rather than a clear framework of how to do it. This is an important consideration as planning for catastrophic risks is only half of the problem—we can’t consistently (or at all) predict black swan events and thus decision making at speed in crises is incredibly (if not more) important, as Covid showed us.
Given decision science research, I query the heavy reliance on expert judgment as a key node to improve the predictive accuracy, as there’s a healthy body of evidence that suggests quality of reasoning as opposed to domain expertise is a better predictor for such accuracy. Your White Paper actually seems to account for this by proxy when it highlights specific reasoning methods to drive improved accuracy (e.g. IDEA framework).
In addition, I’m less sure how beneficial the democratic/deliberation process with citizens is for the risks you are targeting. The examples you note (such as abortion and LGBTQ+ issues) are primarily social issues which lend themselves well to citizen assemblies given they are moral in nature. On the other side, planning policy is quite heavily democratised in the UK and arguably has led to very bad outcomes given wider economic or societal benefits from construction are less tangible than personal concerns around changes to the local area. These externalities aren’t always accurately priced into people’s incentives and thus their judgements aren’t necessarily what’s best for society. Do you see a similar issue for catastrophic risks/how will you mitigate if so?
Thank you James for such a thorough response! Always pleased to see recognition of the still neglected potential of political technologies and existing best practices.
1. We aim to do aspects of the modelling ourselves, but by identifying in literature review existing open access models rather than building them from the data directly, time can be saved. An assembly can be conducted over a few months, as can the modelling be undertaken simultaneously with effective resourcing to parallelise phases of the Process, which is not unusual for a large policy initiative of any kind. I would emphasise that time is not the bottleneck when deciding how, from a meta- or strategic planning level, how to proceed. It is about building capacity so you are ready in future to act quickly. Badly and well prepared administration can both act quickly; the difference in what they chose to have ready is what we are aiming for with this work.
2. No one has combined all three components yet, however we want to test this because we see each success story and underlying logic of the components as mutually reinforcing. Some syntheses of parts of this exist: horizon scanning and public deliberation were advocated for in Kemp et al’s Climate Endgame: https://www.pnas.org/doi/10.1073/pnas.2108146119as well as numerous examples of DMDU (the Room for the River initiative for strategic planning of the Dutch Delta going over 40 years into the future) and participatory processes being reviewed at this year’s DMDU 2023 conference. For example, DMDU is a fundamentally different ontology for modelling; rather than trying consolidative modelling (which is legitimate, but is also often used to confirm biases or flatten a complex space), DMDU accepts that we need to explore many possible futures to increase the likelihood of being prepared for uncertainty. It also works when there are deeper disagreements on what theories or uncertainties to account for:
Citations useful for exploratory modelling include:
- https://www.jstor.org/stable/171847
- https://www.sciencedirect.com/science/article/pii/S2214629622001876
- https://forum.effectivealtruism.org/posts/kCBQHWqbk4Nrns8P7/model-based-policy-analysis-under-deep-uncertainty
3. I would liken the time constraints as pertaining to the tactical level problem, which is still hindered or enabled by the quality of a pre-emptively prepared ‘strategic’ resilience plan. In short, due to these uncertainties, the point isn’t deciding from scratch as the risk is occurring—it is having done enough horizon scanning, scenario planning, and invested enough in resilient infrastructure, backups, failsafes, public education etc. before the time so that a faster dynamic response, as with DAPPs, can occur and be iterated (think nuclear war infomercials from the Cold War—if best practice wasn’t established for surviving a blast, and publicly communicated, there wouldn’t be time to figure this out rapidly on the day). If we don’t put in the time, work, and focus that something like our Process does before time, there will be no easy way to suddenly decide quickly because the ground wouldn’t have been prepared. Similarly, Bayesian inference is only as good as it’s priors, and falls apart with complex low probabilities collapse events where we had very little data to learn from Taleb N’s Statistical Consequences of Fat Tails: https://arxiv.org/ftp/arxiv/papers/2001/2001.10488.pdf
Collective intelligence, and broad and diverse sets of experts—not homogenous or narrow sets—have something to contribute to developing those models. As do the public for developing something that is viable for enforcement, not something likely to lead to social unrest if attempted. As such setting a values driven strategy that can be adjusted with different operational means when events hit is the goal, and as such time is not the only or primary bottleneck. The quality of preparations—which have also been sorely lacking as seen in Britain with the Covid-19 inquiry—is vital. Taiwan had prior experience with SARS, but also an effective test and trace system built up after that. Britain still does not—with better foresight, the need for resilience and robust decision making is seen more clearly, as we might now invest better for the next pandemic. The problem wasn’t only time, it was political partiality (which can be mitigated somewhat through a wider sample of the public, rather than a slim and vested interest beholden political party) and it was a lack of transparency that also allowed for abysmal prioritisation to stay hidden. Matt Boyd’s anticipatory governance paper here is solid on a range of wider recommendations for good existential and catastrophic risk policy making, including deliberation and transparency taken as essential (built from a wide sample of expert elicitation and examples of successful governance): https://ojs.victoria.ac.nz/pq/article/view/7313/6467
As to your last point, there is even more of a moral component to x-risks than perhaps issues of obvious social cleavage; it is a fundamental misstep to assume that how we prioritise, who we identify as more or less vulnerable, and what we want to invest in or pursue despite their risks, is not a moral question. Brushing them under the carpet and assuming we can technocratically address them is not viable. I would again refer here to a Matt Boyd paper on the need for deliberation on values, because even the ‘how’ questions carry deeper ‘why’ aspects to them: https://philarchive.org/rec/BOYERN
Rounds of iterated expert, modelling, and citizen deliberation can help price in exactly those externalities. This is why we don’t just say ‘let’s save time and do just a deliberation, or let’s make it more technical and do just a horizon scan’ - for these issues, by taking a similar amount of time as a longer policy consultation but with broader expert, generalist, complexity modelling, and finally public collective intelligence, we ensure there are phased opportunities to consider multiple levels of knowledge and values. For the greatest risks, we think this is a huge bargain for avoiding stumbling into them or failing to enforce their mitigation due to flawed, secretive, and self-sabotaging approaches to the problem spaces.