Great post! I feel these diagrams will be really useful for clarifying the possible interventions and parts of the existential risks.
Do you think they’ll also serve for comparing different positions on a specific existential risk, like the trajectories in this post? Or do you envision the diagram for a specific risk as a summary of all causal pathways to this risk?
By “comparing different positions on a specific existential risk”, it seems to me that you could mean either:
1. Comparing what different “stages” of a specific risk would be like
e.g., comparing what it’d be like if we’re at the “implementation of hazardous information” vs “harmful events” stage of a engineered pathogen risk
2. Comparing different people’s views on what stage a specific risk is currently at
e.g., identifying that one person believes the information required to develop an engineered pathogen just hasn’t been developed, while another believes that it’s been developed but has yet to be shared or implemented
3. Comparing different people’s views on a specific risk more generally
e.g., identifying that two people roughly agree on the chances an engineered pathogen could be developed, but disagree on how likely it is that it’d be implemented or that a resulting outbreak would result in collapse/extinction, and that’s why they overall disagree about the risk levels
e.g., identifying that two people roughly agree on the overall risks from an engineered pandemic, but this obscures the fact that they disagree, in ways that roughly cancel out, on the probabilities of progression from each stage to the next stage. This could be important because it could help them understand why they advocate for different interventions.
(Note that I just randomly chose to go with pathogen examples here—as I say in the post, these diagrams can be used for a wide range of risks.)
I think that, if these diagrams can be useful at all (which I hope they can!), they can be useful for 1 and 3. And I think perhaps you had 3 in mind, as that’s perhaps most similar to what the state space model you linked to accomplishes. (I’d guess these models could also be useful for 2, but I’m not sure how often informed people would have meaningful disagreements about what stage a specific risk is currently at.)
Hopefully my examples already make it somewhat clear why I think that these diagrams could help with 1 and 3, and why that’s important. Basically, I think most things that help people make their more of their thinking more explicit, or that prompt/force them to do so, will help them identify precisely where they agree and disagree with each other. (I think this also applies to stating one’s probabilities/credences explicitly, as I sort of allude to in passing in a few places here.)
Another way to put that is these things will help or make people “factor out” various inputs into their bottom line conclusions, so we can more easily point to those inputs that seem most uncertain or contestable, or conversely we can realise “Oh, that’s actually a great point—I should add that to my own internal model of the situation”. I think visualisation also generally makes that sort of thing easier and more effective.
And I think these diagrams can also work as “a summary of all causal pathways to this risk”, if I’m interpreting you correctly. For example, you could further flesh out my final diagram from this post (not the Defence in Depth diagram) to represent basically all the major causal pathways to existential catastrophes from bioengineering. And then you could also have people assign probabilities to moving from each stage to each other stage is connects to, or even contest which connections are shown (e.g., suggest how one step could “bypass” what I showed it as connecting to in order to connect to later steps). And then they could debate these things.
(But as I say in the post, I think if we wanted to get quite granular, we’d probably want to ultimately use something like Guesstimate. And there are also various other models and approaches we could use to complement these diagrams.)
I was indeed thinking about 3., even if 1. and 2. are also important. And I get that the main value of these diagrams is to force an explicit and as formal as possible statement to be made.
I guess my question was more about, given two different causal diagrams for the same risk (made by different researchers for example), have you an idea of how to compare them? Like finding the first difference along the causal path, or others means of comparison. This seems important because even with clean descriptions of our views, we can still talk past each other if we cannot see where the difference truly lies.
Hmm, I’m not sure I fully understand what you mean. But hopefully the following somewhat addresses it:
One possibility is that two different researchers might have different ideas of what the relevant causal pathways actually are. For a simple example, one researcher might not think of the possibility that a risk could progress right from the initial harms to the existential catastrophe, without a period of civilizational collapse first, or might think of that but dismiss it as not even worth considering because it seems so unlikely. A different researcher might think that that path is indeed worth considering.
If either of the researchers tried to make an explicit causal diagram of how they think the risk could lead to existential catastrophe, the other one would probably notice that their own thoughts on the matter differ. This would likely help them see where the differences in their views lie, and the researcher who’d neglected that path might immediately say “Oh, good point, hadn’t thought of that!”, or they might discuss why that seems worth considering to one of the researchers but not to the other.
(I would guess that in practice this would typically occur for less obvious paths than that, such as specific paths that can lead to or prevent the development/spread of certain types of information.)
Another possibility is that two different researchers have essentially the same idea of what the relevant causal pathways are, but very different ideas of the probabilities of progression from certain steps to other steps. In that case, merely drawing these diagrams, in the way they’re shown in this post, wouldn’t be sufficient for them to spot why their views differ.
But having the diagrams in front of them could help them talk through how likely they think each particular path or step is. Or they could each assign an actual probability to each path or step. Either way, they should then be able to see why and where their views differ.
In all of these cases, ideally, the researchers would go beyond just noticing where their views differ and instead discuss why each of them believes what they believe about the point on which they differ.
Does that answer your question?
If by “how to compare them” you mean “how to tell which one is better”, then that’s something that this tool alone can’t do. But by facilitating clear, explicit thought and discussion, this tool could potentially help well-informed people form views about which diagrams/models are more valid or useful.
Great post! I feel these diagrams will be really useful for clarifying the possible interventions and parts of the existential risks.
Do you think they’ll also serve for comparing different positions on a specific existential risk, like the trajectories in this post? Or do you envision the diagram for a specific risk as a summary of all causal pathways to this risk?
Thanks! I hope so.
By “comparing different positions on a specific existential risk”, it seems to me that you could mean either:
1. Comparing what different “stages” of a specific risk would be like
e.g., comparing what it’d be like if we’re at the “implementation of hazardous information” vs “harmful events” stage of a engineered pathogen risk
2. Comparing different people’s views on what stage a specific risk is currently at
e.g., identifying that one person believes the information required to develop an engineered pathogen just hasn’t been developed, while another believes that it’s been developed but has yet to be shared or implemented
3. Comparing different people’s views on a specific risk more generally
e.g., identifying that two people roughly agree on the chances an engineered pathogen could be developed, but disagree on how likely it is that it’d be implemented or that a resulting outbreak would result in collapse/extinction, and that’s why they overall disagree about the risk levels
e.g., identifying that two people roughly agree on the overall risks from an engineered pandemic, but this obscures the fact that they disagree, in ways that roughly cancel out, on the probabilities of progression from each stage to the next stage. This could be important because it could help them understand why they advocate for different interventions.
(Note that I just randomly chose to go with pathogen examples here—as I say in the post, these diagrams can be used for a wide range of risks.)
I think that, if these diagrams can be useful at all (which I hope they can!), they can be useful for 1 and 3. And I think perhaps you had 3 in mind, as that’s perhaps most similar to what the state space model you linked to accomplishes. (I’d guess these models could also be useful for 2, but I’m not sure how often informed people would have meaningful disagreements about what stage a specific risk is currently at.)
Hopefully my examples already make it somewhat clear why I think that these diagrams could help with 1 and 3, and why that’s important. Basically, I think most things that help people make their more of their thinking more explicit, or that prompt/force them to do so, will help them identify precisely where they agree and disagree with each other. (I think this also applies to stating one’s probabilities/credences explicitly, as I sort of allude to in passing in a few places here.)
Another way to put that is these things will help or make people “factor out” various inputs into their bottom line conclusions, so we can more easily point to those inputs that seem most uncertain or contestable, or conversely we can realise “Oh, that’s actually a great point—I should add that to my own internal model of the situation”. I think visualisation also generally makes that sort of thing easier and more effective.
And I think these diagrams can also work as “a summary of all causal pathways to this risk”, if I’m interpreting you correctly. For example, you could further flesh out my final diagram from this post (not the Defence in Depth diagram) to represent basically all the major causal pathways to existential catastrophes from bioengineering. And then you could also have people assign probabilities to moving from each stage to each other stage is connects to, or even contest which connections are shown (e.g., suggest how one step could “bypass” what I showed it as connecting to in order to connect to later steps). And then they could debate these things.
(But as I say in the post, I think if we wanted to get quite granular, we’d probably want to ultimately use something like Guesstimate. And there are also various other models and approaches we could use to complement these diagrams.)
Thanks for that very in-depth answer!
I was indeed thinking about 3., even if 1. and 2. are also important. And I get that the main value of these diagrams is to force an explicit and as formal as possible statement to be made.
I guess my question was more about, given two different causal diagrams for the same risk (made by different researchers for example), have you an idea of how to compare them? Like finding the first difference along the causal path, or others means of comparison. This seems important because even with clean descriptions of our views, we can still talk past each other if we cannot see where the difference truly lies.
Hmm, I’m not sure I fully understand what you mean. But hopefully the following somewhat addresses it:
One possibility is that two different researchers might have different ideas of what the relevant causal pathways actually are. For a simple example, one researcher might not think of the possibility that a risk could progress right from the initial harms to the existential catastrophe, without a period of civilizational collapse first, or might think of that but dismiss it as not even worth considering because it seems so unlikely. A different researcher might think that that path is indeed worth considering.
If either of the researchers tried to make an explicit causal diagram of how they think the risk could lead to existential catastrophe, the other one would probably notice that their own thoughts on the matter differ. This would likely help them see where the differences in their views lie, and the researcher who’d neglected that path might immediately say “Oh, good point, hadn’t thought of that!”, or they might discuss why that seems worth considering to one of the researchers but not to the other.
(I would guess that in practice this would typically occur for less obvious paths than that, such as specific paths that can lead to or prevent the development/spread of certain types of information.)
Another possibility is that two different researchers have essentially the same idea of what the relevant causal pathways are, but very different ideas of the probabilities of progression from certain steps to other steps. In that case, merely drawing these diagrams, in the way they’re shown in this post, wouldn’t be sufficient for them to spot why their views differ.
But having the diagrams in front of them could help them talk through how likely they think each particular path or step is. Or they could each assign an actual probability to each path or step. Either way, they should then be able to see why and where their views differ.
In all of these cases, ideally, the researchers would go beyond just noticing where their views differ and instead discuss why each of them believes what they believe about the point on which they differ.
Does that answer your question?
If by “how to compare them” you mean “how to tell which one is better”, then that’s something that this tool alone can’t do. But by facilitating clear, explicit thought and discussion, this tool could potentially help well-informed people form views about which diagrams/models are more valid or useful.
That answers my question, yes. :)