A thanksgiving turkey has an excellent model that predicts the farmer wants him to be safe and happy. But an explanation of thanksgiving traditions tells us a lot more about the risks of slaughter than the number of days the turkey has been fed and protected.
With nuclear war, we have explanations for why nuclear exchange is possible, including as an outcome of a conflict.
Just like with the turkey, we should pay attention to the explanation, not just try to make predictions based on past data.
With all of this, probability terminology is baked into the language and it is hard to speak without incorporating it. With the previous post, it was co-authored, and I wanted to remove that phrase, but concessions were made.
I agree with you, but once again I don’t see the difference between the case of the turkey and nuclear war, versus the case of longtermism or AGI. “With nuclear war, we have explanations for why nuclear exchange is possible, including as an outcome of a conflict.” Just the same with AGI—we have explanations for why AGI seems possible, we have some evidence from scaling laws that describe how AI systems get better when given more resources, and ideas about what might motivate people to create more and more powerful AI systems, and why that might be dangerous, etc.
I am not an academically trained philosopher (rather, an engineer!), so I’m not sure what’s the best way to talk about probability and make it clear what kind of uncertainty we’re talking about. But in all cases, it seems that we should basically use a mixture of empirical evidence based on past experience (where available), and first-principles reasoning about what might be possible in the future. With some things—mathematical theorems are a great example—evidence might be hard to come by, so it might be very difficult to predict with precision. But it doesn’t seem like we are in fundamentally different, “unknowable” terrain—it’s more uncertain than nuclear war risk, which in turn is more uncertain than forecasting things like housing prices or wheat harvests, which in turn is more uncertain than forecasting that the sun will rise tomorrow. They all seem like part of the same spectrum, and the long-term future of civilization seems important enough that it’s worth thinking about even amid high uncertainty.
How about this: let’s split future events into two groups. 1) Events that are not influenced by people and 2) Events that are influenced by people.
In 1, we can create predictive models, use probability, even calculate uncertainty. All the standard rules apply, Bayesian and otherwise.
In 2, we can still create predictive models, but they’ll be nonsensical. That’s because we cannot know how knowledge creation will affect 2. We don’t even need any fancy reasoning, it’s already implied in the definition of terms like knowledge creation and discovery. You can’t discover something before you discover it, before it’s created.
So, up until recently, the bodies of the solar system fell into category 1. We can predict their positions many years hence, as long as people don’t get involved. However, once we are capable, there’s no way now to know what we’ll do with the planets and asteroids in the future. Maybe we’ll find use for some mineral found predominantly in some asteroids, or maybe we’ll use a planet to block heat from the sun as it expands, or maybe we’ll detect some other risk/benefit and make changes accordingly.
This is an extreme example, but it applies across the board. Any time human knowledge creation impacts a system, there’s no way to model that impact before the knowledge is created.
Therefore, longtermism hinges on the idea that we have some idea of how to impact the long term future. But even more than the solar system example, that future will be overwhelmingly dominated by new knowledge, and hence unknowable to us to today, unable to be anticipated.
Sure, we can guess, and in the case of known future threats like nuclear war, we should guess and should try to ameliorate risk. But those problems apply to the very near future as well, they are problems facing us today (that’s why we know a fair bit about them). We shouldn’t waste effort trying to calculate the risk because we can’t do that for items in group 2. Instead, we know from our best explanations that nuclear war is a risk.
In this way the threat of nuclear is like the turkey—if the turkey even hears a rumor about thanksgiving traditions, should it sit down and try to update its priors? Or take the entirely plausible theory seriously, try to test it (have other turkeys been slaughtered? Are there any turkeys over a year old?) And decide if it’s worth it to take some precautions.
There are a few distinctions that might help with your update:
determinism: knowledge of some system of causes now allows prediction of their outcomes until the end of time
closed world: we know all there is to know about the topic. Any search through our knowledge that fails to prove some hypothesis means that the hypothesis is false.
defeasibility: new observations can contradict earlier beliefs and result in withdrawal of earlier beliefs from one’s knowledge.
It seems like your use of the solar system example allows you to assume the first two distinctions apply to knowledge of the solar system. I’m not sure a physicist would agree with your choice of example, but I’m OK with it.
Human reasoning is defeasible, but until an observation provides an update, we do not necessarily consider the unknown beyond making passive observations of the real world.
From my limited understanding of the philosophy behind classic EA epistemics, believing what you know leads to refusing new observations that update your closed world. Thus the emphasis on incomplete epistemic confidence most of the time. So the thinking goes, it ensures that you’re not close-minded to always hold out that you think you might be wrong.
When running predictions, until someone provides a specific new item for a list of alternative outcomes (e.g, a new s-risk), the given list is all that is considered. Probabilities are divided among its alternatives when those alternatives are outcomes. The only exhaustive list of alternatives is one that includes a contradictory option, such as:
A
B
C
not A and not B and not C
and that covers all the possibilities. The interesting options are implicit in that last “not A and not B and not C”. This is not a big deal, since it’s usually the positive statements of options (A, B, or C) that are of interest.
So what’s a discovery? It seems like, in your model, it’s an alternative that is not listed directly. For example, given:
future 1
future 2
not future 1 and not future 2
An unexpected discovery belongs to future 3. All we know about it is that it is not future 1 and not future 2. One way to reframe your line of thought would be to ask:
how can we weight future 3?
A concrete example of discoveries of road surfacing strategies:
road paving that is concrete that absorbs CO2 with X efficiency (20%)
road paving that is made of plastic (5%)
road paving that is not concrete and that is not plastic (50%)
not road paving but serves to provide a road surface (24%)
something better than roads (1%)
That actually looks ridiculous. How do we know that there’s a 1% chance that we discover something better than roads?
In a longtermist framework, reasoning by analogy, lets consider some futures, and this example is fiction, not what I believe:
we make the planet slightly hotter and kill some species (1%)
something bad kills us all (29%)
something bad makes us suffer (15%)
not 1 or 2 or 3 (55%)
Future 4 has a probability of 55%. But future 4 is simply the unknowable future. What in heck is going on here?
If I understand what you’re trying to say, it’s that futures like future 4 in that example cannot be assigned a probability or risk. Furthermore, given that future 4 is a mutually exclusive alternative to futures 1, 2, and 3, those futures cannot be assigned a probability either.
Have I made an error in reasoning or did I misunderstand you?
Basically, predictions about the future are fine as long as they include the caveat “unless we figure out something else.” That caveat can’t be ascribed a meaningful probability because we can’t know discoveries before we discovery them, we can’t know things before we know them.
Well, my basic opinion about forecasting is that probabilities don’t inform the person receiving the forecast. Before you commit to weighting possible outcomes, you commit to at least two mutually exclusive futures, X and not X. So what you supply is a limitation on possible outcomes, either X or not X. At best, you’re aware of mutually exclusive alternative and specific futures. Then you can limit what not X means to something specific, for example, Y. So now you can say, “The future will contain X or Y.” That sort of analysis is enabled by your causal model. As your causal model improves, it becomes easier to supply a list of alternative future outcomes.
However, the future is not a game of chance, and there’s no useful interpretation to supply meaningful weights to the future prediction of any specific outcome, unless the outcomes belong to a game of chance, where you’re predicting rolls of a fair die, choice of a hand from a deck of cards, etc.
What’s worse, that does not limit your feelings about what probabilities apply. Those feelings can seem real and meaningful because they let you talk about lists of outcomes and which you think are more credible.
As a forecaster, I might supply outcomes in a forecast that I consider less credible along with those that I consider more credible. but if you ask me which options I consider credible, I might offer a subset of the list. So in that way weights can seem valuable, because they let you distinguish which you think are more credible and which you can rule out. But the weights also obscure that information because they can scale that credibility in confusing ways.
For example, I believe in outcomes A or B, but I offer A at 30%, B at 30%, C at 20%, D at 10%, and E at 10%. Have I communicated what I intended with my weights, namely, that A and B are credible, that C is somewhat credible, but D and E are not? Maybe I could adjust A and B to 40% and 40%, but now I’m fiddling with the likelihoods of C, D, and E, when all I really mean to communicate is that I like A or B as outcomes and C as an alternate. My probabilities communicate more and differently than I intend. I could make it clear with A and B each at 48% or something, but really now I’m trying to pretend I know what the chances of C, D, and E are, when all I really know about them is that my causal model doesn’t support their production much. I could go back and quantify that somehow, but information with which to do that is not available , so I have to pretend confidence in some estimation of the outcomes C, D, and E. My information is not useless, but it’s not relevant to weighting all possible outcomes against each other. If I’m forced to provide weights for all the listed outcomes, then I’m forced to figure out how to communicate my analysis in terms of weights so that the audience for my forecast understands what I intend to mean.
In general, analyzing causal models that determine possible futures is a distinct activity from weighting those futures. The valuable information is in the causal models and in the selection of futures based on those models. The extra information on epistemic confidence is not useful and pretends more information than a forecaster likely has. I would go as far as two tiers of selections, just to qualify what I think my causal model implies,
“A or B, and if not those, then C, but not D or E”.
Actually, I think someone reading my forecast with weights will just leave with that kind of information anyway. If they try to mathematically apply the weights I chose to communicate my tiers of selections, then they will be led astray, expecting precision when there wasn’t any. They would do better to get details of the causal models involved and determine whether those have any merit, particularly in cases of:
very different forecasts (I forecast A or B, everyone else forecasts D)
a single forecast predicting very different outcomes (A or B are contradictory outcomes)
a homogenous bunch of forecasts (A, B, or A or B)
a heterogenous bunch of forecasts (A, B, C or D, A or B, E)
so basically in all cases. What might distinguish superforecasters is not their grasp of probability or their ability to update bayesian priors or whatever, but rather the applicability of causal models they develop, and what those causal models emphasize as causes and consequences.
That’s the background of my thinking, now here’s how I think it relates to what you’re saying:
If discoveries influence future outcomes in unknown ways, and your information is insufficient to predict all outcomes, then your causal model makes predictions that belong under an assumption of an open world. You are less useful as a predictor of outcomes and more useful as an supplier of possible outcomes. If we are both forecasting, and I supply outcomes A and B; you might supply outcomes C and D; someone else might supply E, F, and G; yet another person might supply H. Our forecasts run from A to H so far, and they are not exhaustive. As forecasters, our job becomes to create lists of plausible futures, not to select from predetermined lists.
I think this is appropriate to conditions where development of knowledge or inventions is a human choice. Any forecast will depend not only on what is plausible under some causal model, but also on what future people want to explore and how they explore it. Forecasts in that scenario can influence the future, so better that they supply options rather than weight them.
A thanksgiving turkey has an excellent model that predicts the farmer wants him to be safe and happy. But an explanation of thanksgiving traditions tells us a lot more about the risks of slaughter than the number of days the turkey has been fed and protected.
With nuclear war, we have explanations for why nuclear exchange is possible, including as an outcome of a conflict.
Just like with the turkey, we should pay attention to the explanation, not just try to make predictions based on past data.
With all of this, probability terminology is baked into the language and it is hard to speak without incorporating it. With the previous post, it was co-authored, and I wanted to remove that phrase, but concessions were made.
I agree with you, but once again I don’t see the difference between the case of the turkey and nuclear war, versus the case of longtermism or AGI. “With nuclear war, we have explanations for why nuclear exchange is possible, including as an outcome of a conflict.” Just the same with AGI—we have explanations for why AGI seems possible, we have some evidence from scaling laws that describe how AI systems get better when given more resources, and ideas about what might motivate people to create more and more powerful AI systems, and why that might be dangerous, etc.
I am not an academically trained philosopher (rather, an engineer!), so I’m not sure what’s the best way to talk about probability and make it clear what kind of uncertainty we’re talking about. But in all cases, it seems that we should basically use a mixture of empirical evidence based on past experience (where available), and first-principles reasoning about what might be possible in the future. With some things—mathematical theorems are a great example—evidence might be hard to come by, so it might be very difficult to predict with precision. But it doesn’t seem like we are in fundamentally different, “unknowable” terrain—it’s more uncertain than nuclear war risk, which in turn is more uncertain than forecasting things like housing prices or wheat harvests, which in turn is more uncertain than forecasting that the sun will rise tomorrow. They all seem like part of the same spectrum, and the long-term future of civilization seems important enough that it’s worth thinking about even amid high uncertainty.
How about this: let’s split future events into two groups. 1) Events that are not influenced by people and 2) Events that are influenced by people.
In 1, we can create predictive models, use probability, even calculate uncertainty. All the standard rules apply, Bayesian and otherwise.
In 2, we can still create predictive models, but they’ll be nonsensical. That’s because we cannot know how knowledge creation will affect 2. We don’t even need any fancy reasoning, it’s already implied in the definition of terms like knowledge creation and discovery. You can’t discover something before you discover it, before it’s created.
So, up until recently, the bodies of the solar system fell into category 1. We can predict their positions many years hence, as long as people don’t get involved. However, once we are capable, there’s no way now to know what we’ll do with the planets and asteroids in the future. Maybe we’ll find use for some mineral found predominantly in some asteroids, or maybe we’ll use a planet to block heat from the sun as it expands, or maybe we’ll detect some other risk/benefit and make changes accordingly.
This is an extreme example, but it applies across the board. Any time human knowledge creation impacts a system, there’s no way to model that impact before the knowledge is created.
Therefore, longtermism hinges on the idea that we have some idea of how to impact the long term future. But even more than the solar system example, that future will be overwhelmingly dominated by new knowledge, and hence unknowable to us to today, unable to be anticipated.
Sure, we can guess, and in the case of known future threats like nuclear war, we should guess and should try to ameliorate risk. But those problems apply to the very near future as well, they are problems facing us today (that’s why we know a fair bit about them). We shouldn’t waste effort trying to calculate the risk because we can’t do that for items in group 2. Instead, we know from our best explanations that nuclear war is a risk.
In this way the threat of nuclear is like the turkey—if the turkey even hears a rumor about thanksgiving traditions, should it sit down and try to update its priors? Or take the entirely plausible theory seriously, try to test it (have other turkeys been slaughtered? Are there any turkeys over a year old?) And decide if it’s worth it to take some precautions.
There are a few distinctions that might help with your update:
determinism: knowledge of some system of causes now allows prediction of their outcomes until the end of time
closed world: we know all there is to know about the topic. Any search through our knowledge that fails to prove some hypothesis means that the hypothesis is false.
defeasibility: new observations can contradict earlier beliefs and result in withdrawal of earlier beliefs from one’s knowledge.
It seems like your use of the solar system example allows you to assume the first two distinctions apply to knowledge of the solar system. I’m not sure a physicist would agree with your choice of example, but I’m OK with it.
Human reasoning is defeasible, but until an observation provides an update, we do not necessarily consider the unknown beyond making passive observations of the real world.
From my limited understanding of the philosophy behind classic EA epistemics, believing what you know leads to refusing new observations that update your closed world. Thus the emphasis on incomplete epistemic confidence most of the time. So the thinking goes, it ensures that you’re not close-minded to always hold out that you think you might be wrong.
When running predictions, until someone provides a specific new item for a list of alternative outcomes (e.g, a new s-risk), the given list is all that is considered. Probabilities are divided among its alternatives when those alternatives are outcomes. The only exhaustive list of alternatives is one that includes a contradictory option, such as:
A
B
C
not A and not B and not C
and that covers all the possibilities. The interesting options are implicit in that last “not A and not B and not C”. This is not a big deal, since it’s usually the positive statements of options (A, B, or C) that are of interest.
So what’s a discovery? It seems like, in your model, it’s an alternative that is not listed directly. For example, given:
future 1
future 2
not future 1 and not future 2
An unexpected discovery belongs to future 3. All we know about it is that it is not future 1 and not future 2. One way to reframe your line of thought would be to ask:
how can we weight future 3?
A concrete example of discoveries of road surfacing strategies:
road paving that is concrete that absorbs CO2 with X efficiency (20%)
road paving that is made of plastic (5%)
road paving that is not concrete and that is not plastic (50%)
not road paving but serves to provide a road surface (24%)
something better than roads (1%)
That actually looks ridiculous. How do we know that there’s a 1% chance that we discover something better than roads?
In a longtermist framework, reasoning by analogy, lets consider some futures, and this example is fiction, not what I believe:
we make the planet slightly hotter and kill some species (1%)
something bad kills us all (29%)
something bad makes us suffer (15%)
not 1 or 2 or 3 (55%)
Future 4 has a probability of 55%. But future 4 is simply the unknowable future. What in heck is going on here?
If I understand what you’re trying to say, it’s that futures like future 4 in that example cannot be assigned a probability or risk. Furthermore, given that future 4 is a mutually exclusive alternative to futures 1, 2, and 3, those futures cannot be assigned a probability either.
Have I made an error in reasoning or did I misunderstand you?
Beautiful! We can’t determine “something we haven’t thought of” as simply “1 - all the things we’ve thought of”.
Basically, predictions about the future are fine as long as they include the caveat “unless we figure out something else.” That caveat can’t be ascribed a meaningful probability because we can’t know discoveries before we discovery them, we can’t know things before we know them.
Well, my basic opinion about forecasting is that probabilities don’t inform the person receiving the forecast. Before you commit to weighting possible outcomes, you commit to at least two mutually exclusive futures, X and not X. So what you supply is a limitation on possible outcomes, either X or not X. At best, you’re aware of mutually exclusive alternative and specific futures. Then you can limit what not X means to something specific, for example, Y. So now you can say, “The future will contain X or Y.” That sort of analysis is enabled by your causal model. As your causal model improves, it becomes easier to supply a list of alternative future outcomes.
However, the future is not a game of chance, and there’s no useful interpretation to supply meaningful weights to the future prediction of any specific outcome, unless the outcomes belong to a game of chance, where you’re predicting rolls of a fair die, choice of a hand from a deck of cards, etc.
What’s worse, that does not limit your feelings about what probabilities apply. Those feelings can seem real and meaningful because they let you talk about lists of outcomes and which you think are more credible.
As a forecaster, I might supply outcomes in a forecast that I consider less credible along with those that I consider more credible. but if you ask me which options I consider credible, I might offer a subset of the list. So in that way weights can seem valuable, because they let you distinguish which you think are more credible and which you can rule out. But the weights also obscure that information because they can scale that credibility in confusing ways.
For example, I believe in outcomes A or B, but I offer A at 30%, B at 30%, C at 20%, D at 10%, and E at 10%. Have I communicated what I intended with my weights, namely, that A and B are credible, that C is somewhat credible, but D and E are not? Maybe I could adjust A and B to 40% and 40%, but now I’m fiddling with the likelihoods of C, D, and E, when all I really mean to communicate is that I like A or B as outcomes and C as an alternate. My probabilities communicate more and differently than I intend. I could make it clear with A and B each at 48% or something, but really now I’m trying to pretend I know what the chances of C, D, and E are, when all I really know about them is that my causal model doesn’t support their production much. I could go back and quantify that somehow, but information with which to do that is not available , so I have to pretend confidence in some estimation of the outcomes C, D, and E. My information is not useless, but it’s not relevant to weighting all possible outcomes against each other. If I’m forced to provide weights for all the listed outcomes, then I’m forced to figure out how to communicate my analysis in terms of weights so that the audience for my forecast understands what I intend to mean.
In general, analyzing causal models that determine possible futures is a distinct activity from weighting those futures. The valuable information is in the causal models and in the selection of futures based on those models. The extra information on epistemic confidence is not useful and pretends more information than a forecaster likely has. I would go as far as two tiers of selections, just to qualify what I think my causal model implies,
“A or B, and if not those, then C, but not D or E”.
Actually, I think someone reading my forecast with weights will just leave with that kind of information anyway. If they try to mathematically apply the weights I chose to communicate my tiers of selections, then they will be led astray, expecting precision when there wasn’t any. They would do better to get details of the causal models involved and determine whether those have any merit, particularly in cases of:
very different forecasts (I forecast A or B, everyone else forecasts D)
a single forecast predicting very different outcomes (A or B are contradictory outcomes)
a homogenous bunch of forecasts (A, B, or A or B)
a heterogenous bunch of forecasts (A, B, C or D, A or B, E)
so basically in all cases. What might distinguish superforecasters is not their grasp of probability or their ability to update bayesian priors or whatever, but rather the applicability of causal models they develop, and what those causal models emphasize as causes and consequences.
That’s the background of my thinking, now here’s how I think it relates to what you’re saying:
If discoveries influence future outcomes in unknown ways, and your information is insufficient to predict all outcomes, then your causal model makes predictions that belong under an assumption of an open world. You are less useful as a predictor of outcomes and more useful as an supplier of possible outcomes. If we are both forecasting, and I supply outcomes A and B; you might supply outcomes C and D; someone else might supply E, F, and G; yet another person might supply H. Our forecasts run from A to H so far, and they are not exhaustive. As forecasters, our job becomes to create lists of plausible futures, not to select from predetermined lists.
I think this is appropriate to conditions where development of knowledge or inventions is a human choice. Any forecast will depend not only on what is plausible under some causal model, but also on what future people want to explore and how they explore it. Forecasts in that scenario can influence the future, so better that they supply options rather than weight them.
I love it. Creating lists of plausible outcomes is very valuable, we can leave alone to idea of assigning probabilities.