I am confused about the precise claim made regarding the Hilbert Hotel and measure theory. When you say “we have no measure over the set of all possible futures”, do you mean that no such measures exist (which would be incorrect without further requirements: https://en.wikipedia.org/wiki/Dirac_measure , https://encyclopediaofmath.org/wiki/Wiener_measure ), or that we don’t have a way of choosing the right measure? If it is the latter, I agree that this is an important challenge, but I’d like to highlight that the situation is not too different from the finite case in which there is still an infinitude of possible measures for a given set to choose from.
(I was also confused by this, and wrote a couple of comments in response. I actually think they don’t add much to the overall discussion, especially now that Vaden has clarified below what kind of argument they were trying to make. But maybe you’re interested given we’ve had similar initial confusions.)
Yup, the latter. This is why the lack-of-data problem is the other core part of my argument. Once data is in the picture, now we can start to get traction. There is something to fit the measure to, something to be wrong about, and a means of adjudicating between which choice of measure is better than which other choice. Without data, all this probability talk is just idol speculation painted with a quantitative veneer.
Ok, makes sense. I think that our ability to make predictions about the future steeply declines with increasing time horizions, but find it somewhat implausible that it would become entirely uncorrelated with what is actually going to happen in finite time. And it does not seem to be the case that data supporting long term predictions is impossible to get by: while it might be pretty hard to predict whether AI risk is going to be a big deal by whatever measure, I can still be fairly certain that the sun will exist in a 1000 years; in part due to a lot of data collection and hypothesis testing done by physicist.
“while it might be pretty hard to predict whether AI risk is going to be a big deal by whatever measure, I can still be fairly certain that the sun will exist in a 1000 years”
These two things are correlated.
They are, but I don’t think that the correlation is strong enough to invalidate my statement. P(sun will exist|AI risk is a big deal) seems quite large to me.
Obviously, this is not operationalized very well...
Yes, there are certain rare cases where longterm prediction is possible. Usually these involve astronomical systems, which are unique because they are cyclical in nature and unusually unperturbed by the outside environment. Human society doesn’t share any of these properties unfortunately, and long term historical prediction runs into the impossibility proof in epistemology anyway.
I don’t think I buy the impossibility proof as predicting future knowledge in a probabilistic manner is possible (most simply, I can predict that if I flip a coin now, that there’s a 50⁄50 chance I’ll know the coin landed on heads/tails in a minute). I think there is some important true point behind your intuition about how knowledge (especially of more complex form than about a coin flip) is hard to predict, but I am almost certain you won’t be able to find any rigorous mathematical proof for this intuition because reality is very fuzzy (in a mathematical sense, what exactly is the difference between the coin flip and knowledge about future technology?) so I’d be a lot more excited about other types of arguments (which will likely only support weaker claims).
I don’t think I buy the impossibility proof as predicting future knowledge in a probabilistic manner is possible (most simply, I can predict that if I flip a coin now, that there’s a 50⁄50 chance I’ll know the coin landed on heads/tails in a minute).
In this example you aren’t predicting future knowledge, you’re predicting that you’ll have knowledge in the future—that is, in one minute, you will know the outcome of the coin flip. I too think we’ll gain knowledge in the future, but that’s very different from predicting the content of that future knowledge today. It’s the difference between saying “sometime in the future we will have a theory that unifies quantum mechanics and general relativity” and describing the details of future theory itself.
I am almost certain you won’t be able to find any rigorous mathematical proof for this intuition
The proof is here: https://vmasrani.github.io/assets/pdf/poverty_historicism_quote.pdf.
(And who said proofs have to be mathematical? Proofs have to be logical—that is, concerned with deducing true conclusions from true premises—not mathematical, although they often take mathematical form.)
The proof [for the impossibility of certain kinds of long-term prediction] is here: https://vmasrani.github.io/assets/pdf/poverty_historicism_quote.pdf.
Note that in that text Popper says:
The argument does not, of course, refute the possibility of every kind of social prediction; on the contrary, it is perfectly compatible with the possibility of testing social theories—for example economic theories—by way of predicting that certain developments will take place under certain conditions. It only refutes the possibility of predicting historical developments to the extent to which they may be influenced by the growth of our knowledge.
And that he rejects only
the possibility of a theoretical history; that is to say, of a historical social science that would correspond to theoretical physics.
My guess is that everyone in this discussion (including MacAskill and Greaves) agree with this, at least as claims about what’s currently possible in practice. On the other hand, it seems uncontroversial that some form of long-run predictions are possible (e.g. above you’ve conceded they’re possible for some astronomical systems).
Thus it seems to me that the key question is whether longtermism requires the kind of predictions that aren’t feasible—or whether longtermism is viable with the sort of predictions we can currently make. And like Flodorner I don’t think that mathematical or logical arguments will be much help with that question.
Why can’t we be longtermists while being content to “predict that certain developments will take place under certain conditions”?
Regarding Popper’s claim that it’s impossible to “predict historical developments to the extent to which they may be influenced by the growth of our knowledge”:
I can see how there might be a certain technical sense in which this is true, though I’m not sufficiently familiar with Popper’s formal arguments to comment in detail.
However, I don’t think the claim can be true in the everyday sense (rather than just for a certain technical sense of “predicting”) that arguably is relevant when making plans for the future.
For example, consider climate change. It seems clear that between now and, say, 2100 our knowledge will grow in various ways that are relevant: we’ll better understand the climate system, but perhaps even more crucially we’ll know more about the social and economic aspects (e.g. how people will to adapt to a warmer climate, how much emission reductions countries will pursue, …) and on how much progress we’ve made with developing various relevant technologies (e.g. renewable energy, batteries, carbon capture and storage, geoengineering, …).
The latter two seem like paradigm examples of things that would be “impossible to predict” in Popper’s sense. But does it follow that regarding climate change we should throw our hands up in the air and do nothing because it’s “impossible to predict the future”? Or that climate change policy faces some deep technical challenge?
Maybe all we are doing when choosing between climate change policies in Popper’s terms is “predicting that certain developments will take place under certain conditions” rather than “predicting historical developments” simpliciter. But as I said, then this to me just suggests that as longtermists we will be just fine using “predictions of certain developments under certain conditions”.
I find it hard to see why there would be a qualitative difference between longtermism (as a practical project) and climate change mitigation which implies that the former is infeasible while the latter is a worthwhile endeavor.
In this example [coin flip] you aren’t predicting future knowledge, you’re predicting that you’ll have knowledge in the future—that is, in one minute, you will know the outcome of the coin flip.
If we’re giving a specific probability distribution for the outcome of the coin flip, it seems like we’re doing more than that:
Consider that we would predict to know the outcome of the coin flip in one minute no matter what we think the odds of heads are.
Therefore, if we do give specific odds (such as 50%), we’re doing more than just saying we’ll know the outcome in the future.
Hi Max_Daniel! I’m sympathetic to both your and Vaden’s arguments, so I may try to bridge the gap on climate change vs. your Christmas party vs. longtermism.
Climate change is a problem now, and we have past data to support projecting already-observed effects into the future. So statements of the sort “if current data projected forward with no notable intervention, the Earth would be uninhabitable in x years.” This statement is reliant on some assumptions about future data vs. past data, but we can be reasonably clear about them and debate them.
Future knowledge will undoubtedly help things and reframe certain problems, but a key point is that we know where to start gathering data on some of the aspects you raise: “how ppl will adapt”, “how can we develop renewable energy or batteries”, etc, because climate change is already a well defined problem. We have current knowledge that will help us get off the ground.
I agree the measure theoretic arguments may prove too much, but the number of people at your Christmas party is an unambiguously posed question for which you have data on how many people you invited, how flaky your friends are, etc.
In both cases, you may use probabilistic predictions, based on a set of assumptions, to compel others to act on climate change or compel yourself to invite more people.
the key question is whether longtermism requires the kind of predictions that aren’t feasible
At the risk of oversimplification by using AI Safety example as a representative longtermist argument, the key difference is that we haven’t created or observed human-level AI, or even those which can adaptively set their own goals.
There are meaningful arguments we can use to compel others to discuss issues of safety (in algorithm development, government regulation, etc). After all, it will be a human process to develop and deploy these AI, and we can set guardrails by focused discussion today.
Vaden’s point seems to be that arguments that rely on expected values or probabilities are of significantly less value in this case. We are not operating in a well-defined problem, with already-available or easily -collectable data, because we haven’t even created the AI.
This seems to be the key point about “predicting future knowledge” being fundamentally infeasible (just as people in 1900 couldn’t meaningfully reason about the internet, let alone make expected utility calculations). Again, we’re not as ignorant as ppl in 1900 and may have a sense this problem is important, but can we actually make concrete progress with respect to killer robots today?
Everyone on this forum may have their own assumptions about the future AI, or climate change for that matter. We may not ever be able to align our priors and sufficiently agree on the future, but for the purposes of planning and allocating resources, the discussion around climate change seems significantly more grounded.
Hi brekels, I think these are fair points. In particular, I think we may be able to agree on the following statement as well as more precise versions of it:
We may not ever be able to align our priors and sufficiently agree on the future, but for the purposes of planning and allocating resources, the discussion around climate change seems significantly more grounded [than the one about e.g. AI safety].
In my view, the key point is that, say, climate change and AI safety differ in degree but not in kind regarding whether we can make probabilistic predictions, should take action now, etc.
In particular, consider the following similarities:
I agree that for climate change we utilize extrapolations of current trends such as “if current data projected forward with no notable intervention, the Earth would be uninhabitable in x years.”—But in principle we can do the same for AI safety, e.g. “if Moore’s Law continued, we could buy a brain-equivalent of compute for $X in Y years.”
Yes, it’s not straightforward to say what a “brain-equivalent of compute” is, or why this matters. But neither is it straightforward to e.g. determine when the Earth becomes “uninhabitable”. (Again, I might concede that the latter notion is in some sense easier to define—my point it just that I don’t see a qualitative difference.)
You say we haven’t yet observed human-level AI. But neither have we observed (at least not directly an on a planetary scale), say, +6 degrees of global warming compared to pre-industrial times. Yes, we have observed anthropogenic climate change, but we’ve also observed AI systems developed by humans including specific failure modes (e.g. misspecified rewards, biased training data, or lack of desired generalization in response to distributional shift).
In various ways it sounds right to me that we have “more data” on climate change, or that the problem of more severe climate change is “more similar” to current climate change than the problem of misaligned transformative AI is to current AI failure modes. But again, to me this seems like “merely” a difference in degree.
Separately, I think that if we try hard to find the most effective intervention to avoid some distant harm (say, one we think would occur in the year 2100, or even 2050), we will have to confront the “less well-defined” and “more uncertain” aspects of the future anyway, no matter whether the harm we’re considering has some relatively well-understood core (such as climate change).
This is because, whether we like it or not, these less well-defined issues such as the future of technology, governance, economic and political systems, etc., as well as interactions with other, less predictable, issues (e.g. migration, war, inequality, …) will make a massive difference to how some apparently predictable harm will in fact affect different people, how we in fact might be able to prevent or respond to it etc.
E.g. it’s not that much use if I can predict how much warming we’d get by 2100 conditional on a certain amount of emissions (though note that even in this seemingly “well-defined” case a lot hinges on which prior over climate sensitivity we use, since that has a large affect on the a posteriori probability of bad tail scenarios—and how to determine that prior isn’t something we can just “read off” from any current observation) if I don’t know for even the year 2050 the state of nuclear fusion, carbon capture and storage, geoengineering, solar cell efficiency, batteries, US-China relations, or whether in the meantime a misaligned AI system killed everyone.
It seems to me that the alternative, i.e. planning based on just those aspects of the future that seem “well-defined” or “predictable”, leads to things like the Population Bomb or Limits to Growth, i.e. things that have a pretty bad track record.
It seems like the proof critically hinges on assertion 2) which is not proven in your link. Can you point me to the pages of the book that contain the proof?
I agree that proofs are logical, but since we’re talking about probabilistic predictions, I’d be very skeptical of the relevance of a proof that does not involve mathematical reasoning,
Yep it’s Chapter 22 of The Open Universe (don’t have a pdf copy unfortunately)