I think the contrast with elections is an important and interesting one. I’ll start by saying that being able to coarse-grain the set of all possible worlds into two possibilities doesn’t mean we should assign both possibilities positive probability. Consider the set of all possible sequences of infinite coin tosses. We can coarse-grain those sequences into two sets: the ones where finitely many coins land heads, and the ones where infinitely many coins lands heads. But, assuming we’re actually going to toss infinitely many coins, and assuming each coin is fair, the first set of sequences has probability zero and the second set has probability one.
In the election case, we have a good understanding of the mechanism by which elections are (hopefully) won. In this simple case with a plurality rule, we just want to know which candidate will get the most votes. So we can define probability distributions over the possible number of votes cast, and probability distributions over possible distributions of those votes to different candidates (where vote distributions are likely conditional on overall turnout), and coarse-grain those various vote distributions into the possibility of each candidate winning. This is a simple case, and no doubt real-world election models have many more parameters, but my point is that we understand the relevant possibility space and how it relates to our outcomes of interest fairly well. I don’t think we have anything like this understanding in the AGI case.
Great, I think we’ve gotten to the crux. I agree we have much worse understanding in the AGI case but I think we easily have enough understanding to assign positive probabilities, and likely substantial ones. I agree more detailed models are ideal but in some cases they’re impractical and you have to do the best you’ve got with the evidence you have. Also, this is a matter of degree and not binary, and I think often people take explicit models too literally/seriously and don’t account enough for model uncertainty e.g. putting too much faith in oversimplified economic models, underestimating how much explicit climate models might be missing out on tail risks or unknown unknowns.
I’d be extremely curious to get your take on why AGI forecasting is so different from the long-term speculative forecasts in the piece Nuno linked above, of which many turned out to be true.
I don’t have a fully-formed opinion here, but for now I’ll just note that the task that the examined futurists are implicitly given is very different from assigning a probability distribution to a variable based on parameters. Rather, the implicit task is to say some stuff that you think will happen. Then we’re judging whether those things happen. But I’m not sure how to translate the output from the task into action. (E.g., Asimov says X will happen, and so we should do Y.)
Agree that these are different; I think they aren’t different enough to come anywhere close to meaning that longtermism can’t be action-guiding though!
Would love to hear more from you when you’ve had a chance to form more of an opinion :)
Edit: also, it seems like one could mostly refute this objection by just finding times when someone did something with the intention of affecting the future in 10-20 years (which many people give some weight to for AGI timelines), and the action had the intended effect? This seems trivial.
I think the contrast with elections is an important and interesting one. I’ll start by saying that being able to coarse-grain the set of all possible worlds into two possibilities doesn’t mean we should assign both possibilities positive probability. Consider the set of all possible sequences of infinite coin tosses. We can coarse-grain those sequences into two sets: the ones where finitely many coins land heads, and the ones where infinitely many coins lands heads. But, assuming we’re actually going to toss infinitely many coins, and assuming each coin is fair, the first set of sequences has probability zero and the second set has probability one.
In the election case, we have a good understanding of the mechanism by which elections are (hopefully) won. In this simple case with a plurality rule, we just want to know which candidate will get the most votes. So we can define probability distributions over the possible number of votes cast, and probability distributions over possible distributions of those votes to different candidates (where vote distributions are likely conditional on overall turnout), and coarse-grain those various vote distributions into the possibility of each candidate winning. This is a simple case, and no doubt real-world election models have many more parameters, but my point is that we understand the relevant possibility space and how it relates to our outcomes of interest fairly well. I don’t think we have anything like this understanding in the AGI case.
Great, I think we’ve gotten to the crux. I agree we have much worse understanding in the AGI case but I think we easily have enough understanding to assign positive probabilities, and likely substantial ones. I agree more detailed models are ideal but in some cases they’re impractical and you have to do the best you’ve got with the evidence you have. Also, this is a matter of degree and not binary, and I think often people take explicit models too literally/seriously and don’t account enough for model uncertainty e.g. putting too much faith in oversimplified economic models, underestimating how much explicit climate models might be missing out on tail risks or unknown unknowns.
I’d be extremely curious to get your take on why AGI forecasting is so different from the long-term speculative forecasts in the piece Nuno linked above, of which many turned out to be true.
I don’t have a fully-formed opinion here, but for now I’ll just note that the task that the examined futurists are implicitly given is very different from assigning a probability distribution to a variable based on parameters. Rather, the implicit task is to say some stuff that you think will happen. Then we’re judging whether those things happen. But I’m not sure how to translate the output from the task into action. (E.g., Asimov says X will happen, and so we should do Y.)
Agree that these are different; I think they aren’t different enough to come anywhere close to meaning that longtermism can’t be action-guiding though!
Would love to hear more from you when you’ve had a chance to form more of an opinion :)
Edit: also, it seems like one could mostly refute this objection by just finding times when someone did something with the intention of affecting the future in 10-20 years (which many people give some weight to for AGI timelines), and the action had the intended effect? This seems trivial.