I’m not an AI Risk expert, so any answer I gave to 1 would just be polluting. Let’s say my probabilities are A and B for a two-parameter Carlsmith Model, and those parameters could be 3% or 33% as per your example. So a simple mean of this situation is A = (3% + 33%)/2 = 18% and B is the same, so simple mean is ~3%. The geometric mean is more like 1%.
The most important point I wanted to get across is that the distribution of probabilities can be important in some contexts. If something important happens to our response at a 1% risk then it is useful to know that we will observe less than 1% risk in 3⁄4 of all possible worlds (ie worlds when A or B are at 3%). In the essay I argue that since strategies for living in a low-risk world are likely to be different from strategies for living in a high-risk world (and both sets of strategies are likely to be different from optimal strategy if we live in a simple-mean medium-risk world), distribution is what matters.
If we agree about that (which I’m not certain we do—I think possibly you are arguing that you can and should always reduce probabilities-of-probabilities to just probabilities?), then I don’t really have a strong position on your other point about geometric mean of odds vs simple mean. The most actionable summary statistic depends on the context. While I think geometric mean of odds is probably the correct summary statistic for this application, I accept that there’s an argument to be had on the point.
I’m not an AI Risk expert, so any answer I gave to 1 would just be polluting
I can understand if you don’t want to state those probabilities publicly. But then I don’t know how to resolve what feels to me like an inconsistency. I think you have to bite one of these two bullets:
Most survey respondents are wrong in (some or most of) their probabilities for the “Conditional on …” questions, and your best guess at (some or most of) these probabilities is much lower.
The probability of AGI catastrophe conditional on being invented is much higher than 1.6%
Which one is it? Or is there a way to avoid both bullets while having consistent beliefs (then I would probably need concrete probabilities to be convinced)?
Hmm… I don’t see a contradiction here. I note you skimmed some of the methods, so it might perhaps help explain the contradiction to read the second half of section 3.3.2?
The bullet I bite is the first—most survey respondents are wrong, because they give point probabilities (which is what I asked for, in fairness) whereas in reality there will be uncertainty over those probabilities. Initiatively we might think that this uncertainty doesn’t matter because it will ‘cancel out’ (ie every time you are uncertain in a low direction relative to the truth I am uncertain in a high direction relative to the truth) but in reality—given specific structural assumptions in the Carlsmith Model—this is not true. In reality, the low-end uncertainty compounds and the high-end uncertainty is neutered, which is why you end up with an asymmetric distribution favouring very low-risk outcomes.
Thanks for biting a bullet, I think I am making progress in understanding your view.
I also realized that part of my “feeling of inconsistency” comes from not having realized that the table in section 3.2 reports geometric mean of odds instead of the average, and where the average would be lower.
Lets say we have a 2-parameter Carlsmith model, where we estimate probabilities P(A) and P(B|A), in order to get to a final estimate of the probability P(A∩B). Lets say we have uncertainty over our probability estimates, and we estimate P(A) using a random variable X, and estimate P(B|A) using a random variable Y. To make the math easier, I am going to assume that X,Y are discrete (I can repeat it for a more general case, eg using densities if requested): We have k possible estimates ai for P(A), and pi:=P(X=ai) is the probability that X assigns the value ai for our estimate of P(A). Similarly, bi are estimates for P(B|A) that Y outputs with probability qi:=P(Y=bi). We also have ∑ki=1pi=∑ki=1qi=1.
Your view seems to be something like “To estimate P(A∩B), we should sample from X and Y, and then compute the geometric mean of odds for our final estimate.”
Sampling from X and Y, we get values ai⋅bi with probability pi⋅qi, and then taking the geometric mean of odds would result in the formula
P(A∩B)=k∏i=1k∏j=1(aibj1−aibj)piqj.
Whereas my view is “We should first collapse the probabilities by taking the mean, and then multiply”, that is we first calculate P(A)=∑ki=1aipi and P(B|A)=∑kj=1biqi, for a final formula of
P(A∩B)=(k∑i=1aipi)(k∑j=1bjqj).
And you are also saying ”P(A∩B)=P(A)⋅P(B|A) is still true, but the above naive estimates for P(A) and P(B) are not good, and should actually be different (and lower than typical survey respondents in the case of AI xrisk estimates).” (I can’t derive a precise formula from your comments or my skim of the article, but I don’t think thats a crucial issue.)
Do I characterize your view roughly right? (Not saying that is your whole view, just parts of it).
I’m not an AI Risk expert, so any answer I gave to 1 would just be polluting. Let’s say my probabilities are A and B for a two-parameter Carlsmith Model, and those parameters could be 3% or 33% as per your example. So a simple mean of this situation is A = (3% + 33%)/2 = 18% and B is the same, so simple mean is ~3%. The geometric mean is more like 1%.
The most important point I wanted to get across is that the distribution of probabilities can be important in some contexts. If something important happens to our response at a 1% risk then it is useful to know that we will observe less than 1% risk in 3⁄4 of all possible worlds (ie worlds when A or B are at 3%). In the essay I argue that since strategies for living in a low-risk world are likely to be different from strategies for living in a high-risk world (and both sets of strategies are likely to be different from optimal strategy if we live in a simple-mean medium-risk world), distribution is what matters.
If we agree about that (which I’m not certain we do—I think possibly you are arguing that you can and should always reduce probabilities-of-probabilities to just probabilities?), then I don’t really have a strong position on your other point about geometric mean of odds vs simple mean. The most actionable summary statistic depends on the context. While I think geometric mean of odds is probably the correct summary statistic for this application, I accept that there’s an argument to be had on the point.
I can understand if you don’t want to state those probabilities publicly. But then I don’t know how to resolve what feels to me like an inconsistency. I think you have to bite one of these two bullets:
Most survey respondents are wrong in (some or most of) their probabilities for the “Conditional on …” questions, and your best guess at (some or most of) these probabilities is much lower.
The probability of AGI catastrophe conditional on being invented is much higher than 1.6%
Which one is it? Or is there a way to avoid both bullets while having consistent beliefs (then I would probably need concrete probabilities to be convinced)?
Hmm… I don’t see a contradiction here. I note you skimmed some of the methods, so it might perhaps help explain the contradiction to read the second half of section 3.3.2?
The bullet I bite is the first—most survey respondents are wrong, because they give point probabilities (which is what I asked for, in fairness) whereas in reality there will be uncertainty over those probabilities. Initiatively we might think that this uncertainty doesn’t matter because it will ‘cancel out’ (ie every time you are uncertain in a low direction relative to the truth I am uncertain in a high direction relative to the truth) but in reality—given specific structural assumptions in the Carlsmith Model—this is not true. In reality, the low-end uncertainty compounds and the high-end uncertainty is neutered, which is why you end up with an asymmetric distribution favouring very low-risk outcomes.
Thanks for biting a bullet, I think I am making progress in understanding your view.
I also realized that part of my “feeling of inconsistency” comes from not having realized that the table in section 3.2 reports geometric mean of odds instead of the average, and where the average would be lower.
Lets say we have a 2-parameter Carlsmith model, where we estimate probabilities P(A) and P(B|A), in order to get to a final estimate of the probability P(A∩B). Lets say we have uncertainty over our probability estimates, and we estimate P(A) using a random variable X, and estimate P(B|A) using a random variable Y. To make the math easier, I am going to assume that X,Y are discrete (I can repeat it for a more general case, eg using densities if requested): We have k possible estimates ai for P(A), and pi:=P(X=ai) is the probability that X assigns the value ai for our estimate of P(A). Similarly, bi are estimates for P(B|A) that Y outputs with probability qi:=P(Y=bi). We also have ∑ki=1pi=∑ki=1qi=1.
Your view seems to be something like “To estimate P(A∩B), we should sample from X and Y, and then compute the geometric mean of odds for our final estimate.”
Sampling from X and Y, we get values ai⋅bi with probability pi⋅qi, and then taking the geometric mean of odds would result in the formula
P(A∩B)=k∏i=1k∏j=1(aibj1−aibj)piqj.
Whereas my view is “We should first collapse the probabilities by taking the mean, and then multiply”, that is we first calculate P(A)=∑ki=1aipi and P(B|A)=∑kj=1biqi, for a final formula of
P(A∩B)=(k∑i=1aipi)(k∑j=1bjqj).
And you are also saying ”P(A∩B)=P(A)⋅P(B|A) is still true, but the above naive estimates for P(A) and P(B) are not good, and should actually be different (and lower than typical survey respondents in the case of AI xrisk estimates).” (I can’t derive a precise formula from your comments or my skim of the article, but I don’t think thats a crucial issue.)
Do I characterize your view roughly right? (Not saying that is your whole view, just parts of it).