I agree that the arith-vs-geo question is basically the crux when it comes to whether this essay should move FF’s ‘fair betting probabilities’ - it sounds like everyone is pretty happy with the point about distributions and I’m really pleased about that because it was the main point I was trying to get across. I’m even more pleased that there is background work going on in the analysis of uncertainty space, because that’s an area where public statements by AI Risk organisations have sometimes lagged behind the state of the art in other risk management applications.
With respect to the crux, I hate to say it—because I’d love to be able to make as robust a claim for the prize as possible—but I’m not sure there is a principled reason for using geomean over arithmean for this application (or vice versa). The way I view it, they are both just snapshots of what is ‘really’ going on, which is the full distribution of possible outcomes given in the graphs / model. By analogy, I would be very suspicious of someone who always argued the arithmean would be a better estimate of central tendency than the median for every dataset / use case! I agree with you the problem of which is best for this particular dataset / use case is subtle, and I think I would characterise it as being a question of whether my manipulations of people’s forecasts have retained some essential ‘forecast-y’ characteristic which means geomean is more appropriate for various features it has, or whether they have been processed into having some sort of ‘outcome-y’ characteristic in which case arithmean is more appropriate. I take your point below in the coin example and the obvious superiority of arithmeans for that application, but my interpretation is that the FF didn’t intend for the ‘fair betting odds’ position to limit discussion about alternate ways to think about probabilities (“Applicants need not agree with or use our same conception of probability”)
However, to be absolutely clear, even if geomean was the right measure of central tendency I wouldn’t expect the judges to pay that particular attention—if all I had done was find a novel way of averaging results then my argument would basically be mathematical sophistry, perhaps only one step better than simply redefining ‘AI Risk’ until I got a result I liked. I think the distribution point is the actually valuable part of the essay, and I’m quite explicit in the essay that neither geomean nor arithmean is a good substitute for the full distribution. While I would obviously be delighted if I could also convince you my weak preference for geomean as a summary statistic was actually robust and considered, I’m actually not especially wedded to the argument for one summary statistic over the other. I did realise after I got my results that the crux for moving probabilities was going to be a very dry debate about different measures of central tendency, but I figured since the Fund was interested in essays on the theme of “a bunch of this AI stuff is basically right, but we should be focusing on entirely different aspects of the problem” (even if they aren’t being strictly solicited for the prize) the distribution bit of the essay might find a readership there anyway.
By the way, I know your four-step argument is intended just as a sketch of why you prefer arithmean for this application, but I do want to just flag up that I think it goes wrong on step 4, because acting according to arithmean probability (or geomean, for that matter) throws away information about distributions. As I mention here and elsewhere, I think the distribution issue is far more important than the geo-vs-arith issue, so while I don’t really feel strongly if I lose the prize because the judges don’t share my intuition that geomean is a slightly better measure of central tendency I would be sad to miss out because the distribution point was misunderstood! I describe in Section 5.2.2 how the distribution implied by my model would quite radically change some funding decisions, probably by more than an argument taking the arithmean to 3% (of course, if you’re already working on distribution issues then you’ve probably already reached those conclusions and so I won’t be changing your mind by making them—but in terms of publicly available arguments about AI Risk I’d defend the case that the distribution issue implies more radical redistribution of funds than changing the arithmean to 1.6%). So I think “act according to that mean probability” is wrong for many important decisions you might want to take—analogous to buying a lot of trousers with 1.97 legs in my example in the essay. No additional comment if that is what you meant though and were just using shorthand for that position.
I’d argue that you need to use a point estimate to decide what bets to make, and that you should make that point estimate by (1) geomean-pooling raw estimates of parameters, (2) reasoning over distributions of all parameters, then (3) taking arithmean of the resulting distribution-over-probabilities and (4) acting according to that mean probability.
I think “act according to that mean probability” is wrong for many important decisions you might want to take—analogous to buying a lot of trousers with 1.97 legs in my example in the essay. No additional comment if that is what you meant though and were just using shorthand for that position.
Clarifying, I do agree that there are some situations where you need something other than a subjective p(risk) to compare EV(value|action A) with EV(value|action B). I don’t actually know how to construct a clear analogy from the 1.97-legged trousers example if the variable we’re meaning is probabilities (though I agree that there are non-analogous examples; VOI for example).
I’ll go further, though, and claim that what really matters is what worlds the risk is distributed over, and that expanding the point-estimate probability to a distribution of probabilities, by itself, doesn’t add any real value. If it is to be a valuable exercise, you have to be careful what you’re expanding and what you’re refusing to expand.
More concretely, you want to be expanding over things your intervention won’t control, and then asking about your intervention’s effect at each point in things-you-won’t-control-space, then integrating back together. If you expand over any axis of uncertainty, then not only is there a multiplicity of valid expansions, but the natural interpretation will be misleading.
For example, say we have a 10% chance of drawing a dangerous ball from a series of urns, and 90% chance of drawing a safe one. If we describe it as (1) “50% chance of 9.9% risk, 50% chance of 10.1% risk” or (2) “50% chance of 19% risk, 50% chance of 1% risk” or (3) “10% chance of 99.1% risk, 90% chance of 0.1% risk”, what does it change our opinion of <intervention A>? (You can, of course, construct a two-step ball-drawing procedure that produces any of these distributions-over-probabilities.)
I think the natural intuition is that interventions are best in (2), because most probabilities of risk are middle-ish, and worst in (3), because probability of risk is near-determined. And this, I think, is analogous to the argument of the post that anti-AI-risk interventions are less valuable than the point-estimate probability would indicate.
But that argument assumes (and requires) that our interventions can only chance the second ball-drawing step, and not the first. So using that argument requires that, in the first place, we sliced the distribution up over things we couldn’t control. (If that is the thing we can control with our intervention, then interventions are best in the world of (3).)
Back to the argument of the original post: You’re deriving a distribution over several p(X|Y) parameters from expert surveys, and so the bottom-line distribution over total probabilities reflects the uncertainty in experts’ opinions on those conditional probabilities. Is it right to model our potential interventions as influencing the resolution of particular p(X|Y) rolls, or as influencing the distribution of p(X|Y) at a particular stage?
I claim it’s possible to argue either side.
Maybe a question like “p(much harder to build aligned than misaligned AGI | strong incentives to build AGI systems)” (the second survey question) is split between a quarter of the experts saying ~0% and three-quarters of the experts saying ~100%. (This extremizes the example, to sharpen the hypothetical analysis.) We interpret this as saying there’s a one-quarter chance we’re ~perfectly safe and a three-quarters chance that it’s hopeless to develop and aligned AGI instead of a misaligned one.
If we interpret that as if God will roll a die and put us in the “much harder” world with three-quarters probability and the “not much harder” world with one-quarters probability, then maybe our work to increase the we get an aligned AGI is low-value, because it’s unlikely to move either the ~0% or ~100% much lower (and we can’t change the die). If this was the only stage, then maybe all of working on AGI risk is worthless.
But “three-quarter chance it’s hopeless” is also consistent with a scenario where there’s a three-quarters chance that AGI development will be available to anyone, and many low-resourced actors will not have alignment teams and find it ~impossible to develop with alignment, but a one-quarter chance that AGI development will be available only to well-resourced actors, who will find it trivial to add on an alignment team and develop alignment. But then working on AGI risk might not be worthless, since we can work on increasing the chance that AGI development is only available to actors with alignment teams.
I claim that it isn’t clear, from the survey results, whether the distribution of experts’ probabilities for each step reflect something more like the God-rolls-a-die model, or different opinions about the default path of a thing we can intervene on. And if that’s not clear, then it’s not clear what to do with the distribution-over-probabilities from the main results. Probably they’re a step forward in our collective understanding, but I don’t think you can conclude from the high chances of low risk that there’s a low value to working on risk mitigation.
I agree that the arith-vs-geo question is basically the crux when it comes to whether this essay should move FF’s ‘fair betting probabilities’ - it sounds like everyone is pretty happy with the point about distributions and I’m really pleased about that because it was the main point I was trying to get across. I’m even more pleased that there is background work going on in the analysis of uncertainty space, because that’s an area where public statements by AI Risk organisations have sometimes lagged behind the state of the art in other risk management applications.
With respect to the crux, I hate to say it—because I’d love to be able to make as robust a claim for the prize as possible—but I’m not sure there is a principled reason for using geomean over arithmean for this application (or vice versa). The way I view it, they are both just snapshots of what is ‘really’ going on, which is the full distribution of possible outcomes given in the graphs / model. By analogy, I would be very suspicious of someone who always argued the arithmean would be a better estimate of central tendency than the median for every dataset / use case! I agree with you the problem of which is best for this particular dataset / use case is subtle, and I think I would characterise it as being a question of whether my manipulations of people’s forecasts have retained some essential ‘forecast-y’ characteristic which means geomean is more appropriate for various features it has, or whether they have been processed into having some sort of ‘outcome-y’ characteristic in which case arithmean is more appropriate. I take your point below in the coin example and the obvious superiority of arithmeans for that application, but my interpretation is that the FF didn’t intend for the ‘fair betting odds’ position to limit discussion about alternate ways to think about probabilities (“Applicants need not agree with or use our same conception of probability”)
However, to be absolutely clear, even if geomean was the right measure of central tendency I wouldn’t expect the judges to pay that particular attention—if all I had done was find a novel way of averaging results then my argument would basically be mathematical sophistry, perhaps only one step better than simply redefining ‘AI Risk’ until I got a result I liked. I think the distribution point is the actually valuable part of the essay, and I’m quite explicit in the essay that neither geomean nor arithmean is a good substitute for the full distribution. While I would obviously be delighted if I could also convince you my weak preference for geomean as a summary statistic was actually robust and considered, I’m actually not especially wedded to the argument for one summary statistic over the other. I did realise after I got my results that the crux for moving probabilities was going to be a very dry debate about different measures of central tendency, but I figured since the Fund was interested in essays on the theme of “a bunch of this AI stuff is basically right, but we should be focusing on entirely different aspects of the problem” (even if they aren’t being strictly solicited for the prize) the distribution bit of the essay might find a readership there anyway.
By the way, I know your four-step argument is intended just as a sketch of why you prefer arithmean for this application, but I do want to just flag up that I think it goes wrong on step 4, because acting according to arithmean probability (or geomean, for that matter) throws away information about distributions. As I mention here and elsewhere, I think the distribution issue is far more important than the geo-vs-arith issue, so while I don’t really feel strongly if I lose the prize because the judges don’t share my intuition that geomean is a slightly better measure of central tendency I would be sad to miss out because the distribution point was misunderstood! I describe in Section 5.2.2 how the distribution implied by my model would quite radically change some funding decisions, probably by more than an argument taking the arithmean to 3% (of course, if you’re already working on distribution issues then you’ve probably already reached those conclusions and so I won’t be changing your mind by making them—but in terms of publicly available arguments about AI Risk I’d defend the case that the distribution issue implies more radical redistribution of funds than changing the arithmean to 1.6%). So I think “act according to that mean probability” is wrong for many important decisions you might want to take—analogous to buying a lot of trousers with 1.97 legs in my example in the essay. No additional comment if that is what you meant though and were just using shorthand for that position.
Clarifying, I do agree that there are some situations where you need something other than a subjective p(risk) to compare EV(value|action A) with EV(value|action B). I don’t actually know how to construct a clear analogy from the 1.97-legged trousers example if the variable we’re meaning is probabilities (though I agree that there are non-analogous examples; VOI for example).
I’ll go further, though, and claim that what really matters is what worlds the risk is distributed over, and that expanding the point-estimate probability to a distribution of probabilities, by itself, doesn’t add any real value. If it is to be a valuable exercise, you have to be careful what you’re expanding and what you’re refusing to expand.
More concretely, you want to be expanding over things your intervention won’t control, and then asking about your intervention’s effect at each point in things-you-won’t-control-space, then integrating back together. If you expand over any axis of uncertainty, then not only is there a multiplicity of valid expansions, but the natural interpretation will be misleading.
For example, say we have a 10% chance of drawing a dangerous ball from a series of urns, and 90% chance of drawing a safe one. If we describe it as (1) “50% chance of 9.9% risk, 50% chance of 10.1% risk” or (2) “50% chance of 19% risk, 50% chance of 1% risk” or (3) “10% chance of 99.1% risk, 90% chance of 0.1% risk”, what does it change our opinion of <intervention A>? (You can, of course, construct a two-step ball-drawing procedure that produces any of these distributions-over-probabilities.)
I think the natural intuition is that interventions are best in (2), because most probabilities of risk are middle-ish, and worst in (3), because probability of risk is near-determined. And this, I think, is analogous to the argument of the post that anti-AI-risk interventions are less valuable than the point-estimate probability would indicate.
But that argument assumes (and requires) that our interventions can only chance the second ball-drawing step, and not the first. So using that argument requires that, in the first place, we sliced the distribution up over things we couldn’t control. (If that is the thing we can control with our intervention, then interventions are best in the world of (3).)
Back to the argument of the original post: You’re deriving a distribution over several p(X|Y) parameters from expert surveys, and so the bottom-line distribution over total probabilities reflects the uncertainty in experts’ opinions on those conditional probabilities. Is it right to model our potential interventions as influencing the resolution of particular p(X|Y) rolls, or as influencing the distribution of p(X|Y) at a particular stage?
I claim it’s possible to argue either side.
Maybe a question like “p(much harder to build aligned than misaligned AGI | strong incentives to build AGI systems)” (the second survey question) is split between a quarter of the experts saying ~0% and three-quarters of the experts saying ~100%. (This extremizes the example, to sharpen the hypothetical analysis.) We interpret this as saying there’s a one-quarter chance we’re ~perfectly safe and a three-quarters chance that it’s hopeless to develop and aligned AGI instead of a misaligned one.
If we interpret that as if God will roll a die and put us in the “much harder” world with three-quarters probability and the “not much harder” world with one-quarters probability, then maybe our work to increase the we get an aligned AGI is low-value, because it’s unlikely to move either the ~0% or ~100% much lower (and we can’t change the die). If this was the only stage, then maybe all of working on AGI risk is worthless.
But “three-quarter chance it’s hopeless” is also consistent with a scenario where there’s a three-quarters chance that AGI development will be available to anyone, and many low-resourced actors will not have alignment teams and find it ~impossible to develop with alignment, but a one-quarter chance that AGI development will be available only to well-resourced actors, who will find it trivial to add on an alignment team and develop alignment. But then working on AGI risk might not be worthless, since we can work on increasing the chance that AGI development is only available to actors with alignment teams.
I claim that it isn’t clear, from the survey results, whether the distribution of experts’ probabilities for each step reflect something more like the God-rolls-a-die model, or different opinions about the default path of a thing we can intervene on. And if that’s not clear, then it’s not clear what to do with the distribution-over-probabilities from the main results. Probably they’re a step forward in our collective understanding, but I don’t think you can conclude from the high chances of low risk that there’s a low value to working on risk mitigation.