Froolow

Karma: 897

Froolow Oct 21, 2022, 8:26 AM
3 points
0 ∶ 0
in reply to: Dan_Keys’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
In practice these numbers wouldn’t perfectly match even if there was no correlation because there is some missing survey data that the SDO method ignores (because naturally you can’t sample data that doesn’t exist). In principle I don’t see why we shouldn’t use this as a good rule-of-thumb check for unacceptable correlation.
The synth distribution gives a geomean of 1.6%, a simple mean of around 9.6%, as per the essay
The distribution of all survey responses multiplied together (as per Alice p1 x Alice p2 x Alice p3) gives a geomean of approx 2.3% and a simple mean of approx 17.3%.
I’d suggest that this implies the SDO method’s weakness to correlated results is potentially depressing the actual result by about 50%, give or take. I don’t think that’s either obviously small enough not to matter or obviously large enough to invalidate the whole approach, although my instinct is that when talking about order-of-magnitude uncertainty, 50% point error would not be a showstopper.

Froolow Oct 20, 2022, 6:03 PM
1 point
0 ∶ 0
in reply to: Erik Jenner’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I’m not sure we actually disagree about the fact on the ground, but I don’t fully agree with the specifics of what you’re saying (if that makes sense). In a general sense I agree the risk of ‘AI is invented and then something bad happens because of that’ is substantially higher than 1.6%. In the specific scenario the Future Fund are interested in for the contest however, I think the scenario is too narrow to say with confidence what would happen on examination of structural uncertainty. I could think of ways in which a more disjunctive structural model could even plausibly diminish the risk of the specific Future Fund catastrophe scenario—for example in models where some of the microdynamics make it easier to misuse AI deliberately. That wouldn’t necessarily change the overall risk of some AI Catastrophe befalling us, but it would be a relevant distinction to make with respect to the Future Fund question which asks about a specific kind of Catastrophe.
Also you’re right the second and third quotes you give are too strong—it should read something like ‘...the actual risk of AI Catastrophe of this particular kind...’ - you’re right that this essay says nothing about AI Catastrophe broadly defined, just the specific kind of catastrophe the Future Fund are interested in. I’ll change that, as it is undesirable imprecision.

Froolow Oct 20, 2022, 10:13 AM
15 points
6 ∶ 0
in reply to: aog’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
Both of the links you suggest are strong philosophical arguments for ‘disjunctive’ risk, but are not actually model schema (although Soares does imply he has such a schema and just hasn’t published it yet). The fact that I only use Carlsmith to model risk is a fair reflection of the state of the literature.
(As an aside, this seems really weird to me—there is almost no community pressure to have people explicitly draw out their model schema in powerpoint or on a piece of paper or something. This seems like a fundamental first step in communicating about AI Risk, but only Carlsmith has really done it to an actionable level. Am I missing something here? Are community norms in AI Risk very different to community norms in health economics, which is where I usually do my modelling?)

Froolow Oct 20, 2022, 9:55 AM
9 points
4 ∶ 0
in reply to: harfe’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I don’t know if a rough analogy might help, but imagine you just bought a house . The realtor warns you that some houses in this neighbourhood have faulty wiring, and your house might randomly set on fire during the 5 years or so you plan to live in it (that is, there is a 10% or whatever chance per year the house sets on fire). There are certain precautions you might take, like investing in a fire blanket and making sure your emergency exits are always clear, but principally buying very good home insurance, at a very high premium.
Imagine then you meet a builder in a bar and he says, “Oh yes, Smith was a terrible electrician and any house Smith built has faulty wiring, giving it a 50% chance of fire each year. If Smith didn’t do your wiring then it is no more risky than any other house, maybe 1% per year”. You don’t actually live in a house with a 10% risk, you live in a house with a 1% or 50% risk. Each of those houses necessitates a different strategy—in a low risk house you can basically take no action, and save money on the premium insurance. In the high risk house you want to basically sell immediately (or replace the wiring completely). One important thing you would want to do straight away is discover if Smith or Jones built your house, which is irrelevant information in the first situation before you met the builder in the bar, where you implicitly have perfect certainty. You might reason inductively—“I saw a fire this year, so it is highly likely I live in a home that Smith built, so I am going to sell at a loss to avoid the fire which will inevitably happen next year” (compared to the first situation where you would just reason you were unlucky)
I totally agree with your final paragraph—to actually do anything with the information there is an asymmetrically distributed ex post AI Risk requires a totally different model. This is not an essay about what to actually do about AI Risk. However hopefully this comment gives perhaps a sketch picture of what might be accomplished when such a model is designed and deployed.

Froolow Oct 20, 2022, 9:46 AM
5 points
0 ∶ 0
in reply to: Steven Byrnes’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I think you’re using a philosophical framework I just don’t recognise here - ‘conjunctive’ and ‘disjunctive’ are not ordinary vocabulary in the sort of statistical modelling I do. One possible description of statistical modelling is that you are aiming to capture relevant insights about the world in a mathematical format so you can test hypotheses about those insights. In that respect, a model is good or bad based on how well its key features reflect the real world, rather than because it takes some particular position on the conjunctive-vs-disjunctive dispute. For example I am very excited to see the results of the MTAIR project, which will use a model a little bit like the below. This isn’t really ‘conjunctive’ or ‘disjunctive’ in any meaningful sense—it tries to multiply probabilities when they should be multiplied and add probabilities when they should be added. This is more like the philosophical framework I would expect modelling to be undertaken in.
I’d add that one of the novel findings of this essay is that if there are ‘conjunctive’ steps between ‘disjunctive’ steps it is likely the distribution effect I find will still apply (that is, given order-of-magnitude uncertainty). Insofar as you agree that 4-ish steps in AI Risk are legitimately conjunctive as per your comment above, we probably materially agree on the important finding of this essay (that the distribution of risk is asymmetrically weighted towards low-risk worlds) even if we disagree about the exact point estimate around which that distribution skews
Small point of clarification—you’re looking at the review table for Carlsmith (2021), which corresponds to Section 4.3.1. The correlation table I produce is for the Full Survey dataset, which corresponds to Section 4.1.1. Perhaps to highlight the difference, in the Full Survey dataset of 42 people; 5 people give exactly one probability <10%, 2 people give exactly two probabilities <10%, 2 people give exactly three probabilities <10% and 1 mega-outlier gives exactly four probabilities <10%. To me this does seem like there is evidence of ‘optimism bias’ / correlation relative to what we might expect to see (which would be closer to 1 person giving exactly 2 probabilities <10% I suppose), but not enough to fundamentally alter the conclusion that low-risk worlds are more likely than high-risk worlds based on community consensus (eg see section 4.3.3)

Froolow Oct 20, 2022, 9:27 AM
3 points
0 ∶ 0
in reply to: kaarel’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
Hope I’ve understood you right! I’ve taken the arithmetic mean of all columns and then computed the product of those arithmetic means. I end up with 9.74%. Again, I think this is slightly different from my model’s estimate of the value because the survey has some missing data which doesn’t occur in the synthetic distribution of the model

Froolow Oct 20, 2022, 8:08 AM
6 points
0 ∶ 0
in reply to: harfe’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
Hmm… I don’t see a contradiction here. I note you skimmed some of the methods, so it might perhaps help explain the contradiction to read the second half of section 3.3.2?
The bullet I bite is the first—most survey respondents are wrong, because they give point probabilities (which is what I asked for, in fairness) whereas in reality there will be uncertainty over those probabilities. Initiatively we might think that this uncertainty doesn’t matter because it will ‘cancel out’ (ie every time you are uncertain in a low direction relative to the truth I am uncertain in a high direction relative to the truth) but in reality—given specific structural assumptions in the Carlsmith Model—this is not true. In reality, the low-end uncertainty compounds and the high-end uncertainty is neutered, which is why you end up with an asymmetric distribution favouring very low-risk outcomes.

Froolow Oct 20, 2022, 8:03 AM
2 points
1 ∶ 0
in reply to: Misha_Yagudin’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
My understanding of what everyone is producing (Carlsmith, Beckstead etc) is their point estimate / most likely probability for some proposition being true. Shifting this point estimate to below 10% would be near enough a prize, but plenty of real-world applications have highish point estimates with a lower bound uncertainty that is very low.
The application where I am most familiar with this effect is clinical trials for oncology drugs; it isn’t uncommon for the point estimate for a drug’s effectiveness to be (say) 50% better than all other drugs on the market, but with a 95% confidence interval that covers no better at all, or even sometimes substantially worse. It seems to me to be quite a radical claim that we have better knowledge of AI Risk across nearly all parameters than we have of an oncology drug across a single parameter following a clinical trial.

Froolow Oct 20, 2022, 7:54 AM
2 points
0 ∶ 0
in reply to: Erik Jenner’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
My apologies if I wasn’t clear enough in the essay—I think there is a very good case for investigating structural uncertainty, it is just that it would require another essay-length treatment to do a decent job with. I hope to be able to produce such a treatment before the contest deadline (and I’ll publish afterwards anyway if this isn’t possible). This essay implicitly treats the model structure as fixed (except for a tiny nod to the issue in 4.3.3) and parameter uncertainty as the only point of contention, but in reality both the model structural uncertainty and parameter uncertainty will contribute to the overall uncertainty.

Froolow Oct 19, 2022, 8:35 PM
5 points
1 ∶ 0
in reply to: Linch’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I dropped 10% from both the low and high end- so the analysis in the results above are the most central 80% of estimates for each parameter (although just eyeballing the data I was left with quite a few >99% probabilities even after dropping the extreme top end)

Froolow Oct 19, 2022, 11:09 AM
1 point
0 ∶ 0
in reply to: Dan_Keys’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
That’s correct—the table gives the geometric mean of odds for each individual line, but then the final line is a simple product of the preceding lines rather than the geometric mean of each individual final estimate. This is a tiny bit naughty of me, because it means I’ve changed my method of calculation halfway through the table—the reason I do this is because it is implicitly what everyone else has been doing up until now (e.g. it is what is done in Carlsmith 2021) , and I want to highlight the discrepancy this leads to.

Froolow Oct 19, 2022, 10:59 AM
3 points
0 ∶ 0
in reply to: kaarel’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I’m not completely sure I understand your request. The screenshot below is the Excel file with the survey results in. Column U is the product of columns N to S. You’d like the geometric mean of odds of column U? This is 0.023, which is approximately 2.3%. This isn’t quite the same as the estimate in my model, I think because there is some missing survey data which isn’t carried over into the model

Froolow Oct 19, 2022, 10:49 AM
2 points
0 ∶ 0
in reply to: kaarel’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
Just a small clarification point I didn’t make clear enough in the essay—my geometric mean is always the geometric mean of odds, converted back into probability because it makes it easier to interpret for most readers. So 1.6% is genuinely the geometric mean of odds, but I take your point that the geometric mean of odds might not be the best summary statistic to use in the first place. To reiterate though, my main argument is that point estimates are misleading in this case regardless of which point estimate you use, and distributions of ex post risk are important to consider.
I’m really sorry, I don’t know what I meant by the reference to Briar Scores either—I’ll change that footnote until I can figure out what I was trying to say.

Froolow Oct 19, 2022, 10:14 AM
17 points
0 ∶ 0
in reply to: Dan_Keys’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I had not thought to do that, and it seems quite sensible (I agree with your point about prima facie worry about low outliers). The results are below.
To my eye, the general mechanism I wanted to defend about is preserved (there is an asymmetric probability of finding yourself in a low-risk world), but the probability of finding yourself in an ultra-low-risk world has significantly lowered, with that probability mass roughly redistributing itself around the geometric mean (which itself has gone up to 7%-ish)
In some sense this isn’t totally surprising—removing the lowest 10% of estimates means that order-of-magnitude uncertainty is only preserved for one of the six parameters in the equation (Containment), so the SDO mechanism doesn’t really apply. I don’t have the subject-specific knowledge to conclude is de-extremising the data in this way is reasonable (do we actually have better-than-order-of-magnitude knowledge about all of these parameters except Containment?), but the analysis you suggest is an important limitation of my results which I had totally overlooked, so thank you for the suggestion.

Froolow Oct 19, 2022, 10:00 AM
3 points
0 ∶ 0
in reply to: Linch’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I think that’s a fair criticism. For all I know, the FF are not at all uncertain about their estimates (or at least not uncertain over order-of-magnitude) and so the SDO mechanism doesn’t come into play. I still think there is value in explicitly and systematically considering uncertainty, even if you end up concluding it doesn’t really matter for your specific beliefs -if only because you can’t be totally confident it doesn’t matter until you have actually done the maths.
I’ve updated the text to replace ‘geometric mean’ with ‘geometric mean of odds’ everywhere it occurs. Thanks so much for the close reading and spotting the error.

Froolow Oct 19, 2022, 9:46 AM
36 points
3 ∶ 0
in reply to: Thomas Kwa’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
This is unquestionably the strongest argument against the SDO method as it applies to AI Risk, and therefore the biggest limitation of the essay. There is really good chance that many of the parameters in the Carlsmith Model are correlated in real life (since basically everything is correlated with everything else by some mechanism), so the important question is whether they are independent enough that what I’ve got here is still plausible. I offer some thoughts on the issue in Section 5.1.
To the best of my knowledge, there is no work making a very strong theoretical claim that any particular element of the Carlsmith Model will be strongly correlated with any other element. I have seen people suggest mechanisms with the implicit claim that if AI is more revolutionary than we expect then there will be correlation between our incentive to deploy it, our desire to expose it to high-impact inputs and our inability to stop it once it tries to disempower us—but I’m pretty confident the validity check in Section 4.3.3 demonstrates that correlation between some parameters doesn’t fundamentally alter conclusions about distributions, although would alter the exact point estimates which were reached.
Practically, I don’t think there is strong evidence that people’s parameters are correlated across estimates to a degree that will significantly alter results. Below is the correlation matrix for the Full Survey estimates with p<0.05 highlighted in green. Obviously I’m once again leaning on the argument that a survey of AI Risk is the same thing as the actual AI Risk, which I think is another weakness of the essay.
This doesn’t spark any major concerns for me—there is more correlation than would be expected by chance, but it seems to be mostly contained within the ‘Alignment turns out to be easy’ step, and as discussed above the mechanism still functions if one or two steps are removed because they are indistinguishable from preceding steps. The fact that there is more positive than negative correlation step is some evidence of the ‘general factor of optimism’ which you describe (because the ‘optimistic’ view is that we won’t deploy AI until we know it is safe, so we’d expect negative correlation on this factor in the table). Overall I think my assumption of independence is reasonable in the sense that the results are likely to be robust to the sorts of correlations I have empirically observed and theoretically seen accounted for, however I do agree with you that if there is a critical flaw in the essay it is likely to be found here.
I don’t quite follow your logic where you conclude that if estimates are correlated then simple mean is preferred—my exploration of the problem suggests that if estimates are correlated to a degree significant enough to affect my overall conclusion then you stop being able to use conventional statistics at all and have to do something fancy like microsimulation. Anecdata—in the specific example you give my intuition is that 0.4% really is a better summary of our knowledge, since otherwise we round off Aida’s position to ‘approximately 1%’ which is several orders of magnitude incorrect. Although as I say above, in the situation you describe above both summary estimates are misleading in different ways and we should look at the distribution—which is the key point I was trying to make in the essay.
What links here?
- Steven Byrnes's comment on ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting by Froolow (19 Oct 2022 19:48 UTC; 2 points)

Froolow 19 Oct 2022 8:47 UTC
3 points
0 ∶ 0
in reply to: harfe’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I’m not an AI Risk expert, so any answer I gave to 1 would just be polluting. Let’s say my probabilities are A and B for a two-parameter Carlsmith Model, and those parameters could be 3% or 33% as per your example. So a simple mean of this situation is A = (3% + 33%)/2 = 18% and B is the same, so simple mean is ~3%. The geometric mean is more like 1%.
The most important point I wanted to get across is that the distribution of probabilities can be important in some contexts. If something important happens to our response at a 1% risk then it is useful to know that we will observe less than 1% risk in ³⁄₄ of all possible worlds (ie worlds when A or B are at 3%). In the essay I argue that since strategies for living in a low-risk world are likely to be different from strategies for living in a high-risk world (and both sets of strategies are likely to be different from optimal strategy if we live in a simple-mean medium-risk world), distribution is what matters.
If we agree about that (which I’m not certain we do—I think possibly you are arguing that you can and should always reduce probabilities-of-probabilities to just probabilities?), then I don’t really have a strong position on your other point about geometric mean of odds vs simple mean. The most actionable summary statistic depends on the context. While I think geometric mean of odds is probably the correct summary statistic for this application, I accept that there’s an argument to be had on the point.

Froolow 19 Oct 2022 8:29 UTC
5 points
0 ∶ 0
in reply to: Ross Rheingans-Yoo🔸’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
I think there are good reasons for preferring geometric mean of odds to simple mean when presenting data of this type, but not good enough that I’d take to the barricades over them. Linch (below) links to the same post I do in giving my reasons to believe this. Overall, however, this is an essay about distributions rather than point estimates so if your main objection is to the summary statistic I used then I think we agree on the material points, but have a disagreement about how the work should be presented.
On the point about betting odds, I note that the contest announcement also states “Applicants need not agree with or use our same conception of probability”. I think the way in which I actually disagree with the Future Fund is more radical than simple means vs geometric mean of odds—I think they ought to stop putting so much emphasis on summary statistics altogether.

Froolow 19 Oct 2022 8:19 UTC
4 points
1 ∶ 0
in reply to: Linch’s comment on: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
This comment is exactly right, although it seems I came across stronger on the point about geometric mean of odds than I intended to. I wanted to say basically exactly what you did in this comment—there are relatively sound reasons to treat geometric mean of odds as the default in this case, but that there was a reasonable argument for simple means too. For example see footnotes 7 and 9 where I make this point. What I wanted to get across was that the argument about simple means vs geometric mean of odds was likely not the most productive argument to be having—point estimates always (necessarily) summarise the underlying distribution of data, and it is dangerous to merely use summary statistics when the distribution itself contains interesting and actionable information
Just for clarity—I use geometric mean of odds, which I then convert back into probability as an additional step (because people are more familiar with probability than odds). If I said anywhere that I took the geometric mean of probabilities then this is a typo and I will correct it!

‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting

Froolow18 Oct 2022 22:54 UTC

111 points

63 comments39 min readEA link

Froolow

‘Dis­solv­ing’ AI Risk – Pa­ram­e­ter Uncer­tainty in AI Fu­ture Forecasting

‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting