If the estimates for the different components were independent, then wouldn’t the distribution of synthetic estimates be the same as the distribution of individual people’s estimates?
Multiplying Alice’s p1 x Bob’s p2 x Carol’s p3 x … would draw from the same distribution as multiplying Alice’s p1 x Alice’s p2 x Alice’s p3 … , if estimates to the different questions are unrelated.
So you could see how much non-independence affects the bottom-line results just by comparing the synthetic distribution with the distribution of individual estimates (treating each individual as one data point and multiplying their 6 component probabilities together to get their p(existential catastrophe)).
Insofar as the 6 components are not independent, the question of whether to use synthetic estimates or just look at the distribution of individuals’ estimates comes down to 1) how much value is there in increasing the effective sample size by using synthetic estimates and 2) is the non-independence that exists something that you want to erase by scrambling together different people’s component estimates (because it mainly reflects reasoning errors) or is it something that you want to maintain by looking at individual estimates (because it reflects the structure of the situation).
In practice these numbers wouldn’t perfectly match even if there was no correlation because there is some missing survey data that the SDO method ignores (because naturally you can’t sample data that doesn’t exist). In principle I don’t see why we shouldn’t use this as a good rule-of-thumb check for unacceptable correlation.
The synth distribution gives a geomean of 1.6%, a simple mean of around 9.6%, as per the essay
The distribution of all survey responses multiplied together (as per Alice p1 x Alice p2 x Alice p3) gives a geomean of approx 2.3% and a simple mean of approx 17.3%.
I’d suggest that this implies the SDO method’s weakness to correlated results is potentially depressing the actual result by about 50%, give or take. I don’t think that’s either obviously small enough not to matter or obviously large enough to invalidate the whole approach, although my instinct is that when talking about order-of-magnitude uncertainty, 50% point error would not be a showstopper.
If the estimates for the different components were independent, then wouldn’t the distribution of synthetic estimates be the same as the distribution of individual people’s estimates?
Multiplying Alice’s p1 x Bob’s p2 x Carol’s p3 x … would draw from the same distribution as multiplying Alice’s p1 x Alice’s p2 x Alice’s p3 … , if estimates to the different questions are unrelated.
So you could see how much non-independence affects the bottom-line results just by comparing the synthetic distribution with the distribution of individual estimates (treating each individual as one data point and multiplying their 6 component probabilities together to get their p(existential catastrophe)).
Insofar as the 6 components are not independent, the question of whether to use synthetic estimates or just look at the distribution of individuals’ estimates comes down to 1) how much value is there in increasing the effective sample size by using synthetic estimates and 2) is the non-independence that exists something that you want to erase by scrambling together different people’s component estimates (because it mainly reflects reasoning errors) or is it something that you want to maintain by looking at individual estimates (because it reflects the structure of the situation).
In practice these numbers wouldn’t perfectly match even if there was no correlation because there is some missing survey data that the SDO method ignores (because naturally you can’t sample data that doesn’t exist). In principle I don’t see why we shouldn’t use this as a good rule-of-thumb check for unacceptable correlation.
The synth distribution gives a geomean of 1.6%, a simple mean of around 9.6%, as per the essay
The distribution of all survey responses multiplied together (as per Alice p1 x Alice p2 x Alice p3) gives a geomean of approx 2.3% and a simple mean of approx 17.3%.
I’d suggest that this implies the SDO method’s weakness to correlated results is potentially depressing the actual result by about 50%, give or take. I don’t think that’s either obviously small enough not to matter or obviously large enough to invalidate the whole approach, although my instinct is that when talking about order-of-magnitude uncertainty, 50% point error would not be a showstopper.