This is unquestionably the strongest argument against the SDO method as it applies to AI Risk, and therefore the biggest limitation of the essay. There is really good chance that many of the parameters in the Carlsmith Model are correlated in real life (since basically everything is correlated with everything else by some mechanism), so the important question is whether they are independent enough that what I’ve got here is still plausible. I offer some thoughts on the issue in Section 5.1.
To the best of my knowledge, there is no work making a very strong theoretical claim that any particular element of the Carlsmith Model will be strongly correlated with any other element. I have seen people suggest mechanisms with the implicit claim that if AI is more revolutionary than we expect then there will be correlation between our incentive to deploy it, our desire to expose it to high-impact inputs and our inability to stop it once it tries to disempower us—but I’m pretty confident the validity check in Section 4.3.3 demonstrates that correlation between some parameters doesn’t fundamentally alter conclusions about distributions, although would alter the exact point estimates which were reached.
Practically, I don’t think there is strong evidence that people’s parameters are correlated across estimates to a degree that will significantly alter results. Below is the correlation matrix for the Full Survey estimates with p<0.05 highlighted in green. Obviously I’m once again leaning on the argument that a survey of AI Risk is the same thing as the actual AI Risk, which I think is another weakness of the essay.
This doesn’t spark any major concerns for me—there is more correlation than would be expected by chance, but it seems to be mostly contained within the ‘Alignment turns out to be easy’ step, and as discussed above the mechanism still functions if one or two steps are removed because they are indistinguishable from preceding steps. The fact that there is more positive than negative correlation step is some evidence of the ‘general factor of optimism’ which you describe (because the ‘optimistic’ view is that we won’t deploy AI until we know it is safe, so we’d expect negative correlation on this factor in the table). Overall I think my assumption of independence is reasonable in the sense that the results are likely to be robust to the sorts of correlations I have empirically observed and theoretically seen accounted for, however I do agree with you that if there is a critical flaw in the essay it is likely to be found here.
I don’t quite follow your logic where you conclude that if estimates are correlated then simple mean is preferred—my exploration of the problem suggests that if estimates are correlated to a degree significant enough to affect my overall conclusion then you stop being able to use conventional statistics at all and have to do something fancy like microsimulation. Anecdata—in the specific example you give my intuition is that 0.4% really is a better summary of our knowledge, since otherwise we round off Aida’s position to ‘approximately 1%’ which is several orders of magnitude incorrect. Although as I say above, in the situation you describe above both summary estimates are misleading in different ways and we should look at the distribution—which is the key point I was trying to make in the essay.
I researched this fairly extensively a few years ago, and it is a true (but maybe misleading, depending on context) claim.
The usual source for this claim is the Sentience Institute, although if you go to a huge amount of effort to check government records by hand you get basically the same number so I’m not worried that the source is somewhat biased. They get the 99% number by using USDA data on the size of farms, and then defining any farm over a certain size as a ‘factory’ farm. This makes sense to me, and is how I’d approach the definitional problem unless I was shown extremely compelling evidence of a farm processing eg 5000 pigs a year using traditional ‘mom and pop’ techniques.
The reason the claim might be misleading is that it is using ‘meat’ as a shorthand for ‘meat animals’ rather than eg ‘carcass weight’. Because the vast majority of farmed animals are chickens, and chickens are overwhelmingly factory farmed when farmed, the result of the Sentience Institute methodology is that it appears the overwhelming percentage of farmed animals are factory farmed. In fact, by carcass weight it is ‘only’ about 90% of meat which is factory farmed.
This could in theory drop a bit lower if you say that the process for factory farming cows is not all that morally relevant for the 2⁄3 of their life they spend in pastures and hence were very exacting with your definitions (ie maybe for the sake of argument we would say something like “85% of meat-by-weight is farmed in a way that would be extremely distressing for the animal” rather than “99% of meat is factory farmed”), but it is hard to get very much lower than this because pigs and chickens are almost exclusively raised in cramped factory conditions and also make up a great deal of the meat we eat.