Why do you think superforecasters who were selected specifically for assigning a low probability to AI x-risk are well described as “a bunch of smart people with no particular reason to be biased”?
For the avoidance of doubt, I’m not upset that the supers were selected in this way, it’s the whole point of the study, made very clear in the write-up, and was clear to me as a participant. It’s just that “your arguments failed to convince randomly selected superforecasters” and “your arguments failed to convince a group of superforecasters who were specifically selected for confidentiality disagreeing with you” are very different pieces of evidence.
One small clarification: the skeptical group was not all superforecasters. There were two domain experts as well. I was one of them.
I’m sympathetic to David’s point here. Even though the skeptic camp was selected for their skepticism, I think we still get some information from the fact that many hours of research and debate didn’t move their opinions. I think there are plausible alternative worlds where the skeptics come in with low probabilities (by construction), but update upward by a few points after deeper engagement reveals holes in their early thinking.
Ok, I slightly overstated the point. This time, the supers selected were not a (mostly) random draw from the set of supers. But they were in the original X-risk tournament, and in that case too, they were not persuaded to change their credences via further interaction with the concerned (that is the X-risk experts.) Then, when we took the more skeptical of them and gave them yet more exposure to AI safety arguments, that still failed to move the skeptics. I think taken together, these two results show that AI safety arguments are not all that persuasive to the average super. (More precisely, that no amount of exposure to them will persuade all supers as a group to the point where they get a median significantly above 0.75% in X-risk by the centuries end.)
Why do you think superforecasters who were selected specifically for assigning a low probability to AI x-risk are well described as “a bunch of smart people with no particular reason to be biased”?
For the avoidance of doubt, I’m not upset that the supers were selected in this way, it’s the whole point of the study, made very clear in the write-up, and was clear to me as a participant. It’s just that “your arguments failed to convince randomly selected superforecasters” and “your arguments failed to convince a group of superforecasters who were specifically selected for confidentiality disagreeing with you” are very different pieces of evidence.
One small clarification: the skeptical group was not all superforecasters. There were two domain experts as well. I was one of them.
I’m sympathetic to David’s point here. Even though the skeptic camp was selected for their skepticism, I think we still get some information from the fact that many hours of research and debate didn’t move their opinions. I think there are plausible alternative worlds where the skeptics come in with low probabilities (by construction), but update upward by a few points after deeper engagement reveals holes in their early thinking.
Ok, I slightly overstated the point. This time, the supers selected were not a (mostly) random draw from the set of supers. But they were in the original X-risk tournament, and in that case too, they were not persuaded to change their credences via further interaction with the concerned (that is the X-risk experts.) Then, when we took the more skeptical of them and gave them yet more exposure to AI safety arguments, that still failed to move the skeptics. I think taken together, these two results show that AI safety arguments are not all that persuasive to the average super. (More precisely, that no amount of exposure to them will persuade all supers as a group to the point where they get a median significantly above 0.75% in X-risk by the centuries end.)