I think your cons are good things to have noted, but here are reasons why two of them might matter less than one might think:
I think the very fact that “It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any” could actually also make this useful for skill-building and testing fit; people will be forced to learn to avoid those failure modes, and “we” (the community, potential future hirers, etc.) can see how well they do so.
E.g., to do this red teaming well, they may have to learn to identify how central an error is to a paper/post’s argument, to think about whether a slightly different argument could reach the same conclusion without needing the questionable premise, etc.
I have personally found that the line between “noticing errors in existing work” and “generating novel research” is pretty blurry.
A decent amount of the research I’ve done (especially some that is unfortunately nonpublic so far) has basically followed the following steps:
“This paper/post/argument seems interesting and important”
“Oh wait, it actually requires a premise that they haven’t noted and that seems questionable” / “It ignores some other pathway by which a bad thing can happen” / “Its concepts/definitions are fuzzy or conflate things in way that may obscure something important”
[I write a post/doc that discusses that issue, provides some analysis in light of this additional premise being required or this other pathway being possible or whatever, and discussing what implications this has—e.g., whether some risk is actually more or less important than we thought, or what new intervention ideas this alternative risk pathway suggests might be useful]
I’d guess that the same could also sometimes happen with this red teaming, especially if that was explicitly encouraged, people were given guidance on how to lean into this more “novel research” element when they notice something potentially major during the red teaming, people were given examples of how that has happened in the past, etc.
I think your cons are good things to have noted, but here are reasons why two of them might matter less than one might think:
I think the very fact that “It’s possible that doing deliberate “red-teaming” would make one predisposed to spot trivial issues rather than serious ones, or falsely identify issues where there aren’t any” could actually also make this useful for skill-building and testing fit; people will be forced to learn to avoid those failure modes, and “we” (the community, potential future hirers, etc.) can see how well they do so.
E.g., to do this red teaming well, they may have to learn to identify how central an error is to a paper/post’s argument, to think about whether a slightly different argument could reach the same conclusion without needing the questionable premise, etc.
I have personally found that the line between “noticing errors in existing work” and “generating novel research” is pretty blurry.
A decent amount of the research I’ve done (especially some that is unfortunately nonpublic so far) has basically followed the following steps:
“This paper/post/argument seems interesting and important”
“Oh wait, it actually requires a premise that they haven’t noted and that seems questionable” / “It ignores some other pathway by which a bad thing can happen” / “Its concepts/definitions are fuzzy or conflate things in way that may obscure something important”
[I write a post/doc that discusses that issue, provides some analysis in light of this additional premise being required or this other pathway being possible or whatever, and discussing what implications this has—e.g., whether some risk is actually more or less important than we thought, or what new intervention ideas this alternative risk pathway suggests might be useful]
Off the top of my head, some useful pieces of public work by other people that I feel could be roughly described as “red teaming that turned into novel research” include A Proposed Adjustment to the Astronomical Waste Argument and The long-term significance of reducing global catastrophic risks
I’d guess that the same could also sometimes happen with this red teaming, especially if that was explicitly encouraged, people were given guidance on how to lean into this more “novel research” element when they notice something potentially major during the red teaming, people were given examples of how that has happened in the past, etc.