especially the observation that successful prediction systems across most domains use cluster not sequence thinking.
I find this “observation” confusing / misleading, given that Holden defines cluster thinking as aggregating decisions from multiple perspectives. This is very different from aggregating the predictions of multiple models. The evidence of “success” he cites only applies to the latter (where “success” is with respect to Brier scores and such), not the former.
And this is practically relevant: If you aggregate multiple models but then maximize EV under the aggregated model, you don’t get the “sandboxing” property Holden claims cluster thinking satisfies. The fanatical/Pascalian model will still dominate the EV calculation.
(ETA: As an aside on sequence thinking / cluster thinking generally, I wish these discussions made it very clear whether we’re taking ST/CT as (1) different normative standards for good epistemology / decision-making per se, vs. as (2) different procedures for satisfying a given epistemological / decision-theoretic standard. Cf. “criterion of rightness vs. decision procedure” in ethics. This would be helpful for clarifying what’s meant by claims like “cluster thinking is how ‘successful’ prediction systems operate”. I’ve been assuming (2), here, FWIW.)
Thanks for the intriguing pushback, part of why I kept bringing this up over the years was to surface this kind of counterargument, upvoted. Flagging for myself later to look into the evidence base behind
The evidence of “success” he cites only applies to the latter (where “success” is with respect to Brier scores and such), not the former.
because I’d always assumed it was “obviously” the former (wrongly it seems), since the latter seemed non-robust in the sense Dan Luu looked into (cf. “you really have to understand things”, which multi-model aggregations are not).
I find this “observation” confusing / misleading, given that Holden defines cluster thinking as aggregating decisions from multiple perspectives. This is very different from aggregating the predictions of multiple models. The evidence of “success” he cites only applies to the latter (where “success” is with respect to Brier scores and such), not the former.
And this is practically relevant: If you aggregate multiple models but then maximize EV under the aggregated model, you don’t get the “sandboxing” property Holden claims cluster thinking satisfies. The fanatical/Pascalian model will still dominate the EV calculation.
(ETA: As an aside on sequence thinking / cluster thinking generally, I wish these discussions made it very clear whether we’re taking ST/CT as (1) different normative standards for good epistemology / decision-making per se, vs. as (2) different procedures for satisfying a given epistemological / decision-theoretic standard. Cf. “criterion of rightness vs. decision procedure” in ethics. This would be helpful for clarifying what’s meant by claims like “cluster thinking is how ‘successful’ prediction systems operate”. I’ve been assuming (2), here, FWIW.)
Thanks for the intriguing pushback, part of why I kept bringing this up over the years was to surface this kind of counterargument, upvoted. Flagging for myself later to look into the evidence base behind
because I’d always assumed it was “obviously” the former (wrongly it seems), since the latter seemed non-robust in the sense Dan Luu looked into (cf. “you really have to understand things”, which multi-model aggregations are not).
I’ve also assumed (2) FWIW.