I like that this post has set out the sketch of a theory of organisation truthfulness. In particular “In worlds where motivated reasoning is commonplace, we’d expect to see:
Red-teaming will discover errors that systematically slant towards an organization’s desired conclusion.
Deeper, more careful reanalysis of cost-effectiveness or impact analyses usually points towards lower rather than higher impact.”
Presumably, in worlds where motivated reasoning is rare, red-teaming will discover errors that slant towards and away from an organisation’s desired conclusion and deeper, more careful reanalysis of cost-effectiveness points towards lower and higher impact equally often.
I think this is first-order correct (and what my post was trying to get at). Second-order, I think there’s at least one important caveat (which I cut from my post) with just tallying total number (or importance-weighted number of) errors towards versus away from the desired conclusion as a proxy for motivated reasoning. Namely, you can’t easily differentiate “motivated reasoning” biases from perfectly innocent traditional optimizer’s curse.
Suppose an organization is considering 20 possible interventions and do initial cost-effectiveness analyses for each of them. If they have a perfectly healthy and unbiased epistemic process, then the top 2 interventions that they’ve selected from that list would a) in expectation be better than the other 18 and b) in expectation will have more errors slanted towards higher impact rather than lower impact.
If they then implement the top 2 interventions and do an impact assessment 1 year later, then I think it’s likely the original errors (not necessarily biases) from the initial assessment will carry through.
External red-teamers will then discover that these errors are systematically biased upwards, but at least on first blush “naive optimizer’s curse issues” looks importantly different in form, mitigation measures, etc, from motivated reasoning concerns.
I think it’s likely that either formal Bayesian modeling or more qualitative assessments can allow us to differentiate the two hypotheses.
Here’s one possible way to distinguish the two: Under the optimizer’s curse + judgement stickiness scenario retrospective evaluation should usually take a step towards the truth, though it could be a very small one if judgements are very sticky! Under motivated reasoning, retrospective evaluation should take a step towards the “desired truth” (or some combination of truth an desired truth, if the organisation wants both).
Thanks for your extensions! Worth pondering more.
I think this is first-order correct (and what my post was trying to get at). Second-order, I think there’s at least one important caveat (which I cut from my post) with just tallying total number (or importance-weighted number of) errors towards versus away from the desired conclusion as a proxy for motivated reasoning. Namely, you can’t easily differentiate “motivated reasoning” biases from perfectly innocent traditional optimizer’s curse.
Suppose an organization is considering 20 possible interventions and do initial cost-effectiveness analyses for each of them. If they have a perfectly healthy and unbiased epistemic process, then the top 2 interventions that they’ve selected from that list would a) in expectation be better than the other 18 and b) in expectation will have more errors slanted towards higher impact rather than lower impact.
If they then implement the top 2 interventions and do an impact assessment 1 year later, then I think it’s likely the original errors (not necessarily biases) from the initial assessment will carry through.
External red-teamers will then discover that these errors are systematically biased upwards, but at least on first blush “naive optimizer’s curse issues” looks importantly different in form, mitigation measures, etc, from motivated reasoning concerns.
I think it’s likely that either formal Bayesian modeling or more qualitative assessments can allow us to differentiate the two hypotheses.
Here’s one possible way to distinguish the two: Under the optimizer’s curse + judgement stickiness scenario retrospective evaluation should usually take a step towards the truth, though it could be a very small one if judgements are very sticky! Under motivated reasoning, retrospective evaluation should take a step towards the “desired truth” (or some combination of truth an desired truth, if the organisation wants both).