Thanks, Joey. Really appreciate you taking the time to engage on these questions.
To be clear, I’m not seriously suggesting ignoring all research from before the decision. I’m just saying that mathematically, an independent test needs its backrest data to exclude all calibration data.
It strikes me that there are broadly 3 buckets of risk / potential failure:
Execution risk—this is significant and you can only find out by trying, but you only really know if you’re being successful with the left hand side of the theory of change
Logic risk—having an external organisation take a completely fresh view should solve most of this
Evidence risk—even with an external organisation marking your homework, they are still probably drawing on the same pool of research and that might suffer from survivorship bias
Thanks, Joey. Really appreciate you taking the time to engage on these questions.
To be clear, I’m not seriously suggesting ignoring all research from before the decision. I’m just saying that mathematically, an independent test needs its backrest data to exclude all calibration data.
It strikes me that there are broadly 3 buckets of risk / potential failure:
Execution risk—this is significant and you can only find out by trying, but you only really know if you’re being successful with the left hand side of the theory of change
Logic risk—having an external organisation take a completely fresh view should solve most of this
Evidence risk—even with an external organisation marking your homework, they are still probably drawing on the same pool of research and that might suffer from survivorship bias