It would be silly to create this model and then not change my mind based on the evidence it presents.
Suggested way of thinking about this: your brain has an intuitive model of the situation, informed by your thinking & research. Creating a formal quantitative model gives you a way to cross-check your intuitive model. If your formal model and your intuitive model differ, you’d want to find some way to average them, examine the disparity, or perhaps collect additional models with which to do more cross-checking. Sometimes simpler models can even be better than complicated models, as in statistics when complicated models are more prone to overfitting. Possibly related post.
Or here’s Daniel Kahneman in Thinking Fast and Slow, talking about a disciplined method he helped the Israeli military come up with for evaluating potential soldiers:
The interviewers came close to mutiny. These bright young people were displeased to be ordered, by someone hardly older than themselves, to switch off their intuition and focus entirely on boring factual questions. One of them complained, “You are turning us into robots!” So I compromised. “Carry out the interview exactly as instructed,” I told them, “and when you are done, have your wish: close your eyes, try to imagine the recruit as a soldier, and assign him a score on a scale of 1 to 5.”
Several hundred interviews were conducted by this new method, and a few months later we collected evaluations of the soldiers’ performance from the commanding officers of the units to which they had been assigned. The results made us happy. As Meehl’s book had suggested, the new interview procedure was a substantial improvement over the old one. The sum of our six ratings predicted soldiers’ performance much more accurately than the global evaluations of the previous interviewing method, although far from perfectly. We had progressed from “completely useless” to “moderately useful.”
The big surprise to me was that the intuitive judgment that the interviewers summoned up in the “close your eyes” exercise also did very well, indeed just as well as the sum of the six specific ratings. I learned from this finding a lesson that I have never forgotten: intuition adds value even in the justly derided selection interview, but only after a disciplined collection of objective information and disciplined scoring of separate traits. I set a formula that gave the “close your eyes” evaluation the same weight as the sum of the six trait ratings. A more general lesson that I learned from this episode was do not simply trust intuitive judgment—your own or that of others—but do not dismiss it, either.
The takeaway for me is that if an intelligent person has made a disciplined effort to review all the relevant literature with an open mind, I’m inclined to give their intuitive judgements credence similar to what I would give a quantitative model. You referenced base rate neglect in your previous post—I agree humans have weaknesses, but quantitative models can have weaknesses too.
You might be able to use this tool to generate images for LaTeX formulas. (In Google Chrome: right-click the image, then “copy image address”)
if an intelligent person has made a disciplined effort to review all the relevant literature with an open mind, I’m inclined to give their intuitive judgements credence similar to what I would give a quantitative model.
I trust the person who has read the literature more than the layperson and I trust the intuitions of the person who attempts to build a linear model out of the literature even more.
Thanks for the comments, John. I pretty much agree with what you’re saying here—we should use a combination of intuition and explicit models. I believe most people rely way too much on intuition and not enough on explicit models. This project was an attempt at improving models and making them more reliable.
Cool stuff! Some comments after a quick skim:
Suggested way of thinking about this: your brain has an intuitive model of the situation, informed by your thinking & research. Creating a formal quantitative model gives you a way to cross-check your intuitive model. If your formal model and your intuitive model differ, you’d want to find some way to average them, examine the disparity, or perhaps collect additional models with which to do more cross-checking. Sometimes simpler models can even be better than complicated models, as in statistics when complicated models are more prone to overfitting. Possibly related post.
Or here’s Daniel Kahneman in Thinking Fast and Slow, talking about a disciplined method he helped the Israeli military come up with for evaluating potential soldiers:
The takeaway for me is that if an intelligent person has made a disciplined effort to review all the relevant literature with an open mind, I’m inclined to give their intuitive judgements credence similar to what I would give a quantitative model. You referenced base rate neglect in your previous post—I agree humans have weaknesses, but quantitative models can have weaknesses too.
You might be able to use this tool to generate images for LaTeX formulas. (In Google Chrome: right-click the image, then “copy image address”)
I trust the person who has read the literature more than the layperson and I trust the intuitions of the person who attempts to build a linear model out of the literature even more.
Thanks for the comments, John. I pretty much agree with what you’re saying here—we should use a combination of intuition and explicit models. I believe most people rely way too much on intuition and not enough on explicit models. This project was an attempt at improving models and making them more reliable.