[NB: I talked briefly to Greg in person about this last weekend, but felt it might be valuable to put this up here anyway for the purpose of public discussion / testing my beliefs.]
I have mixed feelings about this.
On the one hand, I really like the framing of reality being underpowered in certain contexts, and I think this post does a good job of explaining why this is often the case. I think the observation that we often have a lot of tacit data about the world that is hard to fit into explicit models but can nevertheless make non-data-driven expert predictions perform better than change is well-made and well-taken.
Nevertheless, I feel that in a great many cases, non-quantitative, intuitive, first-principles-heavy analyses of the world very often fail; that their rate of failure may often be poorly correlated with their apparent compellingness; that non-quantitative experts overestimate the explanatory power of their work at least as much as (and probably more than) more data-driven analysts; and that a shift towards more explicit, quantitative, data-driven approaches is often among the best ways to distinguish real knowledge about the world from the pseudo-knowledge that I think is rampant in many fields of human enquiry.
As an example: I have several friends who are academic historians, and from time to time we’ve talked about cliometrics/data-driven approaches to history. The general attitude seems to be “yeah, seems cool if you can do it, but just try that in [my period of study]. There’s no way you could build a decent quantitative model with that little data.” While I’ve generally been too tactful to say this to their faces, my response to these sorts of claims has historically been that if the data is too sparse to do meaningful analysis on, it’s probably also too sparse to draw any other conclusions more general than a simple existence proof (“this thing happened once”). Or, more pithily, “if you don’t have enough data to know things, you should just admit it”.
I now suspect this is too strong a stance, but I still think there is some important truth in it. My feeling is that there may well be “good reasons why expert communities in some areas haven’t tried to use data explicitly to answer problems in their field”, but there are also many bad reasons, among the most common of which is that very few people in that field have strong quantitative skills. I suspect that experts in data-poor fields often lack the epistemic modesty or statistical know-how to admit the consequences of that paucity.
[NB: I talked briefly to Greg in person about this last weekend, but felt it might be valuable to put this up here anyway for the purpose of public discussion / testing my beliefs.]
I have mixed feelings about this.
On the one hand, I really like the framing of reality being underpowered in certain contexts, and I think this post does a good job of explaining why this is often the case. I think the observation that we often have a lot of tacit data about the world that is hard to fit into explicit models but can nevertheless make non-data-driven expert predictions perform better than change is well-made and well-taken.
Nevertheless, I feel that in a great many cases, non-quantitative, intuitive, first-principles-heavy analyses of the world very often fail; that their rate of failure may often be poorly correlated with their apparent compellingness; that non-quantitative experts overestimate the explanatory power of their work at least as much as (and probably more than) more data-driven analysts; and that a shift towards more explicit, quantitative, data-driven approaches is often among the best ways to distinguish real knowledge about the world from the pseudo-knowledge that I think is rampant in many fields of human enquiry.
As an example: I have several friends who are academic historians, and from time to time we’ve talked about cliometrics/data-driven approaches to history. The general attitude seems to be “yeah, seems cool if you can do it, but just try that in [my period of study]. There’s no way you could build a decent quantitative model with that little data.” While I’ve generally been too tactful to say this to their faces, my response to these sorts of claims has historically been that if the data is too sparse to do meaningful analysis on, it’s probably also too sparse to draw any other conclusions more general than a simple existence proof (“this thing happened once”). Or, more pithily, “if you don’t have enough data to know things, you should just admit it”.
I now suspect this is too strong a stance, but I still think there is some important truth in it. My feeling is that there may well be “good reasons why expert communities in some areas haven’t tried to use data explicitly to answer problems in their field”, but there are also many bad reasons, among the most common of which is that very few people in that field have strong quantitative skills. I suspect that experts in data-poor fields often lack the epistemic modesty or statistical know-how to admit the consequences of that paucity.