Ok this me explaining on part of my thought process.
I’m pretty sceptical of macroeconomic theory. I think we mostly don’t understand how inflation works, DSGE models (the forefront of macroeconomic theory) mostly don’t have very good predictive power, we don’t really understand how economic growth works for instance. So even if someone shows me a new macro paper that proposes some new theory and attempts to empirically verify it with both micro and macro data I’ll shrug and eh probably wrong.
But this is so radically far ahead epistemically than something like the evaporative cooling model. We have thousands of datapoints for macro data and tens (?) of millions of micro data, macro models are actively used by commercial and central banks so get actual feedback on their predictions and they’re still not very good. Even in microeconomics where we have really a lot of data and a lot of quasi-random variation, we got something as basic as the effect of the minimum wage on unemployment wrong until we started doing good causal inference work, despite the minium wage effect being predicted by a model which worked very well in other domains (i.e supply and demand.)
If when I read an econ paper I need high quality casual inference to belive the theory it’s offering me, and even thousands of datapoints aren’t enough to properly specify test a model it’s unclear to me why I should have a lower standard of evidence for other social science research. The evaporative cooling model isn’t supported by
High-quality casual inference
Any regression at all
More than 4 data points
In-depth case studies or ethnographies
Regular application by practitioners who get good results using it
If I read a social science paper which didn’t have any of these things I’d just ignore it—as it is I mostly ignore anything that doesn’t have some combination of high-quality causal inference or a large literature of low-medium quality causal inference and observational studies reporting similar effects.
This is like a very hard version of this take—in practice because we have to make decisions in the actual world I use social science with less good empirical foundations—but I just have limited trust in that sort of thing. But man even rcts sometimes don’t replicate or scale.
Thanks for clarifying further, and some of that rationale does make sense (e.g. it’s important to critically look at the assumptions in models, and how data was collected).
I still think your conclusion/dismissal is too strong, particularly given social science is very broad (much more so than the economics examples given here), some things are inherently harder to model accurately than others, and if experts in a given field have certain approaches the first question I would ask is ‘why’.
It’s better to approach these things with humility and an open mind, particularly given how important the problems are that EA is trying to tackle.
“Almost all social science is wrong” is a very strong assertion without evidence to back it up, and I think such over-generalizations are unhelpful.
Ok this me explaining on part of my thought process.
I’m pretty sceptical of macroeconomic theory. I think we mostly don’t understand how inflation works, DSGE models (the forefront of macroeconomic theory) mostly don’t have very good predictive power, we don’t really understand how economic growth works for instance. So even if someone shows me a new macro paper that proposes some new theory and attempts to empirically verify it with both micro and macro data I’ll shrug and eh probably wrong.
But this is so radically far ahead epistemically than something like the evaporative cooling model. We have thousands of datapoints for macro data and tens (?) of millions of micro data, macro models are actively used by commercial and central banks so get actual feedback on their predictions and they’re still not very good. Even in microeconomics where we have really a lot of data and a lot of quasi-random variation, we got something as basic as the effect of the minimum wage on unemployment wrong until we started doing good causal inference work, despite the minium wage effect being predicted by a model which worked very well in other domains (i.e supply and demand.)
If when I read an econ paper I need high quality casual inference to belive the theory it’s offering me, and even thousands of datapoints aren’t enough to properly specify test a model it’s unclear to me why I should have a lower standard of evidence for other social science research. The evaporative cooling model isn’t supported by
High-quality casual inference
Any regression at all
More than 4 data points
In-depth case studies or ethnographies
Regular application by practitioners who get good results using it
If I read a social science paper which didn’t have any of these things I’d just ignore it—as it is I mostly ignore anything that doesn’t have some combination of high-quality causal inference or a large literature of low-medium quality causal inference and observational studies reporting similar effects.
This is like a very hard version of this take—in practice because we have to make decisions in the actual world I use social science with less good empirical foundations—but I just have limited trust in that sort of thing. But man even rcts sometimes don’t replicate or scale.
Thanks for clarifying further, and some of that rationale does make sense (e.g. it’s important to critically look at the assumptions in models, and how data was collected).
I still think your conclusion/dismissal is too strong, particularly given social science is very broad (much more so than the economics examples given here), some things are inherently harder to model accurately than others, and if experts in a given field have certain approaches the first question I would ask is ‘why’.
It’s better to approach these things with humility and an open mind, particularly given how important the problems are that EA is trying to tackle.
I’ve just commented on your EA forum post, and there’s quite a lot of overlap and further comments seemed more relevant there compared to this post: https://forum.effectivealtruism.org/posts/WYktRSxq4Edw9zsH9/be-less-trusting-of-intuitive-arguments-about-social?commentId=GATZcZbh9kKSQ6QPu