I’d like to see more deference to the evidence as you say, which isn’t the same as to the experts themselves, but more deference to either theory or common sense is an invitation for motivated reasoning and for most people in most contexts would, I suspect, be a step backward.
But ultimately I’d like to see this question solved by some myopic empiricism! We know experts are not great forecasters; we know they’re pretty good at knowing which studies in their fields will or not replicate. More experiments like that are what will tell us how much to defer to them.
Thanks for sharing that piece, it’s a great counterpoint. I have a few thoughts in response.
Strevens argues that myopic empiricism drives people to do useful experiments which they perhaps might not have done if they stuck to theory. This seems to have been true in the case of physics. However, there are also a mountain of cases of wasted research effort, some of them discussed in my post. The value of information from eg most studies on the minimum wage and observational nutritional epidemiology is miniscule in my opinion. Indeed, it’s plausible that the majority of social science research is wasted money, per the claims of the meta-science movement.
I agree that it’s not totally clear if it would be positive if in general people tried to put more weight on theory and common sense. But some reliance on theory and common sense is just unavoidable. So, this is a question of how much reliance we put on that, not whether to do it at all. For example, to make judgements about whether we should act on the evidence of whether masks work, we need to make judgments about the external validity of studies, which necessarily involves making some theoretical judgements about the mechanism by which masks work, which the empirical studies confirm. The true logical extension of myopic empiricism is the inability to infer anything from any study. “We showed that one set of masks worked in a series of studies in US in 2020, but we don’t have a study of this other set of masks works in Manchester in 2021, so we don’t know whether they work”.
I tend to think it would be positive if scientists gave up on myopic empiricism and shifted to being more explicitly Bayesian.
Thanks for this. I don’t agree for scientists, at least in their published work, but I do agree that to an extent it’s of course inevitable to bring in various other forms of reasoning to make subjective assessments that allow for inferences. So I think we’re mostly arguing over extent.
My argument would basically be:
Science made great progress when it agreed to focus its argument on empirical evidence and explanations of that evidence.
Economics has (in my opinion) made great progress in moving from a focus on pure deduction and theory (akin to the Natural Philosophers pre-science) and focused more on generating careful empirical evidence (especially about cause and effect). Theory’s job is then to explain that evidence. (Supply and demand models don’t do a good job of explaining the minimum wage evidence that’s been developed over more than a decade.)
In forecasting, starting with base rates (a sort of naive form of empiricism) is best practice. Likewise, naive empiricism seems to work in business, sports, etc. Even descriptive and correlational data appears practically very useful.
Therefore, I’d like to see EA and the rationality community stay rooted in empiricism as much as possible. That’s not always an option of course, but empirically driven processes seem to beat pure deduction much of the time, even when the data available doesn’t meet every standard of statistical inference. This still leaves plenty of room for the skilled Bayesian to weight things well, vet the evidence, depart from it when warranted etc.
I’ve not mentioned experts even once. I find the question of when and how much to defer to be quite difficult and I have no strong reaction to what I think is your view on that. My concern is with your justification of it. Anything that moves EA/rationality away from empiricism worries me. A bunch of smart people debating and deducing, largely cut off from observation and experiment, is a recipe for missing the mark. I know that’s not truly what you’re suggesting but that’s where I’m coming from.
(Finally, I grant that the scale of the replication crisis, or put another way the social science field in question, matters a lot and I’ve not addressed it.)
2. I would disagree on economics. I view the turn of economics towards high causal identification and complete neglect of theory as a major error, for reasons I touch on here. The discipline has moved from investigating important things to trivial things with high causal identification. The trend towards empirical behavioural economics is also in my view a fad with almost no practical usefulness. (To reiterate my point on the minimum wage—the negative findings are almost certainly false: it is what you would expect to find for a small treatment effect and noisy data in observational studies. Before reading the literature and believing that the effect of a minimum wage increase of $3 is small but negative, I would still expect to find a lot of studies finding no effect because empirical research is not very good, so one should not update much on those negative findings. If you think the demand curve for low skilled labour is vertical, then the phenomenon of ~0 effect on native US wages after a massive influx of low skilled labour from Cuba is inexplicable. And: the literature is very mixed—it’s not like all the studies find no effect, that is a misconception)
3. I agree that focusing on base rates is important but that doesn’t seem to get at the myopic empiricism issue. For example, the base rate of vaccine efficacy dropping off a cliff after 22 days is very low, but that was not established in the initial Astra Zeneca study. To form that judgement, one needs evidence from other domains, which myopic empiricists ignore.
4. I’m not sure where we disagree there. I don’t think EAs should stay rooted in empiricism if that means ‘form judgements only on the basis of the median published scientific study’, which is the view I criticise. I’m not saying we should become less empirical—I think we should take account of theory but also empirical evidence from other domains, which as I discuss many other experts refuse to do in some cases.
I’m not saying that we should be largely cut off from observation and experiment and should just deduce from theory. I’m saying that the myopic empiricist approach is not the right one.
Makes sense on 3 and 4. Out of curiosity, what would change your mind on the minimum wage? If you don’t find empirical economics valuable nor the views of experts (or at least don’t put much stock in them) how would you decide whether supply and demand was better or worse theory than an alternative? The premises underlying traditional economic models are clearly not fully 100% always-and-everywhere true, so their conclusions need not be either. How do you decide about a theory’s accuracy or usefulness if not by reference to evidence or expertise?
I think I would find it very hard to update on the view that the minimum wage reduces demand for labour. Maybe if there were an extremely well done RCT showing no effect from a large minimum wage increase of $10, I would update. Incidentally, here is discussion of an RCT on the minimum wage which illustrates where the observational studies might be going wrong. The RCT shows that employers reduced hours worked, which wouldn’t show up in observational studies, which mainly study disemployment effects
I am very conscious of the fact that almost everyone I have ever tried to convince of this view on the minimum wage remains wholly unmoved. I should make it clear that I am in favour of redistribution through tax credits, subsidies for childcare and that kind of thing. I think the minimum wage is not a smart way to help lower income people.
Maybe to try and see if I understand I should try to answer: it’d be a mix of judgment, empirical evidence (but much broader than the causal identification papers), deductive arguments that seem to have independent force, and maybe some deference to an interdisciplinary group with good judgment, not necessarily academics?
Philosopher Michael Strevens argues that what you’re calling myopic empiricism is what has made science so effective: https://aeon.co/essays/an-irrational-constraint-is-the-motivating-force-in-modern-science
I’d like to see more deference to the evidence as you say, which isn’t the same as to the experts themselves, but more deference to either theory or common sense is an invitation for motivated reasoning and for most people in most contexts would, I suspect, be a step backward.
But ultimately I’d like to see this question solved by some myopic empiricism! We know experts are not great forecasters; we know they’re pretty good at knowing which studies in their fields will or not replicate. More experiments like that are what will tell us how much to defer to them.
Thanks for sharing that piece, it’s a great counterpoint. I have a few thoughts in response.
Strevens argues that myopic empiricism drives people to do useful experiments which they perhaps might not have done if they stuck to theory. This seems to have been true in the case of physics. However, there are also a mountain of cases of wasted research effort, some of them discussed in my post. The value of information from eg most studies on the minimum wage and observational nutritional epidemiology is miniscule in my opinion. Indeed, it’s plausible that the majority of social science research is wasted money, per the claims of the meta-science movement.
I agree that it’s not totally clear if it would be positive if in general people tried to put more weight on theory and common sense. But some reliance on theory and common sense is just unavoidable. So, this is a question of how much reliance we put on that, not whether to do it at all. For example, to make judgements about whether we should act on the evidence of whether masks work, we need to make judgments about the external validity of studies, which necessarily involves making some theoretical judgements about the mechanism by which masks work, which the empirical studies confirm. The true logical extension of myopic empiricism is the inability to infer anything from any study. “We showed that one set of masks worked in a series of studies in US in 2020, but we don’t have a study of this other set of masks works in Manchester in 2021, so we don’t know whether they work”.
I tend to think it would be positive if scientists gave up on myopic empiricism and shifted to being more explicitly Bayesian.
Thanks for this. I don’t agree for scientists, at least in their published work, but I do agree that to an extent it’s of course inevitable to bring in various other forms of reasoning to make subjective assessments that allow for inferences. So I think we’re mostly arguing over extent.
My argument would basically be:
Science made great progress when it agreed to focus its argument on empirical evidence and explanations of that evidence.
Economics has (in my opinion) made great progress in moving from a focus on pure deduction and theory (akin to the Natural Philosophers pre-science) and focused more on generating careful empirical evidence (especially about cause and effect). Theory’s job is then to explain that evidence. (Supply and demand models don’t do a good job of explaining the minimum wage evidence that’s been developed over more than a decade.)
In forecasting, starting with base rates (a sort of naive form of empiricism) is best practice. Likewise, naive empiricism seems to work in business, sports, etc. Even descriptive and correlational data appears practically very useful.
Therefore, I’d like to see EA and the rationality community stay rooted in empiricism as much as possible. That’s not always an option of course, but empirically driven processes seem to beat pure deduction much of the time, even when the data available doesn’t meet every standard of statistical inference. This still leaves plenty of room for the skilled Bayesian to weight things well, vet the evidence, depart from it when warranted etc.
I’ve not mentioned experts even once. I find the question of when and how much to defer to be quite difficult and I have no strong reaction to what I think is your view on that. My concern is with your justification of it. Anything that moves EA/rationality away from empiricism worries me. A bunch of smart people debating and deducing, largely cut off from observation and experiment, is a recipe for missing the mark. I know that’s not truly what you’re suggesting but that’s where I’m coming from.
(Finally, I grant that the scale of the replication crisis, or put another way the social science field in question, matters a lot and I’ve not addressed it.)
2. I would disagree on economics. I view the turn of economics towards high causal identification and complete neglect of theory as a major error, for reasons I touch on here. The discipline has moved from investigating important things to trivial things with high causal identification. The trend towards empirical behavioural economics is also in my view a fad with almost no practical usefulness. (To reiterate my point on the minimum wage—the negative findings are almost certainly false: it is what you would expect to find for a small treatment effect and noisy data in observational studies. Before reading the literature and believing that the effect of a minimum wage increase of $3 is small but negative, I would still expect to find a lot of studies finding no effect because empirical research is not very good, so one should not update much on those negative findings. If you think the demand curve for low skilled labour is vertical, then the phenomenon of ~0 effect on native US wages after a massive influx of low skilled labour from Cuba is inexplicable. And: the literature is very mixed—it’s not like all the studies find no effect, that is a misconception)
3. I agree that focusing on base rates is important but that doesn’t seem to get at the myopic empiricism issue. For example, the base rate of vaccine efficacy dropping off a cliff after 22 days is very low, but that was not established in the initial Astra Zeneca study. To form that judgement, one needs evidence from other domains, which myopic empiricists ignore.
4. I’m not sure where we disagree there. I don’t think EAs should stay rooted in empiricism if that means ‘form judgements only on the basis of the median published scientific study’, which is the view I criticise. I’m not saying we should become less empirical—I think we should take account of theory but also empirical evidence from other domains, which as I discuss many other experts refuse to do in some cases.
I’m not saying that we should be largely cut off from observation and experiment and should just deduce from theory. I’m saying that the myopic empiricist approach is not the right one.
Makes sense on 3 and 4. Out of curiosity, what would change your mind on the minimum wage? If you don’t find empirical economics valuable nor the views of experts (or at least don’t put much stock in them) how would you decide whether supply and demand was better or worse theory than an alternative? The premises underlying traditional economic models are clearly not fully 100% always-and-everywhere true, so their conclusions need not be either. How do you decide about a theory’s accuracy or usefulness if not by reference to evidence or expertise?
I think I would find it very hard to update on the view that the minimum wage reduces demand for labour. Maybe if there were an extremely well done RCT showing no effect from a large minimum wage increase of $10, I would update. Incidentally, here is discussion of an RCT on the minimum wage which illustrates where the observational studies might be going wrong. The RCT shows that employers reduced hours worked, which wouldn’t show up in observational studies, which mainly study disemployment effects
I am very conscious of the fact that almost everyone I have ever tried to convince of this view on the minimum wage remains wholly unmoved. I should make it clear that I am in favour of redistribution through tax credits, subsidies for childcare and that kind of thing. I think the minimum wage is not a smart way to help lower income people.
Maybe to try and see if I understand I should try to answer: it’d be a mix of judgment, empirical evidence (but much broader than the causal identification papers), deductive arguments that seem to have independent force, and maybe some deference to an interdisciplinary group with good judgment, not necessarily academics?