Thanks for this. I don’t agree for scientists, at least in their published work, but I do agree that to an extent it’s of course inevitable to bring in various other forms of reasoning to make subjective assessments that allow for inferences. So I think we’re mostly arguing over extent.
My argument would basically be:
Science made great progress when it agreed to focus its argument on empirical evidence and explanations of that evidence.
Economics has (in my opinion) made great progress in moving from a focus on pure deduction and theory (akin to the Natural Philosophers pre-science) and focused more on generating careful empirical evidence (especially about cause and effect). Theory’s job is then to explain that evidence. (Supply and demand models don’t do a good job of explaining the minimum wage evidence that’s been developed over more than a decade.)
In forecasting, starting with base rates (a sort of naive form of empiricism) is best practice. Likewise, naive empiricism seems to work in business, sports, etc. Even descriptive and correlational data appears practically very useful.
Therefore, I’d like to see EA and the rationality community stay rooted in empiricism as much as possible. That’s not always an option of course, but empirically driven processes seem to beat pure deduction much of the time, even when the data available doesn’t meet every standard of statistical inference. This still leaves plenty of room for the skilled Bayesian to weight things well, vet the evidence, depart from it when warranted etc.
I’ve not mentioned experts even once. I find the question of when and how much to defer to be quite difficult and I have no strong reaction to what I think is your view on that. My concern is with your justification of it. Anything that moves EA/rationality away from empiricism worries me. A bunch of smart people debating and deducing, largely cut off from observation and experiment, is a recipe for missing the mark. I know that’s not truly what you’re suggesting but that’s where I’m coming from.
(Finally, I grant that the scale of the replication crisis, or put another way the social science field in question, matters a lot and I’ve not addressed it.)
2. I would disagree on economics. I view the turn of economics towards high causal identification and complete neglect of theory as a major error, for reasons I touch on here. The discipline has moved from investigating important things to trivial things with high causal identification. The trend towards empirical behavioural economics is also in my view a fad with almost no practical usefulness. (To reiterate my point on the minimum wage—the negative findings are almost certainly false: it is what you would expect to find for a small treatment effect and noisy data in observational studies. Before reading the literature and believing that the effect of a minimum wage increase of $3 is small but negative, I would still expect to find a lot of studies finding no effect because empirical research is not very good, so one should not update much on those negative findings. If you think the demand curve for low skilled labour is vertical, then the phenomenon of ~0 effect on native US wages after a massive influx of low skilled labour from Cuba is inexplicable. And: the literature is very mixed—it’s not like all the studies find no effect, that is a misconception)
3. I agree that focusing on base rates is important but that doesn’t seem to get at the myopic empiricism issue. For example, the base rate of vaccine efficacy dropping off a cliff after 22 days is very low, but that was not established in the initial Astra Zeneca study. To form that judgement, one needs evidence from other domains, which myopic empiricists ignore.
4. I’m not sure where we disagree there. I don’t think EAs should stay rooted in empiricism if that means ‘form judgements only on the basis of the median published scientific study’, which is the view I criticise. I’m not saying we should become less empirical—I think we should take account of theory but also empirical evidence from other domains, which as I discuss many other experts refuse to do in some cases.
I’m not saying that we should be largely cut off from observation and experiment and should just deduce from theory. I’m saying that the myopic empiricist approach is not the right one.
Makes sense on 3 and 4. Out of curiosity, what would change your mind on the minimum wage? If you don’t find empirical economics valuable nor the views of experts (or at least don’t put much stock in them) how would you decide whether supply and demand was better or worse theory than an alternative? The premises underlying traditional economic models are clearly not fully 100% always-and-everywhere true, so their conclusions need not be either. How do you decide about a theory’s accuracy or usefulness if not by reference to evidence or expertise?
I think I would find it very hard to update on the view that the minimum wage reduces demand for labour. Maybe if there were an extremely well done RCT showing no effect from a large minimum wage increase of $10, I would update. Incidentally, here is discussion of an RCT on the minimum wage which illustrates where the observational studies might be going wrong. The RCT shows that employers reduced hours worked, which wouldn’t show up in observational studies, which mainly study disemployment effects
I am very conscious of the fact that almost everyone I have ever tried to convince of this view on the minimum wage remains wholly unmoved. I should make it clear that I am in favour of redistribution through tax credits, subsidies for childcare and that kind of thing. I think the minimum wage is not a smart way to help lower income people.
Maybe to try and see if I understand I should try to answer: it’d be a mix of judgment, empirical evidence (but much broader than the causal identification papers), deductive arguments that seem to have independent force, and maybe some deference to an interdisciplinary group with good judgment, not necessarily academics?
Thanks for this. I don’t agree for scientists, at least in their published work, but I do agree that to an extent it’s of course inevitable to bring in various other forms of reasoning to make subjective assessments that allow for inferences. So I think we’re mostly arguing over extent.
My argument would basically be:
Science made great progress when it agreed to focus its argument on empirical evidence and explanations of that evidence.
Economics has (in my opinion) made great progress in moving from a focus on pure deduction and theory (akin to the Natural Philosophers pre-science) and focused more on generating careful empirical evidence (especially about cause and effect). Theory’s job is then to explain that evidence. (Supply and demand models don’t do a good job of explaining the minimum wage evidence that’s been developed over more than a decade.)
In forecasting, starting with base rates (a sort of naive form of empiricism) is best practice. Likewise, naive empiricism seems to work in business, sports, etc. Even descriptive and correlational data appears practically very useful.
Therefore, I’d like to see EA and the rationality community stay rooted in empiricism as much as possible. That’s not always an option of course, but empirically driven processes seem to beat pure deduction much of the time, even when the data available doesn’t meet every standard of statistical inference. This still leaves plenty of room for the skilled Bayesian to weight things well, vet the evidence, depart from it when warranted etc.
I’ve not mentioned experts even once. I find the question of when and how much to defer to be quite difficult and I have no strong reaction to what I think is your view on that. My concern is with your justification of it. Anything that moves EA/rationality away from empiricism worries me. A bunch of smart people debating and deducing, largely cut off from observation and experiment, is a recipe for missing the mark. I know that’s not truly what you’re suggesting but that’s where I’m coming from.
(Finally, I grant that the scale of the replication crisis, or put another way the social science field in question, matters a lot and I’ve not addressed it.)
2. I would disagree on economics. I view the turn of economics towards high causal identification and complete neglect of theory as a major error, for reasons I touch on here. The discipline has moved from investigating important things to trivial things with high causal identification. The trend towards empirical behavioural economics is also in my view a fad with almost no practical usefulness. (To reiterate my point on the minimum wage—the negative findings are almost certainly false: it is what you would expect to find for a small treatment effect and noisy data in observational studies. Before reading the literature and believing that the effect of a minimum wage increase of $3 is small but negative, I would still expect to find a lot of studies finding no effect because empirical research is not very good, so one should not update much on those negative findings. If you think the demand curve for low skilled labour is vertical, then the phenomenon of ~0 effect on native US wages after a massive influx of low skilled labour from Cuba is inexplicable. And: the literature is very mixed—it’s not like all the studies find no effect, that is a misconception)
3. I agree that focusing on base rates is important but that doesn’t seem to get at the myopic empiricism issue. For example, the base rate of vaccine efficacy dropping off a cliff after 22 days is very low, but that was not established in the initial Astra Zeneca study. To form that judgement, one needs evidence from other domains, which myopic empiricists ignore.
4. I’m not sure where we disagree there. I don’t think EAs should stay rooted in empiricism if that means ‘form judgements only on the basis of the median published scientific study’, which is the view I criticise. I’m not saying we should become less empirical—I think we should take account of theory but also empirical evidence from other domains, which as I discuss many other experts refuse to do in some cases.
I’m not saying that we should be largely cut off from observation and experiment and should just deduce from theory. I’m saying that the myopic empiricist approach is not the right one.
Makes sense on 3 and 4. Out of curiosity, what would change your mind on the minimum wage? If you don’t find empirical economics valuable nor the views of experts (or at least don’t put much stock in them) how would you decide whether supply and demand was better or worse theory than an alternative? The premises underlying traditional economic models are clearly not fully 100% always-and-everywhere true, so their conclusions need not be either. How do you decide about a theory’s accuracy or usefulness if not by reference to evidence or expertise?
I think I would find it very hard to update on the view that the minimum wage reduces demand for labour. Maybe if there were an extremely well done RCT showing no effect from a large minimum wage increase of $10, I would update. Incidentally, here is discussion of an RCT on the minimum wage which illustrates where the observational studies might be going wrong. The RCT shows that employers reduced hours worked, which wouldn’t show up in observational studies, which mainly study disemployment effects
I am very conscious of the fact that almost everyone I have ever tried to convince of this view on the minimum wage remains wholly unmoved. I should make it clear that I am in favour of redistribution through tax credits, subsidies for childcare and that kind of thing. I think the minimum wage is not a smart way to help lower income people.
Maybe to try and see if I understand I should try to answer: it’d be a mix of judgment, empirical evidence (but much broader than the causal identification papers), deductive arguments that seem to have independent force, and maybe some deference to an interdisciplinary group with good judgment, not necessarily academics?