In this post, I discuss the extent to which we may appeal to the object-level reasons when forming beliefs. I argue that the object-level reasons should sometimes play a role in determining our credences, even if that is only via our choice of epistemic peers and superiors. Thanks to Gregory Lewis, Stefan Schubert, Aidan Goth, Stephen Clare and Johannes Ackva for comments and discussion.
1. Introduction
There was a lively discussion in EA recently about the case for and against epistemic modesty. On epistemic modesty, one’s all-things-considered credences (the probability one puts on different propositions) should be determined, not by consideration of the object-level arguments, but rather by the credences of one’s epistemic peers and superiors.
Suppose I am trying to form a judgement about the costs and benefits of free trade. On epistemic modesty, my credence should be determined, not by my own impression of the merits of the object-level arguments for and against free trade (regarding issues such as comparative advantage and the distributional effects of tariffs), but rather by what the median expert (e.g. the median trade economist) believes about free trade. Even if my impressions of the arguments lean Trumpian, epistemic modesty requires me to defer to the experts. There is thus a difference between (1) my own personal impressions and (2) the all-things-considered credences I ought to have.
In this way, epistemic modesty places restrictions on the extent to which agents may permissibly appeal to object-level reasons when forming credences. When we are deciding what to believe about an issue, meta-level (non-object level considerations) about the epistemic merits of the expert group are very important.[1] These meta-level considerations include:
Time: These putative experts have put significant time into thinking about the question
Ability: They are selected for high cognitive ability
Scrutiny: Their work has been subject to substantial scrutiny from their peers
Numbers: They are numerous.
On epistemic modesty, a strong argument for the view that we should defer to trade economists on trade is that they are smart people, who put a lot of time into thinking about the topic, have been subject to significant external scrutiny, and they are numerous, which produces a wisdom of crowds effect. These factors are major advantages that trade economists have over me with respect to this topic, which suggests that the aggregate of economists are >10x more likely to be correct than me. So, on this topic, deference seems like a reasonable strategy.
1.1. Do the object-level arguments matter at all?
However, on epistemic modesty, the object-level arguments are not completely irrelevant to the all-things-considered credences I ought to have. Before we get into why, it is useful to distinguish two types of object-level arguments:
People’s object-level arguments on propositions relevant to people’s epistemic virtue on p, not including object-level arguments about p.
People’s object-level arguments and credences about p.
Suppose that our aim is to assess some proposition, such as the efficacy of sleeping pills. When we are choosing peers and superiors on this question, people’s object-level reasoning on medicine in general is relevant to their epistemic virtue. If we learn that someone has good epistemics on medical questions and knows the literature well (picture a Bayesian health economist), then that would be a reason to upgrade their epistemic virtue on the question of whether sleeping pills work. If we learn that someone has in the past appealed to homeopathic arguments about medicine, then that would be a reason to discount their epistemic virtue on the efficacy of sleeping pills. Thus, 1 is relevant to people’s epistemic virtue, and so is relevant to our all-things-considered credences on p.
This point is worth emphasising: even on epistemic modesty, consideration of the object-level arguments can be important for your all-things-considered credence, although the effect is indirect and comes via peer selection.
However, on some strong versions of epistemic modesty, 2 is not relevant to our assessment of people’s epistemic virtue on p. We must select people’s epistemic virtue on a proposition p in advance of considering their object-level reasons and verdict on p. Having selected trade economists as my epistemic superior on the benefits of trade, I cannot demote them merely because their object-level arguments on trade seem implausible. Call this the Object-level Reasons Restriction
Object-level Reasons Restriction = Your own impressions of the object-level reasons regarding p cannot be used to determine people’s epistemic virtue on p.[2]
2. Problems with the Object-level Reasons Restriction
In this section, I will argue against the Object-level Reasons Restriction using an argument from symmetry: if type 1 reasons are admissible when evaluating epistemic virtue, then type 2 reasons are as well.
Suppose I am forming beliefs about the efficacy of sleeping pills. When I am choosing peers and superiors on this question, on epistemic modesty I am allowed to take people’s object-level reasoning on medicine in general: I can justifiably exclude homeopaths from my peer group, for example. But if I choose Geoff as my peer on the efficacy of sleeping pills and he starts making homeopathic arguments about them, then I am not allowed to demote him from my peer group. I see no reason for this asymmetry, so I see no reason to accept the Object-level Reasons Restriction. If the object-level arguments about medicine in general are relevant to one’s epistemic virtue on the efficacy of sleeping pills, then the object-level reasons about the efficacy of sleeping pills are also relevant.
Suppose I have selected Geoff and Hilda as equally good truth-trackers on the efficacy of sleeping pills, and I then get the information that Geoff appeals to homeopathic arguments. From a Bayesian point of view, this is clearly an update away from him being as good a truth-tracker on the efficacy of sleeping pills as Hilda.
I envision two main responses to this, one from epistemic egoism, and one from rule epistemology.
2.1. Epistemic egoism
Firstly, it might be argued that by downgrading Geoff’s epistemic virtue, I in effect put extra weight on my own beliefs merely because they are mine. Since we are both peers, I should treat each of us equally.
I don’t think this argument works. My argument for demoting Geoff is not “my belief that his reasoning is bad” but rather “his reasoning is bad, which I believe”.[3] These two putative reasons are different. One can see this by using the following counterfactual test. Imagine a hypothetical world where I believe that Geoff’s reasoning is bad, but I am wrong. Do I, in the actual world, believe that I should still demote Geoff in the imagined world? No I do not. If I am wrong, I should not demote. So, my reason for downgrading is not my belief that the reasoning is bad, but is rather the proposition that the reasoning is bad, which I believe. Therefore, there is no objectionable epistemic egoism here.
In the sleeping pills case, when I demote someone from my peer group because they appeal to homeopathy, my reason to demote them is not my belief that homeopathy is unscientific, it is the fact that homeopathy is unscientific.
2.2. Rule epistemology
Another possible response is to say that I do better by following the rule of not downgrading according to the object-level reasons, so I should follow that rule rather than trying to find the exceptions. My response to this is that I do even better by following that rule except when it comes to Geoff. There is a difference between trying (and maybe failing) to justifiably downgrade experts and justifiably downgrading experts. If it is asked - ‘why think you should downgrade in this case?’, then a good answer is just to refer to the object-level reasons. This is a good answer regarding sleeping pills, just as it is, as everyone agrees, regarding medical expertise in general.
It is true that the rule of demoting people on the basis of the object-level reasons is liable to abuse. A mercantilist could use this kind of reasoning to demote all trade economists to be his epistemic inferiors. However, the problem here is bad object-level reasoning and epistemic virtue assessment, not the plausibility of the Object-level Reasons Restriction. As we have seen, proponents of epistemic modesty agree that object-level reasons are sometimes admissible when we are assessing people’s epistemic virtue. For example, the Object-level Reasons Restriction would not rule out the mercantilist using his views on trade as a reason to demote economists on other questions in economics. This is also an error, but one that merely stems from mundane bad reasoning, which we have non-modest reasons to care about.
What is usually going on when mercantilists dismiss the expert consensus on trade is (1) they simply don’t understand the arguments for and against free trade; and (2) that many of them also simply do not know that there is such a strong expert consensus on trade. This is simply inept reasoning, not an argument for never considering the object-level arguments when picking one’s peers.
3. Where does this leave us?
I have argued that it is sometimes appropriate to appeal to the object-level arguments on p when deciding on people’s epistemic virtue on p. I illustrated this with (I hope) a relatively clear case involving a rogue homeopath.
The arguments here are potentially practically important. Any sophisticated assessment of some topic will involve appealing to the object-level reasons to give more weight to the views of some putative experts who appear to do equally well on the meta-level considerations.
This is the approach that GiveWell takes when assessing the evidence on interventions in global health. Their approach is not to just take the median view in the literature on some question, but rather to filter out certain parts of the literature on the basis of general methodological quality. In their post on Common Problems with Formal Evaluations, they say they are hesitant to use non-RCT evidence because of selection effects and publication bias.[4] In this way, object-level arguments play a role in filtering the body of evidence that GiveWell responds to. Nonetheless, their approach is still modest in the sense that the prevailing view found among studies of sufficiently high quality plays a major role in determining their all-things-considered view.
In his review of the effect of saving lives on fertility, David Roodman rules out certain types of evidence, such as cross-country regressions and some of the instrumental variables studies, and puts more weight on other quasi-experimental studies. In this way, his final credence is determined by the expert consensus as weighted by the object-level reasons, rather than the crude median of the aggregate of putative experts.
Continental philosophers—Foucauldians, Hegelians, Marxists, postmodernists—do pretty well on the meta-level considerations—time, general cognitive ability, scrutiny and numbers. But they do poorly at object-level reasoning. This is a good reason to give them less weight than analytic philosophers when forming credences about philosophy.[5]
The examples above count against an ‘anything goes’ approach to assessments of the object-level reasons; the object-level reasoning still has to be done well. David Roodman and GiveWell put incredible amounts of intellectual heft into filtering the evidence. Epistemic peerhood and superiority are relative, and Roodman and GiveWell set a high bar. The point here is just that the object-level reasons are sometimes admissible.
The object-level reasons and the meta-level considerations should each play a role in assessments of epistemic virtue. This can justify positions that seem immodest. Some examples that are top of mind for me:
We should put less weight on estimates of climate sensitivity that update from a uniform prior, as I argued here.
We should almost all ignore nutritional epidemiology and just follow an enlightened common sense prior on nutrition.
We should often not defer to experts who use non-Bayesian epistemology, where this might make a difference relative to the prevailing (eg frequentist or scientistic) epistemology. E.g. this arguably played a role in early mainstream expert scepticism towards face masks as a tool to prevent COVID transmission.
[1] I avoid calling these ‘outside view’ reasons because the object-level reasons might also be outside view/base rate-type reasons. It might be that the people who do well on the meta-level reasons ignore the ‘outside view’ reasons. Witness, for example, the performance of many political scientists in Tetlock’s experiments.
[2] This is how I interpret Greg Lewis’ account of epistemic modesty: “One rough heuristic for strong modesty is this: for any question, find the plausible expert class to answer that question (e.g. if P is whether to raise the minimum wage, talk to economists). If this class converges on a particular answer, believe that answer too. If they do not agree, have little confidence in any answer. Do this no matter whether one’s impression of the object level considerations that recommend (by your lights) a particular answer.”
[3] David Enoch, “Not Just a Truthometer: Taking Oneself Seriously (but Not Too Seriously) in Cases of Peer Disagreement,” Mind 119, no. 476 (January 10, 2010): sec. 7, https://doi.org/10.1093/mind/fzq070.
[4] I have some misgivings about the RCT-focus of GiveWell’s methodology, but I think the general approach of filtering good and bad studies on the basis of methodological quality is correct.
[5] I think a reasonable case could be made for giving near zero weight to the views of Continental philosophers on difficult philosophical topics.
How modest should you be?
In this post, I discuss the extent to which we may appeal to the object-level reasons when forming beliefs. I argue that the object-level reasons should sometimes play a role in determining our credences, even if that is only via our choice of epistemic peers and superiors. Thanks to Gregory Lewis, Stefan Schubert, Aidan Goth, Stephen Clare and Johannes Ackva for comments and discussion.
1. Introduction
There was a lively discussion in EA recently about the case for and against epistemic modesty. On epistemic modesty, one’s all-things-considered credences (the probability one puts on different propositions) should be determined, not by consideration of the object-level arguments, but rather by the credences of one’s epistemic peers and superiors.
Suppose I am trying to form a judgement about the costs and benefits of free trade. On epistemic modesty, my credence should be determined, not by my own impression of the merits of the object-level arguments for and against free trade (regarding issues such as comparative advantage and the distributional effects of tariffs), but rather by what the median expert (e.g. the median trade economist) believes about free trade. Even if my impressions of the arguments lean Trumpian, epistemic modesty requires me to defer to the experts. There is thus a difference between (1) my own personal impressions and (2) the all-things-considered credences I ought to have.
In this way, epistemic modesty places restrictions on the extent to which agents may permissibly appeal to object-level reasons when forming credences. When we are deciding what to believe about an issue, meta-level (non-object level considerations) about the epistemic merits of the expert group are very important.[1] These meta-level considerations include:
Time: These putative experts have put significant time into thinking about the question
Ability: They are selected for high cognitive ability
Scrutiny: Their work has been subject to substantial scrutiny from their peers
Numbers: They are numerous.
On epistemic modesty, a strong argument for the view that we should defer to trade economists on trade is that they are smart people, who put a lot of time into thinking about the topic, have been subject to significant external scrutiny, and they are numerous, which produces a wisdom of crowds effect. These factors are major advantages that trade economists have over me with respect to this topic, which suggests that the aggregate of economists are >10x more likely to be correct than me. So, on this topic, deference seems like a reasonable strategy.
1.1. Do the object-level arguments matter at all?
However, on epistemic modesty, the object-level arguments are not completely irrelevant to the all-things-considered credences I ought to have. Before we get into why, it is useful to distinguish two types of object-level arguments:
People’s object-level arguments on propositions relevant to people’s epistemic virtue on p, not including object-level arguments about p.
People’s object-level arguments and credences about p.
Suppose that our aim is to assess some proposition, such as the efficacy of sleeping pills. When we are choosing peers and superiors on this question, people’s object-level reasoning on medicine in general is relevant to their epistemic virtue. If we learn that someone has good epistemics on medical questions and knows the literature well (picture a Bayesian health economist), then that would be a reason to upgrade their epistemic virtue on the question of whether sleeping pills work. If we learn that someone has in the past appealed to homeopathic arguments about medicine, then that would be a reason to discount their epistemic virtue on the efficacy of sleeping pills. Thus, 1 is relevant to people’s epistemic virtue, and so is relevant to our all-things-considered credences on p.
This point is worth emphasising: even on epistemic modesty, consideration of the object-level arguments can be important for your all-things-considered credence, although the effect is indirect and comes via peer selection.
However, on some strong versions of epistemic modesty, 2 is not relevant to our assessment of people’s epistemic virtue on p. We must select people’s epistemic virtue on a proposition p in advance of considering their object-level reasons and verdict on p. Having selected trade economists as my epistemic superior on the benefits of trade, I cannot demote them merely because their object-level arguments on trade seem implausible. Call this the Object-level Reasons Restriction
Object-level Reasons Restriction = Your own impressions of the object-level reasons regarding p cannot be used to determine people’s epistemic virtue on p.[2]
2. Problems with the Object-level Reasons Restriction
In this section, I will argue against the Object-level Reasons Restriction using an argument from symmetry: if type 1 reasons are admissible when evaluating epistemic virtue, then type 2 reasons are as well.
Suppose I am forming beliefs about the efficacy of sleeping pills. When I am choosing peers and superiors on this question, on epistemic modesty I am allowed to take people’s object-level reasoning on medicine in general: I can justifiably exclude homeopaths from my peer group, for example. But if I choose Geoff as my peer on the efficacy of sleeping pills and he starts making homeopathic arguments about them, then I am not allowed to demote him from my peer group. I see no reason for this asymmetry, so I see no reason to accept the Object-level Reasons Restriction. If the object-level arguments about medicine in general are relevant to one’s epistemic virtue on the efficacy of sleeping pills, then the object-level reasons about the efficacy of sleeping pills are also relevant.
Suppose I have selected Geoff and Hilda as equally good truth-trackers on the efficacy of sleeping pills, and I then get the information that Geoff appeals to homeopathic arguments. From a Bayesian point of view, this is clearly an update away from him being as good a truth-tracker on the efficacy of sleeping pills as Hilda.
I envision two main responses to this, one from epistemic egoism, and one from rule epistemology.
2.1. Epistemic egoism
Firstly, it might be argued that by downgrading Geoff’s epistemic virtue, I in effect put extra weight on my own beliefs merely because they are mine. Since we are both peers, I should treat each of us equally.
I don’t think this argument works. My argument for demoting Geoff is not “my belief that his reasoning is bad” but rather “his reasoning is bad, which I believe”.[3] These two putative reasons are different. One can see this by using the following counterfactual test. Imagine a hypothetical world where I believe that Geoff’s reasoning is bad, but I am wrong. Do I, in the actual world, believe that I should still demote Geoff in the imagined world? No I do not. If I am wrong, I should not demote. So, my reason for downgrading is not my belief that the reasoning is bad, but is rather the proposition that the reasoning is bad, which I believe. Therefore, there is no objectionable epistemic egoism here.
In the sleeping pills case, when I demote someone from my peer group because they appeal to homeopathy, my reason to demote them is not my belief that homeopathy is unscientific, it is the fact that homeopathy is unscientific.
2.2. Rule epistemology
Another possible response is to say that I do better by following the rule of not downgrading according to the object-level reasons, so I should follow that rule rather than trying to find the exceptions. My response to this is that I do even better by following that rule except when it comes to Geoff. There is a difference between trying (and maybe failing) to justifiably downgrade experts and justifiably downgrading experts. If it is asked - ‘why think you should downgrade in this case?’, then a good answer is just to refer to the object-level reasons. This is a good answer regarding sleeping pills, just as it is, as everyone agrees, regarding medical expertise in general.
It is true that the rule of demoting people on the basis of the object-level reasons is liable to abuse. A mercantilist could use this kind of reasoning to demote all trade economists to be his epistemic inferiors. However, the problem here is bad object-level reasoning and epistemic virtue assessment, not the plausibility of the Object-level Reasons Restriction. As we have seen, proponents of epistemic modesty agree that object-level reasons are sometimes admissible when we are assessing people’s epistemic virtue. For example, the Object-level Reasons Restriction would not rule out the mercantilist using his views on trade as a reason to demote economists on other questions in economics. This is also an error, but one that merely stems from mundane bad reasoning, which we have non-modest reasons to care about.
What is usually going on when mercantilists dismiss the expert consensus on trade is (1) they simply don’t understand the arguments for and against free trade; and (2) that many of them also simply do not know that there is such a strong expert consensus on trade. This is simply inept reasoning, not an argument for never considering the object-level arguments when picking one’s peers.
3. Where does this leave us?
I have argued that it is sometimes appropriate to appeal to the object-level arguments on p when deciding on people’s epistemic virtue on p. I illustrated this with (I hope) a relatively clear case involving a rogue homeopath.
The arguments here are potentially practically important. Any sophisticated assessment of some topic will involve appealing to the object-level reasons to give more weight to the views of some putative experts who appear to do equally well on the meta-level considerations.
This is the approach that GiveWell takes when assessing the evidence on interventions in global health. Their approach is not to just take the median view in the literature on some question, but rather to filter out certain parts of the literature on the basis of general methodological quality. In their post on Common Problems with Formal Evaluations, they say they are hesitant to use non-RCT evidence because of selection effects and publication bias.[4] In this way, object-level arguments play a role in filtering the body of evidence that GiveWell responds to. Nonetheless, their approach is still modest in the sense that the prevailing view found among studies of sufficiently high quality plays a major role in determining their all-things-considered view.
In his review of the effect of saving lives on fertility, David Roodman rules out certain types of evidence, such as cross-country regressions and some of the instrumental variables studies, and puts more weight on other quasi-experimental studies. In this way, his final credence is determined by the expert consensus as weighted by the object-level reasons, rather than the crude median of the aggregate of putative experts.
Continental philosophers—Foucauldians, Hegelians, Marxists, postmodernists—do pretty well on the meta-level considerations—time, general cognitive ability, scrutiny and numbers. But they do poorly at object-level reasoning. This is a good reason to give them less weight than analytic philosophers when forming credences about philosophy.[5]
The examples above count against an ‘anything goes’ approach to assessments of the object-level reasons; the object-level reasoning still has to be done well. David Roodman and GiveWell put incredible amounts of intellectual heft into filtering the evidence. Epistemic peerhood and superiority are relative, and Roodman and GiveWell set a high bar. The point here is just that the object-level reasons are sometimes admissible.
The object-level reasons and the meta-level considerations should each play a role in assessments of epistemic virtue. This can justify positions that seem immodest. Some examples that are top of mind for me:
We should put less weight on estimates of climate sensitivity that update from a uniform prior, as I argued here.
We should almost all ignore nutritional epidemiology and just follow an enlightened common sense prior on nutrition.
We should often not defer to experts who use non-Bayesian epistemology, where this might make a difference relative to the prevailing (eg frequentist or scientistic) epistemology. E.g. this arguably played a role in early mainstream expert scepticism towards face masks as a tool to prevent COVID transmission.
[1] I avoid calling these ‘outside view’ reasons because the object-level reasons might also be outside view/base rate-type reasons. It might be that the people who do well on the meta-level reasons ignore the ‘outside view’ reasons. Witness, for example, the performance of many political scientists in Tetlock’s experiments.
[2] This is how I interpret Greg Lewis’ account of epistemic modesty: “One rough heuristic for strong modesty is this: for any question, find the plausible expert class to answer that question (e.g. if P is whether to raise the minimum wage, talk to economists). If this class converges on a particular answer, believe that answer too. If they do not agree, have little confidence in any answer. Do this no matter whether one’s impression of the object level considerations that recommend (by your lights) a particular answer.”
[3] David Enoch, “Not Just a Truthometer: Taking Oneself Seriously (but Not Too Seriously) in Cases of Peer Disagreement,” Mind 119, no. 476 (January 10, 2010): sec. 7, https://doi.org/10.1093/mind/fzq070.
[4] I have some misgivings about the RCT-focus of GiveWell’s methodology, but I think the general approach of filtering good and bad studies on the basis of methodological quality is correct.
[5] I think a reasonable case could be made for giving near zero weight to the views of Continental philosophers on difficult philosophical topics.