Or: âUnderrated/âoverratedâ discourse is itself overrated.
BLUF: âX is overratedâ, âY is neglectedâ, âZ is a weaker argument than people thinkâ, are all species of second-order evaluations: we are not directly offering an assessment of X, Y, or Z, but do so indirectly by suggesting another assessment, offered by someone else, needs correcting up or down.
I recommend everyone cut this habit down ~90% in aggregate for topics they deem important, replacing the great majority of second-order evaluations with first-order evaluations. Rather than saying whether you think X is over/âunder rated (etc.) just try and say how good you think X is.
The perils of second-order evaluation
Suppose I say âI think forecasting is underratedâ. Presumably I mean something like:
I think forecasting should be rated this highly (e.g. 8â10 or whatever)
I think others rate forecasting lower than this (e.g. 5â10 on average or whatever)
So I think others are not rating forecasting highly enough.
Yet whether âForecasting is overratedâ is true or not depends on more than just âhow good is forecasting?â It is confounded by questions of which âothersâ I have in mind, and what their views actually are. E.g.:
Maybe you disagree with meâyou think forecasting is overratedâbut it turns out we basically agree on how good forecasting is. Our apparent disagreement arises because you happen to hang out in more pro-forecasting environments than I do.
Or maybe we hang out in similar circles, but we disagree in how to assess the prevailing vibes. We basically agree on how good forecasting is, but differ on what our mutual friends tend to really think about it.
(Obviously, you could also get specious agreement of two-wrongs-make-a-right variety: you agree with me forecasting is underrated despite having a much lower opinion of it than I do, because you assess third parties having an even lower opinion still)
These are confounders as they confuse the issue we (usually) care about: how good or bad forecasting is, not the inaccuracy of others nor in which direction they err re. how good they think forecasting is.
One can cut through this murk by just assessing the substantive issue directly. I offer my take on how good forecasting is: if folks agree with me, it seems people generally werenât over or under- rating forecasting after all. If folks disagree, we can figure outâin the course of figuring out how good forecasting isâwhether one of us is over/âunder rating it versus the balance of reason, not versus some poorly scribed subset of prevailing opinion. No phantom third parties to the conversation are neededâor helpful toâthis exercise.
In praise of (kind-of) objectivity, precision, and concreteness
This is easier said than done. In the forecasting illustration above, I stipulated âmarks out of tenâ as an assessment of the âtrue valueâ. This is still vague: if I say forecasting is â8/â10â, that could mean a wide variety of thingsâincluding basically agreeing with you despite you giving a different number to me. What makes something 8â10 versus 7â10 here?
It is still a step in the right direction. Although my â8/â10â might be essentially the same as your â7/â10â, there probably some substantive difference between 8â10 and 5â10, or 4â10 and 6â10. It is still better than second order evaluation, which adds another source of vagueness: although saying for myself forecasting is X/â10 is tricky, it is still harder to do this exercise on someone elseâs (or everyone elseâs) behalf.
And we need not stop there. Rather than some singular measure like âmarks out of 10â for âforecastingâ as a whole, maybe we have some specific evalution or recommendation in mind. Perhaps: âMost members of the EA community should have a Metaculus or Good Judgement account they forecast on regularlyâ, or âForecasting interventions are the best opportunities in the improving institutional decision-making cause areaâ, or âForecasting should pay well enough that skilled practitioners can realistically âgo proâ, vs. it remaining universally an amateur sportâ. Or whatever else.
We thus approach substantive propositions (or proposals), and can avoid a mire of a purely verbal disagreementâor vaguely adversarial vibing.
Caveats
(Tl;dr: Iâm right.)
Sometimes things arenât that ambiguous
The risk I highlight of âAlice thinks X is overrated, Bob thinks it is underratedâbut they basically agree on X, but disagree on what other people think about itâ can sometimes be remote. One example is if someone has taken the trouble to clearly and precisely spell out where they stand themselves. Just saying âIâd take the over/âunder on what they thinkâ could be poor epistemic sportsmanship (all too easy to criticise something specific whilst sheltering in generalities yourself), and could do to be more precise (how much over? etc.) but at least there is an actual difference, and you can be reliably placed to a region on the number line.
Another example is where you are really sure you are an outlier vs. ~ everyone else: you rate something so highly or lowly that ~ everyone elseâwhoever they areâis under/âoverrating it by your lights. This will typically be reserved for ones hottest, most extreme, and iconoclastic takes. In principle, this should be rare. In practice, it can be the prelude to verbal clickbait: âlooking after your kids is overratedâ better be elaborated with something at least as spicy as Caplanâs views on parenting, rather than some milquetoast climbdown along the lines of âparents should take care of themselves tooâ or whatever.
Even here, trying to say how much can be clearer if your view really is âa hell of a lotâ. âBuffy the Vampire Slayer is criminally underratedâ could merely mean I place it a cut above other ~naughties TV serials. Yet if I really think things like, âSeason 5 of Buffy alone places it on the highest summits of artistic achievement, and the work as a whole makes a similar contribution to television as Beethovenâs Grosse Fuge does to classical musicâ I should say so, such that listeners are clear in which ballpark I am in, and how far I am departing from common sense.
Updates and pricing in
Overrated/âunderrated can have a different goal than offering an overall assessment. It could instead be a means of introducing a new argument for or against X. E.g. perhaps what I could mean by âforecasting is underratedâ is something like âI have found a new consideration in favour of forecasting, so folksâwho are not aware of it yetâneed to update upwards from wherever they were beforehand.â
This is better, but still not great. (E.g.) âX is underrated because Râ at least gives a locus for discussion (R? ÂŹR?), but second-order considerations can still confound. Although R may be novel to the speaker, others may at least be dimly aware of it, or some R* nearby to it, so perhaps they have already somewhat âpriced inâ R for the all things considered assessment. âI think the strength of R pro/âcon X is under/âoverestimated by othersâ has the familiar problems outlined above.
Saying how muchâthe now familiar remedyâremains effective. (E.g.) âI think R drops the value of X by 5%/â50%/â99%â or whatever clearly signals the strength of consideration you are assigning to R, and sidesteps issues of trying to assess whether someone else (in the conversation or not) are aware of or are appropriately incorporating R into their deliberations.
Cadenza
As before, this greater precision is not a free lunch: it takes both more space on the page to write and more time in the brain to think through. Also as before, there are times when this extra effort is a waste. If I assert âTaylor Swift is overratedâ to my sister, and she asserts âBach is overrated [sic][1]â in turn, neither the subject matter warrantsânor the conversational purpose well-served byâa careful pseudo-quantitative quasi-objective disquisition into the musical merit of each. Low-res âLess/âmore than someone thinksâ remarks are also fine for a bunch of other circumstances. Usually unimportant ones.
Yet also as before, sometimes there is a real matter which really matters, sometimes we want our words to amount to substantial work not idle talk, and sometimes we at least aspire to be serious people striving to say something serious about something serious. For such Xs, it is rare for there to be disagreement about whether a given issue is relevant to X, ditto whether its direction is âproâ or âconâ X, but rather its magnitude: how much it counts âproâ or âconâ X, and so where the overall balance of reason lies re. X all things considered, where all the things to be considered are all various degrees of âkinda, but...â, which need to be all weighed together.[2]
In these cases that count, something like counting needs to be attempted in natural language, despite its inadequacy for the task. Yet although (e.g.) â8/â10â, âmaybe this cuts 20% off the overall value of Xâ (etc.) remain imperfect, more/âless statements versus some usually vague comparator is even worse. Simply put: underrated/âoverrated is a peregrination, not a prolegomenon, for the project of proper precisification.[3]
Reality is concrete; its machinations, exact. When it is important to talk about it, our words should try to be the same.
Say how much, not more or less versus someone else
Or: âUnderrated/âoverratedâ discourse is itself overrated.
BLUF: âX is overratedâ, âY is neglectedâ, âZ is a weaker argument than people thinkâ, are all species of second-order evaluations: we are not directly offering an assessment of X, Y, or Z, but do so indirectly by suggesting another assessment, offered by someone else, needs correcting up or down.
I recommend everyone cut this habit down ~90% in aggregate for topics they deem important, replacing the great majority of second-order evaluations with first-order evaluations. Rather than saying whether you think X is over/âunder rated (etc.) just try and say how good you think X is.
The perils of second-order evaluation
Suppose I say âI think forecasting is underratedâ. Presumably I mean something like:
I think forecasting should be rated this highly (e.g. 8â10 or whatever)
I think others rate forecasting lower than this (e.g. 5â10 on average or whatever)
So I think others are not rating forecasting highly enough.
Yet whether âForecasting is overratedâ is true or not depends on more than just âhow good is forecasting?â It is confounded by questions of which âothersâ I have in mind, and what their views actually are. E.g.:
Maybe you disagree with meâyou think forecasting is overratedâbut it turns out we basically agree on how good forecasting is. Our apparent disagreement arises because you happen to hang out in more pro-forecasting environments than I do.
Or maybe we hang out in similar circles, but we disagree in how to assess the prevailing vibes. We basically agree on how good forecasting is, but differ on what our mutual friends tend to really think about it.
(Obviously, you could also get specious agreement of two-wrongs-make-a-right variety: you agree with me forecasting is underrated despite having a much lower opinion of it than I do, because you assess third parties having an even lower opinion still)
These are confounders as they confuse the issue we (usually) care about: how good or bad forecasting is, not the inaccuracy of others nor in which direction they err re. how good they think forecasting is.
One can cut through this murk by just assessing the substantive issue directly. I offer my take on how good forecasting is: if folks agree with me, it seems people generally werenât over or under- rating forecasting after all. If folks disagree, we can figure outâin the course of figuring out how good forecasting isâwhether one of us is over/âunder rating it versus the balance of reason, not versus some poorly scribed subset of prevailing opinion. No phantom third parties to the conversation are neededâor helpful toâthis exercise.
In praise of (kind-of) objectivity, precision, and concreteness
This is easier said than done. In the forecasting illustration above, I stipulated âmarks out of tenâ as an assessment of the âtrue valueâ. This is still vague: if I say forecasting is â8/â10â, that could mean a wide variety of thingsâincluding basically agreeing with you despite you giving a different number to me. What makes something 8â10 versus 7â10 here?
It is still a step in the right direction. Although my â8/â10â might be essentially the same as your â7/â10â, there probably some substantive difference between 8â10 and 5â10, or 4â10 and 6â10. It is still better than second order evaluation, which adds another source of vagueness: although saying for myself forecasting is X/â10 is tricky, it is still harder to do this exercise on someone elseâs (or everyone elseâs) behalf.
And we need not stop there. Rather than some singular measure like âmarks out of 10â for âforecastingâ as a whole, maybe we have some specific evalution or recommendation in mind. Perhaps: âMost members of the EA community should have a Metaculus or Good Judgement account they forecast on regularlyâ, or âForecasting interventions are the best opportunities in the improving institutional decision-making cause areaâ, or âForecasting should pay well enough that skilled practitioners can realistically âgo proâ, vs. it remaining universally an amateur sportâ. Or whatever else.
We thus approach substantive propositions (or proposals), and can avoid a mire of a purely verbal disagreementâor vaguely adversarial vibing.
Caveats
(Tl;dr: Iâm right.)
Sometimes things arenât that ambiguous
The risk I highlight of âAlice thinks X is overrated, Bob thinks it is underratedâbut they basically agree on X, but disagree on what other people think about itâ can sometimes be remote. One example is if someone has taken the trouble to clearly and precisely spell out where they stand themselves. Just saying âIâd take the over/âunder on what they thinkâ could be poor epistemic sportsmanship (all too easy to criticise something specific whilst sheltering in generalities yourself), and could do to be more precise (how much over? etc.) but at least there is an actual difference, and you can be reliably placed to a region on the number line.
Another example is where you are really sure you are an outlier vs. ~ everyone else: you rate something so highly or lowly that ~ everyone elseâwhoever they areâis under/âoverrating it by your lights. This will typically be reserved for ones hottest, most extreme, and iconoclastic takes. In principle, this should be rare. In practice, it can be the prelude to verbal clickbait: âlooking after your kids is overratedâ better be elaborated with something at least as spicy as Caplanâs views on parenting, rather than some milquetoast climbdown along the lines of âparents should take care of themselves tooâ or whatever.
Even here, trying to say how much can be clearer if your view really is âa hell of a lotâ. âBuffy the Vampire Slayer is criminally underratedâ could merely mean I place it a cut above other ~naughties TV serials. Yet if I really think things like, âSeason 5 of Buffy alone places it on the highest summits of artistic achievement, and the work as a whole makes a similar contribution to television as Beethovenâs Grosse Fuge does to classical musicâ I should say so, such that listeners are clear in which ballpark I am in, and how far I am departing from common sense.
Updates and pricing in
Overrated/âunderrated can have a different goal than offering an overall assessment. It could instead be a means of introducing a new argument for or against X. E.g. perhaps what I could mean by âforecasting is underratedâ is something like âI have found a new consideration in favour of forecasting, so folksâwho are not aware of it yetâneed to update upwards from wherever they were beforehand.â
This is better, but still not great. (E.g.) âX is underrated because Râ at least gives a locus for discussion (R? ÂŹR?), but second-order considerations can still confound. Although R may be novel to the speaker, others may at least be dimly aware of it, or some R* nearby to it, so perhaps they have already somewhat âpriced inâ R for the all things considered assessment. âI think the strength of R pro/âcon X is under/âoverestimated by othersâ has the familiar problems outlined above.
Saying how muchâthe now familiar remedyâremains effective. (E.g.) âI think R drops the value of X by 5%/â50%/â99%â or whatever clearly signals the strength of consideration you are assigning to R, and sidesteps issues of trying to assess whether someone else (in the conversation or not) are aware of or are appropriately incorporating R into their deliberations.
Cadenza
As before, this greater precision is not a free lunch: it takes both more space on the page to write and more time in the brain to think through. Also as before, there are times when this extra effort is a waste. If I assert âTaylor Swift is overratedâ to my sister, and she asserts âBach is overrated [sic][1]â in turn, neither the subject matter warrantsânor the conversational purpose well-served byâa careful pseudo-quantitative quasi-objective disquisition into the musical merit of each. Low-res âLess/âmore than someone thinksâ remarks are also fine for a bunch of other circumstances. Usually unimportant ones.
Yet also as before, sometimes there is a real matter which really matters, sometimes we want our words to amount to substantial work not idle talk, and sometimes we at least aspire to be serious people striving to say something serious about something serious. For such Xs, it is rare for there to be disagreement about whether a given issue is relevant to X, ditto whether its direction is âproâ or âconâ X, but rather its magnitude: how much it counts âproâ or âconâ X, and so where the overall balance of reason lies re. X all things considered, where all the things to be considered are all various degrees of âkinda, but...â, which need to be all weighed together.[2]
In these cases that count, something like counting needs to be attempted in natural language, despite its inadequacy for the task. Yet although (e.g.) â8/â10â, âmaybe this cuts 20% off the overall value of Xâ (etc.) remain imperfect, more/âless statements versus some usually vague comparator is even worse. Simply put: underrated/âoverrated is a peregrination, not a prolegomenon, for the project of proper precisification.[3]
Reality is concrete; its machinations, exact. When it is important to talk about it, our words should try to be the same.
[sic]
Cf. my previously expressed (and still maintained) allergy towards âcruxâ âcruxyâ, etc.
Peccavi