Say how much, not more or less versus someone else

Gregory Lewis🔾28 Dec 2023 22:24 UTC
100 points
10 comments5 min readEA link

Or: “Underrated/​overrated” discourse is itself overrated.

BLUF: “X is overrated”, “Y is neglected”, “Z is a weaker argument than people think”, are all species of second-order evaluations: we are not directly offering an assessment of X, Y, or Z, but do so indirectly by suggesting another assessment, offered by someone else, needs correcting up or down.

I recommend everyone cut this habit down ~90% in aggregate for topics they deem important, replacing the great majority of second-order evaluations with first-order evaluations. Rather than saying whether you think X is over/​under rated (etc.) just try and say how good you think X is.

The perils of second-order evaluation

Suppose I say “I think forecasting is underrated”. Presumably I mean something like:

  1. I think forecasting should be rated this highly (e.g. 8⁄10 or whatever)

  2. I think others rate forecasting lower than this (e.g. 5⁄10 on average or whatever)

  3. So I think others are not rating forecasting highly enough.

Yet whether “Forecasting is overrated” is true or not depends on more than just “how good is forecasting?” It is confounded by questions of which ‘others’ I have in mind, and what their views actually are. E.g.:

  • Maybe you disagree with me—you think forecasting is overrated—but it turns out we basically agree on how good forecasting is. Our apparent disagreement arises because you happen to hang out in more pro-forecasting environments than I do.

  • Or maybe we hang out in similar circles, but we disagree in how to assess the prevailing vibes. We basically agree on how good forecasting is, but differ on what our mutual friends tend to really think about it.

  • (Obviously, you could also get specious agreement of two-wrongs-make-a-right variety: you agree with me forecasting is underrated despite having a much lower opinion of it than I do, because you assess third parties having an even lower opinion still)

These are confounders as they confuse the issue we (usually) care about: how good or bad forecasting is, not the inaccuracy of others nor in which direction they err re. how good they think forecasting is.

One can cut through this murk by just assessing the substantive issue directly. I offer my take on how good forecasting is: if folks agree with me, it seems people generally weren’t over or under- rating forecasting after all. If folks disagree, we can figure out—in the course of figuring out how good forecasting is—whether one of us is over/​under rating it versus the balance of reason, not versus some poorly scribed subset of prevailing opinion. No phantom third parties to the conversation are needed—or helpful to—this exercise.

In praise of (kind-of) objectivity, precision, and concreteness

This is easier said than done. In the forecasting illustration above, I stipulated ‘marks out of ten’ as an assessment of the ‘true value’. This is still vague: if I say forecasting is ‘8/​10’, that could mean a wide variety of things—including basically agreeing with you despite you giving a different number to me. What makes something 8⁄10 versus 7⁄10 here?

It is still a step in the right direction. Although my ‘8/​10’ might be essentially the same as your ‘7/​10’, there probably some substantive difference between 8⁄10 and 5⁄10, or 4⁄10 and 6⁄10. It is still better than second order evaluation, which adds another source of vagueness: although saying for myself forecasting is X/​10 is tricky, it is still harder to do this exercise on someone else’s (or everyone else’s) behalf.

And we need not stop there. Rather than some singular measure like ‘marks out of 10’ for ‘forecasting’ as a whole, maybe we have some specific evalution or recommendation in mind. Perhaps: “Most members of the EA community should have a Metaculus or Good Judgement account they forecast on regularly”, or “Forecasting interventions are the best opportunities in the improving institutional decision-making cause area”, or “Forecasting should pay well enough that skilled practitioners can realistically ‘go pro’, vs. it remaining universally an amateur sport”. Or whatever else.

We thus approach substantive propositions (or proposals), and can avoid a mire of a purely verbal disagreement—or vaguely adversarial vibing.

Caveats

(Tl;dr: I’m right.)

Sometimes things aren’t that ambiguous

The risk I highlight of ‘Alice thinks X is overrated, Bob thinks it is underrated—but they basically agree on X, but disagree on what other people think about it’ can sometimes be remote. One example is if someone has taken the trouble to clearly and precisely spell out where they stand themselves. Just saying “I’d take the over/​under on what they think” could be poor epistemic sportsmanship (all too easy to criticise something specific whilst sheltering in generalities yourself), and could do to be more precise (how much over? etc.) but at least there is an actual difference, and you can be reliably placed to a region on the number line.

Another example is where you are really sure you are an outlier vs. ~ everyone else: you rate something so highly or lowly that ~ everyone else—whoever they are—is under/​overrating it by your lights. This will typically be reserved for ones hottest, most extreme, and iconoclastic takes. In principle, this should be rare. In practice, it can be the prelude to verbal clickbait: “looking after your kids is overrated” better be elaborated with something at least as spicy as Caplan’s views on parenting, rather than some milquetoast climbdown along the lines of ‘parents should take care of themselves too’ or whatever.

Even here, trying to say how much can be clearer if your view really is ‘a hell of a lot’. “Buffy the Vampire Slayer is criminally underrated” could merely mean I place it a cut above other ~naughties TV serials. Yet if I really think things like, “Season 5 of Buffy alone places it on the highest summits of artistic achievement, and the work as a whole makes a similar contribution to television as Beethoven’s Grosse Fuge does to classical music” I should say so, such that listeners are clear in which ballpark I am in, and how far I am departing from common sense.

Updates and pricing in

Overrated/​underrated can have a different goal than offering an overall assessment. It could instead be a means of introducing a new argument for or against X. E.g. perhaps what I could mean by ‘forecasting is underrated’ is something like “I have found a new consideration in favour of forecasting, so folks—who are not aware of it yet—need to update upwards from wherever they were beforehand.”

This is better, but still not great. (E.g.) “X is underrated because R” at least gives a locus for discussion (R? ¬R?), but second-order considerations can still confound. Although R may be novel to the speaker, others may at least be dimly aware of it, or some R* nearby to it, so perhaps they have already somewhat ‘priced in’ R for the all things considered assessment. “I think the strength of R pro/​con X is under/​overestimated by others” has the familiar problems outlined above.

Saying how much—the now familiar remedy—remains effective. (E.g.) “I think R drops the value of X by 5%/​50%/​99%” or whatever clearly signals the strength of consideration you are assigning to R, and sidesteps issues of trying to assess whether someone else (in the conversation or not) are aware of or are appropriately incorporating R into their deliberations.

Cadenza

As before, this greater precision is not a free lunch: it takes both more space on the page to write and more time in the brain to think through. Also as before, there are times when this extra effort is a waste. If I assert “Taylor Swift is overrated” to my sister, and she asserts “Bach is overrated [sic][1]” in turn, neither the subject matter warrants—nor the conversational purpose well-served by—a careful pseudo-quantitative quasi-objective disquisition into the musical merit of each. Low-res ‘Less/​more than someone thinks’ remarks are also fine for a bunch of other circumstances. Usually unimportant ones.

Yet also as before, sometimes there is a real matter which really matters, sometimes we want our words to amount to substantial work not idle talk, and sometimes we at least aspire to be serious people striving to say something serious about something serious. For such Xs, it is rare for there to be disagreement about whether a given issue is relevant to X, ditto whether its direction is ‘pro’ or ‘con’ X, but rather its magnitude: how much it counts ‘pro’ or ‘con’ X, and so where the overall balance of reason lies re. X all things considered, where all the things to be considered are all various degrees of ‘kinda, but...’, which need to be all weighed together.[2]

In these cases that count, something like counting needs to be attempted in natural language, despite its inadequacy for the task. Yet although (e.g.) “8/​10”, “maybe this cuts 20% off the overall value of X” (etc.) remain imperfect, more/​less statements versus some usually vague comparator is even worse. Simply put: underrated/​overrated is a peregrination, not a prolegomenon, for the project of proper precisification.[3]

Reality is concrete; its machinations, exact. When it is important to talk about it, our words should try to be the same.

  1. ^

    [sic]

  2. ^

    Cf. my previously expressed (and still maintained) allergy towards ‘crux’ ‘cruxy’, etc.

  3. ^

    Peccavi

Gregory Lewis🔾28 Dec 2023 22:24 UTC
100 points
10 comments5 min readEA link