In EA, there appears to be an interest in “good judgment,” sometimes also called “rationality.”
There is also interest in forecasting.
My question is, what are the concrete, operationalized differences between skill at forecasting vs having good judgment?
I’m not asking this question facetiously. For example, the parent company/organization of Superforecasting brands itself as the “Good Judgment Project.”
But at the same time, when I think about “being good at forecasting” and “having good judgment,” I often think of many different qualities. So how can we cleanly separate the two?
Maybe I’ve misunderstood but in my humble opinion, and limited experience, forecasting is just a tiny tiny fraction of good judgement, (maybe about 1% depending on how broad you define forecasting). It can be useful, but somewhat overrated by the EA community.
Other aspects of good judgment may include things like:
Direction setting
Agenda setting
Being conscious of when you change direction part way through a judgement
Understanding the range of factors that are important to a judgement
Knowing how long to spend and how much effort to invest in a judgement
Brainstorming
Creative thinking
Solutions design
Research skills
Information processing
Group dynamics for consensus building or finding challenge
Knowing who to trust
Drawing analogies to other similar situations
Knowing when analagies are likely to be valid
Good intuition
Methords for digging into intuitions
Ability to test and moderate your intuition
Scenario planning (distinct from foresight?)
Horizen scanning (distinct from foresight?)
Foresight and predictions
Robust decion making
A range of models of the world which can inform the judgment
Good heuristics
Systems thinking
Self-awareness
Ability to adjust for unknown unknowns
Seeking evidence that contradicts the way you may want to go
Understanding and counteracting other biases
Understand statistics
Accounting for statistical issues like regression to mean or optimisers curse
Making quantitative comparisons
Weighing up pros and cons
Other generic decision making tools that can be applied, of which there are lots
Specific decion making tools applicable to specific situations
Knowing which of the above is most relevant to a judgement
Ability to bring all of the above together
Speed at bringing all the above together
Preparing for and understating the consequences of having made the wrong judgement
Ability to relearn and update judement later with new evidence
Etc
I think it would be clearer to put many of these under different categories than to lump everything under judgement. In my post I also cover the following, and try to sketch how they’re different:
Intelligence
Decision-making
Strategy
I should have maybe mentioned creativity as another category.
I also contrast ‘using judgement’ with alternatives like statistical analysis; applying best practice; quantitative models etc., though you might draw on these in making your judgement.
Thank Ben super useful.
@Linch I was taking a very very broad view of judgment.
Ben’s post is much better and breaks things done in a much nicer way.
I also made a (not particularly successful) stab at explaining some aspects of not-foresight driven judgement here: https://forum.effectivealtruism.org/posts/znaZXBY59Ln9SLrne/how-to-think-about-an-uncertain-future-lessons-from-other#Story_1__RAND_and_the_US_military
Thanks a lot for the answer! A lot of the things you put into “other” (which is a very long list, btw!) are things I’d put under “forecasting.” I wonder where the crux is?
Some examples (non-exhaustive) of things I consider to be closer to “forecasting” than “not forecasting.”
I also understand all of these as very important to forecasting.
If you’re good at forecasting it’s reasonable to expect you’ll be above average at reasoning or decision making tasks that require making predictions.
But judgment is potentially different. In “Prediction Machines” Agrawal et al separate judgment and prediction as two distinct parts of decision making where the former involves weighing tradeoffs. That’s harder to measure but a potentially distinct way to think about the difference between judgment and forecasting. They have a theoretical paper on this decision making model too.
I think I agree with this answer.
To answer my own question, here is my best guess for how “good judgment” is different from “skill at forecasting.”
Good judgment can roughly be divided within 2 mostly distinct clusters:
Forming sufficiently good world models given practical constraints.
Making good decisions on the basis of such (often limited) models.
Forecasting is only directly related to the former, and not the later (though presumably there are some general skills that are applicable to both). In addition, within the “forming good world models” angle, good forecasting is somewhat agnostic to important factors like:
Group epistemics. There are times where it’s less important whether an individual has the right world models but that your group has access to the right plethora of models.
It may be the case that it’s practically impossible for a single individual to hold all of them, so specialization is necessary.
Asking the right questions. Having the world’s lowest Brier score on something useless is in some sense impressive, but it’s not very impactful compared to being moderately accurate on more important questions.
Correct contrarianism. As a special case of the above two points, in both science and startups, it is often (relatively) more important to be right about things that others are wrong about than it is to be right about everything other people are right about.
___
Note that “better world models” vs “good decisions based on existing models” isn’t the only possible ontology to break up “good judgment.”
- Owen uses understanding of the world vs heuristics.
- In the past, I’ve used intelligence vs wisdom.
From the post you refer to:
Calibrated estimates for future events is the goal of forecasting, and while model-building and knowledge are valuable for this, I think they’re valuable in other ways, too. I think another component of good judgement is being able to judge which problems to work on in the first place and how much effort and resources to put into them, falling under instrumental rationality. You need to decide which problems to apply your forecasting skills to, and I don’t think this is a forecasting problem.
Also, my understanding is that forecasting is specific to predicting possible future events, and would not include having reasonable views on fundamental research questions, e.g. about consciousness, in physics, in normative ethics, etc..
(I suppose you could try to forecast the answers of experts or even hypothetical experts for fundamental research questions, but experts can be wrong, and this seems like a pretty unusual application and ad hoc way to get at fundamental research questions.)