I definitely agree we can look at qualitative data for hypothesis generation (after all, n=1 is still an existence proof). But Iâd generally recommend breadth-first rather than depth-first if weâre trying to adduce considerations.
For many/âmost sorts of policy decisions although we may find a case of X (some factor) --> Y (some desirable outcome), we can probably also find cases of ÂŹX --> Y and X --> ÂŹY. E.g., contrasting with what happened with prospect theory, there are also cases where someone happened on an important breakthrough with much less time/âeffort, or where people over-committed to an intellectual dead-end (naturally, partisans of X or ÂŹX tend to be good at cultivating sets of case-studies which facially support the claim it leads to Y.)
I generally see getting a steer of the correlation of X and Y (so the relative abundance of (ÂŹ/â)X --> (ÂŹ/â)Y across a broad reference class as more valuable than determining whether in a given case (even one which seems nearby to the problem weâre interested in) X really was playing a causal role in driving Y. Problems of selection are formidable, but I take the problems of external validity to tend even worse (and worse enough to make the former have a better ratio of insight:resources).
Thus Iâd be much more interested to see (e.g.) a wide survey of cases which suggests movements prone to in-fighting tend to be less successful than an in depth look of how in-fighting caused the destruction of a nearby analogue to the EA community. Ditto the âmacroâ in macrohistory being at least partly about trying to adduce takeaways across history, as well as trying to divine its big contours.
And although I think work like this is worthwhile to attempt, I think in some instances we may come to learn that reality is so underpowered that thereâs essentially no point doing research (e.g. maybe large bits of history are just ultra-chaotic, so all we can ever see is noise).
I agree with your points, but from my perspective they somewhat miss the mark.
Specifically, your discussion seems to assume that we have a fixed, exogenously given set of propositions or factors X, Y, âŚ, and that our sole task is to establish relations of correlation and causation between them. In this context, I agree on preferring âwide surveysâ etc.
However, in fact, doing research also requires the following tasks:
Identify which factors X, Y, ⌠to consider in the first place.
Refine the meaning of the considered factors X, Y, ⌠by clarifying their conceptual and hypothesized empirical relationships to other factors.
Prioritize which of the myriads of possible correlational or causal relationships between the factors X, Y, ⌠to test.
I think that depth can help with these three tasks in ways in which breadth canât.
For instance, in Willâs example, my guess is that the main value of considering the history of Objectivism does not come from moving my estimate for the strength of the hypothesis âX = romantic involvement between movement leaders â Y = movement collapsesâ. Rather, the source of value is including âromantic involvement between movement leadersâ into the set of factors Iâm considering in the first place. Only then am I able to investigate its relation to outcomes of interests, whether by a âwide survey of casesâ or otherwise. Moreover, I might only have learned about the potential relevance of âromantic involvement between movement leadersâ by looking at some depth into the history of Objectivism. (I know very little about Objectivism, and so donât know if this is true in this instance; itâs certainly possible that the issue of romantic involvement between Objectivist leaders is so well known that it would be mentioned in any 5-sentence summary one would encounter during a breadth-first process. But it also seems possible that itâs not, and Iâm sure I could come up with examples where the interesting factor was buried deeply.)
My model here squares well with your observation that a âcommon feature among superforecasters is they read a lotâ, and in fact makes a more specific prediction: I expect that weâd find that superforecasters read a fair amount (say, >10% of their total reading) of deep, small-n case studiesâfor example, historical accounts of a single war, economic policy, or biographies.
[My guess is that my comment is largely just restating Willâs points from his above comment in other words.]
(FWIW, I think some generators of my overall model here are:
Frequently experiencing disagreements I have with others, especially around AI timelines and takeoff scenarios, as noticing a thought like âUh⌠I just think your overall model of the world lacks depth and detail.â rather than âWait, Iâve read about 50 similar cases, and only 10 of them are consistent with your claimâ.
Semantic holism, or at least some of the arguments usually given in its favor.
Some intuitive and fuzzy sense that, in the terminology of this Julia Galef post, being a âHayekianâ has worked better for me than being a âPlannerâ, including for making epistemic progress.
Some intuitive and fuzzy sense of what Iâve gotten out of âdeepâ versus âbroadâ reading. E.g. my sense is that reading Robert Caroâs monumental, >1,300-page biography of New York city planner Robert Moses has had a significant impact on my model of how individuals can attain political power, albeit by adding a bunch of detail and drawing my attention to factors I previously wouldnât have considered rather than by providing evidence for any particular hypothesis.)
Thanks, Will!
I definitely agree we can look at qualitative data for hypothesis generation (after all, n=1 is still an existence proof). But Iâd generally recommend breadth-first rather than depth-first if weâre trying to adduce considerations.
For many/âmost sorts of policy decisions although we may find a case of X (some factor) --> Y (some desirable outcome), we can probably also find cases of ÂŹX --> Y and X --> ÂŹY. E.g., contrasting with what happened with prospect theory, there are also cases where someone happened on an important breakthrough with much less time/âeffort, or where people over-committed to an intellectual dead-end (naturally, partisans of X or ÂŹX tend to be good at cultivating sets of case-studies which facially support the claim it leads to Y.)
I generally see getting a steer of the correlation of X and Y (so the relative abundance of (ÂŹ/â)X --> (ÂŹ/â)Y across a broad reference class as more valuable than determining whether in a given case (even one which seems nearby to the problem weâre interested in) X really was playing a causal role in driving Y. Problems of selection are formidable, but I take the problems of external validity to tend even worse (and worse enough to make the former have a better ratio of insight:resources).
Thus Iâd be much more interested to see (e.g.) a wide survey of cases which suggests movements prone to in-fighting tend to be less successful than an in depth look of how in-fighting caused the destruction of a nearby analogue to the EA community. Ditto the âmacroâ in macrohistory being at least partly about trying to adduce takeaways across history, as well as trying to divine its big contours.
And although I think work like this is worthwhile to attempt, I think in some instances we may come to learn that reality is so underpowered that thereâs essentially no point doing research (e.g. maybe large bits of history are just ultra-chaotic, so all we can ever see is noise).
I agree with your points, but from my perspective they somewhat miss the mark.
Specifically, your discussion seems to assume that we have a fixed, exogenously given set of propositions or factors X, Y, âŚ, and that our sole task is to establish relations of correlation and causation between them. In this context, I agree on preferring âwide surveysâ etc.
However, in fact, doing research also requires the following tasks:
Identify which factors X, Y, ⌠to consider in the first place.
Refine the meaning of the considered factors X, Y, ⌠by clarifying their conceptual and hypothesized empirical relationships to other factors.
Prioritize which of the myriads of possible correlational or causal relationships between the factors X, Y, ⌠to test.
I think that depth can help with these three tasks in ways in which breadth canât.
For instance, in Willâs example, my guess is that the main value of considering the history of Objectivism does not come from moving my estimate for the strength of the hypothesis âX = romantic involvement between movement leaders â Y = movement collapsesâ. Rather, the source of value is including âromantic involvement between movement leadersâ into the set of factors Iâm considering in the first place. Only then am I able to investigate its relation to outcomes of interests, whether by a âwide survey of casesâ or otherwise. Moreover, I might only have learned about the potential relevance of âromantic involvement between movement leadersâ by looking at some depth into the history of Objectivism. (I know very little about Objectivism, and so donât know if this is true in this instance; itâs certainly possible that the issue of romantic involvement between Objectivist leaders is so well known that it would be mentioned in any 5-sentence summary one would encounter during a breadth-first process. But it also seems possible that itâs not, and Iâm sure I could come up with examples where the interesting factor was buried deeply.)
My model here squares well with your observation that a âcommon feature among superforecasters is they read a lotâ, and in fact makes a more specific prediction: I expect that weâd find that superforecasters read a fair amount (say, >10% of their total reading) of deep, small-n case studiesâfor example, historical accounts of a single war, economic policy, or biographies.
[My guess is that my comment is largely just restating Willâs points from his above comment in other words.]
(FWIW, I think some generators of my overall model here are:
Frequently experiencing disagreements I have with others, especially around AI timelines and takeoff scenarios, as noticing a thought like âUh⌠I just think your overall model of the world lacks depth and detail.â rather than âWait, Iâve read about 50 similar cases, and only 10 of them are consistent with your claimâ.
Semantic holism, or at least some of the arguments usually given in its favor.
Some intuitive and fuzzy sense that, in the terminology of this Julia Galef post, being a âHayekianâ has worked better for me than being a âPlannerâ, including for making epistemic progress.
Some intuitive and fuzzy sense of what Iâve gotten out of âdeepâ versus âbroadâ reading. E.g. my sense is that reading Robert Caroâs monumental, >1,300-page biography of New York city planner Robert Moses has had a significant impact on my model of how individuals can attain political power, albeit by adding a bunch of detail and drawing my attention to factors I previously wouldnât have considered rather than by providing evidence for any particular hypothesis.)