Reality is often underpowered

[Gregory Lewis🔸]

Introduction

When I worked as a doctor, we had a lecture by a paediatric haematologist, on a condition called Acute Lymphoblastic Leukaemia. I remember being impressed that very large proportions of patients were being offered trials randomising them between different treatment regimens, currently in clinical equipoise, to establish which had the edge. At the time, one of the areas of interest was, given the disease tended to have a good prognosis, whether one could reduce treatment intensity to reduce the long term side-effects of the treatment whilst not adversely affecting survival.

On a later rotation I worked in adult medicine, and one of the patients admitted to my team had an extremely rare cancer,^[1] with a (recognised) incidence of a handful of cases worldwide per year. It happened the world authority on this condition worked as a professor of medicine in London, and she came down to see them. She explained to me that treatment for this disease was almost entirely based on first principles, informed by a smattering of case reports. The disease unfortunately had a bleak prognosis, although she was uncertain whether this was because it was an aggressive cancer to which current medical science has no answer, or whether there was an effective treatment out there if only it could be found.

I aver that many problems EA concerns itself with are closer to the second story than the first. That in many cases, sufficient data is not only absent in practice but impossible to obtain in principle. Reality is often underpowered for us to wring the answers from it we desire.

Big units of analysis, small samples

The main driver of this problem for ‘EA topics’ is that the outcomes of interest have units of analysis for which the whole population (leave alone any sample from it) is small-n: e.g. outcomes at the level of a whole company, or a whole state, or whole populations. For these big unit of analysis/​small sample problems, RCTs face formidable in principle challenges:

1. Even if by magic you could get (e.g.) all countries on earth to agree to randomly allocate themselves to policy X or Y, this is merely a sample size of ~200. If you’re looking at companies relevant to cage-free campaigns, or administrative regions within a given state, this can easily fall another order of magnitude.

2. These units of analysis tend highly heterogeneous, almost certainly in ways that affect the outcome of interest. Although the key ‘selling point’ of the RCT is it implicitly controls for all confounders (even ones you don’t know about), this statistical control is a (convex) function of sample size, and isn’t hugely impressive at ~ 100 per arm: it is well within the realms of possibility for the randomisation happen to give arms with unbalanced allocation of any given confounding factor.

3. ‘Roughly’ (in expectation) balanced intervention arms are unlikely to be good enough in cases where the intervention is expected to have much less effect on the outcome than other factors (e.g. wealth, education, size, whatever), thus an effect size that favours one arm or the other can be alternatively attributed to one of these.

4. Supplementing this raw randomisation by explicitly controlling for confounders you suspect (cf. block randomisation, propensity matching, etc.) has limited value when don’t know all the factors which plausibly ‘swamp’ the likely intervention effect (i.e. you don’t have a good predictive model for the outcome but-for the intervention tested). In any case, they tend to trade-off against the already scarce resource of sample size.

These ‘small sample’ problems aren’t peculiar to RCTs, but endemic to all other empirical approaches. The wealth of econometric and quasi-experimental methods (e.g. IVs, regression discontinuity analysis), still run up against hard data limits, as well those owed to in whatever respect they fall short of the ‘ideal’ RCT set-up (e.g. imperfect instrumentation, omitted variable bias, nagging concerns about reverse causation). Qualitative work (case studies, etc.) have the same problems even if other ones (e.g. selection) loom larger.

Value of information and the margin of common-sense

None of this means such work has zero value—big enough effect sizes can still be reliably detected, and even underpowered studies still give us information. But we may learn very little on the margin of common sense. Suppose we are interested in ‘what makes social movements succeed or fail?’ and we retrospectively assess a (somehow) representative sample of social movements. It seems plausible the results of this investigation is the big (and plausibly generalisable) hits may prove commonsensical (e.g. “Social movements are more likely to grow if members talk to other people about the social movement”), whilst the ‘new lessons’ remain equivocal and uncertain.

We should expect to see this if we believe the distribution of relevant effect sizes is heavy-tailed, with most of the variance in (say) social movement success owed to a small number of factors, with the rest comprised of a large multitude of smaller effects. In such case, modest increases in information (e.g. from small sample data) may bring even more modest increases in either explaining the outcome or identifying what contributes to it:

Toy example, where we propose a roughly pareto distribution of effect size among contributory factors. The largest factors (which nonetheless explain a minority of the variance) may prove to be obvious to the naked eye (blue). Adding in the accessible data may only slightly lower detection threshold, with modest impacts on identifying further factors (green) and overall accuracy. The great bulk of the variance remains in virtue of a large ensemble of small factors which cannot be identified (red). Note that detection threshold tends to have diminishing returns with sample size.

The scientific revolution for doing good?

The foregoing should not be read as general scepticism to using data. The triumphs of evidence-based medicine, although not unalloyed, have been substantial, and there remain considerable gains that remain on the table (e.g. leveraging routine clinical practice). The ‘randomista’ trend in international development is generally one to celebrate, especially (as I understand) it increasingly aims to isolate factors that have credible external validity. The people who run cluster-randomised, stepped-wedge, and other study designs with big units of analysis are not ignorant of their limitations, and can deploy these judiciously.

But it should temper our enthusiasm about how many insights we can glean by getting some data and doing something sciency to it.^[2] The early successes of EA in global health owes a lot to this being one of the easier areas to get crisp, intersubjective and legible answers from a wealth of available data. For many to most other issues, data-driven demonstration of ‘what really works’ will never be possible.

We see that people do better than chance (or better than others) in terms of prediction and strategic judgement. Yet, at least judging by the superforecasters (this writeup by AI impacts is an excellent overview), how they do is much more indirectly data-driven: one may have to weigh between several facially-relevant ‘base rates’, adjusting these rates by factors where the coefficient may be estimated by role in loosely analogous cases, and so forth.^[3] Although this process may be informed by statistical and numerical literacy (e.g. decomposition, ‘fermi-ization’), it seems to me the main action going on ‘under the hood’ is developing a large (and implicit, and mostly illegible) set of gestalts and impressions to determine how to ‘weigh’ relevant data that is nonetheless fairly remote to the question at issue.^[4]

Three final EA takeaways:

1. Most who (e.g.) write up a case study or a small-sample analysis tend to be well aware of the limitations of their work. Nonetheless I think it is worth paying more attention to how these bear on overall value of information before one embarks on these pieces of work. Small nuggets of information may not be worth the time to excavate even when the audience are ideal reasoners. As they aren’t, one risks them (or yourself) over-weighing their value when considering problems which should demand tricky aggregation of a multitude of data sources.

2. There can be good reasons why expert communities in some areas haven’t tried to use data explicitly to answer problems in their field. In these cases, the ‘calling card’ of EA-style analysis of doing this anyway can be less of a disruptive breakthrough and more a stigma of intellectual naivete.

3. In areas where ‘being driven by the data’ isn’t a huge advantage, it can be hard to identify an ‘edge’ that the EA community has. There are other candidates: investigating topics neglected by existing work, better aligned incentives, etc. We should be sceptical of stories which boil down a generalized ‘EA exceptionalism’.

---

1. ↩︎

   Its name escapes me, although arguably including it would risk deductive disclosure. To play it safe I’ve obfuscated some details.

2. ↩︎

   And statistics and study design generally prove hard enough that experts often go wrong. Given the EA community’s general lack of cultural competence in these areas, I think their (generally amateur) efforts at the same have tended to fare worse.

3. ↩︎

   I take as supportive evidence a common feature among superforecasters is they read a lot—not just in areas closely relevant to their forecasts, but more broadly across history, politics, etc.

4. ↩︎

   Something analogous happens in other areas of ‘expert judgement’, whereby experts may not be able to explain why they made a given determination. We know that this implicit expert judgement can be outperformed by simple ‘reasoned rules’. My suspicion, however, is it still performs better than chance (or inexpert judgement) when such rules are not available.

[William_MacAskill]

Thanks Greg—I really enjoyed this post.

I don’t think that this is what you’re saying, but I think if someone drew the lesson from your post that, when reality is underpowered, there’s no point in doing research into the question, that would be a mistake.

When I look at tiny-n sample sizes for important questions (e.g.: “How have new ideas made major changes to the focus of academic economics?” or “Why have social movements collapsed in the past?”), I generally don’t feel at all like I’m trying to get a p<0.05 ; it feels more like hypothesis generation. So when I find out that Kahneman and Tversky spent 5 years honing the article Prospect Theory into a form that could be published in an economics journal, I think “wow, ok, maybe that’s the sort of time investment that we should be thinking of”. Or when I see social movements collapse because of in-fighting (e.g. pre-Copenhagen UK climate movement), or romantic disputes between leaders (e.g. Objectivism), then—insofar as we just want to take all the easy wins to mitigate catastrophic risks to the EA community—I know that this risk is something to think about and focus on for EA.

For these sorts of areas, the right approach seems to be granular qualitative research—trying to really understand in depth what happened in some other circumstance, and then think through what lessons that entail for the circumstance you’re interested in. I think that, as a matter of fact, EA does this quite a lot when relevant. (E.g. Grace on Szilard, or existing EA discussion of previous social movements). So I think this gives us extra reason to push against the idea that “EA-style analysis” = “quant-y RCT-esque analysis” rather than “whatever research methods are most appropriate to the field at hand”. But even on qualitative research I think the “EA mindset” can be quite distinctive—certainly I think, for example, that a Bayesian-heavy approach to historical questions, often addressing counterfactual questions, and looking at those issues that are most interesting from an EA perspective (e.g. how modern-day values would be different if Christianity had never taken off), would be really quite different from almost all existing historical research.

[Gregory Lewis🔸]

Thanks, Will!

I definitely agree we can look at qualitative data for hypothesis generation (after all, n=1 is still an existence proof). But I’d generally recommend breadth-first rather than depth-first if we’re trying to adduce considerations.

For many/​most sorts of policy decisions although we may find a case of X (some factor) --> Y (some desirable outcome), we can probably also find cases of ¬X --> Y and X --> ¬Y. E.g., contrasting with what happened with prospect theory, there are also cases where someone happened on an important breakthrough with much less time/​effort, or where people over-committed to an intellectual dead-end (naturally, partisans of X or ¬X tend to be good at cultivating sets of case-studies which facially support the claim it leads to Y.)

I generally see getting a steer of the correlation of X and Y (so the relative abundance of (¬/​)X --> (¬/​)Y across a broad reference class as more valuable than determining whether in a given case (even one which seems nearby to the problem we’re interested in) X really was playing a causal role in driving Y. Problems of selection are formidable, but I take the problems of external validity to tend even worse (and worse enough to make the former have a better ratio of insight:resources).

Thus I’d be much more interested to see (e.g.) a wide survey of cases which suggests movements prone to in-fighting tend to be less successful than an in depth look of how in-fighting caused the destruction of a nearby analogue to the EA community. Ditto the ‘macro’ in macrohistory being at least partly about trying to adduce takeaways across history, as well as trying to divine its big contours.

And although I think work like this is worthwhile to attempt, I think in some instances we may come to learn that reality is so underpowered that there’s essentially no point doing research (e.g. maybe large bits of history are just ultra-chaotic, so all we can ever see is noise).

[Max_Daniel]

I agree with your points, but from my perspective they somewhat miss the mark.

Specifically, your discussion seems to assume that we have a fixed, exogenously given set of propositions or factors X, Y, …, and that our sole task is to establish relations of correlation and causation between them. In this context, I agree on preferring “wide surveys” etc.

However, in fact, doing research also requires the following tasks:

• Identify which factors X, Y, … to consider in the first place.

• Refine the meaning of the considered factors X, Y, … by clarifying their conceptual and hypothesized empirical relationships to other factors.

• Prioritize which of the myriads of possible correlational or causal relationships between the factors X, Y, … to test.

I think that depth can help with these three tasks in ways in which breadth can’t.

For instance, in Will’s example, my guess is that the main value of considering the history of Objectivism does not come from moving my estimate for the strength of the hypothesis “X = romantic involvement between movement leaders → Y = movement collapses”. Rather, the source of value is including “romantic involvement between movement leaders” into the set of factors I’m considering in the first place. Only then am I able to investigate its relation to outcomes of interests, whether by a “wide survey of cases” or otherwise. Moreover, I might only have learned about the potential relevance of “romantic involvement between movement leaders” by looking at some depth into the history of Objectivism. (I know very little about Objectivism, and so don’t know if this is true in this instance; it’s certainly possible that the issue of romantic involvement between Objectivist leaders is so well known that it would be mentioned in any 5-sentence summary one would encounter during a breadth-first process. But it also seems possible that it’s not, and I’m sure I could come up with examples where the interesting factor was buried deeply.)

My model here squares well with your observation that a “common feature among superforecasters is they read a lot”, and in fact makes a more specific prediction: I expect that we’d find that superforecasters read a fair amount (say, >10% of their total reading) of deep, small-n case studies—for example, historical accounts of a single war, economic policy, or biographies.

[My guess is that my comment is largely just restating Will’s points from his above comment in other words.]

(FWIW, I think some generators of my overall model here are:

• Frequently experiencing disagreements I have with others, especially around AI timelines and takeoff scenarios, as noticing a thought like “Uh… I just think your overall model of the world lacks depth and detail.” rather than “Wait, I’ve read about 50 similar cases, and only 10 of them are consistent with your claim”.

• Semantic holism, or at least some of the arguments usually given in its favor.

• Some intuitive and fuzzy sense that, in the terminology of this Julia Galef post, being a “Hayekian” has worked better for me than being a “Planner”, including for making epistemic progress.

• Some intuitive and fuzzy sense of what I’ve gotten out of “deep” versus “broad” reading. E.g. my sense is that reading Robert Caro’s monumental, >1,300-page biography of New York city planner Robert Moses has had a significant impact on my model of how individuals can attain political power, albeit by adding a bunch of detail and drawing my attention to factors I previously wouldn’t have considered rather than by providing evidence for any particular hypothesis.)

[Davidmanheim]

Very much agree with the key points, which are related to what I wrote here.

My unsatisfying conclusion was that there are three approaches when facing an “appropriately underpowered” question:

1. Don’t try to answer these questions empirically, use other approaches.
   If data cannot resolve the problem to the customary “standard” of p<0.05, then use qualitative approaches or theory driven methods instead.

2. Estimate the effect and show that it is statistically non-significant.
   This will presumably be interpreted as the effect having a small or insignificant practical effect, despite the fact that that isn’t how p-values work.

3. Do a Bayesian analysis with comparisons of different prior beliefs to show how the posterior changes.
   This will not alter the fact that there is too little data to convincingly show an answer, and is difficult to explain. Properly uncertain prior beliefs will show that the answer is still uncertain after accounting for the new data, but will perhaps shift the estimated posterior slightly to the right, and narrow the distribution.

I also strongly agree with Will’s comment that this doesn’t (always) imply that we shouldn’t do such work, it just means that we’re doing qualitative work, which as he suggests, can be valuable in different ways.

[Will Bradshaw]

[NB: I talked briefly to Greg in person about this last weekend, but felt it might be valuable to put this up here anyway for the purpose of public discussion /​ testing my beliefs.]

I have mixed feelings about this.

On the one hand, I really like the framing of reality being underpowered in certain contexts, and I think this post does a good job of explaining why this is often the case. I think the observation that we often have a lot of tacit data about the world that is hard to fit into explicit models but can nevertheless make non-data-driven expert predictions perform better than change is well-made and well-taken.

Nevertheless, I feel that in a great many cases, non-quantitative, intuitive, first-principles-heavy analyses of the world very often fail; that their rate of failure may often be poorly correlated with their apparent compellingness; that non-quantitative experts overestimate the explanatory power of their work at least as much as (and probably more than) more data-driven analysts; and that a shift towards more explicit, quantitative, data-driven approaches is often among the best ways to distinguish real knowledge about the world from the pseudo-knowledge that I think is rampant in many fields of human enquiry.

As an example: I have several friends who are academic historians, and from time to time we’ve talked about cliometrics/​data-driven approaches to history. The general attitude seems to be “yeah, seems cool if you can do it, but just try that in [my period of study]. There’s no way you could build a decent quantitative model with that little data.” While I’ve generally been too tactful to say this to their faces, my response to these sorts of claims has historically been that if the data is too sparse to do meaningful analysis on, it’s probably also too sparse to draw any other conclusions more general than a simple existence proof (“this thing happened once”). Or, more pithily, “if you don’t have enough data to know things, you should just admit it”.

I now suspect this is too strong a stance, but I still think there is some important truth in it. My feeling is that there may well be “good reasons why expert communities in some areas haven’t tried to use data explicitly to answer problems in their field”, but there are also many bad reasons, among the most common of which is that very few people in that field have strong quantitative skills. I suspect that experts in data-poor fields often lack the epistemic modesty or statistical know-how to admit the consequences of that paucity.

[Stefan_Schubert]

Great post; thanks for this, Greg.

Regarding what effect sizes we can observe using “common sense”: this may vary with domain. Historically, people seem to have had better common sense-intuitions about human psychology than about medicine. That made the value of scientific studies greater in the latter domain. (Though I’m less sure of the relative accuracy of contemporary expert intuitions in psychology and medicine.)

Thus the benefits of additional studies in a domain depends both on whether that domain is underpowered, and on how reliable our common sense-intuitions are in that domain.

“Common sense” is obviously infused with insights from past scientific studies. Some findings, like the “System 1-System 2″-distinction, become part of common sense (at least in some social circles). (Obviously this is still more true of experts’ intuitions.) Since this tendency should tend to increase the accuracy of common sense, it further reduces what we should expect to learn from additional data on a specific question.

[Tom_Beggs]

Thanks for the post, Greg!

I think there are a couple of things worth mentioning that may allay some of the “small sample” cases somewhat. It’s true, for example, that we generally never have the ability to randomize countries into treatments, and that there simply aren’t enough countries to really test study designs that have a large number of conditions. However, we have some other options in our pocket for cases such as those. If we don’t really need to stay at the country level, we can do things like zoom in and randomize quite large groups of people within the country of interest. We can also use techniques like multilevel modeling to look at the effects at both levels at the same time. We also have quasi-experimental designs: in smoking cessation campaigns, for instance, researchers have used a pre-post design for each country or region: see what the baseline tobacco use is in that region, introduce your campaign, and see how smoking changes.

Now, this is not a perfect design. The country hasn’t been randomly assigned and there’s no control group. And we have the threats of maturation as well: what if people in that country would have cut back for another reason even without our campaign? That said, when we see a positive result in one country, and then two, and then five, ten, and fifteen different countries, we gain more confidence that it is not simply spurious. We can also compare what’s happening in the campaign countries to somewhat similar countries that did not get the intervention across the same timeframe. This is not a perfect control group. But it can help us to have more confidence in what we’re seeing.

So with this one counter-example, I’m basically arguing that we shouldn’t be thinking “RCT or bust.” Reality is simply too messy. But we still have a large number of tools at our disposal that will give us very good information. It’s not perfect information, but it’s the best we can do. And we can learn a whole lot from it.

(Another point worth making is that we can use meta-analyses to help determine what the true effect size may be across a number of smaller studies. Each study on its own may be under-powered, but if we have even five or ten of them, we can get a much, much better estimate. This approach can also help control for things like regional differences and failures of randomization.)

So as Will mentions, I think we should be working on a case-by-case basis to determine what the strongest possible research design would be in each case. We should also weigh the cost of collecting that best-case evidence against the cost of other possible research designs in order to find the right blend of methodological rigor and real-world practicality.

[AlasdairGives]

I think this is a great post and makes a really important point. Thanks for posting

[Matt_Lerner]

Some very interesting thoughts here. I think your final points are excellent, particularly #2. It does seem that experts in some fields have a hard-won humility about the ability of data to answer the central questions in their fields, and that perhaps we should use this as a sort of prior guideline for distributing future research resources.

I just want to note that I think the focus on sample size here is somewhat misplaced. N = 200 is by no means a crazily small sample size for an RCT, particularly when units are villages, administrative units, etc. As you note, suitably large effect sizes are reliably statistically distinguishable from zero in this context. This is true even with considerably smaller samples—even N = 20! Randomizations even of small samples are relatively unlikely to be unbalanced on confounders, and the p-values yielded by now-common methods like randomization inference express exactly this likelihood. To me—and I mean this exclusively in the context of rigorously designed and executed RCTs—this concern can be addressed by greater attention to the actual size of resulting p-values: our threshold for accepting the non-null finding of a high-variance, small-sample RCT should perhaps be some very much lower value.

It is true that when there is high variance across units, statistically significant effects are necessarily large; this can obviously lead to some misleading results. Your point is well-taken in this context: if, for example, there are only 20 administrative units in country X, and we are able to randomize some educational intervention across units that could plausibly increase graduation rates only by 1%, but the variance in graduation rates across units is 5%, well, we’re unlikely to find anything useful. But it remains statistically possible to do so given a strong enough effect!

[Gregory Lewis🔸]

I think I would stick to my guns on the sample size point, although I think you would agree with it if I had expressed it better in the OP.

I agree with you sample sizes of 200 (or 20, or less) can be good enough depending on the context. My core claim is these contexts do not obtain for lots of EA problems: the units vary a lot, the variance is explained by other factors than the one we’re interested in, and the variance explained by the intervention/​factor of interest will be much smaller (i.e. high variance across units, small effect sizes).

[My intuition driving the confounders point is the balancing these doesn’t look feasible if they are sufficiently heavy-tailed (e.g. take all countries starting with A-C, and randomly assign to two arms, these arms will tend to have very large differences in (say) mean GDP), and the implied premise being lots of EA problems will be ones where factors like these are expected to have greater effect than the upper bound on the intervention. I may be off-base.]

[Matt_Lerner]

Thanks for responding. I’ve now reread your post (twice) and I feel comfortable in saying that I twisted myself up reading it the first time around. I don’t think my comment is directly relevant to the point you’re making, and I’ve retracted it. The point is well-taken, and I think it holds up.

[OllieBase]

Writing something brief to ensure this gets into the final stage—I recall reading this post, thinking it captured a very helpful insight and regularly recalling the title when I see claims based on weak data. Thanks Greg!

[gavintaylor]

I read an article about using logic to fill in the gaps around sparse or weak data that reminded me of this post. The article is focused on health science, but I think the idea is relevant to development as well.

https://​​nutritionalrevolution.org/​​2019/​​06/​​30/​​the-case-for-coming-to-conclusions-based-on-weak-evidence/​​

[TomBill]

Thanks for this, gavintaylor!

[Aaron Gertler 🔸]

This post was awarded an EA Forum Prize; see the prize announcement for more details.

My notes on what I liked about the post, from the announcement:

In this post, Lewis makes a powerful argument that we ought to pay more attention when we find ourselves working with whatever data we can scrounge from data-poor environments, and consider other ways of developing our judgments and predictions.

Some elements of this post I especially appreciated:

• The author’s points are applicable to work in many different cause areas, and he explicitly points out ways in which they are more or less applicable depending on the problem at hand.

• He opens with a memorable story before making his general points (I expect that this practice will often make Forum posts more memorable, and thus more likely to be applied when they matter).

• Rather than simply identifying a problem, he points out ways in which we might be able to overcome it, including a section with “final EA takeaways”; I love to see posts that, when relevant, end with a set of actionable suggestions.

[gavintaylor]

I recently stumbled onto this article supporting the use of both serendipitous and planned case studies.

https://​​onlinelibrary.wiley.com/​​doi/​​abs/​​10.1046/​​j.1365-2753.1998.00011.x

This is related to clinical practice, but again the ideas may be relevant to development. The authors note that case-studies are particularly useful to clinicians who are might be in a good position to look for patients fitting into a specific population during their routine practice—I wonder if the same concept could be applied to field staff in development projects. For instance, developmental ‘case-studies’ probably won’t generate generalizable results, but they could be helpful in tailoring an RCT validated intervention to a specific population.

[jtm]

Thanks for a great post, Greg! Loved this quote:

But it should temper our enthusiasm about how many insights we can glean by getting some data and doing something sciency to it.

-Joshua