Summary of Cartwright and Hardie’s “Evidence-based Policy: A practical guide to doing it better”

Summary

RCTs are good for pro­duc­ing claims like “It worked there.”. But what we re­ally care about is “Will it work here?”. The stan­dard an­swer to this ques­tion is ex­ter­nal val­idity, but EBP dis­ap­proves of this an­swer for a va­ri­ety of rea­sons. In­stead, they pro­pose that we an­swer the ques­tion of gen­er­al­iza­tion by think­ing about causal prin­ci­ples, causal roles and sup­port fac­tors. To find sup­port fac­tors and the right causal prin­ci­ple, we must en­gage in, re­spec­tively, hori­zon­tal search and ver­ti­cal search up and down the lad­der of ab­strac­tion.

(This origi­nally ap­peared on my blog. There are some niceties (al­ter­na­tively: an­noy­ances) there that are miss­ing here.)

Vignette

The Tamil Nadu In­te­grated Nutri­tion Pro­ject (TINP) was a pro­gram to re­duce child malnu­tri­tion among the ru­ral poor of In­dia’s Tamil Nadu State. Core el­e­ments of the pro­gram were: nu­tri­tion coun­sel­ing for preg­nant moth­ers and sup­ple­men­tary food for the most de­prived young chil­dren. TINP suc­ceeded in re­duc­ing malnu­tri­tion sub­stan­tially (Weav­ing 1995). See­ing this suc­cess, poli­cy­mak­ers in Bangladesh launched the Bangladesh In­te­grated Nutri­tion Pro­ject (BINP) mod­eled on TINP. Un­for­tu­nately, six years later, child­hood malnu­tri­tion in Bangladesh con­tinued un­abated (Weav­ing 1995).

Why did BINP fail where TINP suc­ceeded?[1] One of the main prob­lems was that, while moth­ers did in­deed learn about child­hood nu­tri­tion, moth­ers of­ten weren’t the pri­mary de­ci­sion mak­ers in mat­ters of nu­tri­tion. In ru­ral Bangladesh, men typ­i­cally do the shop­ping. And if a mother lived with her mother-in-law, that mother-in-law was the fi­nal au­thor­ity on is­sues within the women’s do­main. Be­cause the de­ci­sion-mak­ers hadn’t re­ceived BINP’s nu­tri­tion coun­sel­ing, they of­ten used sup­ple­men­tal food from BINP as a sub­sti­tute and re­al­lo­cated other food away from mother and child.

Effectiveness

The prob­lem: Generalization

Even in the ab­sence of that holy grail—the RCT—there’s con­sid­er­able con­fi­dence that TINP worked. But BINP didn’t. Ev­i­dence-Based Policy takes prob­lems like this as it’s cen­tral con­cern. How do we move from “It worked there.”—effi­cacy—to “It will work here.”—effec­tive­ness?

The stan­dard solu­tion: Ex­ter­nal validity

The stan­dard way of think­ing about this prob­lem is ex­ter­nal val­idity. Briefly, a study is in­ter­nally valid when it pro­vides strong rea­sons to be­lieve its con­clu­sions. A study has ex­ter­nal val­idity when it can be gen­er­al­ized to other con­texts—differ­ent times, places and pop­u­la­tions.

But EBP dis­dains ex­ter­nal val­idity. Claims of ex­ter­nal val­idity usu­ally take a shape like “Study A gives us rea­son to be­lieve that in­ter­ven­tion B—which worked on pop­u­la­tion C at time and place D—will also work on similar (to pop­u­la­tion C) pop­u­la­tion E at similar (to time and place D) time and place F.” But the word “similar” is do­ing all the work here. What does it mean?

“Similar” can’t mean “iden­ti­cal”—then all stud­ies would be pointless be­cause we would never have ex­ter­nal val­idity and could never gen­er­al­ize. But “similar” also shouldn’t be con­strued too per­mis­sively. If you in­sist that the pop­u­la­tion of Man­hat­tan­ites and the pop­u­la­tion of ru­ral Ti­be­tans are “similar” be­cause they both con­sist of hu­mans liv­ing in com­mu­ni­ties with hi­er­ar­chies of es­teem un­der a sys­tem of hege­monic cap­i­tal­ism on planet Earth, you’ll be find your­self per­pet­u­ally sur­prised when your in­ter­ven­tions fail to repli­cate.

Fur­ther­more, similar­ity means rad­i­cally differ­ent things in differ­ent con­texts. If the origi­nal study is about re­duc­ing Alzheimer’s for high risk pop­u­la­tions, similar­ity means biomed­i­cal similar­ity and cer­tain el­derly peo­ple in ru­ral Ti­bet may in fact be more similar to cer­tain el­derly peo­ple in Man­hat­tan than ei­ther sub­pop­u­la­tion is to their neigh­bors. On the other hand, if the study is about the welfare effects of ex­po­sure to per­va­sive ad­ver­tis­ing, ru­ral Ti­bet and Man­hat­tan count as pretty dis­similar.

So “similar” has to mean similar in the right ways and to the right de­gree[2]. The ex­ter­nal val­idity claim then be­comes some­thing like “Study A gives us rea­son to be­lieve that in­ter­ven­tion B—which worked on pop­u­la­tion C at time and place D—will also work on the right pop­u­la­tion E at the right time and place F.” But this is pretty tau­tolog­i­cal. To be a use­ful tool, ex­ter­nal val­idity should trans­form a hard prob­lem into a sim­pler one. But it turns out that, once we un­pack things, it’s hard to know what “similar” means other than “right” and we’re back where we started—we have to rely on out­side knowl­edge to know if we can trans­late “It worked there.” to “It will work here.”.

The Ev­i­dence-based Policy solution

To get a good ar­gu­ment from “it works some­where” to “it will work here” facts about causal prin­ci­ples here and there are needed.

Terms

Causal principle

A causal prin­ci­ple pro­vides a re­li­able, sys­tem­atic con­nec­tion be­tween cause and effect. Be­cause the world is com­pli­cated, causal prin­ci­ples of­ten don’t take on a sim­ple form. In­stead, they are char­ac­ter­ized by In­suffi­cient but Ne­c­es­sary parts of an Un­nec­es­sary but Suffi­cient part (INUS). If I want to reach a lichess.org ELO of 1700, ex­cel­lent knowl­edge of open­ings may nec­es­sary but in­suffi­cient be­cause I would also need at least pass­able mid­dlegame and endgame abil­ity to finish out the game. How­ever, all those nec­es­sary fac­tors (ex­cel­lent open­ing, pass­able mid­dlegame and endgame), even if col­lec­tively suffi­cient, aren’t col­lec­tively nec­es­sary—I could take an en­tirely differ­ent route to 1500 by over­com­ing mediocre open­ing play with ex­cel­lent mid­dlegame play or by learn­ing how to cheat with a chess en­g­ine. Schemat­i­cally[3], this looks like a Boolean for­mula: (A AND B AND C) OR (B AND D) OR (E AND F AND G) or H where A, B, C, etc. can be ei­ther true or false. In or­der for the whole propo­si­tion to be true, one of the dis­juncts in paren­the­ses must be true. That means each dis­junct (e.g. A AND B AND C) cor­re­sponds to an un­nec­es­sary but suffi­cient part. In or­der for any given dis­junct to be true, each of its atoms must be true so each atom (e.g. A) cor­re­sponds to an in­suffi­cient but nec­es­sary part.

Causal factor

EBP calls each one of the atoms in the Boolean for­mula—each fea­ture of the world that the causal prin­ci­ple speci­fies as im­por­tant—a causal fac­tor. So, to go back to our ex­am­ple, ex­cel­lent open­ing play, pass­able mid­dlegame and endgame, abil­ity to cheat with a chess en­g­ine, etc. are all causal fac­tors.

Diagram labeling key terms on chess example

Diagram labeling key terms on Boolean formula example

Causal role and sup­port factors

EBP then goes on to em­pha­size causal roles and sup­port fac­tors—a dis­tinc­tion I ini­tially found some­what con­fus­ing. How­ever, my cur­rent un­der­stand­ing is that they are re­ally both ways of talk­ing about causal fac­tors (atoms). Any given in­ter­ven­tion fo­cuses on one or a few causal fac­tors. EBP de­scribes these causal fac­tors that are the sub­ject of ma­nipu­la­tion as hav­ing a causal role. The causal fac­tors speci­fied by the causal prin­ci­ple that aren’t the fo­cus of the in­ter­ven­tion (but are jointly nec­es­sary with the fo­cused causal fac­tor) are deemed sup­port fac­tors. So whether we talk about a causal fac­tor with the lan­guage “causal role” or with the lan­guage “sup­port fac­tor” de­pends only on the fo­cus of in­ter­ven­tion. We can in­ter­pret “X plays a causal role” as “X is a causal fac­tor ma­nipu­lated by our in­ter­ven­tion” and “Y is a sup­port fac­tor” as “Y is a nec­es­sary causal fac­tor not ma­nipu­lated by our in­ter­ven­tion”.[4]

In our chess ex­am­ple, if I started mem­o­riz­ing open­ings from an open­ing book, I would be aiming to make my open­ings play a pos­i­tive causal role. The other causal fac­tors like pass­able mid­dle and endgame—while con­sid­er­ing this open­ing in­ter­ven­tion—would be sup­port fac­tors. Learn­ing how to cheat with a chess en­g­ine is a causal fac­tor in the over­all causal prin­ci­pal, but it’s not a sup­port fac­tor be­cause it’s not rele­vant to the in­ter­ven­tion we’re fo­cus­ing on. In our Boolean for­mula, if I’m try­ing to change A from ‘false’ to ‘true’, I’m hop­ing to make ‘A’ play a pos­i­tive causal role. While fo­cus­ing on A, I con­sider B and C to be sup­port fac­tors. D, E, F, G and H are causal fac­tors but not sup­port fac­tors for A.

Thesis

Now that we’ve defined the key terms of EBP[5], we can pre­sent the the­sis of EBP. The the­sis is that to feel con­fi­dent that “It will work here.” based on ev­i­dence that “It worked there.” re­quires an ar­gu­ment like this:

Effec­tive­ness argument

  1. The policy worked there (i.e., it played a pos­i­tive causal role in the causal prin­ci­ples that hold there and the sup­port fac­tors nec­es­sary for it to play this pos­i­tive role there were pre­sent for at least some in­di­vi­d­u­als there).

  2. The policy can play the same causal role here as there.

  3. The sup­port fac­tors nec­es­sary for the policy to play a pos­i­tive causal role here are in place for at least some in­di­vi­d­u­als here post-im­ple­men­ta­tion.

Con­clu­sion. The policy will work here.

Mak­ing the effec­tive­ness argument

If we have a effi­ca­cious in­ter­ven­tion (“It worked there.”), how can we fill in the rest of the effec­tive­ness ar­gu­ment above? EBP ad­vo­cates for what they call ver­ti­cal and hori­zon­tal search.

Ver­ti­cal search is about find­ing the right for­mu­la­tion for a causal sub­prin­ci­ple[6]. Any pu­ta­tive causal sub­prin­ci­ple can be stated at differ­ent lev­els of ab­strac­tion. In or­der of in­creas­ing ab­strac­tion, we can have causal sub­prin­ci­ples like:

  1. This ham­mer claw makes it pos­si­ble to ex­tract this nail.

  2. Ham­mer claws make it pos­si­ble to ex­tract nails.

  3. Lev­ers make it pos­si­ble to trans­form a small force ex­erted over a large dis­tance into a large force ex­erted over a small dis­tance.

  4. Sim­ple ma­chines make it pos­si­ble to do things we oth­er­wise couldn’t via me­chan­i­cal ad­van­tage.

Even if all these for­mu­la­tions are true, some are more use­ful than oth­ers. 1 is too spe­cific to be of much use. 2 is use­ful in cases just like ours while 3 pro­vides in­sights and may al­low us to gen­er­al­ize to novel situ­a­tions. 4 may be too ab­stract and re­quires ex­per­tise to turn into prac­ti­cal ad­vice—if you’re strug­gling to pull out a nail and some­one offers the ad­vice that “Sim­ple ma­chines provide me­chan­i­cal ad­van­tage.”, you may not be eter­nally grate­ful for their sage ad­vice.

It’s worth mak­ing ex­plicit here that it won’t always be ob­vi­ous how gen­er­al­iz­able causal sub­prin­ci­ples are. In the vi­gnette at the be­gin­ning, it might have sounded em­i­nently plau­si­ble that the causal sub­prin­ci­ple to be learned from the suc­cess­ful TINP was “If we in­crease the nu­tri­tional knowl­edge of moth­ers, then child­hood malnu­tri­tion will be re­duced.” It’s only af­ter the BINP—de­signed on this prin­ci­ple—failed that it be­came ap­par­ent that the prin­ci­ple is bet­ter phrased as “If we in­crease the nu­tri­tional knowl­edge of those in charge of child­hood nu­tri­tion, then child­hood malnu­tri­tion will be re­duced.” In Tamil Nadu, these two prin­ci­ples are the same. It’s only in Bangladesh where fathers and moth­ers-in-law play a more im­por­tant role in child­hood nu­tri­tion that these two prin­ci­ples di­verge and find­ing the cor­rect level of ab­strac­tion be­comes cru­cial.

So gen­er­ally there’s a ten­sion be­tween mak­ing causal sub­prin­ci­ples con­crete enough to be use­ful and ab­stract enough to be true in new cir­cum­stances. This is why EBP ad­vo­cates care­ful ver­ti­cal search up and down the lad­der of ab­strac­tion.

Hori­zon­tal search is about iden­ti­fy­ing the full set of sup­port fac­tors nec­es­sary for an in­ter­ven­tion to suc­ceed. Be­cause sup­port fac­tors aren’t the tar­get of an in­ter­ven­tion, it’s easy to miss them. But if they’re miss­ing, the in­ter­ven­tion will fail all the same.

For ex­am­ple, in child­hood malnu­tri­tion in­ter­ven­tions like TINP and BINP, care­ful search could un­cover ad­di­tional sup­port fac­tors like:

  • There’s an ad­e­quate amount of food available for pur­chase (if not deal­ing with sub­sis­tence farm­ers)

  • It’s pos­si­ble to con­struct a nu­tri­tious diet with lo­cally available foods

  • The par­a­site load is low enough that a suffi­cient quan­tity of nu­tri­ents can be absorbed

In practice

To its credit, EBP rec­og­nizes that the al­ter­na­tive pro­ce­dure it ad­vo­cates is not a drop-in re­place­ment. Devolu­tion and dis­cre­tion will be cen­tral to the new world or­der. Ten­den­tiously, the book would have us sup­plant a me­chan­i­cal method (RCTs, clear­inghouses, ex­ter­nal val­idity) with a much fuzzier one de­mand­ing more ex­per­tise and dis­cre­tion of poli­cy­mak­ers. But, EBP ar­gues, this is change is nec­es­sary be­cause the me­chan­i­cal method sim­ply isn’t up to the task.

What’s wrong with the ideas of ex­ter­nal val­idity and similar­ity is that they in­vite you to stop think­ing. [...] We do not think that it is pos­si­ble to pro­duce un­am­bigu­ous rules for pre­dict­ing the re­sults of so­cial poli­cies. So, we do not think that we can pro­duce these rules. So in our world, those who make the de­ci­sions will have to de­liber­ate us­ing their judg­ment and dis­cre­tion.

Efficacy

Though the book is mainly about the prob­lem of effec­tive­ness it also has a sec­tion on effi­cacy. In par­tic­u­lar, it talks about causal in­fer­ence us­ing RCTs and al­ter­na­tives.

Ran­dom­ized con­trol­led trials

RCTs are the holy grail for causal in­fer­ence (well, meta-analy­ses and/​or sys­tem­atic re­views of RCTs). EBP pro­poses that this is be­cause RCTs are “self-val­i­dat­ing”. By this, they mean that RCTs are some­thing of a magic causal­ity black box—perform the right rit­u­als pro­ce­dures and out pops a causal claim. There’s no need to have a de­tailed un­der­stand­ing of mechanism, con­text, pop­u­la­tion or any­thing at all in the do­main.

Most of the al­ter­na­tive mechanisms of causal in­fer­ence re­quire real sub­ject mat­ter ex­per­tise. Some of these al­ter­na­tives in­clude:

Con­trol­ling for con­founds : You must know the full set of con­founds to con­trol for them. This is hard (Gor­don 2019).

In­stru­men­tal vari­ables : You must know that there’s no other causal link­age be­tween your in­stru­ment and your out­come vari­able.

Causal graph : You must know the full[7] causal struc­ture.

So what sets RCTs apart is not their abil­ity to make causal claims—it’s that they can do so with­out ex­pert do­main knowl­edge. But EBP ar­gues that ex­pert do­main knowl­edge will be re­quired any­way to make claims of effec­tive­ness. If ex­pert do­main knowl­edge is re­quired to as­sess effec­tive­ness, we might as well de­sanc­tify RCTs and al­low other tech­niques when as­sess­ing effi­cacy. (At least, I think this is a rea­son­able con­nec­tion to make. EBP doesn’t make this con­nec­tion quite as ex­plicit.)

Omitted

Th­ese are the things I’ve left out of the sum­mary (and re­mem­bered to note here—I don’t guaran­tee ex­haus­tive­ness, blog name to the con­trary).

What is an ar­gu­ment?

There’s a whole sec­tion de­scribing war­rants, ev­i­dence, the struc­ture of ar­gu­ments, etc. I didn’t need to read this and I as­sume that’s true for most other read­ers of the book or this sum­mary.

Causal cakes

One of the cen­tral ex­plana­tory metaphors in EBP is a “causal cake”. I omit­ted this be­cause I found it supremely un­helpful. There’s no real rea­son to read it, but here is a list of my com­plaints:

  • The book ad­mits “The cake is just a pic­ture of the list [of causal fac­tors].”.

  • The book says that it prefers a cake-and-in­gre­di­ent metaphor to a pie-and-slice metaphor be­cause each of the in­gre­di­ents is in­te­gral to a cake while you still have a lot of pie left if you take out one slice. All the images of causal cakes have them sliced up into wedges in clear con­tra­dic­tion of the afore­men­tioned ra­tio­nale.

  • Things be­come a bit dizzy­ing when EBP starts talk­ing about “ac­ti­vat­ing” one of the mul­ti­ple cakes that make up a causal prin­ci­ple. The metaphor feels a bit thin—like frost­ing but­ter scraped over too much cake bread.

How to think

A whole chap­ter of EBP is de­voted to what I’d say are fairly gen­eral tools for think­ing. It lists four strate­gies in­tended to help in con­struct­ing a ro­bust effec­tive­ness ar­gu­ment. How­ever, since the chap­ter is not 1) in­te­gral to the rest of the book, 2) an ex­haus­tive list­ing of all such think­ing tools, 3) es­pe­cially rev­e­la­tory, I have not cov­ered it in depth here. The four strate­gies men­tioned are:

  • pre-mortem

  • think­ing step-by-step and think­ing backwards

  • “If it is to work, by what means will it do so?”

  • quick exit trees (a var­i­ant of de­ci­sion trees)

Ev­i­dence-rank­ing schemes

EBP also has a full sec­tion on ev­i­dence-rank­ing schemes and policy clear­inghouses like the What Works Net­work. While I think these are in­ter­est­ing and valuable re­sources, the dis­cus­sion doesn’t seem es­sen­tial to the EBP’s core the­sis.

Fidelity

The sec­tion on fidelity when im­ple­ment­ing an effi­ca­cious in­ter­ven­tion doesn’t seem to cover much that wasn’t already dis­cussed when talk­ing about ver­ti­cal search and ex­ter­nal val­idity.

Works cited

Gor­don, Brett R, Flo­rian Zet­telmeyer, Neha Bhar­gava, and Dan Chap­sky. 2019. “A Com­par­i­son of Ap­proaches to Ad­ver­tis­ing Mea­sure­ment: Ev­i­dence from Big Field Ex­per­i­ments at Face­book.” Mar­ket­ing Science 38 (2). INFORMS: 193–225.

Rohrer, Ju­lia M. 2018. “Think­ing Clearly About Cor­re­la­tions and Cau­sa­tion: Graph­i­cal Causal Models for Ob­ser­va­tional Data.” Ad­vances in Meth­ods and Prac­tices in Psy­cholog­i­cal Science 1 (1). SAGE Publi­ca­tions Sage CA: Los An­ge­les, CA: 27–42.

Weav­ing, Rachel V. 1995. “Tamil Nadu and Child Nutri­tion : A New Assess­ment.” World Bank. http://​​doc­u­ments.wor­ld­bank.org/​​cu­rated/​​en/​​841071468258312355/​​pdf/​​28513.pdf.


  1. I get the im­pres­sion this is very much a sim­plified ren­di­tion of these in­ter­ven­tions for illus­tra­tive pur­poses. EBP even hints that there’s a great deal more con­tro­versy and com­plex­ity than they ini­tially pre­sent. ↩︎

  2. Also, ex­ter­nal val­idity’s de­mand for “similar” is too con­ser­va­tive. It’s pos­si­ble to imag­ine con­texts that differ in some in­ter­ven­tion-rele­vant way where you’d still be happy to re­peat the in­ter­ven­tion. You prob­a­bly shouldn’t say: “Oh, sorry, can’t do it. Direct cash trans­fers of $50 a month are only helpful for those with an an­nual in­come of $2,000. They wouldn’t work for those with an an­nual in­come of $1,000. $1,000 isn’t similar to $2,000—it’s only half as much.” Ul­ti­mately, what we want is not a “similar” con­text but a con­text which is at least as fa­vor­able for the in­ter­ven­tion. ↩︎

  3. Of course, al­most all real world causal prin­ci­ples can’t be ex­pressed as Boolean for­mu­las this sim­ple. I’ve cho­sen a sim­plified ex­am­ple for ped­a­gog­i­cal pur­poses and hope you can see or trust that similar dy­nam­ics arise with more com­pli­cated causal prin­ci­ples. ↩︎

  4. Per­haps my ini­tial con­fu­sion now makes sense to you. Sup­port fac­tors also play a causal role by the plain mean­ing of “causal role”—if they were coun­ter­fac­tu­ally ab­sent, a differ­ent effect would re­sult. EBP just seems to have im­bued the phrase “causal role” with a spe­cial (and, in my opinion, con­fus­ing) mean­ing. ↩︎

  5. An­noy­ingly, EBP offers no suc­cinct, up­front defi­ni­tion of “causal role” that I can find (Believe me, I’ve looked). ↩︎

  6. EBP uses the term “causal prin­ci­ple” in two differ­ent ways. The first is the way we’ve already out­lined—a speci­fi­ca­tion of the full (pos­si­bly dis­junc­tive) causal struc­ture re­spon­si­ble for an effect. The sec­ond us­age of “causal prin­ci­ple” is to de­scribe the re­la­tion­ship be­tween one set of jointly suffi­cient causes and their effect. To avoid con­fu­sion, I use the term “causal sub­prin­ci­ple” for this sec­ond con­cept.

    To be more ex­plicit, a causal prin­ci­ple gov­erns causal fac­tors A through H in our Boolean for­mula. A causal sub­prin­ci­ple gov­erns any of the dis­juncts like (A AND B AND C). A causal prin­ci­ple gov­erns all the causal fac­tors about reach­ing an ELO of 1500 while a causal sub­prin­ci­ple gov­erns all the causal fac­tors about reach­ing 1500 via im­prov­ing knowl­edge of open­ings with a pass­able mid­dle and endgame. ↩︎

  7. Where “full” means “any ad­di­tional vari­able that ei­ther di­rectly or in­di­rectly causally af­fects at least two vari­ables already in­cluded in the DAG should be in­cluded” (Rohrer 2018). ↩︎

No comments.