Lant Pritchett’s “smell test”: is your impact evaluation asking questions that matter?

Link post

Lant Pritch­ett is a de­vel­op­ment economist and mas­ter of back­handed com­pli­ments. I’m always look­ing for new frames to use when I think about causes, and his “smell test” fits the bill.

In short: Think about coun­tries that have been suc­cess­ful, eco­nom­i­cally. What are some things these coun­tries do? And does the de­vel­op­ment pro­gram you fa­vor ac­tu­ally make de­vel­op­ing coun­tries more similar to de­vel­oped coun­tries?

This is, of course, not a fully gen­eral ar­gu­ment against RCT-driven in­ter­ven­tions in the GiveWell mold. But I found it an in­ter­est­ing sup­ple­ment to the Fo­rum’s re­cent de­bate around eco­nomic growth re­search.


In 2006 I was in West Ben­gal with a World Bank team and was ask­ing ques­tions of a group of women about a “liveli­hoods” pro­gram that built and fi­nanced women’s self-help groups as a means of in­creas­ing women’s pro­duc­tivity and in­comes. After ask­ing them ques­tions for an hour or so I asked them if they had any ques­tions for me or the team—af­ter all, they had been so gra­cious to an­swer our nosey ques­tions we would be rude to not al­low them to ask us any­thing they wanted to know. After an awk­ward silence, one woman said “You all are from coun­tries that are much richer and do­ing much bet­ter than our coun­try so your coun­try’s women’s self-help groups must also be much bet­ter, tell us how women’s self-help groups work in your coun­try.”

I’m Amer­i­can. Along on the team was a Ger­man woman, an­other man from New Zealand, and a woman from the UK. We all looked at each other blankly as none of us had any idea whether there even were at any time in our coun­tries’ his­tory such a thing as “women’s self-help groups” in our coun­tries (much less gov­ern­ment pro­gram for pro­mot­ing them). We also had no idea how to ex­plain that, yes, all of our coun­tries are now de­vel­oped but no, all of our coun­tries did this with­out a ma­jor role from women’s self-help groups at any time (or if there were a role we de­vel­op­ment ex­perts were col­lec­tively ig­no­rant of it), but yes, women’s self-help groups pro­mote de­vel­op­ment.

My four-fold “smell test” for what is im­por­tant to development

I have a four-fold crite­ria for whether some­thing is po­ten­tially an im­por­tant de­ter­mi­nant of de­vel­op­ment, or more nar­rowly, just eco­nomic growth, and I am happy if “thing X” that I am propos­ing is “good for de­vel­op­ment” can satisfy all four (and then can move on from these sim­ple facts about po­ten­tial im­por­tance to tease out com­pli­cated ques­tions of prox­i­mal, dis­tal, and re­verse causal­ity).

One, coun­tries differ in their level of de­vel­op­ment by an or­der of mag­ni­tude. Coun­tries that are de­vel­oped should have more of thing X than coun­tries that aren’t. If Den­mark and Canada don’t have more of thing X than Mali or Nepal I am kind of sus­pi­cious.

Two, since now de­vel­oped coun­tries are al­most an or­der of mag­ni­tude more de­vel­oped than they were in 1870 I am happy if there is more of thing X in de­vel­oped coun­tries now than 140 years ago. If Ger­many and Ja­pan don’t have more of thing X (or at least the same amount) than they did in 1870 I am kind of sus­pi­cious.

Three, since over the pe­riod since 1950 some coun­tries have seen their de­vel­op­ment im­prove in­cred­ibly rapidly and oth­ers have seen al­most no progress I am happy if thing X is more preva­lent in rapid de­vel­op­ment suc­cesses than in de­vel­op­ment failures. If Korea and Taiwan don’t have more of thing X than Haiti and Nige­ria then I am kind of sus­pi­cious.

Four, since coun­tries change in their pace of de­vel­op­ment (and this is par­tic­u­larly true of eco­nomic growth, less so of hu­man de­vel­op­ment in­di­ca­tors) dra­mat­i­cally over time, I am happy if there is more of thing X in a coun­try in pe­ri­ods when de­vel­op­ment progress is rapid than in pe­ri­ods when de­vel­op­ment progress is slow. If China doesn’t have more of thing X af­ter 1978 than be­fore 1978 (as growth ac­cel­er­ated by 3.3 ppa) or if Cote d’Ivoire doesn’t have less of thing X af­ter 1978 than be­fore 1978 (as growth de­cel­er­ated by 3.7 ppa) then I am kind of sus­pi­cious.

Th­ese four of course don’t re­solve the de­bates or de­tails about the re­spec­tive roles of macroe­co­nomic man­age­ment, policy ap­proaches to ex­ter­nal mar­kets (e.g. trade, cap­i­tal, ideas), se­cu­rity of prop­erty rights, in­fras­truc­ture, ac­cu­mu­la­tion of hu­man cap­i­tal, tech­nolog­i­cal change, ca­pa­bil­ity in the product space, or “in­sti­tu­tions” (or, more deeply, what is cause and what is con­se­quence amongst these el­e­ments them­selves). But nearly all con­tenders in de­bates about eco­nomic growth or de­vel­op­ment more broadly pass 2 or 3—and some­times all four—of these “smell tests” of at least po­ten­tially be­ing an im­por­tant de­ter­mi­nant.

Are in­ter­ven­tions be­ing eval­u­ated im­por­tant for de­vel­op­ment?

Eva Vi­valt (2014) has writ­ten a pa­per that is so good it de­serves sev­eral blog posts to dis­cuss its in­ter­est­ing find­ings. She and her team have asked the im­por­tant ques­tion about the gen­er­al­iz­abil­ity of the find­ings from “rigor­ous im­pact eval­u­a­tions” (in­clud­ing RCTs). In or­der to do her team sur­veyed 621 pa­pers (not all of which could be used in her anal­y­sis). That is an im­pres­sive num­ber. Sup­pose typ­i­cal pro­duc­tivity of an aca­demic or re­search economist is three origi­nal com­pleted pa­pers per year. Then 621 pa­pers is 207 per­son/​years of re­search. Alter­na­tively think of in­clu­sive cost (op­por­tu­nity cost of re­searcher time plus money costs) per im­pact eval­u­a­tion.

I would en­courage you to fill in this table with the 20 pro­grams on which Vi­valt (2014) finds enough rigor­ous im­pact eval­u­a­tions for com­par­i­son.

After the table is filled in (don’t cheat or I’ll send a nudge mo­bile phone re­minder to al­ter your be­hav­ior) ask your­self: why has much of the best and bright­est tal­ent of a gen­er­a­tion of de­vel­op­ment economists been de­voted to pro­duc­ing rigor­ous im­pact eval­u­a­tions about these 20 top­ics?