Rachel Glennerster is the Chief Economist of DFID, the UK’s ministry for coordinating international development. In this conversation with Nathan Labenz, she discusses the most important lessons she’s learned about development and what it really means for a study’s result to “generalize”.
A transcript of the conversation between Rachel and Nathan, which we have lightly edited for clarity, is below. You can also watch this talk on YouTube, or read its transcript on effectivealtruism.org.
The Conversation
Nathan: Thank you for being here. I’m very excited to have this conversation. I’m going to do my best Rob Wiblin impression and start with what is his traditional first question, which is, what are you working on at the moment and why do you think it’s especially important?
Rachel: I’m doing lots of different work at DFID, but let me talk a little bit about some research work I’m doing, which is evaluating a mass media campaign, a radio campaign, run by Development Media International, which is an NGO here in the UK. They’re doing a family planning program on radio in Burkina Faso. I think it’s really important to look at radio and mass media, because it’s a very cheap way to reach a large number of people and you can make sure that the message is accurate and consistent. The problem is that it’s very hard to evaluate radio for exactly those reasons, because one radio program, which is millions of people at the same time, it’s very hard to randomize.
Now, it happens that Burkina Faso is kind of one of the few places in the world where one can evaluate this effectively. It’s also true that at DFID, we’re worrying a lot about how it’s an area that hasn’t had a lot of UK interest until recently and also, family planning is a hugely important issue, because if you get the demographic transition right, it can have incredible benefits for women, for economic development, and for the health of children.
--
Nathan: Why is it that it’s more testable in Burkina Faso, is that a language group issue?
Rachel: It’s a complex series of factors, that means that you have a lot of different radio stations across Burkina Faso, which indeed have different languages, and so there’s less spillover. So you can randomize at the level of the radio station. You also have people who are so poor that they can’t afford radios. We’re randomly handing out radios to women who don’t have radios. We have kind of two levels of randomization, both at the radio station and within a radio area. Some women already have radios, some women are given radios, and some aren’t. It’s kind of a conglomeration of things that allow you to be able to test.
--
Nathan: Well, we’ll probably come back to that in a minute, when we get to the section on randomized control trials in general. We’re going to try to cover a variety of areas here, including your views on careers and policy, advice for audience members who are interested in policy careers, and the role of evidence in general in aid work. But I thought we might also go back to the beginning of your career, and you could give us a sketch of how your career has developed. You studied economics as an undergraduate, and then began your career at the Treasury, and then it’s taken a lot of different turns from there. Tell us all about that.
Rachel: I think there’s some benefit in telling my story. It sounds very self centered to talk about my story, but I think it’s useful to show that careers don’t always have to be one-directional, and they can take many turns. I think it’s interesting to be doing this in an 80,000 hours podcast, where people are thinking really seriously about what’s the best thing to do, because in some ways, my career has been a bit random. But I think it’s both, right? We have to think seriously about what our next step is and also realize that stuff happens in life.
As an undergraduate, I was very interested in development economics, and in thinking about how I could contribute to addressing the issues of global poverty. And I spent a summer traveling around Kenya, talking to lots of people in the aid sector, and got really depressed and thought: A, lots of things don’t seem to be working, and B, what do I know? What honestly could I bring to these complicated issues? I’m 21, I don’t know anything about Kenya. I decided instead to go into the Government Economic Service, which is a special training position within the government that trains you to use analytical skills to address policy issues.
I did that at the Treasury, I worked on domestic policy, I worked on reform of the health service, trade policy, monetary policy, all sorts of different policies. And I got fantastic training in how to use analytical skills to help us think more rationally about policy decisions. And then I went to get more education. I went to Harvard, and met my future husband, and my life got thrown up in the air because I was convinced I was going to stay in the Government Economics Service for the rest of my life. I loved it.
Michael, my future husband, was American. After failing to persuade him to move to the UK, I had to look for something to do in the US. Then I went back to my old love of development, and having, I think, gained some skills, I worked to represent the UK at the IMF and World Bank. I learned a lot about international institutions and worked on financial regulation in Russia as a way to move to Boston, where Michael was. That all went up in smoke, because a big scandal hit Harvard and its work in Russia. I went back to the IMF and I realized I needed more economics knowledge to really make a difference.
I went back and did a PhD, but I did it part-time. I was constantly on the border between academia and policy work. I think that’s a really interesting and important nexus, where, when I was within government, I was taking academic work and explaining and translating it into policy needs: How can we use this academic work to do our policy work better? Then I moved to J-PAL, where I was on the other side of the fence. I was in academia, but I was helping academia translate what it was doing into the policy world.
I helped found J-PAL MIT, which promoted the use of randomized trials. I think the key thing that J-PAL did was to work really hard on this policy/academic nexus and say: “How do we make sure that this academic work is being translated into policy change?” That’s the thing that’s been common throughout my work. And so yet again, I’ve jumped back onto the other side of the bridge, and I’m now working at the Department for International Development back in the Government Economic Service that I started in all that time ago, as chief economist. And again, I’m helping to bring academic insights and research insights into decisions in government. There’s so much need for that translation across this border.
--
Nathan: I think that’s interesting, as you said, for an 80,000 hours podcast audience that’s thinking a lot about their careers. But also, as we sit here at EA Global London, I think a lot of attendees are trying to figure out how they can shape their own careers to have the most impact. It strikes me that one of the central questions people are asking themselves often is, should I go and do direct work now? Or should I focus on upscaling myself and becoming a more powerful person in whatever domain? It seems like you have climbed those two ladders at different points in your career, is that a model that you would recommend to others?
Rachel: First of all, let me just say that upscaling is not only in academia. You don’t just do it in graduate school. I think the place where I learned the most, as I said, was probably those first two years in the Government Economic Service. I was never trained to write until someone literally went line by line through my work in government to help me write more clearly, and that’s been incredibly important in conveying ideas. You learn technical things when you go to college and graduate school, and later you learn other skills, like how to influence people, how to get things done, and how to run effective organizations.
You need all of those skills if you want to make a difference in the world. Different people will prefer skills that are more on the technical side or more on the “influencing people” side. Think about where your best skills are. I wrote a blog post a while ago about whether economists should go into policy or academia, and I listed the skills that you need in those different areas. I didn’t say one is better than the other or one is harder than the other; it’s just different. Think about building on your natural skills and acquiring new skills, both within institutions and through more formal training.
I do think that people often underestimate the number of skills you learn from being in an effective organization. A lot of people want to start their own organization, and that’s great. But I have seen people start organizations without having really experienced an effective organization. When I arrived at J-PAL, we were three people, and when I left there were 350 people. My deputy and I had both come from the civil service. We had really strong views about how you run an effective organization, and I think that was absolutely critical in building J-PAL.
--
Nathan: Yeah, I think that’s fascinating and very apt. My background is in the software startup world, which is quite different. But often young people will ask me, how can they start their own company or whatever. And I always say, “Work for a great company, before you even think about trying to start one of your own, because there’s a lot that you don’t want to have to reinvent from scratch or try to derive from first principles.” You mentioned randomized controlled trials, and that’s definitely a big subject in your work. Before we dive into the current state of that debate, here’s another historical question.
You started out in an era when effectiveness in aid was much less of a concern. People cared more about just giving and feeling good about it and hoping for the best, or maybe not even really worrying about what happened downstream. How would you sketch the kind of intellectual trajectory of aid and development work over the last three decades?
Rachel: Great. Big questions. I think it’s worth looking at two different trajectories. There’s the trajectory of the aid sector, the development sector, and then there’s the trajectory of academic research on development. and those are a bit different. One of the nice things about RCTs is that they’ve brought those two really closely together. I think the trajectory of the aid sector has been one in which you say, there was not enough emphasis in my view about understanding really rigorously what works. There were a lot of different theories and views about what we should be doing and you see these big swings and fads—for example, the view that development was mainly about investment in physical capital.
Under this theory, the reason we needed aid was that countries didn’t have enough investment, which meant that we should be building stuff. Places like India had five-year plans, and they built steel factories, and then there was a big swing toward a new popular view: “No, we need to worry about human capital and not just steel.” But people aren’t benefiting from human capital programs like job training; they’re still malnourished, they’re not learning. And you saw these big swings and interest about what we should be doing for development, but not a lot of it was especially data-driven. There has been a really big change in the last 10 years or so, especially within the DFID and within the World Bank, to really seriously think about the evidence behind the decisions that we’re making.
One of the reasons I moved to DFID is it has been one of the agencies that I think has changed the most to constantly be examining itself. At the moment, we’re going through the process of looking at what we’re doing and saying, “Is it really evidence-based? What does the new research say? What should we stop doing, because the new evidence is saying we shouldn’t be doing it?” And that’s an important sea change in the aid world.
One of the biggest changes in research, in academia and economics, is just that there are now a lot more people thinking about the developing world, and that’s great. A lot of them are doing RCTs, but if you look at the data, actually, there are just as many non-RCT workers as there were before. Within economics, development used to be a bit off the side; most economists didn’t think about it. But people have realized that the questions in development are actually very similar to the questions in other bits of economics, and we should be learning from each other.
--
Nathan: A lot of behavioral economics lessons came from development, and they are now being taken up and used and learned from in rich countries. I think that it’s been really important to see lessons going in both directions; development has helped us understand that there are a lot of similarities between people, and there’s a lot we can learn from each other.
Moving to RCTs in general, and the state of debate around how much we should rely on them: You mentioned that it’s kind of a 50⁄50 split right now, in today’s work. Do you think that’s an inappropriate split? Do you think that it should be all RCTs? What do you think is the right balance as we try to figure out what is obviously a very complicated world?
Rachel: I think it’s really important to say that all of us who have worked on randomized trials have never suggested that this is the only methodology that you should use. Sometimes it’s held up as a straw person that we go around saying: “This is the only methodology.” But nobody who’s done RCTs has ever thought that they are the only the right approach. I think the right way to see things is that you have a toolbox of ways to answer questions. The right tool depends on the question that you’re asking.
I think we need good descriptive work to understand what the problems are. A lot of development programs fail because they’re trying to solve a problem that doesn’t exist. They’re just solving their own problem. The first really important thing you’ve got to do is to understand what the issue is in any given area. If we’re worried about girls not going to school because of menstruation, let’s start by finding out whether they actually don’t go to school more when they’re menstruating! That’s a really basic, obvious thing, but we actually need more of that kind of thinking; understanding the problem is a really important first step.
RCTs are useful for answering really specific questions, but I think the best RCTs are the ones that test a theory. They test something that’s more generalizable than just “does this program work?” They ask a question about human beings.
Here’s an example. I did a project looking at how to improve immunization rates in India. Only 3% of kids in a certain part of India were getting fully immunized. And given that immunization is one of the most effective things that you could do, so that rate is just appallingly low. There were a number of theories about why that could be. A lot of people said, “Well, people here don’t trust the doctors.” That is, not doctors, because you rarely get doctors in rural India, but nurses and clinics. They don’t trust their formal health system.
There were also other theories. The clinics were often closed, so is that the problem? Is nurse absenteeism the problem?
We had read all this behavioral economics literature, and behavioral economics tells us that people will be happy to get their kids immunized, but they’d rather do it tomorrow. We set up an RCT with one arm that provided just good service, making sure that without fail, there was someone to immunize your child when you reached the clinic. The other arm did the same thing, but also provided a small incentive.
Yes, we were testing a program, but we were also asking a more fundamental question, which is “why don’t people get their kids immunized?” And what we saw in the data is that a lot of people got their kid immunized once, but failed to keep coming back until the end of the immunization schedule. Fixing the supply problem increased the number of people getting the first shot and the second shot. But it failed to fix this persistence problem. However, the incentive worked to help people persist until the end.
By the way, that project was completely impossible to scale. Our “incentive” program involved handing out lentils in the middle of Rajasthan. Nobody showed up, which shows how good economists are at designing logistics. It was a disaster; we learned a lot, but you would never want to actually do a program like this.
A colleague of mine did something similar with another program in Rajasthan, where we ended up improving teachers’ attendance by putting cameras into their classrooms. Again, the logistics were a nightmare, but the project tested a theory. And so once you have the test done, you can think about other questions, like “how do we implement this at scale?”
--
Nathan: I noticed that the illustrations on your website show you in the field as well as in a more academic setting, which is a clear signal that you believe in actually going to places and handing out the lentils (as it were). It seems like you have an on-the-ground, intuitive, firsthand understanding which allows you to generate a number of theories, and then you can test one of those theories and find an effect and then think about, “Okay, now how can we scale this up in such a way that we can simplify the logistics and not have to handle all the lentils personally?”
Rachel: That seems pretty sound. I think people worry about how that then transfers to another context. Could we take that result and say “the incentive will work for persistence”, and take that to another culture or another place and expect similar results? Speak to that a little bit: How can you take your experience when you’re pretty confident that it works in one place and try to generalize it to other places?
I think the discussion around generalizability is really confused. Let me try and explain how I think about it.
People often ask, “Does this result generalize?” To which I respond, “What result? What aspect of this are you asking about?”
I think there are three different main ways in which you need to think about whether something generalizes. The first: Is the problem the same in other places? An intervention won’t generalize to a place that doesn’t have the same problem. In the case of India, the problem was that you have people getting the first immunization, but not persisting to the end of the schedule.
And that’s really easy, that’s something that you can test. You can look at data and say: “Well, in this country, people are willing to get the first immunization, but they don’t manage to persist to the end. And in other places, they’re not getting the first immunization at all.” The second question: “If people have the same problem, do they respond to solutions in a similar way? Does a small incentive help people in a different context to persist in something that they want to do, but don’t manage to do consistently?”
In this case, there is tons of evidence that incentives work well. It’s not all from lentils and vaccines; it’s from across many different kinds of programs. The general finding that small nudges are useful to fight procrastination is very generalizable. Another example: If you charge people for preventative health care, even a nominal amount, you will see a big decline in takeup; that has been consistently found for different kinds of preventative care in different countries.
The last question: “Can I implement something similar elsewhere?” And that is not something that automatically generalizes. We worked with an extremely good NGO in Rajasthan. Their people would absolutely turn up. They could get the lentils there. The lentils were not stolen. None of this would necessarily be the case, if we worked with another organization or government. Also, we wouldn’t want to use lentils if we did the program in New York, right? Lentils wouldn’t be seen as a particularly exciting incentive in other places.
People confuse these three things and lump them into “this has generalized”. Well, basic fundamental principles of human behavior generalize, but logistics don’t. So that is where you need to spend a lot of time understanding the local context and how to run logistics locally.
Coming back to the beginning of your question: I spend less time on the ground now than I would like. But in the past, I spent a lot of time, which is absolutely critical to understanding local context for anyone thinking about working in development.
Even more importantly, you need to partner with people who really understand the area. Nearly, always, when I work in these countries, I am working with a partner who has worked there for many years. And then we build a relationship where they trust me because I’m willing to spend time and effort to understand their problems, and I trust them because I’ve spent enough time to know that they really know the local context. I would also say that it’s a lot easier to spend time on the ground when you’re younger and less established, so get out there!
--
Nathan: How long do you typically spend in a place? I’m reminded of… I don’t know if it’s a joke, or a proverb, but the idea that a guy who goes to China for a week thinks he knows a little bit about China. After a month, he thinks he knows a lot about China. And then, after a year, he realizes he knows nothing about China. How long do you feel you have to be in a place to build the right relationships and have the level of trust that you need to actually be effective?
Rachel: Again, part of it is based on partnerships. I don’t always have to be there all the time if I’m building relationships and talking to people regularly. I haven’t done what I would advise other people to do, because I didn’t go into development initially. I didn’t spend a year or two on the ground as a student in my early 20s, which I wish I had done
I’ve been working in Sierra Leone since 2004. I have worked with colleagues there who have been there all their lives. It’s that kind of repeated interaction that builds knowledge over time. It helps to start building connections in person, but by checking in with those connections regularly over the phone or on short visits, you can rely on them without having to spend a lot of additional time on the ground..
In the end, the answer is: “A lot of time over the years.” This is not something that can be accomplished quickly at all.
But then, some of that translates when you go to other countries. I think there’s an understanding that sets in when you’ve learned about any one developing country quite well. There’s a lot of really interesting work in behavioral economics now about how the pressure of poverty changes how people think and make decisions, and about the constraints that they face. It’s just very hard for us to understand just how different people’s lives are, and just how constrained the environment is in which they have to make decisions.
And in some ways, once you get that once, it really helps with the next context. Now, that doesn’t help you with, “Oh, my God, you should never do your slides in green, because that’s the color of one of the political parties, and they’ll think that you’re linked with that political party.” I mean, there are local nuances that you will not get, that are not transferable from one country to another. But I think some of the most basic things that you get after you spend some time is this more fundamental understanding: “Okay, I get why people are making these decisions.” When you haven’t had much sleep, or when you haven’t had enough to eat, you just make decisions differently.
It’s not great to be asking people to spend a lot of time going to community meetings when they have so much on their plate. Keep in mind all the things that are done for us in a rich society: we have chlorine put in our water, we get reminders, we can’t send our kids to school unless they’ve been immunized. All of this stuff, all of these decisions, they’re just made for us. We need to understand that really poor people in poor environments have to make those decisions and do those things for themselves.
In Sierra Leone, they have to mend the roads themselves. They have to do all the local public services that we get, like trash removal. All these things that we just automatically do, they have to do for themselves. A lot of those things are common across many different societies. So invest in one place, and some of what you learn will translate. But stay open and aware that you need to learn from others about, as I say, the nuances of the local context, like which colors not to use. Remember that you have to be careful of that sort of thing; ask locally before you put your foot in your mouth.
--
Nathan: That discussion of poverty and its effects reconciles a couple things I’ve been wrestling with.You seem to be sort of advocating for a pretty universalist view of human nature. At the same time, I’m well aware of the WEIRD phenomenon, and at least through my filter bubble, it seems that I should be very careful about generalizing from studies or results conducted in, say, Ivy League classrooms. First of all, maybe you reject that notion of the WEIRD phenomenon, but if it is real, maybe poverty is the key common factor among the people you work with?
Rachel: You have to go up a level of abstraction, and then things generalize more. For procrastination, we procrastinate over different things in different contexts, but we all procrastinate, that’s one example. There are differences in the decisions we have to make on a day-to-day basis, but some of the ways in which we as human beings respond to those decisions are very common. The difference is that things are just harder for the poor.
All the things that we fail at, all the ways we’re not very good at making decisions, they’re still much easier for us, because we’ve had a good night’s sleep and we have enough food. It’s very well-established now that people do more short-term thinking when they’re hungry. Even the same farmers in Kenya will make different decisions before and after the harvest. They’ll be more short-termist and fall into more behavioral economics traps before, when they haven’t had enough to eat.
We have to be aware of those pressures on people. Why aren’t people saving money? Why aren’t they taking the good investment option? Well, if you understand all the other constraints they face, that becomes more understandable. In my mind, it’s completely consistent that we as humans are very similar across very different contexts. But the poor just have it harder.
Now, obviously, there are other really important differences that you have to take into account. For example, one of the things that is different across contexts, which you really have to think about carefully, is gender. Constraints on women in different places are very different, and you have to take that into account when you’re designing programs.
I do a lot of work in Bangladesh, where mobility is highly constricted for women and for adolescent girls. When you’re designing a program, you have to understand that women won’t just be able to walk to get somewhere, because it will be very hard for them to be allowed outside their household on their own and it’s hard to persuade other people to take them. There are different practical constraints that you have to take into account in any local context.
But that’s not about humans being different! That comes back to logistics again. As I said, there are three levels: What’s the problem? What’s the underlying human mechanism? And what are the local constraints around logistics? Let’s keep those three boxes separate and think about them separately. Our program designs will be much more practical if we do that.
--
Nathan: I think that’s a great transition to the next set of questions that I have on the importance of policy reform and a growth-oriented agenda. Tyler Cowen, who was a recent guest on the 80,000 Hours podcast, just published a book in which he basically argues that our number-one focus should be on economic growth, because that’s where almost all of the good that we enjoy comes from, subject to some constraints around general human rights.
It seems like you would probably agree with the emphasis on growth. At the same time, some have argued that the focus on RCTs and sort of what has been called the “aid effectiveness craze” is focusing our attention on small issues that may be distracting us from the bigger questions of broad economic growth and societal progress. Do you think that is a valid worry? How do you trade off between small-scale and society-wide capability-building?
Rachel: I think there are a number of different things going on here.
First, I need to object to the characterization of RCTs as “aid effectiveness”. Most RCT work is not focused on aid. Most of the money that goes into poverty relief is money spent by people in developing countries, both governments and individuals. And actually, if you look at most of the people doing RCTs, they don’t think that their audience is aid donors. Their audience is the government of India or the government of Brazil, and to some extent, big companies or other groups of individuals in those countries. Because that’s where the money is, to be honest.
Let’s remember: There’s aid and there’s development, and aid is only ever one small part of development. I agree that improving the policies of developing-country governments is a hugely important way to impact global poverty. The RCT craze is not about aid effectiveness; it’s about government effectiveness, poverty effectiveness. So that’s one slight quibble.
Then there’s the heart of your question, which is policy versus working on small questions. And then should we be thinking: “Well do RCTs work on small questions?” And also: “How do we think about long-term development versus working on improving someone’s health or education right now?”
Again, I think those are two different questions. As I was explaining before, I actually think that RCTs should not be seen as testing specific programs. They should be seen as testing big questions that can then influence policy. You might test a specific project on education, but in doing so, you would aim to learn something more general.
For example, a lot of work on education has suggested that the most effective thing we can do is to focus on the learning within the classroom. It’s not about more money or more textbooks, even though that’s what governments spend their money on. They spend it on teachers and textbooks, mainly teachers. But having more of these things doesn’t actually improve learning. Instead, RCTs within the Indian education system have suggested that the most important problem is that the Indian curriculum is too difficult for most students.
If you just look at the descriptive data, you’ll see that in an average Indian ninth-grade classroom, none of the kids are even close to the ninth-grade curriculum. They’re testing at somewhere between second grade and sixth grade. No wonder they’re not learning very much, because the only thing that the teacher has to do by law in India is complete the curriculum, even if the kids have no idea what they’re talking about.
When RCT testing was done on very specific interventions, all of the ones that worked were those that taught material at a level that the kids could actually understand. The lesson for the Indian government, if they were ever to agree to this, is “change your curriculum”. Yes, you’re testing little things, but you’re coming out with big answers. And that’s what people like Angus Deaton, who came up with some of those critiques, don’t seem to understand.
Now, the final part, and I think the hardest part, is economic development versus, say, working on health and education. At DFID, we have shifted a lot of emphasis relatively recently into trying to do more on economic transformation, under the recognition that the biggest reductions in poverty, as you say, have come from transforming a country’s economic policy. For example, the big opening up of India and China towards more market-oriented economies—and I’m not saying market solve everything, they absolutely don’t—but when you’ve got a system as screwed-up as Communist China, markets can move you a long way, and can really help transform the economy. The same happened in India, and you saw massive reductions in poverty thanks to a move towards a slightly more sensible economic policy.
When I was recently doing my kind of ranking of the most effective things that DFID could do, we were saying, “Well, if there were cases of countries that are as screwed up as China...”
Where things are that screwed up, helping countries move toward effective economic management will be the most effective thing that we can do for poverty. You can’t easily do that as an outside donor. I’d say that Ethiopia at the moment is going through tremendous reform, and we really ought to be focusing attention on helping Ethiopia through that transition. There’s tremendous potential for growth, and they’re fundamentally changing policy there in ways that could be really beneficial to the poor. So jump on those opportunities when you see them; we can’t make them happen, that’s something the developing country has to decide to do themselves, but we should help them as much as we can.
What do you do to promote economic development in countries that are going through this type of fundamental reform process? Sometimes you can nudge them a bit in the right direction, help improve trade policy, reduce trade barriers, and so on. But to be honest, in a lot of countries, it’s not entirely obvious what you can do to promote economic development.
We need a lot more research, a lot more understanding about how to do that, because I absolutely agree that it’s fundamental. But we don’t always have all the tools that we need to make economic transformation happen. And now, think about our own countries: It’s not like we only ever worry about economics and development. We also worry about health and education. Because we don’t grow in order to have more money, we grow so that we can have better lives. We want to make sure that more money translates into actually better lives.
We need to take opportunities for economic development growth when we can, where there’s an opportunity. But we also need to be working on health and education, not least because we know that those things are really important for economic development, right? We know that there are high productivity improvements if kids are given the right nutrition early on; that’s about a 10% return to investing in education. To some extent, you can’t have economic transformation without the building blocks of human capital. In the classic economic growth model, there’s human capital and physical capital. If you want growth, you need to be working on both of those things.
--
Nathan: I wouldn’t be doing my job if I didn’t get to the famous section of these podcasts where we do the “Overrated/Underrated” list. I’ll give you a number of prompts, and you can respond with overrated or underrated. And of course, you’re free to pass on any of them if you don’t have a strong view, or would rather just avoid the topic. And then maybe we’ll circle back to a little bit more career advice for the audience, as we close.
The first item: Charter cities as a means of promoting the sort of growth that we’re talking about.
Rachel: I’m not a fan of charter cities, but I don’t think anyone else is either, apart from one Nobel Prize winner.
Nathan: How about going along to get along with your colleagues?
Rachel: I think it’s really important to learn how to influence and how to get along with your colleagues if you’re going to make change, so: underrated.
Nathan: Starting a business in the developing world.
Rachel: Probably underrated. Social entrepreneurship, overrated. Business, underrated.
Nathan: And how would you draw a line between those?
Rachel: Social entrepreneurship is… I don’t want to get in trouble for sort of dumping on some specific things. But many of those businesses don’t take off in a big way, partly because potential buyers don’t have a lot of money. I think you can have a much bigger impact by working in big organizations. There’s a lot of evidence that businesses in the developing world are really badly managed, and that there are a lot of improvements that could be made. And basically, people want jobs. They don’t want money to create their own businesses; they want jobs. So getting effective private-sector businesses working in these countries is really important. I know people who set up businesses after many years working in development, and I think that’s great.
Nathan: How about cash transfers?
Rachel: Cash transfers, I think we rate very high—appropriately. People have been a little bit down on them recently, because of some recent work saying the long term impacts of one transfer weren’t as good as people had hoped. But I think when you look at the literature as a whole, I think cash is very positive, including long-term benefits. And even if the control group eventually catches up, getting people out of poverty earlier is still really beneficial.
Nathan: Okay, how about gene drives for mosquitos and other disease-carrying insects?
Rachel: Okay, I’m going to pause on that. I don’t know enough about that.
Nathan: Genetically modified and CRISPR crops?
Rachel: I don’t know about CRISPR crops, but I’m a big fan of GM crops, particularly improved agricultural varieties in the developing world. Those are hugely beneficial. Some of that you can get without GM, but I think we’re probably a little bit paranoid about GM.
Nathan: How about cracking down on tax havens or other sources of illicit financial flows?
Rachel: Underrated. We should do more of that.
Nathan: What’s the mechanism by which that benefits everyone?
Rachel: A huge amount of money flows out of developing countries into tax havens. That’s a big problem in terms of fueling corruption. There’s a big opportunity to expose the bad deals that are done with bad governments in developing countries, which are often arranged in the developed world. We ought to be doing more to stop it, and I’m pleased to say that DFID is working in that area.
Nathan: Micronutrient supplementation?
Rachel: Micronutrients? Underrated. Supplementation? we still need more work on that. Because the way we’re putting micronutrients out at the moment doesn’t seem to be working very well. Anemia is probably underrated as a huge, huge problem. It really affects productivity and cognition. We haven’t quite figured out how to address it, though.
Rachel: Yeah, macroeconomic policy is really important, but we’ve actually kind of figured it out. If you look at inflation, it used to be a major problem. When I was doing development economics, half of the course was about how to deal with hyperinflation. Virtually nobody has hyperinflation anymore. That’s a really major success that we don’t talk about enough.
Nathan: Okay, couple more, how about preregistration?
Rachel: I think that’s overrated. And that’s a little funny coming from me, because I wrote one of the papers saying that we ought to do more of it in economics and now I’m finding some of the downsides. Yeah, there’s a big move to register in advance what your analysis is going to be. But sometimes tying your hands is not actually a good idea, so we need to be a bit careful about preregistration. Preanalysis plans, which say “this is exactly how I’m going to analyze my data when it comes out”, can be a problem, because when you look at the data and new circumstances have arisen, it may be really important to change how you’re doing analysis. Plus, I’ve found that journal referees hate it.
Nathan: Do you think that people are not doing that extended analysis, or that it’s being unfairly dismissed as a result of the preregistration?
Rachel: I understand that people worry that you run a trial, and then test your results on 50 different outcomes and promote the one that had a positive effect. Most academic work doesn’t work quite like that, because your referees force you to show 50 robustness checks, and you don’t get passed if only one of them had a positive effect. I think we need to rely a bit more on theory. Theory tells you which things should go together; I think theory can be as an effective way of looking at the data and pulling out patterns, while also somewhat tying your hands. It might even be a more effective way of tying your hands than preanalysis plans. I’m not saying you should never make those plans, but they’re not the simple answer that people thought they were.
Nathan: Okay, two more overrated/underrated questions as we begin to transition to your career advice to close things out. How about reading the news or the newspaper?
Rachel: Reading the news of the kind that you already support, we should be doing less of. Reading things that shock you, or that come from a different perspective, I think we don’t do enough of. I try and read about African politics, for example. It’s not on our news normally, but it’s really interesting. I find it really helpful to read news from serious people who cover African politics; it brings a different perspective. But I think that often, when we read the news, we read things that confirm what we already knew, and that’s not very helpful.
Nathan: Last one for overrated/underrated. How about postgraduate degrees?
Rachel: I think it just depends on what you’re trying to do with your life.
--
Nathan: Let’s get a bit more practical for the last couple of minutes, as we try to give some useful advice to the audience. At DFID, what are the skill sets that you find to be in short supply, and wish more young people were developing today?
Rachel: This is less true of DFID, but I think the development sector in general could really benefit from the skills of the AI community, in the sense of good hard analysis, linked with a passion to make change. The UK’s Government Economic Service and Government Statistical Service produce exactly that kind of hard analysis. And there’s actually a big demand for them, and government in general is desperate for more people to go into the Government Economic Service.
Anyone interested in economics and policy and putting economics to good use should definitely be looking at the Government Economic Service, especially DFID. I think that ability to take a hard look at numbers and think about how they could be used to answer the questions that people have is desperately needed within policy. It’s a really important part of global poverty work. NGOs too. You rarely find good implementing NGOs who are very good at kind of analysis.
Take a look, for example, at the work that Caitlin Tulloch is doing with the IRC, of just taking the data and figuring out the cost structure of what they’re doing and what’s driving variance in costs. NGOs produce lots of data, and they don’t know what to do with most of it. But it helps to have someone really good analytically who works with them to help them understand and use that data effectively.
--
Nathan: Two last questions to close things out. I don’t know how well you know the EA movement. But based on your knowledge, do you have a sense for how EAs most misunderstand or most often get wrong about global health and development?
Rachel: The old saying that genius is 1% inspiration and 99% perspiration is so true in development. Success comes from refining the logistics of making something work. Even once we figure out what the problem is, and we figure out there’s a solution that has generalized, actually making it work at scale on the ground requires infinite amounts of testing and figuring it out and testing it again, which doesn’t necessarily mean randomized trials. I mean, testing can mean that we tried an information blitz and we put up posters and the next morning none of the posters were there. That’s testing. The hard sweat and toil of making something work in dysfunctional societies cannot be underestimated.
One of the things that I meant to say in your earlier question about the evolution of the RCT movement: I think that one of the biggest things that it’s done in academia is bringing academics out into the world. People working on RCTs are basically running large, implementing organizations in developing countries. We’re hiring tons of people, we’re trying to get things from A to B, we’re building stuff—and the amount of insight you’re getting from trying to run a big organization in a dysfunctional country is unbelievable, and it helps generate new questions. I think that’s what I think of anything, that’s what people misunderstand. There’s a lot of discussion up here about the blood, sweat and tears of making the trains run on time.
--
Nathan: Okay, last question for today. What are the best decisions that you’ve made in your career? And you’ve scattered advice throughout this conversation, but what are the top recommendations you would give to EAs who want to make a difference in your line of work, perhaps as civil servants?
Rachel: I’m pretty proud of the work that I did at J-PAL. It was kind of a crazy thing to give up my job at the IMF and go start this organization. As I said, we started with three people, and we had $300,000 from MIT when we started. But the lesson there was that it really was possible to connect what’s coming out of research with practical policy. That work isn’t for everyone, but I think it’s an extremely important area that people should at least consider whether they are interested in.
Because another thing that happens within policy is that people remember what they did at university, but then don’t keep up on the latest literature and get further and further away from what we know now. And I feel one of the things that’s great about the EA community is that you’re endlessly curious. You will keep trying to get up to speed on the latest thinking and the latest evidence. And so if you are in policy, you will be constantly wanting to improve things, and constantly willing to reach out and go the extra mile and read the extra paper and find out what the evidence is saying, and that is so desperately needed in policy.
Rachel Glennerster: Fireside chat (2018)
Link post
Rachel Glennerster is the Chief Economist of DFID, the UK’s ministry for coordinating international development. In this conversation with Nathan Labenz, she discusses the most important lessons she’s learned about development and what it really means for a study’s result to “generalize”.
A transcript of the conversation between Rachel and Nathan, which we have lightly edited for clarity, is below. You can also watch this talk on YouTube, or read its transcript on effectivealtruism.org.
The Conversation
Nathan: Thank you for being here. I’m very excited to have this conversation. I’m going to do my best Rob Wiblin impression and start with what is his traditional first question, which is, what are you working on at the moment and why do you think it’s especially important?
Rachel: I’m doing lots of different work at DFID, but let me talk a little bit about some research work I’m doing, which is evaluating a mass media campaign, a radio campaign, run by Development Media International, which is an NGO here in the UK. They’re doing a family planning program on radio in Burkina Faso. I think it’s really important to look at radio and mass media, because it’s a very cheap way to reach a large number of people and you can make sure that the message is accurate and consistent. The problem is that it’s very hard to evaluate radio for exactly those reasons, because one radio program, which is millions of people at the same time, it’s very hard to randomize.
Now, it happens that Burkina Faso is kind of one of the few places in the world where one can evaluate this effectively. It’s also true that at DFID, we’re worrying a lot about how it’s an area that hasn’t had a lot of UK interest until recently and also, family planning is a hugely important issue, because if you get the demographic transition right, it can have incredible benefits for women, for economic development, and for the health of children.
--
Nathan: Why is it that it’s more testable in Burkina Faso, is that a language group issue?
Rachel: It’s a complex series of factors, that means that you have a lot of different radio stations across Burkina Faso, which indeed have different languages, and so there’s less spillover. So you can randomize at the level of the radio station. You also have people who are so poor that they can’t afford radios. We’re randomly handing out radios to women who don’t have radios. We have kind of two levels of randomization, both at the radio station and within a radio area. Some women already have radios, some women are given radios, and some aren’t. It’s kind of a conglomeration of things that allow you to be able to test.
--
Nathan: Well, we’ll probably come back to that in a minute, when we get to the section on randomized control trials in general. We’re going to try to cover a variety of areas here, including your views on careers and policy, advice for audience members who are interested in policy careers, and the role of evidence in general in aid work. But I thought we might also go back to the beginning of your career, and you could give us a sketch of how your career has developed. You studied economics as an undergraduate, and then began your career at the Treasury, and then it’s taken a lot of different turns from there. Tell us all about that.
Rachel: I think there’s some benefit in telling my story. It sounds very self centered to talk about my story, but I think it’s useful to show that careers don’t always have to be one-directional, and they can take many turns. I think it’s interesting to be doing this in an 80,000 hours podcast, where people are thinking really seriously about what’s the best thing to do, because in some ways, my career has been a bit random. But I think it’s both, right? We have to think seriously about what our next step is and also realize that stuff happens in life.
As an undergraduate, I was very interested in development economics, and in thinking about how I could contribute to addressing the issues of global poverty. And I spent a summer traveling around Kenya, talking to lots of people in the aid sector, and got really depressed and thought: A, lots of things don’t seem to be working, and B, what do I know? What honestly could I bring to these complicated issues? I’m 21, I don’t know anything about Kenya. I decided instead to go into the Government Economic Service, which is a special training position within the government that trains you to use analytical skills to address policy issues.
I did that at the Treasury, I worked on domestic policy, I worked on reform of the health service, trade policy, monetary policy, all sorts of different policies. And I got fantastic training in how to use analytical skills to help us think more rationally about policy decisions. And then I went to get more education. I went to Harvard, and met my future husband, and my life got thrown up in the air because I was convinced I was going to stay in the Government Economics Service for the rest of my life. I loved it.
Michael, my future husband, was American. After failing to persuade him to move to the UK, I had to look for something to do in the US. Then I went back to my old love of development, and having, I think, gained some skills, I worked to represent the UK at the IMF and World Bank. I learned a lot about international institutions and worked on financial regulation in Russia as a way to move to Boston, where Michael was. That all went up in smoke, because a big scandal hit Harvard and its work in Russia. I went back to the IMF and I realized I needed more economics knowledge to really make a difference.
I went back and did a PhD, but I did it part-time. I was constantly on the border between academia and policy work. I think that’s a really interesting and important nexus, where, when I was within government, I was taking academic work and explaining and translating it into policy needs: How can we use this academic work to do our policy work better? Then I moved to J-PAL, where I was on the other side of the fence. I was in academia, but I was helping academia translate what it was doing into the policy world.
I helped found J-PAL MIT, which promoted the use of randomized trials. I think the key thing that J-PAL did was to work really hard on this policy/academic nexus and say: “How do we make sure that this academic work is being translated into policy change?” That’s the thing that’s been common throughout my work. And so yet again, I’ve jumped back onto the other side of the bridge, and I’m now working at the Department for International Development back in the Government Economic Service that I started in all that time ago, as chief economist. And again, I’m helping to bring academic insights and research insights into decisions in government. There’s so much need for that translation across this border.
--
Nathan: I think that’s interesting, as you said, for an 80,000 hours podcast audience that’s thinking a lot about their careers. But also, as we sit here at EA Global London, I think a lot of attendees are trying to figure out how they can shape their own careers to have the most impact. It strikes me that one of the central questions people are asking themselves often is, should I go and do direct work now? Or should I focus on upscaling myself and becoming a more powerful person in whatever domain? It seems like you have climbed those two ladders at different points in your career, is that a model that you would recommend to others?
Rachel: First of all, let me just say that upscaling is not only in academia. You don’t just do it in graduate school. I think the place where I learned the most, as I said, was probably those first two years in the Government Economic Service. I was never trained to write until someone literally went line by line through my work in government to help me write more clearly, and that’s been incredibly important in conveying ideas. You learn technical things when you go to college and graduate school, and later you learn other skills, like how to influence people, how to get things done, and how to run effective organizations.
You need all of those skills if you want to make a difference in the world. Different people will prefer skills that are more on the technical side or more on the “influencing people” side. Think about where your best skills are. I wrote a blog post a while ago about whether economists should go into policy or academia, and I listed the skills that you need in those different areas. I didn’t say one is better than the other or one is harder than the other; it’s just different. Think about building on your natural skills and acquiring new skills, both within institutions and through more formal training.
I do think that people often underestimate the number of skills you learn from being in an effective organization. A lot of people want to start their own organization, and that’s great. But I have seen people start organizations without having really experienced an effective organization. When I arrived at J-PAL, we were three people, and when I left there were 350 people. My deputy and I had both come from the civil service. We had really strong views about how you run an effective organization, and I think that was absolutely critical in building J-PAL.
--
Nathan: Yeah, I think that’s fascinating and very apt. My background is in the software startup world, which is quite different. But often young people will ask me, how can they start their own company or whatever. And I always say, “Work for a great company, before you even think about trying to start one of your own, because there’s a lot that you don’t want to have to reinvent from scratch or try to derive from first principles.” You mentioned randomized controlled trials, and that’s definitely a big subject in your work. Before we dive into the current state of that debate, here’s another historical question.
You started out in an era when effectiveness in aid was much less of a concern. People cared more about just giving and feeling good about it and hoping for the best, or maybe not even really worrying about what happened downstream. How would you sketch the kind of intellectual trajectory of aid and development work over the last three decades?
Rachel: Great. Big questions. I think it’s worth looking at two different trajectories. There’s the trajectory of the aid sector, the development sector, and then there’s the trajectory of academic research on development. and those are a bit different. One of the nice things about RCTs is that they’ve brought those two really closely together. I think the trajectory of the aid sector has been one in which you say, there was not enough emphasis in my view about understanding really rigorously what works. There were a lot of different theories and views about what we should be doing and you see these big swings and fads—for example, the view that development was mainly about investment in physical capital.
Under this theory, the reason we needed aid was that countries didn’t have enough investment, which meant that we should be building stuff. Places like India had five-year plans, and they built steel factories, and then there was a big swing toward a new popular view: “No, we need to worry about human capital and not just steel.” But people aren’t benefiting from human capital programs like job training; they’re still malnourished, they’re not learning. And you saw these big swings and interest about what we should be doing for development, but not a lot of it was especially data-driven. There has been a really big change in the last 10 years or so, especially within the DFID and within the World Bank, to really seriously think about the evidence behind the decisions that we’re making.
One of the reasons I moved to DFID is it has been one of the agencies that I think has changed the most to constantly be examining itself. At the moment, we’re going through the process of looking at what we’re doing and saying, “Is it really evidence-based? What does the new research say? What should we stop doing, because the new evidence is saying we shouldn’t be doing it?” And that’s an important sea change in the aid world.
One of the biggest changes in research, in academia and economics, is just that there are now a lot more people thinking about the developing world, and that’s great. A lot of them are doing RCTs, but if you look at the data, actually, there are just as many non-RCT workers as there were before. Within economics, development used to be a bit off the side; most economists didn’t think about it. But people have realized that the questions in development are actually very similar to the questions in other bits of economics, and we should be learning from each other.
--
Nathan: A lot of behavioral economics lessons came from development, and they are now being taken up and used and learned from in rich countries. I think that it’s been really important to see lessons going in both directions; development has helped us understand that there are a lot of similarities between people, and there’s a lot we can learn from each other.
Moving to RCTs in general, and the state of debate around how much we should rely on them: You mentioned that it’s kind of a 50⁄50 split right now, in today’s work. Do you think that’s an inappropriate split? Do you think that it should be all RCTs? What do you think is the right balance as we try to figure out what is obviously a very complicated world?
Rachel: I think it’s really important to say that all of us who have worked on randomized trials have never suggested that this is the only methodology that you should use. Sometimes it’s held up as a straw person that we go around saying: “This is the only methodology.” But nobody who’s done RCTs has ever thought that they are the only the right approach. I think the right way to see things is that you have a toolbox of ways to answer questions. The right tool depends on the question that you’re asking.
I think we need good descriptive work to understand what the problems are. A lot of development programs fail because they’re trying to solve a problem that doesn’t exist. They’re just solving their own problem. The first really important thing you’ve got to do is to understand what the issue is in any given area. If we’re worried about girls not going to school because of menstruation, let’s start by finding out whether they actually don’t go to school more when they’re menstruating! That’s a really basic, obvious thing, but we actually need more of that kind of thinking; understanding the problem is a really important first step.
RCTs are useful for answering really specific questions, but I think the best RCTs are the ones that test a theory. They test something that’s more generalizable than just “does this program work?” They ask a question about human beings.
Here’s an example. I did a project looking at how to improve immunization rates in India. Only 3% of kids in a certain part of India were getting fully immunized. And given that immunization is one of the most effective things that you could do, so that rate is just appallingly low. There were a number of theories about why that could be. A lot of people said, “Well, people here don’t trust the doctors.” That is, not doctors, because you rarely get doctors in rural India, but nurses and clinics. They don’t trust their formal health system.
There were also other theories. The clinics were often closed, so is that the problem? Is nurse absenteeism the problem?
We had read all this behavioral economics literature, and behavioral economics tells us that people will be happy to get their kids immunized, but they’d rather do it tomorrow. We set up an RCT with one arm that provided just good service, making sure that without fail, there was someone to immunize your child when you reached the clinic. The other arm did the same thing, but also provided a small incentive.
Yes, we were testing a program, but we were also asking a more fundamental question, which is “why don’t people get their kids immunized?” And what we saw in the data is that a lot of people got their kid immunized once, but failed to keep coming back until the end of the immunization schedule. Fixing the supply problem increased the number of people getting the first shot and the second shot. But it failed to fix this persistence problem. However, the incentive worked to help people persist until the end.
By the way, that project was completely impossible to scale. Our “incentive” program involved handing out lentils in the middle of Rajasthan. Nobody showed up, which shows how good economists are at designing logistics. It was a disaster; we learned a lot, but you would never want to actually do a program like this.
A colleague of mine did something similar with another program in Rajasthan, where we ended up improving teachers’ attendance by putting cameras into their classrooms. Again, the logistics were a nightmare, but the project tested a theory. And so once you have the test done, you can think about other questions, like “how do we implement this at scale?”
--
Nathan: I noticed that the illustrations on your website show you in the field as well as in a more academic setting, which is a clear signal that you believe in actually going to places and handing out the lentils (as it were). It seems like you have an on-the-ground, intuitive, firsthand understanding which allows you to generate a number of theories, and then you can test one of those theories and find an effect and then think about, “Okay, now how can we scale this up in such a way that we can simplify the logistics and not have to handle all the lentils personally?”
Rachel: That seems pretty sound. I think people worry about how that then transfers to another context. Could we take that result and say “the incentive will work for persistence”, and take that to another culture or another place and expect similar results? Speak to that a little bit: How can you take your experience when you’re pretty confident that it works in one place and try to generalize it to other places?
I think the discussion around generalizability is really confused. Let me try and explain how I think about it.
People often ask, “Does this result generalize?” To which I respond, “What result? What aspect of this are you asking about?”
I think there are three different main ways in which you need to think about whether something generalizes. The first: Is the problem the same in other places? An intervention won’t generalize to a place that doesn’t have the same problem. In the case of India, the problem was that you have people getting the first immunization, but not persisting to the end of the schedule.
And that’s really easy, that’s something that you can test. You can look at data and say: “Well, in this country, people are willing to get the first immunization, but they don’t manage to persist to the end. And in other places, they’re not getting the first immunization at all.”
The second question: “If people have the same problem, do they respond to solutions in a similar way? Does a small incentive help people in a different context to persist in something that they want to do, but don’t manage to do consistently?”
In this case, there is tons of evidence that incentives work well. It’s not all from lentils and vaccines; it’s from across many different kinds of programs. The general finding that small nudges are useful to fight procrastination is very generalizable. Another example: If you charge people for preventative health care, even a nominal amount, you will see a big decline in takeup; that has been consistently found for different kinds of preventative care in different countries.
The last question: “Can I implement something similar elsewhere?” And that is not something that automatically generalizes. We worked with an extremely good NGO in Rajasthan. Their people would absolutely turn up. They could get the lentils there. The lentils were not stolen. None of this would necessarily be the case, if we worked with another organization or government. Also, we wouldn’t want to use lentils if we did the program in New York, right? Lentils wouldn’t be seen as a particularly exciting incentive in other places.
People confuse these three things and lump them into “this has generalized”. Well, basic fundamental principles of human behavior generalize, but logistics don’t. So that is where you need to spend a lot of time understanding the local context and how to run logistics locally.
Coming back to the beginning of your question: I spend less time on the ground now than I would like. But in the past, I spent a lot of time, which is absolutely critical to understanding local context for anyone thinking about working in development.
Even more importantly, you need to partner with people who really understand the area. Nearly, always, when I work in these countries, I am working with a partner who has worked there for many years. And then we build a relationship where they trust me because I’m willing to spend time and effort to understand their problems, and I trust them because I’ve spent enough time to know that they really know the local context. I would also say that it’s a lot easier to spend time on the ground when you’re younger and less established, so get out there!
--
Nathan: How long do you typically spend in a place? I’m reminded of… I don’t know if it’s a joke, or a proverb, but the idea that a guy who goes to China for a week thinks he knows a little bit about China. After a month, he thinks he knows a lot about China. And then, after a year, he realizes he knows nothing about China. How long do you feel you have to be in a place to build the right relationships and have the level of trust that you need to actually be effective?
Rachel: Again, part of it is based on partnerships. I don’t always have to be there all the time if I’m building relationships and talking to people regularly. I haven’t done what I would advise other people to do, because I didn’t go into development initially. I didn’t spend a year or two on the ground as a student in my early 20s, which I wish I had done
I’ve been working in Sierra Leone since 2004. I have worked with colleagues there who have been there all their lives. It’s that kind of repeated interaction that builds knowledge over time. It helps to start building connections in person, but by checking in with those connections regularly over the phone or on short visits, you can rely on them without having to spend a lot of additional time on the ground..
In the end, the answer is: “A lot of time over the years.” This is not something that can be accomplished quickly at all.
But then, some of that translates when you go to other countries. I think there’s an understanding that sets in when you’ve learned about any one developing country quite well. There’s a lot of really interesting work in behavioral economics now about how the pressure of poverty changes how people think and make decisions, and about the constraints that they face. It’s just very hard for us to understand just how different people’s lives are, and just how constrained the environment is in which they have to make decisions.
And in some ways, once you get that once, it really helps with the next context. Now, that doesn’t help you with, “Oh, my God, you should never do your slides in green, because that’s the color of one of the political parties, and they’ll think that you’re linked with that political party.” I mean, there are local nuances that you will not get, that are not transferable from one country to another. But I think some of the most basic things that you get after you spend some time is this more fundamental understanding: “Okay, I get why people are making these decisions.” When you haven’t had much sleep, or when you haven’t had enough to eat, you just make decisions differently.
It’s not great to be asking people to spend a lot of time going to community meetings when they have so much on their plate. Keep in mind all the things that are done for us in a rich society: we have chlorine put in our water, we get reminders, we can’t send our kids to school unless they’ve been immunized. All of this stuff, all of these decisions, they’re just made for us. We need to understand that really poor people in poor environments have to make those decisions and do those things for themselves.
In Sierra Leone, they have to mend the roads themselves. They have to do all the local public services that we get, like trash removal. All these things that we just automatically do, they have to do for themselves. A lot of those things are common across many different societies. So invest in one place, and some of what you learn will translate. But stay open and aware that you need to learn from others about, as I say, the nuances of the local context, like which colors not to use. Remember that you have to be careful of that sort of thing; ask locally before you put your foot in your mouth.
--
Nathan: That discussion of poverty and its effects reconciles a couple things I’ve been wrestling with.You seem to be sort of advocating for a pretty universalist view of human nature. At the same time, I’m well aware of the WEIRD phenomenon, and at least through my filter bubble, it seems that I should be very careful about generalizing from studies or results conducted in, say, Ivy League classrooms. First of all, maybe you reject that notion of the WEIRD phenomenon, but if it is real, maybe poverty is the key common factor among the people you work with?
Rachel: You have to go up a level of abstraction, and then things generalize more. For procrastination, we procrastinate over different things in different contexts, but we all procrastinate, that’s one example. There are differences in the decisions we have to make on a day-to-day basis, but some of the ways in which we as human beings respond to those decisions are very common. The difference is that things are just harder for the poor.
All the things that we fail at, all the ways we’re not very good at making decisions, they’re still much easier for us, because we’ve had a good night’s sleep and we have enough food. It’s very well-established now that people do more short-term thinking when they’re hungry. Even the same farmers in Kenya will make different decisions before and after the harvest. They’ll be more short-termist and fall into more behavioral economics traps before, when they haven’t had enough to eat.
We have to be aware of those pressures on people. Why aren’t people saving money? Why aren’t they taking the good investment option? Well, if you understand all the other constraints they face, that becomes more understandable. In my mind, it’s completely consistent that we as humans are very similar across very different contexts. But the poor just have it harder.
Now, obviously, there are other really important differences that you have to take into account. For example, one of the things that is different across contexts, which you really have to think about carefully, is gender. Constraints on women in different places are very different, and you have to take that into account when you’re designing programs.
I do a lot of work in Bangladesh, where mobility is highly constricted for women and for adolescent girls. When you’re designing a program, you have to understand that women won’t just be able to walk to get somewhere, because it will be very hard for them to be allowed outside their household on their own and it’s hard to persuade other people to take them. There are different practical constraints that you have to take into account in any local context.
But that’s not about humans being different! That comes back to logistics again. As I said, there are three levels: What’s the problem? What’s the underlying human mechanism? And what are the local constraints around logistics? Let’s keep those three boxes separate and think about them separately. Our program designs will be much more practical if we do that.
--
Nathan: I think that’s a great transition to the next set of questions that I have on the importance of policy reform and a growth-oriented agenda. Tyler Cowen, who was a recent guest on the 80,000 Hours podcast, just published a book in which he basically argues that our number-one focus should be on economic growth, because that’s where almost all of the good that we enjoy comes from, subject to some constraints around general human rights.
It seems like you would probably agree with the emphasis on growth. At the same time, some have argued that the focus on RCTs and sort of what has been called the “aid effectiveness craze” is focusing our attention on small issues that may be distracting us from the bigger questions of broad economic growth and societal progress. Do you think that is a valid worry? How do you trade off between small-scale and society-wide capability-building?
Rachel: I think there are a number of different things going on here.
First, I need to object to the characterization of RCTs as “aid effectiveness”. Most RCT work is not focused on aid. Most of the money that goes into poverty relief is money spent by people in developing countries, both governments and individuals. And actually, if you look at most of the people doing RCTs, they don’t think that their audience is aid donors. Their audience is the government of India or the government of Brazil, and to some extent, big companies or other groups of individuals in those countries. Because that’s where the money is, to be honest.
Let’s remember: There’s aid and there’s development, and aid is only ever one small part of development. I agree that improving the policies of developing-country governments is a hugely important way to impact global poverty. The RCT craze is not about aid effectiveness; it’s about government effectiveness, poverty effectiveness. So that’s one slight quibble.
Then there’s the heart of your question, which is policy versus working on small questions. And then should we be thinking: “Well do RCTs work on small questions?” And also: “How do we think about long-term development versus working on improving someone’s health or education right now?”
Again, I think those are two different questions. As I was explaining before, I actually think that RCTs should not be seen as testing specific programs. They should be seen as testing big questions that can then influence policy. You might test a specific project on education, but in doing so, you would aim to learn something more general.
For example, a lot of work on education has suggested that the most effective thing we can do is to focus on the learning within the classroom. It’s not about more money or more textbooks, even though that’s what governments spend their money on. They spend it on teachers and textbooks, mainly teachers. But having more of these things doesn’t actually improve learning. Instead, RCTs within the Indian education system have suggested that the most important problem is that the Indian curriculum is too difficult for most students.
If you just look at the descriptive data, you’ll see that in an average Indian ninth-grade classroom, none of the kids are even close to the ninth-grade curriculum. They’re testing at somewhere between second grade and sixth grade. No wonder they’re not learning very much, because the only thing that the teacher has to do by law in India is complete the curriculum, even if the kids have no idea what they’re talking about.
When RCT testing was done on very specific interventions, all of the ones that worked were those that taught material at a level that the kids could actually understand. The lesson for the Indian government, if they were ever to agree to this, is “change your curriculum”. Yes, you’re testing little things, but you’re coming out with big answers. And that’s what people like Angus Deaton, who came up with some of those critiques, don’t seem to understand.
Now, the final part, and I think the hardest part, is economic development versus, say, working on health and education. At DFID, we have shifted a lot of emphasis relatively recently into trying to do more on economic transformation, under the recognition that the biggest reductions in poverty, as you say, have come from transforming a country’s economic policy. For example, the big opening up of India and China towards more market-oriented economies—and I’m not saying market solve everything, they absolutely don’t—but when you’ve got a system as screwed-up as Communist China, markets can move you a long way, and can really help transform the economy. The same happened in India, and you saw massive reductions in poverty thanks to a move towards a slightly more sensible economic policy.
When I was recently doing my kind of ranking of the most effective things that DFID could do, we were saying, “Well, if there were cases of countries that are as screwed up as China...”
Where things are that screwed up, helping countries move toward effective economic management will be the most effective thing that we can do for poverty. You can’t easily do that as an outside donor. I’d say that Ethiopia at the moment is going through tremendous reform, and we really ought to be focusing attention on helping Ethiopia through that transition. There’s tremendous potential for growth, and they’re fundamentally changing policy there in ways that could be really beneficial to the poor. So jump on those opportunities when you see them; we can’t make them happen, that’s something the developing country has to decide to do themselves, but we should help them as much as we can.
What do you do to promote economic development in countries that are going through this type of fundamental reform process? Sometimes you can nudge them a bit in the right direction, help improve trade policy, reduce trade barriers, and so on. But to be honest, in a lot of countries, it’s not entirely obvious what you can do to promote economic development.
We need a lot more research, a lot more understanding about how to do that, because I absolutely agree that it’s fundamental. But we don’t always have all the tools that we need to make economic transformation happen. And now, think about our own countries: It’s not like we only ever worry about economics and development. We also worry about health and education. Because we don’t grow in order to have more money, we grow so that we can have better lives. We want to make sure that more money translates into actually better lives.
We need to take opportunities for economic development growth when we can, where there’s an opportunity. But we also need to be working on health and education, not least because we know that those things are really important for economic development, right? We know that there are high productivity improvements if kids are given the right nutrition early on; that’s about a 10% return to investing in education. To some extent, you can’t have economic transformation without the building blocks of human capital. In the classic economic growth model, there’s human capital and physical capital. If you want growth, you need to be working on both of those things.
--
Nathan: I wouldn’t be doing my job if I didn’t get to the famous section of these podcasts where we do the “Overrated/Underrated” list. I’ll give you a number of prompts, and you can respond with overrated or underrated. And of course, you’re free to pass on any of them if you don’t have a strong view, or would rather just avoid the topic. And then maybe we’ll circle back to a little bit more career advice for the audience, as we close.
The first item: Charter cities as a means of promoting the sort of growth that we’re talking about.
Rachel: I’m not a fan of charter cities, but I don’t think anyone else is either, apart from one Nobel Prize winner.
Nathan: How about going along to get along with your colleagues?
Rachel: I think it’s really important to learn how to influence and how to get along with your colleagues if you’re going to make change, so: underrated.
Nathan: Starting a business in the developing world.
Rachel: Probably underrated. Social entrepreneurship, overrated. Business, underrated.
Nathan: And how would you draw a line between those?
Rachel: Social entrepreneurship is… I don’t want to get in trouble for sort of dumping on some specific things. But many of those businesses don’t take off in a big way, partly because potential buyers don’t have a lot of money. I think you can have a much bigger impact by working in big organizations. There’s a lot of evidence that businesses in the developing world are really badly managed, and that there are a lot of improvements that could be made. And basically, people want jobs. They don’t want money to create their own businesses; they want jobs. So getting effective private-sector businesses working in these countries is really important. I know people who set up businesses after many years working in development, and I think that’s great.
Nathan: How about cash transfers?
Rachel: Cash transfers, I think we rate very high—appropriately. People have been a little bit down on them recently, because of some recent work saying the long term impacts of one transfer weren’t as good as people had hoped. But I think when you look at the literature as a whole, I think cash is very positive, including long-term benefits. And even if the control group eventually catches up, getting people out of poverty earlier is still really beneficial.
Nathan: Okay, how about gene drives for mosquitos and other disease-carrying insects?
Rachel: Okay, I’m going to pause on that. I don’t know enough about that.
Nathan: Genetically modified and CRISPR crops?
Rachel: I don’t know about CRISPR crops, but I’m a big fan of GM crops, particularly improved agricultural varieties in the developing world. Those are hugely beneficial. Some of that you can get without GM, but I think we’re probably a little bit paranoid about GM.
Nathan: How about cracking down on tax havens or other sources of illicit financial flows?
Rachel: Underrated. We should do more of that.
Nathan: What’s the mechanism by which that benefits everyone?
Rachel: A huge amount of money flows out of developing countries into tax havens. That’s a big problem in terms of fueling corruption. There’s a big opportunity to expose the bad deals that are done with bad governments in developing countries, which are often arranged in the developed world. We ought to be doing more to stop it, and I’m pleased to say that DFID is working in that area.
Nathan: Micronutrient supplementation?
Rachel: Micronutrients? Underrated. Supplementation? we still need more work on that. Because the way we’re putting micronutrients out at the moment doesn’t seem to be working very well. Anemia is probably underrated as a huge, huge problem. It really affects productivity and cognition. We haven’t quite figured out how to address it, though.
Nathan: Improving developing countries’ macroeconomic policy. We’ve kind of covered that.
Rachel: Yeah, macroeconomic policy is really important, but we’ve actually kind of figured it out. If you look at inflation, it used to be a major problem. When I was doing development economics, half of the course was about how to deal with hyperinflation. Virtually nobody has hyperinflation anymore. That’s a really major success that we don’t talk about enough.
Nathan: Okay, couple more, how about preregistration?
Rachel: I think that’s overrated. And that’s a little funny coming from me, because I wrote one of the papers saying that we ought to do more of it in economics and now I’m finding some of the downsides. Yeah, there’s a big move to register in advance what your analysis is going to be. But sometimes tying your hands is not actually a good idea, so we need to be a bit careful about preregistration. Preanalysis plans, which say “this is exactly how I’m going to analyze my data when it comes out”, can be a problem, because when you look at the data and new circumstances have arisen, it may be really important to change how you’re doing analysis. Plus, I’ve found that journal referees hate it.
Nathan: Do you think that people are not doing that extended analysis, or that it’s being unfairly dismissed as a result of the preregistration?
Rachel: I understand that people worry that you run a trial, and then test your results on 50 different outcomes and promote the one that had a positive effect. Most academic work doesn’t work quite like that, because your referees force you to show 50 robustness checks, and you don’t get passed if only one of them had a positive effect. I think we need to rely a bit more on theory. Theory tells you which things should go together; I think theory can be as an effective way of looking at the data and pulling out patterns, while also somewhat tying your hands. It might even be a more effective way of tying your hands than preanalysis plans. I’m not saying you should never make those plans, but they’re not the simple answer that people thought they were.
Nathan: Okay, two more overrated/underrated questions as we begin to transition to your career advice to close things out. How about reading the news or the newspaper?
Rachel: Reading the news of the kind that you already support, we should be doing less of. Reading things that shock you, or that come from a different perspective, I think we don’t do enough of. I try and read about African politics, for example. It’s not on our news normally, but it’s really interesting. I find it really helpful to read news from serious people who cover African politics; it brings a different perspective. But I think that often, when we read the news, we read things that confirm what we already knew, and that’s not very helpful.
Nathan: Last one for overrated/underrated. How about postgraduate degrees?
Rachel: I think it just depends on what you’re trying to do with your life.
--
Nathan: Let’s get a bit more practical for the last couple of minutes, as we try to give some useful advice to the audience. At DFID, what are the skill sets that you find to be in short supply, and wish more young people were developing today?
Rachel: This is less true of DFID, but I think the development sector in general could really benefit from the skills of the AI community, in the sense of good hard analysis, linked with a passion to make change. The UK’s Government Economic Service and Government Statistical Service produce exactly that kind of hard analysis. And there’s actually a big demand for them, and government in general is desperate for more people to go into the Government Economic Service.
Anyone interested in economics and policy and putting economics to good use should definitely be looking at the Government Economic Service, especially DFID. I think that ability to take a hard look at numbers and think about how they could be used to answer the questions that people have is desperately needed within policy. It’s a really important part of global poverty work. NGOs too. You rarely find good implementing NGOs who are very good at kind of analysis.
Take a look, for example, at the work that Caitlin Tulloch is doing with the IRC, of just taking the data and figuring out the cost structure of what they’re doing and what’s driving variance in costs. NGOs produce lots of data, and they don’t know what to do with most of it. But it helps to have someone really good analytically who works with them to help them understand and use that data effectively.
--
Nathan: Two last questions to close things out. I don’t know how well you know the EA movement. But based on your knowledge, do you have a sense for how EAs most misunderstand or most often get wrong about global health and development?
Rachel: The old saying that genius is 1% inspiration and 99% perspiration is so true in development. Success comes from refining the logistics of making something work. Even once we figure out what the problem is, and we figure out there’s a solution that has generalized, actually making it work at scale on the ground requires infinite amounts of testing and figuring it out and testing it again, which doesn’t necessarily mean randomized trials. I mean, testing can mean that we tried an information blitz and we put up posters and the next morning none of the posters were there. That’s testing. The hard sweat and toil of making something work in dysfunctional societies cannot be underestimated.
One of the things that I meant to say in your earlier question about the evolution of the RCT movement: I think that one of the biggest things that it’s done in academia is bringing academics out into the world. People working on RCTs are basically running large, implementing organizations in developing countries. We’re hiring tons of people, we’re trying to get things from A to B, we’re building stuff—and the amount of insight you’re getting from trying to run a big organization in a dysfunctional country is unbelievable, and it helps generate new questions. I think that’s what I think of anything, that’s what people misunderstand. There’s a lot of discussion up here about the blood, sweat and tears of making the trains run on time.
--
Nathan: Okay, last question for today. What are the best decisions that you’ve made in your career? And you’ve scattered advice throughout this conversation, but what are the top recommendations you would give to EAs who want to make a difference in your line of work, perhaps as civil servants?
Rachel: I’m pretty proud of the work that I did at J-PAL. It was kind of a crazy thing to give up my job at the IMF and go start this organization. As I said, we started with three people, and we had $300,000 from MIT when we started. But the lesson there was that it really was possible to connect what’s coming out of research with practical policy. That work isn’t for everyone, but I think it’s an extremely important area that people should at least consider whether they are interested in.
Because another thing that happens within policy is that people remember what they did at university, but then don’t keep up on the latest literature and get further and further away from what we know now. And I feel one of the things that’s great about the EA community is that you’re endlessly curious. You will keep trying to get up to speed on the latest thinking and the latest evidence. And so if you are in policy, you will be constantly wanting to improve things, and constantly willing to reach out and go the extra mile and read the extra paper and find out what the evidence is saying, and that is so desperately needed in policy.