Two Big-Picture Critiques of GiveWell’s Approach, and Six Lessons from Their Recent Work (Elie Hassenfeld on The 80,000 Hours Podcast)
Over at The 80,000 Hours Podcast we just published an interview that is likely to be of particular interest to people who identify as involved in the effective altruism community: Elie Hassenfeld on Two Big-Picture Critiques of GiveWell’s Approach, and Six Lessons from Their Recent Work.
You can click through for the audio, a full transcript, and related links. Below is the episode summary and some key excerpts.
Episode summary
It strikes me that there’s more of a risk of doing harm here, by assuming that we do have the answer and pushing economic policy in a certain direction. There’s just a lot of opportunity for unintended consequences of pushing countries to do things that are different even if we knew how to do it. All that said, I do think the critique still stands, because ideal GiveWell would have said, “We spent a year on this because it’s an important idea”. - Elie Hassenfeld |
GiveWell is one of the world’s best-known charity evaluators, with the goal of “searching for the charities that save or improve lives the most per dollar.” It mostly recommends projects that help the world’s poorest people avoid easily prevented diseases, like intestinal worms or vitamin A deficiency.
But should GiveWell, as some critics argue, take a totally different approach to its search, focusing instead on directly increasing subjective well-being, or alternatively, raising economic growth?
Today’s guest — cofounder and CEO of GiveWell, Elie Hassenfeld — is proud of how much GiveWell has grown in the last five years. Its ‘money moved’ has quadrupled to around $600 million a year.
Its research team has also more than doubled, enabling them to investigate a far broader range of interventions that could plausibly help people an enormous amount for each dollar spent. That work has led GiveWell to support dozens of new organisations, such as Kangaroo Mother Care, MiracleFeet, and Dispensers for Safe Water.
But some other researchers focused on figuring out the best ways to help the world’s poorest people say GiveWell shouldn’t just do more of the same thing, but rather ought to look at the problem differently.
Currently, GiveWell uses a range of metrics to track the impact of the organisations it considers recommending — such as ‘lives saved,’ ‘household incomes doubled,’ and for health improvements, the ‘quality-adjusted life year.’ To compare across opportunities, it then needs some way of weighing these different types of benefits up against one another. This requires estimating so-called “moral weights,” which Elie agrees is far from the most mature part of the project.
The Happier Lives Institute (HLI) has argued that instead, GiveWell should try to cash out the impact of all interventions in terms of improvements in subjective wellbeing. According to HLI, it’s improvements in wellbeing and reductions in suffering that are the true ultimate goal of all projects, and if you quantify everyone on this same scale, using some measure like the wellbeing-adjusted life year (WELLBY), you have an easier time comparing them.
This philosophy has led HLI to be more sceptical of interventions that have been demonstrated to improve health, but whose impact on wellbeing has not been measured, and to give a high priority to improving lives relative to extending them.
An alternative high-level critique is that really all that matters in the long run is getting the economies of poor countries to grow. According to this line of argument, hundreds of millions fewer people live in poverty in China today than 50 years ago, but is that because of the delivery of basic health treatments? Maybe a little), but mostly not.
Rather, it’s because changes in economic policy and governance in China allowed it to experience a 10% rate of economic growth for several decades. That led to much higher individual incomes and meant the country could easily afford all the basic health treatments GiveWell might otherwise want to fund, and much more besides.
On this view, GiveWell should focus on figuring out what causes some countries to experience explosive economic growth while others fail to, or even go backwards. Even modest improvements in the chances of such a ‘growth miracle’ will likely offer a bigger bang-for-buck than funding the incremental delivery of deworming tablets or vitamin A supplements, or anything else.
Elie sees where both of these critiques are coming from, and notes that they’ve influenced GiveWell’s work in some ways. But as he explains, he thinks they underestimate the practical difficulty of successfully pulling off either approach and finding better opportunities than what GiveWell funds today.
In today’s in-depth conversation, Elie and host Rob Wiblin cover the above, as well as:
The research that caused GiveWell to flip from not recommending chlorine dispensers as an intervention for safe drinking water to spending tens of millions of dollars on them.
What transferable lessons GiveWell learned from investigating different kinds of interventions, like providing medical expertise to hospitals in very poor countries to help them improve their practices.
Why the best treatment for premature babies in low-resource settings may involve less rather than more medicine.
The high prevalence of severe malnourishment among children and what can be done about it.
How to deal with hidden and non-obvious costs of a programme, like taking up a hospital room that might otherwise have been used for something else.
Some cheap early treatments that can prevent kids from developing lifelong disabilities, which GiveWell funds.
The various roles GiveWell is currently hiring for, and what’s distinctive about their organisational culture.
Get this episode by subscribing to our podcast on the world’s most pressing problems and how to solve them: type ‘80,000 Hours’ into your podcasting app. Or read the transcript below.
Producer: Keiran Harris
Audio mastering: Simon Monsour and Ben Cordell
Transcriptions: Katy Moore
Highlights
The subjective wellbeing approach in contrast with GiveWell’s approach
Elie Hassenfeld: First I think it would be helpful for me to just explain what GiveWell is doing today, which is we cash everything out either in terms of increased ability to consume (i.e. people have more money) or reductions in disability-adjusted life years — some of which are health-related and some are mortality-related.
But I very much take the point that subjective wellbeing is an important consideration. We don’t view the two outcomes we use today as the only outcomes that make sense. They’re just the two outcomes that we’ve been able to use to date. I do think over time, as we continue to grow and increase the size of our team, we’ll be in a position to include more factors explicitly in that analysis.
I think the pro of subjective wellbeing measures is that it’s one more angle to use to look at the effectiveness of a programme. It seems to me it’s an important one, and I would like us to take it into consideration.
I think the downside, or the reasons not to, might be that on one level, I think it can just be harder to measure. A death is very straightforward: we know what has happened. And the measures of subjective wellbeing are squishier in ways that it makes it harder to really know what it is. Also, I think some people might say, “I really value reducing suffering and therefore I choose subjective wellbeing.” I think other people might say, “I think these measures are telling me something that is not part of my ‘view of the good,’ and I don’t want to support that.” That would cause someone to want to leave it out of their calculus and the donations they’re making.
In some ideal world, I would love for GiveWell to be able to offer options for donors who have different philosophical perspectives about what they want to achieve. Obviously, GiveWell institutionally also needs to have a view, because there’s funds that come to us directly. But ideally, in the future vision of GiveWell, for people who have subjective wellbeing as their core focus, other moral values, or maybe even a very different tradeoff between increasing income and reducing disability-adjusted life years (or increasing DALYs, maybe, depending on how you think about it), those are programmes we’d like to be able to bring to donors and let them choose.
Because we’re not trying to add value by being particularly good philosophically. That’s not part of GiveWell’s comparative advantage. It would be better if we could, where donors want it, allow them to use their own judgements to make decisions.
The value of saving a life when that life is going to be very difficult
Rob Wiblin: I think to most people, it’s intuitive that it’s more valuable to save the life of someone who feels that they’re really flourishing and is super glad to be alive than it is to save the life of someone who thinks their life is barely worth living, who maybe doesn’t even care that much whether they live or die.
It could be useful to use some numbers to make it a bit clearer how this might end up affecting your relative priorities here. If you imagine people scoring their quality of life out of 10, that’s kind of the standard subjective wellbeing scale. Let’s say that we use the number 3 as the number at which someone is rating their existence as neutral, with the good and bad things in their life cancelling out: that’s kind of a typical answer for what people say would be the neutral point for them if they were scoring themselves.
If someone is going to report a quality of life of 4 out of 10 for the rest of their lives, then from a wellbeing-adjusted life year, a WELLBY, point of view, then it’s equally valuable to them to prevent them from dying as it is to increase their wellbeing permanently by one point out of 10. That would be from 4 to 5 in this case. On the other hand, if someone reports a quality of life of 5 out of 10, then from a WELLBY point of view, it’s twice as valuable to save their life as to increase their wellbeing permanently by one point — in this case from 5 to 6 — because the difference from 3 to 5 is twice as great as from 5 to 6.
The Happier Lives Institute notes that many people in very poor countries — who otherwise might die of malaria in the absence of additional antimalarial bednets — have unsurprisingly pretty challenging lives with plenty of hardship in them. That, as I understand it, suggests that to them it’s more likely to be cost effective to make people’s lives better than to make them longer or less equal.
What do you and GiveWell make of that line of argument?
Elie Hassenfeld: I think the place I want to start is this is a case where I feel most strongly that I would want to hear from the people themselves in low-income countries about this topic. Because if you kind of draw out this line of reasoning, it leads you to the conclusion that there is a very high proportion of people living in low-income countries who would choose death over continued living, based on their self-reported life satisfaction.
That’s a very uncomfortable conclusion, but maybe more importantly, one that is so counterintuitive that I feel the need to follow up on it before accepting it at face value. That may be a somewhat minor point about where you draw the line on the scale, but still, in this case, I think the maybe purely emotional urge I have is to say that doesn’t quite seem like it could be right. Intellectually, I know it could be right — therefore I need to follow up on it, because it’s so inconsistent with my starting point for what people would say.
Rob Wiblin: Yeah, it definitely can get uncomfortable or weird. Or maybe if you were surveying people on their subjective wellbeing, and you really said, “If you score yourself a 2, we’re going to take it that you actually mean that you would rather not be alive right now,” then maybe people would reassess. Because an interesting thing is that when you survey people, almost everywhere in the world, even people in serious poverty almost always say that they think their life is better than not existing, and they want to continue surviving and so on.
I’ve heard some philosophers say that that kind of intuition that we all have about how great it is to continue existing might be suspicious, because we might have evolved to have that attitude. We necessarily almost have to evolve to have that attitude, even if our lives are very unpleasant. That kind of bias might affect all of us. But I’m not really too keen to go there, and I feel extremely uncomfortable. If someone says that saving their life is really valuable, I’m inclined to take that at face value and to trust that over some subjective wellbeing survey.
Elie Hassenfeld: Right. I think that discomfort is a good starting point, though not an ending point. Certainly something that we are very committed to internally — one of our company values or whatever you want to call it — is truth-seeking. What we mean by that is we’re going to have the hard conversations, and keep digging to try to get the answer that is correct, as far as we can see it. Therefore, in this case, I would say I am very suspicious of philosophising and reaching a conclusion that seems extremely counterintuitive and then running with it. But we’re a place that wants to go deeper and be open to strange conclusions. Or maybe I should say it differently, like: conclusions that seem strange to us today that will not seem strange to us in the future once we’ve spent more time with them and done more research on them.
Whether economic policy is what really matters overwhelmingly
Elie Hassenfeld: I think I want to start with the parts of the critique that I take on board, and what I think we would ideally be doing differently, but then move into the critiques of the critique that I have and where I think it maybe is overstating its case.
The part of this critique that I really like, and I’ve been thinking about recently, is that I don’t think that we at GiveWell have put enough time into finding ways to explore the space of possibilities in this area, given its potential importance. I think that is something that I don’t regret historically — I’ll tell you why — but I do think going forward, as we’ve grown and as we continue to grow, I’d like to be in a position where we’ve explored this enough to have a really great answer, which either is we’re doing this in this area or we’re not, because of this pretty compelling reason.
I think one of the things that explains GiveWell’s history, largely, is that GiveWell did something very unique by going very deep on charitable interventions and understanding them very well. A lot of how we’ve grown is by sticking to that core pretty intensively over a long period of time, while we expand out in many of the ways that we’ve talked about today. I think in some ways that is our greatest institutional strength and maybe our greatest institutional weakness. We’ve been very focused on maintaining quality and rigour, and that has been very hard as we’ve grown a lot. I think we’ve been successful at it, and also it has made us more deliberate in the approach that we take to things — and I think that’s a fair characterisation of GiveWell.
So when there have been ideas that are more outside of our bailiwick, I think we’ve been just less effective at engaging with them. Just looking at the trajectory we’ve been on in the last three years and how we’ve expanded, when I look out five more years with our growth, I think we will be in a much better position to be engaging more seriously with these ideas. Maybe that’s the institutional critique and what I think we could do differently. But I’m happy to move on and engage more substantively with the ideas.
—-
Rob Wiblin: Is it maybe the case that there’s just fewer organisations who perceive this is their goal, this is their direct mission in the developing world, relative to how many health-related organisations there are?
Elie Hassenfeld: Maybe, but I think it’s also a question of how you would attack this philanthropically — like I also wonder how neglected this space truly is. There’s the World Bank, IMF, there’s other institutions. There are the Washington think tanks that are definitely focused on economic growth, and academics who focus on macroeconomics and how we can improve low income country conditions.
Dispensers for Safe Water
Elie Hassenfeld: So in many parts of low-income countries, people don’t have access to clean water, and drinking unclean water can lead to diarrhoeal disease, which most importantly leads to death among children under five. This intervention puts a small chlorine dispenser near a water point, so that when someone comes to collect water from a spring, a pipe, they quickly push down on the chlorine dispenser into the jerry can that holds the water. That puts chlorine into the storage container that they then carry home.
This intervention is potentially much more effective than other attempts at chlorination in the past, because the individual collecting water only needs to remember to put chlorine in their container one time, right at collection. Also, the chlorine remains effective while the water is in the container once they bring it back home. So it reduces the need, say, for an alternative programme, which would require someone to go to the store to purchase chlorine tablets, or get them from a nonprofit, have them at home, and then use them each time they choose to consume water. And that easier behavioural intervention makes it more effective.
Rob Wiblin: Yeah. I’m not sure how much I would sanitise my water if I had to stick something in it every time I poured a glass of water. Sounds super annoying.
What’s the prima facie or conceptual, high-level case for why this wouldn’t just be good, but it could be amazing and one of the best things for you to fund?
Elie Hassenfeld: It’s that unclean water leads to a great deal of mortality in low-income countries. Having a diarrhoeal disease not only can lead directly to death, but can exacerbate malnutrition, which itself is a risk factor for death from other infectious diseases. So it’s a major problem.
Then, this is a very low technology type of intervention. It’s very simple. I’ve used it — I’ve visited this in Kenya, and it just requires pushing down on this thing to deliver chlorine. So it’s easy to implement and easy to monitor and follow up on; it’s easy to check. You put that all together, and it’s a fairly low-cost programme that has a direct effect on a major public health problem globally.
Rob Wiblin: Yeah. Just so I can picture it in my head, people are getting a big bottle of water from a well, or from a common tap, and then they have to stick a little chlorine tablet in it? Or is it kind of a chlorine spray that they stick in the bottle once they’re done?
Elie Hassenfeld: Basically, imagine that the person is carrying a jerry can. This is often a yellow, several-gallon container. They’re bringing that container up to maybe a pipe, or even without a pipe, just a spring that water is flowing from. Right next to that water point, there’s a stand that’s maybe two to three feet high with a little plastic container that holds liquid chlorine. You press down on the pump one time, almost like a soap pump that you’d find in a public bathroom, and out of that pump comes the appropriate amount of chlorine for that jerry can. So it’s just dispensing it directly into the water container.
How to avoid attributing deaths incorrectly
Rob Wiblin: Could these issues be quite widespread in investigations that you and other groups do of other interventions and other programmes? So each individual study of whatever other interventions finds no effect, but then if you added them all together, you’d find that there was a large effect? Or maybe if they focus on malaria, they look at deaths from malaria — but in fact it’s had a much larger effect on mortality than what’s apparent, because people are attributing deaths incorrectly. Did you worry about that?
Elie Hassenfeld: It’s definitely something that we’re very focused on. For a long time, when we’ve looked at malaria data, we’ve focused when we can on all-cause mortality. And of the randomised trials that were done on malaria nets, historically a large number were on malaria rates, but a number were on all-cause mortality as a whole, because of this reason exactly.
One of the lessons we took away from this is that, years ago, when we first did this analysis, we insufficiently brought our conclusions to experts outside. I think, had we done that, it’s possible that they would have raised this question in 2019 and we would have more quickly updated because we would have realised that we were too narrowly assessing the impact of the intervention. And that is a change we’ve made with some of the other programmes (which I think we’ll talk about in a minute): we’ve taken our estimates to outsiders and they’ve helped us see a broader picture of what they might be doing so we can home in on the best possible estimate we can.
Rob Wiblin: If I recall, another thing that Kremer did (who was the person who did this early aggregation of all of these different studies, and tipped you off that maybe you would want to take another look at this): he had access to a bunch of data that wasn’t entirely public, or maybe some studies that hadn’t come out yet that allowed him to get a larger sample and notice this. Is that a common problem? That studies get done and either the data is not published yet for a long time, or perhaps you don’t have access to the specific numbers that you need from that study in order to aggregate it to get a clearer picture?
Elie Hassenfeld: I think it is a pretty common issue. And in many cases, when we go deeper on analysis, we’re doing that via reaching out to authors and getting the underlying data itself, so we can understand what’s happening. We’ve done that a number of times historically. Mostly we’re relying on publicly available data because the time costs involved in trying to track down that data and get more of it are high, relative to just relying on the data that’s already out there.
Bridging the gap between abstract arguments and ways to actually move forward
Rob Wiblin: I think [some listeners might] say, at least in the cases of countries going backwards massively, we know things that countries shouldn’t do that quite consistently lead them to have economic disasters — like causing hyperinflation is one of them. They might say, even if we don’t know what the very best policy is, we at least know some things that are clearly bad, and maybe more effort should be put into preventing those, given how catastrophic they are. Do you want to react to that one?
Elie Hassenfeld: Off the cuff, they also seem like the countries that are hardest to influence. If it’s so well known, then why are they doing it? Well, they’re probably doing it because leadership in the country does not have their population’s best interests in mind. That seems like quite a challenge for philanthropy to address.
Rob Wiblin: Yeah, I think that’s probably my biggest concern with this line of argument, which in general I’m quite sympathetic to. Like you, I think there’s a lot to it, but I feel that often it’s not appreciating that there’s reasons that countries have bad policy. Very often it’s not merely just a mistake; it’s because of the political settlement within a country and who has power. And coming in and telling people that they could be richer if they change their policy one way or another — the elites often don’t want to implement those policies because they think it would weaken their position one way or another, or at least they’re not suffering from the poverty. There’s this whole other angle of political economy, trying to understand how countries end up with the policies that they do, given how the political system works.
Elie Hassenfeld: That’s why I think ultimately, where I think GiveWell has something to add to this conversation — many of the conversations we’ve had — is to say that we can look at it from the 10,000-foot view or the 50,000-foot view. That’s important because it can help us decide where to put our resources. It’s hard to figure out what’s true from such a high level.
I think to some extent what makes me really excited about our work, why I think it’s really cool, is that we’re trying to be good about thinking at the 50,000-foot level, but then dig all the way in and ask: What can we do in this case about this problem? When I think about this specifically, I have absolutely no idea what to give money to to improve economic growth in country A, B, or C. But I can imagine a proximate step of finding people to spend time on this for a while and see what they come back with.
Having watched a lot of different types of programmes over many years — from GiveWell, from Open Philanthropy — more research often leads to new ideas. And so we’re excited, I’m excited, about our opportunity to support work like that, because it can bridge this gap between very abstract arguments — where there’s good arguments on both sides — to find opportunities to actually move things forward.
Linkpost from the HLI blog
Major points
I’m writing from the perspective of the Happier Lives Institute. We’re delighted that HLI’s work and the subjective wellbeing approach featured so prominently in this podcast. It was a really high-quality conversation, and kudos to host Rob Wiblin for doing such an excellent job putting forward our point of view. Quite a few people have asked us what we thought, so I’ve written up some comments.
I split those into four main comments and a number of minor ones. To preview, the main comments are:
We’re delighted, but surprised, to hear GiveWell are now so positive about the SWB approach; we’re curious to know what changed their mind.
HLI and GiveWell disagree on what does the most good based on differences of how to interpret the evidence; we’d be open to an ‘adversarial collaboration’ to see if we can iron out those differences.
We’d love to do more research, but we’re currently funding constrained. If you – GiveWell or anyone else – want to see it, please consider supporting us!
Finally, Rob, it’s about time you had us on the podcast!
Main points
1. We’re delighted, but surprised, to hear GiveWell are now so positive about the SWB approach; we’re curious to know what changed their mind.
Elie Hassenfeld says the differences in opinion between HLI and GiveWell aren’t because HLI cares about SWB and GiveWell does not, but down to differences of opinion interpreting the data[1]. This is great news – we’re glad to see major decision-makers like GiveWell taking happiness seriously – but it is also news to us!
Listeners of the podcast may not know this, but I (as a PhD student) and then HLI have been publicly advocating for SWB since about 2017 (e.g., 1, 2). I/we have also done this privately, with a number of organisations, including GiveWell, who I spoke to about once a year. Whilst lots of people were sympathetic, I could not seem to interest the various GiveWell staff I talked to. That’s why I was surprised when earlier this year, GiveWell made its first public written comment on SWB and was tentatively in favour; Elie’s podcast this week seemed more positive.
So, we’re curious to know how thinking in GiveWell changed on this. This is of interest to us but I’m sure others would like to know how change happens inside large organisations.
2. HLI and GiveWell disagree on what does the most good based on differences of how to interpret the evidence; we’d be open to an ‘adversarial collaboration’ to see if we can iron out those differences.
Elie explained that the reason GiveWell doesn’t recommend StrongMinds[2] which HLI does recommend, is due to differences in the interpretation of the empirical data. Effectively, what GiveWell did was look at our numbers, then apply some subjective adjustments on factors they thought were off. We previously wrote a long response to GiveWell’s assessment and don’t want to get stuck into all those weeds here. Elie says – and we agree! – that reasonable people can really disagree on how to interpret the evidence. That’s why we’d be interested in an ‘adversarial collaboration’ to see if we can resolve them. I can see three areas of disagreement.
First, on the general theoretical issue of whether and how to make subjective adjustments to evidence. GiveWell are prepared to make adjustments, even if they’re not sure exactly how big they should be. For example, Elie says he’s unsure about the 20% reduction for ‘experimenter demand effect’. Our current view is to be very reluctant to make adjustments without clear evidence of what size is justified. Our reluctance is motivated by cases such as this: these ‘GiveWellian haircuts’ can really add up and change the results, so the conclusion ends up being more on the researcher’s interpretation of the evidence than the evidence itself. But we’re not sure how to think about it either!
A second, potentially more tractable issue, is clarifying what evidence would change our mind about the specific issues. For instance, if there was a well conducted RCT that directly estimated the household spillover effects of psychotherapy in a setting similar to StrongMinds, we’d likely largely adopt that estimate.
A third issue is deworming. Elie and Rob discuss HLI’s reassessment of GiveWell’s deworming numbers, for which GiveWell very generously awarded us a prize. However, GiveWell haven’t commented – on the podcast or elsewhere – on our follow-up work, which finds the available evidence suggests that there are no statistically significant long-term effects of deworming on SWB. This uses the exact same studies that GiveWell relies on; it indicates that GiveWell aren’t as bought into about SWB as Elie sounds or they haven’t integrated this evidence yet.
3. We’d love to do more research, but we’re currently funding constrained. If you – GiveWell or anyone else – want to see it, please consider supporting us!
As a small organisation – we’re just 4 researchers – we were pleased to see our ideas and work is influencing the wider discussion about how to do have the biggest impact. As the podcast highlights, HLI brings a different (reasonable) perspective, we have provided important checks on other’s work, we provide unique expertise in philosophy and wellbeing measurement, and we have managed to push difficult issues up to the top of the agenda[3], [4], [5]. We think we ‘punch above our weight’.
Elie mentions a number of areas where he’d love to see more research including in SWB and the difficult question of how to put numbers on the value of saving a life. We think we’d be very well placed to do this work, for the reasons given above; we’re not sure anyone else will do it, either (we understand GiveWell don’t have immediate plans, for instance). However, we don’t have the capacity to do more and we can’t expand due to funding constraints. So, we’d love donors to step forward and support us!
Of course we’re biased, but we believe we’re a very high leverage, cost-effective funding opportunity for donors who want to see top-quality research that changes the paradigm on global wellbeing and how to do the most good. Please donate here or get in touch at Michael@happierlivesinstitute.org. We’re currently finalising our research agenda for 2023-4 (available on request).
4. Finally, Rob, it’s about time you had us on the podcast!
We’ve got much more to say about topics covered, plus other issues besides: longtermism, moral uncertainty, etc. (Rob has said we’re on the list, but it might take a while because of the whole AI thing that’s been blowing up; which seems fair).
“I think ultimately what it comes down to is we have a different interpretation of the empirical data — meaning we look at the same empirical data and reach different conclusions about what it means for the likely impact of the programme in the real world.”
An organisation that treats depression at scale and is currently our top recommendation.
“…I think one of the things that HLI has done effectively is just ensure that [tradeoffs between saving and extending lives] is on people’s minds. I mean, without a doubt their work has caused us to engage with it more than we otherwise might have. Similar to some of the questions you were asking earlier, like, “Why doesn’t institution X see that it should do whatever?” Well, because it’s kind of hard, and sometimes you need another organisation to be pushing it in front of you. I think that’s really good that they’ve done that, because it’s clearly an important area that we want to learn more about, and I think could eventually be more supportive of in the future.”
“Yeah, they went extremely deep on our deworming cost-effectiveness analysis and pointed out an issue that we had glossed over, where the effect of the deworming treatment degrades over time. We had seen that degrading, and the way we had treated it, I should say, was that that’s just a noisy estimate, and we just took the average estimate persisting over the long run. Their critique convinced us that we should at least incorporate some probability that the effect is degrading into our overall model, and that shifted our overall assessment of deworming down by a small amount. Had we taken their correction on board in the past, it would have meant a few million dollars that we would have given elsewhere instead of deworming. Their published critique, I think we didn’t agree with the headline result that they reached, but we were really grateful for that critique, and I thought it catalysed us to launch this Change Our Mind Contest. And also it was a great example of the engagement that we’re getting from being transparent. That we can say, “Here’s our decisions, here’s why they could point to an error, and it changes our mind.” That was really cool, and we were really grateful for it.”
“I think the pro of subjective wellbeing measures is that it’s one more angle to use to look at the effectiveness of a programme. It seems to me it’s an important one, and I would like us to take it into consideration.”
Linkpost from the HLI blog
Minor points
These respond to bits of the discussion in the order they happened.
1. On the meaning of SWB
Rob and Elie jump into discussing SWB without really defining it. Subjective wellbeing is an umbrella term that refers to self-assessments of life. It’s often broken down into various elements, each of which can be measured separately. There are (1) experiential measures, how you feel during your life – this is closest to ‘happiness’ in the ordinary use of the word; (2) evaluative measure are an assess of life a a whole; the most common is life satisfaction; (3) ‘eudaimonic’ measures of how meaningful life is. These can give quite similar answers. As an example, see the image below for what the UK’s Office of National Statistics asks (which it does to about 300,000 people each year!) and the answers people give.
2. On the wellbeing life-year, aka the WELLBY
The way the discussion is framed, you might think HLI invented the WELLBY, or we’re the only people using it. That gives us too much credit: we didn’t and we’re not! Research into subjective wellbeing – happiness – using surveys has been going on for decades now and we’re making use of that. The idea of the WELLBY isn’t particularly radical – it’s a natural evolution of QALYs and DALYs – although the first use seems of the term seems have only been in 2015 (1, 2). The UK government has had WELLBYs as a part of their official toolkit for policy appraisal since 2021.
It is true that HLI is one of the first (if not the first) organisations to actually try to do WELLBY cost-effectiveness; although the UK government has this ‘on the books’, our understanding is it’s not being implemented yet.
3. Is SWB ‘squishy’ and hard to measure?
Elie: I think the downside [of measuring SWB], or the reasons not to, might be that on one level, I think it can just be harder to measure. A death is very straightforward: we know what has happened. And the measures of subjective wellbeing are squishier in ways that it makes it harder to really know what it is
As noted, there are different components of SWB: happiness is not the same thing as life satisfaction. I don’t think either of these are that squishy or that we don’t know what they are; they are different things. I don’t think measuring them isn’t hard: you can just ask “how happy are you, 0-10” or “how satisfied are you with your life nowadays, 0-10”! People find it easier to answer questions about their SWB than their income, if you look at non-response rates (OECD 2013). Of course, they are a measure of something subjective, but that’s the whole point. I don’t know how happy you feel: you need to tell me!
4. Does anyone’s view of the good not include SWB?
Elie: I think some people might say, “I really value reducing suffering and therefore I choose subjective wellbeing.” I think other people might say, “I think these measures are telling me something that is not part of my ‘view of the good,’ and I don’t want to support that.”
What constitutes someone’s wellbeing, that is, ultimately makes their life go well for them? In philosophy, there are three standard answers (see our explainer here). What matters is (1) feeling good – happiness, (2) having your desires met – life satisfaction, roughly – or (3) objective goods, such as knowledge, beauty, justice, etc, plus maybe (1) and/or (2). It would be a pretty wild view of wellbeing where people’s subjective experience of life didn’t matter at all, or in any way! It might not be all that matters, but that’s another thing.
5. On organisational comparative advantage
Elie: Because we’re not trying to add value by being particularly good philosophically. That’s not part of GiveWell’s comparative advantage.
If I can be forgiven for tooting our horn, I do see HLI’s comparative advantage as being particularly philosophically rigorous, as well as really understanding wellbeing science (I’m a philosopher; the other researchers are social scientists). We’re certainly much less experienced than GiveWell at understanding how well organisations implement their programmes.
6. On moral weights
Elie: I think this is an area — moral weights — where I don’t feel the same way. I don’t think this is a mature part of GiveWell. Instead, this is a part of GiveWell that has a huge amount of room for improvement
We could be talked into helping with this! In 2020 we explored how SWB would change GiveWell’s moral weights (GiveWell didn’t respond at the time). We subsequently have been doing more work on how to compare life-saving to life-improving interventions, including a survey looking at the ‘neutral point’ and other issues about interpreting SWB data.
7. On the challenges of using SWB given data limitations
Rob Wilbin: So just the number of studies that you can draw on if you’re strictly only going to consider subjective wellbeing is much lower.
I think another thing that really bites is that subjective wellbeing outcomes are really at the end of the chain of all of these different factors about someone’s life — their income, their health, their education, their relationships, all of these different factors.
We see data limitations as the biggest single challenge for SWB. The sort of data we’d want is scarce in low-income countries. We’ve started to talk to organisation working in LIC and to encourage them to collect data. Our capacity to do this is, however, very limited, but we would expand it if we had the resources. We found enough to compare mental health to cash transfers etc. (and even then we had to convert from mental health scores to SWB scores) but we expect to find much less data to look at other interventions.
On the complex causal chain, part of the virtue of SWB is that, if you have the SWB data, you don’t need to guess at how much all the different changes an intervention makes to someone’s life affects their wellbeing: you can just look at their answers, and they tell you! Take an education programme. It might change someone’s life in all sorts of ways. But if you’ve done an RCT and measured SWB, you can see the impact without needing to identify where it came from.
Elie: Maybe there’s reason to give credence to the measures that are easier to deal with and easier to know that you’ve done something good and made someone’s life better.
All this raises a good question: what should we do if we don’t have the data we want? As Daniel Kahneman reminds us, we should distinguish two concerns about measurement (and judgement). One is noise, the other is bias.
From Kahneman et al. (2016).
One way to put Elie and Rob’s concern is that SWB is a noisy measure. Now, if you have loads of data, you should use a noisy measure over a biased one because all the noise will average out. However, if you don’t have much data, and you have a choice between (C on the figure) a non-noisy, biased measure or (B on the figure) a noisy, non-biased one, you could sensibly conclude you’d get closer to the bull’s eye with (B).
Here’s a hypothetical example that brings this thought. Imagine we’re evaluating a new intervention. There is a n = 20,000 RCT that shows it doubles income for pennies. If we convert the income effects to WELLBYs it’s much more cost-effective than StrongMinds. But there’s an n = 100 RCT with SWB that shows it has a slightly negative but very very imprecisely measured effect on SWB. In this case, I think we’d mostly go with the income evidence (more technically: we’d combine the uncertainty of the income estimate with the conversion to SWB estimate and then combine both the income-converted and the SWB evidence in a bayesian manner).
But the issue is this. How do you know how biased your measures are? You need to establish bias by reference to a ‘ground truth’ of accuracy – how far are they from the bull’s eye? I’d argue that, when it comes to measuring impact, SWB data is the least-bad ground truth: you learn how important income, unemployment, health etc are by seeing their relationship to SWB. Hence, in the above example, I’d be inclined to go with the income data because there’s so much evidence already income does improve SWB! If the full sweep of available data showed income had no effect, I wouldn’t conclude that the income data from the hypothetical example was evidence the programme was effective. Of course there will be gaps in our evidence, and sometimes we have to guess, but we should try to avoid doing that.
8. On the tricky issue of the value of saving a life
It’s too long to quote in full, but I don’t think Rob or Elie quite captured what HLI’s view is on these issues, so let me try here. The main points are these; see our report here.
(1) comparing quality to quantity of life is difficult and complicated; there isn’t just ‘one simple way’ to do it. There are a couple of key issues which are discussed in the philosophical literature which haven’t, for whatever reason, made it into the discussions by GiveWell, effective altruists, etc.
(2) One of these issues, the one Elie and Rob focus on, is the ‘neutral point’: where on a 0-10 life satisfaction should count as equivalent to non-existence? We think it’s not obvious and merits further research. So far, there’s been basically no work on this, which is why we’ve been looking into it.
(3) how you answer these philosophical questions and make quite a big difference to the relative priorities of life-saving vs life-improving interventions. We got into that in a big report we did at the end of 2022, where we found that going from one extreme of ‘reasonable’ opinion to the other changes the cost-effectiveness of AMF by about 10 times.
(4) HLI doesn’t have a ‘house view’ on these issues and, if possible, we’ll avoid taking one! We think that’s for donors to decide.
(5) GiveWell does take a ‘house view’ on how to make this comparison. We’ve pointed out that GiveWell’s view is at the ‘most favourable to saving lives, least favourable to improving lives’ end of the spectrum, and that (on our estimates) treating depression does good more if you hold even slightly less favourable assumptions. This shouldn’t really need saying, but philosophy matters!