What do DALYs capture?
INTRODUCTION
There are many different causes that require our attention, but because our resources are limited, we need to decide which ones should go first. Within health, we use Health-Adjusted Life Years (HALYs) to help us decide which interventions to prioritize. Health is not the only determinant of wellbeing we care about. There may be value in building broader metrics that also encompass some of the other factors, but health is definitely an important one, so that is why it is be the focus of this article.
HALYs capture morbidity and mortality: morbidity is how life with that disease compares to life in full health (the amount of life years left weighted by the severity of the disease or “Years of Life Lived with Disability“); and mortality is the number of years by which the patient’s life has been shortened because of the disease, taking life expectancy as a reference (“Years of Life Lost”).
Two of the most widely used types of HALYS are Quality-Adjusted Life Years (QALYs), and Disability-Adjusted Life Years (DALYs). They are conceptually very similar, but QALYs capture the benefits of health interventions, and hence we want as many of them as possible, whereas DALYs capture the losses caused by a health state, so we want to minimize them [1]. QALYs are more widely used, but DALYs are more relevant here because they are the ones used in development, and hence we will focus on them. Here, ‘disability weights’ will be used as a synonym for DALYs.
There are several methods to elicit disability weights (e.g., standard gamble, visual analogue scale, person trade-off), but the most popular is the Time Trade-Off (TTO). Respondents are asked to think how many years in full health (x) are equivalent to a longer time (t) in a poor health state. Utility of full health is assigned to be 1, and the utility of the poor health state is then x/t. These questions are posed either to the members of the general population or to experts, when the former is not possible. Health states are described using instruments such as the EQ-5D (other examples include the SF-6D, and Health Utilities Index (HUI)). EQ-5D uses five dimensions to describe health states (mobility, self-care, usual activities, pain/discomfort, and anxiety/depression). Each dimension has three levels ((1) no problems, (2) some problems, (3) extreme problems). The digits for the five dimensions make up the score that describes the health state. For instance, the best possible health state would be represented by “11111”.
DALYs are useful in that they help us make comparisons across health interventions, but they have important limitations too. Before we make decisions based on them, we should make sure that we also understand what they they may be misrepresenting or not capturing at all:
-
As a result of the elicitation process, DALYs may misrepresent the relative importance of mental compared to physical health.
-
DALYs do not capture indirect effects of health interventions, and thus they could be missing a very important part of the picture.
1) DALYs MISREPRESENT MENTAL HEALTH
We may miss an opportunity for increasing people’s wellbeing if we do not think critically about how well DALYs capture the prevalence and impact of mental health with respect to physical health. There are two factors that contribute towards this misrepresentation: first, the types of questions people are asked; and second, the answers they give.
The National Institute for Health Care Excellence (NICE) and other agencies recommend using EQ-5D as the instrument to elicit people’s preferences over health states. Its dimension composition may not be a good reflection of what actually matters to people (Dolan, 2011). In particular, the fact that only 1 out of its 5 dimensions is explicitly about mental health, and that anxiety and depression are pooled together into one item. When we ask people about health with preference-based methods, we get one answer (“physical functioning and pain matter as much to people, and sometimes more, as mental health when they are asked to risk death or trade off life years”), whereas when we ask them directly about what we are interested in, their happiness, the picture we obtain is different (“mental health and vitality appear to be most strongly associated with happiness, whilst physical functioning and pain are not so strongly associated with happiness”). In short, “the dimensions of health privileged by the EQ-5D and SF-6D may not be those that most affect people’s lives” (direct quotes from Dolan, 2011).
The second factor is linked to the elicitation of disability weights. In order to estimate DALYs, we survey people and ask them to predict how different health states would be. This prediction is susceptible to affective forecasting errors, which affect the evaluation of physical and mental health differently.
The focusing illusion makes people give more weight in their judgement to attributes that are more notable. When people think of their lives in their current health state and compare them with a life with an illness with salient physical symptoms, the ways in which these symptoms would affect their lives are easier to think of than they would be if they had a mental illness. This makes that physical health problems are judged to be worse than they actually are, and mental health problems are judged to be less bad than they actually are. Despite of people’s predictions, there is evidence asymptomatic conditions such as hypertension are correlated with less happiness (Blanchflower & Oswald, 2008).
The impact bias makes people overestimate the length and intensity of future emotional states, and so they exaggerate how bad it will be in a certain health state.
Simultaneously, they ignore that after a while, their happiness levels will go back to their pre-condition levels; this is known as hedonic adaptation. Dolan (2011) reviews evidence on this phenomenon and quotes a study by Hurst and colleagues (1994) where they found that people with either chronic health conditions or a physical disability showed “considerable levels of adaptation to these conditions”.
However, mental health conditions are among the most resistant to adaptation. Dolan and Kahneman (2008) attribute this to the fact that these kinds of conditions are “part-time experiences”, in that they only affect wellbeing when attention is drawn to the limitation they impose, whereas mental health problems are “full-time in their attention seeking and impact on our lives”.
In addition to this, people also underestimate how much, after becoming physically or functionally disabled, they would adapt (“learning and acquiring new skills in order to regain functionality”), cope (“adjusting your expectations about your performance to reduce the gap between expected and actual functionality”), and adjust (“changing one’s life plans so that those dimensions that are not affected by the disability become more important”) (Solomon & Murray, 2002, referenced by Brock & Wikler, 2006).
Because of all of this, assessments of physical health issues may be overstating how bad their impact on wellbeing is, compared to mental health issues, and so more resources will be devoted to their treatment and prevention, while mental illnesses may be under-catered for.
2) DALYs MISS INDIRECT EFFECTS
DALYs capture the direct health loss of caused by a given disease, but they may be underestimating its overall detrimental effects because they don’t account for indirect effects. If we care about the effect of health interventions to the broader society, then DALYs, which focus on the effect to the individual, may not be giving us an accurate picture. Accounting for indirect effects may change the picture of which health interventions should be prioritized.
In addition to the actual disease symptoms, other health problems may be alleviated too if the disease is treated. For instance, Miller, Paschall, and Svendsen (2008) found evidence that patients with co-morbidities that involve severe mental illnesses and another condition (such as heart disease) experience higher mortality ratios than their counterparts without the co-morbidities. Hence, treating one of those diseases could make the other one less bad. Also, the effects of some illnesses, with time, could also cascade and affect other dimensions of patients’ health, increasing its negative consequences. For instance, losing some physical functionality could impact vitality (these dimensions are part of the SF-6D instrument).
Diseases can also impact the health, lifestyle, or economic prospects of people around the patient. If the disease is transmittable, not treating it increases the chances that more people will get the disease, and that would multiply its negative effects. Severe illnesses, such as Alzheimer’s, can significantly alter the patient’s family and friends’ lifestyles (Dolan, 2011). Also, when patients do not survive the disease, this causes a great amount of pain and suffering to the people who knew them.
There are four ways through which improved health fosters economic development (World Bank, 1993). The first one has to do with opportunity costs: better health frees up the resources that would otherwise have been used to care for the patient. Second, better health translates into gains in worker productivity, who also miss less work days, and have increased chances of obtaining better-paying jobs. Third, when some diseases are controlled, people can exploit natural resources that were inaccessible beforehand. This was the case for some areas of Sri Lanka when malaria was tackled, and Uganda when river blindness was fought with insecticides and medication. Last, better health is translated into economic gains through education: school enrollment, ability to learn, and participation by girls will be higher.
These indirect effects vary across regions with economic, ethical, cultural and social differences. For example, being blind in countries like Niger will impair your ability to make a living, and that could lead to malnutrition, and premature death. In the UK, on the other side, the first years may be difficult, but after that it would not affect other areas of your health or have such an impact in your life as it would in Niger.
Not accounting for these differences make that DALYs underweight health losses in poor countries. First, for the same health intervention, people in poorer countries have more potential to benefit from the indirect effects. This is because “they are typically most handicapped by ill health and [they are the ones] who stand to gain the most from the development of underutilized natural resources” (World Bank, 1993). Second, if the intervention is not implemented, they are also the ones that have more to lose, as their income is mostly dependent on physical labour rather than cognitive abilities, and often they do not have a savings safety net to fall back on. And third, indirect health negative consequences are larger for them too: “when a family’s breadwinner becomes ill, other members of the household may at first cope by working harder themselves and by reducing consumption, perhaps even of food. Both adjustments can harm the health of the whole family”.
OTHER ISSUES
The reason why DALYs are estimated by surveying members of the general population is that they are intended to reflect their preferences. However, DALYs have been criticised because they capture the benefit of health interventions but disregard how they are distributed across the affected population, which is something most people care about. Focusing on maximizing health in the aggregate but disregarding equity concerns can lead to distributions that look unacceptable to most people. An example of this is the Oregon case (Brock & Wikler, 2003), where treating a very prevalent but low impact condition (performing 150 teeth capping) was seen as more valuable than giving an appendectomy, which is a life-saving intervention with a great impact to the person who receives it.
QALYs and DALYs are slightly different in this. QALYs do not give preferential treatment to anyone depending on the severity of their illness or personal characteristics (such as age, sex, level of deprivation, or their role in society, and others). This, known as QALY egalitarianism, is considered to be fair because everyone gets the same opportunities. Distributing QALYs according to this principle can lead to QALY losses for some, but as long as they are compensated by QALY gains for others, there will be a net efficiency gain and society as a whole will be better off (Whitehead & Ali, 2010). DALYs, on the other side, do favour people in some age groups by applying age discounting.
In the 2006 edition of the Disease Control Priorities (DCP) report (Jamison et al., 2006), the age weights were “zero at birth, ignoring health losses from still birth prior to live birth; reach a maximum at age 25; and decline almost to zero at advanced age”. In the 2013 edition, which is the latest revision of DALYs, constant age weighting (treating all years alike) was used.
There is little evidence that one way of discounting is better than the other one, but some people argue in favour of having some kind of discounting for the following two reasons. First, to account for the fact that quality of life may depend on age. Second, to reflect the effect of health improvements on others. In particular, the fact that individuals in their productive years usually have young and/or elderly people that depend on them emotionally, physically, and financially. This argument has been criticised because it discriminates individuals depending in their social and economic value to others. This criterion is not linked to health, and also, it would justify outcomes that most people would consider unfair. For instance, it would justify that between a rich and a poor patient with the same medical needs, treating the rich was prioritized because they are more socially productive than the poor.
Another way of incorporating distributional concerns into DALYs would be to use time discounting. This would make benefits in the future less attractive and so it would give an advantage to ‘present patients’ over ‘future patients’. The first argument in favour of doing this is consistency (treating benefits in the same way that we treat costs). Discounting is also supported in order to reflect general uncertainty about the future, opportunity costs, negative health effects that could cascade if the patients are not treated immediately, and people’s time preferences (this argument has been contested by evidence of how time preferences vary depending on the elicitation method (Frederick, 2003), and the implications that discounting would have on our preference of the past over the present – i.e., “discounting time at a 1% rate […] a single day of Tutankhamen’s life would have been more valuable than the entire lives of all 7,000,000,000 humans alive today put together” (Ord and Wiblin, n.d.). And finally, discounting would avoid paradoxes such as the Keeler and Cretin Paradox (1983) [2] and the infinite benefit of eradicating diseases, which would justify any finite cost [3].
The main criticism to discounting is that it violates intergenerational justice. Is it ethical to confer less value to increasing someone’s wellbeing just because it happens in the distant future rather than now? Another argument against discounting is that it systematically disadvantages programs with benefits that take time to be accrued (such as vaccination programs or unhealthy behaviour change – i.e., start exercising now to not to get a coronary disease later on). And last, there is a concern that applying a discount factor would be double-discounting, given that some of the elicitation methods (TTO, for example) are already capturing at least some of these uncertainties.
The above is not an exhaustive discussion of all the criticism to DALYs, but is intended to give an overview of some of the points that are currently being debated. Alternative approaches to value health such as using wellbeing measures have been suggested as a solution to some of these problems.
CONCLUSION
DALYs measure health. But they miss, or misjudge, some important factors. First, DALYs are biased towards physical health. The instruments used for eliciting them and affective forecasting errors cause mental health to be underrepresented. Second, DALYs fail to capture various indirect effects. These include indirect health consequences for the patient, consequences for people around them, and economic impacts. Some of these effects have a stronger effect in poorer countries, and that is also unaccounted, biasing DALYs towards richer countries. Alternative ways of valuing health (e.g., using wellbeing measures) are currently being explored.
[1] There are two other main differences between QALYs and DALYs. First, QALYs describe health states in terms of a few dimensions and DALYs describe specific diseases. This implies that QALYs can account for co-morbidities but DALYs cannot. Second, DALYs incorporate age discounting, but for QALYs that would need to be done in an additional step. DALYs assign a different value to a year of life extension of the same quality, depending on the age at which an individual receives it; specifically, life extension for individuals during their adult productive work years is assigned greater value than a similar period of life extension for infants and young children or the elderly (Brock & Wikler, 2006).
[2] “If you were faced with the choice of spending X dollars now to achieve a certain health benefit, or investing it and spending it a year later, you should invest it because a year from now you’ll have more money to spend and can achieve a greater benefit. But then why not delay two years, etc? The paradox is that infinite delay is called for by this logic. Discounting of future health benefits potentially solves the problem. You have more money to spend, but if future health benefits are valued less, you aren’t necessarily getting more for your dollar by delaying.”
[3] Ord and Wiblin say that this would technically only be true if humanity was expected to survive until infinity and never to come up with an alternative cure for smallpox. “A more realistic benefit appraisal of this situation is that the vaccine would contribute to eradicate it earlier, rather than preventing it to be “a menace for billions of years”.
REFERENCES
Blanchflower, D. G., & Oswald, A. J. (2008). Hypertension and happiness across nations. Journal of health economics, 27(2), 218-233.
Brazier, J., & Tsuchiya, A. (2015). Improving cross-sector comparisons: going beyond the health-related QALY. Applied health economics and health policy, 13(6), 557-565.
Brock, D., & Wikler, D. (2006). Ethical issues in resource allocation, research, and new product development. Disease control priorities in developing countries, 2, 259-60.
Dolan, P. (2011). Using happiness to value health. London: Office of Health Economics.
Dolan, P., & Kahneman, D. (2008). Interpretations of utility and their implications for the valuation of health. The Economic Journal, 118(525), 215-234.
Frederick, S. (2003). Measuring intergenerational time preference: Are future lives valued less?. Journal of Risk and Uncertainty, 26(1), 39-53.
Hurst, N.P., Jobanputra, M., Hunter, M., Lambert, M., Lockhead, A. and Brown, H. (1994) Validity of EuroQol—a generic health status instrument—in patients with rheumatoid arthritis. Rheumatology. 33(7), 655–662.
Jamison, D. T.; Breman, J. G.; Measham, A. R.; Alleyne, G.; Claeson, M.; Evans, D. B.; Jha, P.; Mills, A.; Musgrove, P. (2006). Disease Control Priorities in Developing Countries, Second Edition. Washington, DC: World Bank and Oxford University Press. Retrieved from: https://openknowledge.worldbank.org/handle/10986/7242
Keeler, E. B., & Cretin, S. (1983). Discounting of life-saving and other nonmonetary effects. Management science, 29(3), 300-306.
Miller, B. J., Paschall III, C. B., & Svendsen, D. P. (2008). Mortality and medical comorbidity among patients with serious mental illness. Focus, 6(2), 239-245.
Ord, T., & Wiblin, R. (n.d.) Should we discount future health benefits when considering cost-effectiveness? Retrieved from: https://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/discounting-health2.pdf
Solomon, J. A., & Murray, C. J. L. (2002). A conceptual framework for understanding adaptation, coping and adjustment in health state valuations. Summary Measures of Population Health, 11.
Whitehead, S. J., & Ali, S. (2010). Health outcomes in economic evaluation: the QALY and utilities. British medical bulletin, 96(1), 5-21.
World Bank (1993). World Development Report 1993: Investing in Health. New York: Oxford University Press. Retrieved from https://openknowledge.worldbank.org/handle/10986/5976
I only skimmed this, but I think the majority of EAs don’t actually look into the how and why of GiveWell’s recommendations. And even less go into the processes and publications that lead to the numbers that GiveWell eventually uses. An indirect result is that GiveWell doesn’t get as much feedback as it could likely benefit from, and too many EAs can’t speak to M&E professionals in international development at a meaningful level.
What’s explained here, and alluded to here, as well as the criticisms, is important basic info for many EAs who are unfamiliar with it. The various methodologies for costing and discounting (both included here and others), in particular, are definitely worth investigating further for those who haven’t.
Thanks for posting this, it’s a really thorough write up of the issue.
I wrote a bit about this about a year ago where I argued effective altruism is overlooking happiness and I’m pleased to see you reached the same conclusions (I also found the the Dolan 2011 paper very persuasive)! I think your analysis was 1. much more substantial than mine and 2. didn’t hide the information in an additional document people had to go find (on reflection, that was a mistake on my part). Where I think this criticism of DALYs leads us, in our quest to do the most good, is towards mental health as a substantial cause area and away from physical health.
As a separate point: this post does raise the more general worry about the effectiveness with which information gets shared in EA circles. I’d looked into this before and there’s some duplication of effort here: if I’d found a way to make my research better known, the author might have researched something else instead. To be clear, I mean this in no way as a criticism of the author, I think it’s just unfortunate. It’s not the first time I’ve come across this phenomenon in the EA world either and I may make a post on the general problem soon.
Thanks for your comments, Michael!
You raise a good point. I think a possible way of making communication better may be by labelling posts with keywords? That way, everyone could find very easily everything that has been posted about a topic. I am not sure that would have helped in this case, though. My goal with this post was to be comprehensive about what is not so good about DALYs. I did as much reading as I could and wrote more extensively about the factors that seemed more important. Mental health came up as one of these factors (I read your article and I was persuaded, so I decided to dig deeper), so I did not feel that I could leave it out, even if there was already something written about it. You may be right in that this may have been inefficient, and in the future, I think referencing posts rather than writing new ones may be a better alternative. Something definitely worth keeping in mind.
Great post!
Nitpick:
I think “11111” usually refers to full health. (cf. the “EQ-5D Value Sets: Inventory, Comparative Review and User Guide” by Szende, Oppe & Devlin, 2007).
As part of a bigger project on descriptive (population) ethics, I’ve been working on a literature review of health economics. It also contains a section on the EQ-5D and its weaknesses. Here some excerpts:
(Incorporating the TTO lead-time approach can easily overcome this problem.)
Anway, you write:
I couldn’t agree more.
IMHO, another big problem is the evaluation of states worse than death (SWD) (and states of severe mental illness such as depression arguably belong in this category). For example, most studies don’t even allow for SWD assessments. Furthermore, most researchers transform negative evaluations, limiting them to a lower bound of −1. Assuming that people with a history of mental illness more often evaluate health states indicating severe mental illness as highly negative (i.e. give utilities as lower than −1), then this ex-post transformation causes their judgments to have less influence than the judgments of uninformed people who underestimate the severity of mental illness.
I discuss this problem, as well as other problems, in much greater detail in my doc.
I plan on publishing the doc within the next months, but if you’re interested I’m happy to send you a link to the current version.
Some nitpicks in turn!
I don’t think this follows. If these states are impossible (I don’t disagree) then they’ll never come in real life, so it won’t matter what people say in the TTOs. As long as people make sensible judgements about the health states that actually occur, it doesn’t matter what they say in impossible ones. I think you should push the fact they don’t make sensible judgements in general - affective forecasting stuff, etc.
Curious. Hmm. IIRC, DALYs and QALYs don’t have a neutral point: 1 is healthy, 0 is dead, but it’s not specified where between 0 and 1 is neutral. Is neutral 0.5? 0? Unless you know where neutral is you can’t specify the minimum point on the scale, because it doesn’t make sense.
What would −1 mean here? DALYs and QALYs aren’t well-being scales and can’t straightforwardly be interpreted as such.
Good point. But I wonder whether they reinterpret the meanings of some of the dimensions of the ED-Q5 in order to make sense of some of the health states they are asked to rate.
Agree.
This depends on the study. I’m afraid it will take me a couple of paragraphs to explain the methodology, but I hope you’ll bear with me :)
The literature review by Tilling et al. (2010) concluded that only 8% of all TTO studies even allow for subjects to rate health states as worse than death (i.e. as below 0), so for the vast majority of studies, the minimum point on the scale is indeed 0. I think this is problematic since e.g. health states like 33333 (if they are permanent) are probably worse than death for many, maybe even most people.
Of the few TTO studies that allow for negative values, the protocols by Torrence et al. (1982) and Dolan (1997) are used by almost all of them. Below a quote by Tilling et al. (2010), describing these two methods:
These two TTO protocols, in theory, would allow for extremely negative (and even infinite) negative values. Tilling et al. (2010) explain:
How do researchers respond? Again, I’ll quote Tilling et al. (2010, emphasis mine):
In the two most commonly used TTO protocols, the smallest unit of time the TTO procedure iterates toward for SWD is 1 year. Consequently, the lower bound is −9. (Sometimes, the smallest united of time is 3 months, so the lowest possible value is −39.)
To give a concrete example: The subject is indifferent between A) living for 2 years in full health and for 8 years in health state 33333 and B) dying immediately. Thus, the value for health state 33333, for this subject, is − 8⁄2 = − 4.
Now almost all researchers then transform these values, such that the lowest possible value is −1. In my view, this is somewhat arbitrary.
Below some quotes by Devlin et al. (2011) on the matter:
...
And here another quote by Tilling et al (2010):
I hope this explains my previous comment.
References:
Devlin, N. J., Tsuchiya, A., Buckingham, K., & Tilling, C. (2011, 02). A uniform time trade off method for states better and worse than dead: Feasibility study of the ‘lead time’ approach. Health Economics, 20(3), 348-361.
Dolan, P. (1997). Modeling Valuations for EuroQol Health States. Medical Care, 35(11), 1095-1108.
Tilling, C., Devlin, N., Tsuchiya, A., & Buckingham, K. (2010, 09). Protocols for Time Tradeoff Valuations of Health States Worse than Dead: A Literature Review. Medical Decision Making, 30(5), 610-619.
Torrance, G. W., Boyle, M. H., & Horwood, S. P. (1982, 12). Application of Multi-Attribute Utility Theory to Measure Social Preferences for Health States. Operations Research, 30(6), 1043-1069.
Categorizing quality of life based on personal testimony is a challenging task. The reasons you listed show many specific problems, and more generally, human judgement is fickle and error-prone. For instance, Thinking Fast and Slow claims that we are loss-averse and that we overweight the cost of losing something. I wonder, then, how the responses of perceived quality of life differ between people who were born with particular illnesses (like blindness) and people that suffered from it later in life.
The inherent fallacies in human judgement cause me to wonder if it can ever be a reliable source to quantify the effect of illnesses. At the risk of being hyper-pragmatic, perhaps we should attempt to quantify the effect of illnesses by only considering the degree to which the illness impacts a person’s ability to provide useful social function.
Of course, this approach also has many inherent issues. For one, meaningfully quantifying this would be incredibly challenging if not infeasible. It would also likely weight the value of the rich much higher than the value of the poor.
If you don’t think you can quantify QoL by self-reports, I’m not sure how you’re going to be able to quantify useful social functions instead!
FWIW, measuring happiness turns out to be basically fine. You might like this article on the topic which discusses it: http://journals.sagepub.com/doi/10.1111/j.1745-6916.2007.00030.x
Maybe a relevant post from the past.
http://effective-altruism.com/ea/pu/we_care_about_walys_not_qalys/
“QALYs only measure health, and health is not all that matters. Most effective altruists care about increasing the number of “WALYs” or well-being adjusted life years, where health is just one component of wellbeing.”
Keep in mind that there are some differences between DALYs and QALYs, for example see the discussion in https://academic.oup.com/heapol/article/21/5/402/578296/Calculating-QALYs-comparing-QALY-and-DALY