If you’re talking to two people, one with a small cut and another with multiple sclerosis, everyone present will agree that having multiple sclerosis is much worse. If you offered the two of them some magic option that would restore exactly one of them to full health, they would probably be able to agree on who should get it. But in general, how should we compare across people to figure out whose situation is worse, who would benefit more from treatment and who, everything else being equal, should be treated first? This example was easy because the difference was nice and large, but what do we do in harder cases?
One is to ask people questions like “if there were a surgery that could restore you to full health (without improving your lifespan) but had a 20% chance of killing you, would you take it?” If they say “yes” then this indicates that this disability, for this person, is more than 20% as bad as being dead. Ask these “standard gamble” questions to a lot of people with a lot of disabilities, varying the percentages, and you could build up a list of how bad different ones are, all on a common scale.
This would useful for balancing projects against each other, figuring out what to focus on, and generally setting funding priorities. Unfortunately people are really bad at answering questions like this. Mostly we’re just bad at thinking about percentages and chances of bad things happening, but you also wonder about the problems of asking someone with, say, “Schizophrenia: acute state” to answer this sort of question.
You could fix this by asking people about “time tradeoffs”. For example, you could ask someone with a disability about whether they would take a medicine that would restore them to full health for a year even if it took two years off their life. Alternatively you can ask people, generally public health professionals, if given the choice between curing 1000 people with disability X and 2000 people with disability Y which one they would choose. These “person tradeoffs” get us out of needing to ask about probabilities, which means we can probably trust the numbers more, but we’re stuck either with collecting data from people with the disabilities in question (hard work, maybe the disability affects mental function) or trusting that public health experts fully understand what it’s like to have different disabilities (seems unlikely). And even if we did decide that we were only going to collect data from people actually affected by a disability, remember that in many cases they can’t actually give the comparison we want because they haven’t experienced both having and not having the disability (ex: blindness from birth).
The first Global Burden of Disease Report (pdf) attempted to collect these weights for a large number of different diseases. They got a panel of public health experts to come to Geneva and through discussion around “person tradeoffs” came to consensus first on weights for 22 “indicator diseases”. Then they agreed on weights for the several hundred remaining diseases by comparing them to these anchor conditions.
You might worry that the final weights would be really strongly affected by the particular consensus the Geneva group happened to get for the 22 anchors, but they ran nine other attempts with different experts and the average of those attempts correlates pretty well with the Geneva results:
For the 2010 update to the Global Burden of Disease weights (pdf, also see the appendix pdf) they decided to take an entirely different approach. Instead of asking experts to figure out tradeoffs they asked lots of people in several countries (Indonesia, Peru, USA, Bangladesh, Tanzania, plus a ‘global’ internet survey) to do lots of comparisons where given two people they would say which one was healthier. This makes a lot of sense, asking lots of regular people, and the question is much simpler to answer. On the other hand, while I’m not sure experts are all that good at estimating how bad it is to have various disabilities I would expect regular people to be even worse at it. Still, there was at least pretty good correlation between countries:
But hold on: how did they turn a large number of responses where people said one disability was more or less healthy than another into weightings on a 0-1 scale where 0 is full health and 1 is death? It turns out that a quarter (n=4000) of the people who took the survey on the internet were also asked the “person tradeoff” style questions used in the 1990 version, which they called “population health equivalence questions”. So they first determined an ordering from most to least healthy using their large quantity of comparison data, and then used the tradeoff data to map this ordering onto the “0=healthy 1=death” line.
This means that when we look at the cross-country correlations above we’re only seeing their agreement on the relative ordering of conditions, not on the absolute differences. If people in Indonesia on average think that the worst disabilities are only 10% as bad as being dead while people in Peru think they’re 90% as bad, this wouldn’t keep them from having perfect correlation on a chart like this. Which is kind of a problem, because we need more than an ordering for prioritization.
It turns out that this method of estimation actually gives pretty different results from the one used in the earlier version: [1]
Yes, there’s a correlation, but it’s pretty weak. And it’s probably not just about the fifteen years between when most of the first estimates were made and when most of the second were; these aren’t disabilities that are quickly changing. This indicates that what we’re trying to measure is just not that well captured by the measurements we’re making.
So, a summary. Getting good answers means asking people questions they’re not good at thinking about, or that they are good at thinking about but don’t have the right experience to be able to answer. It’s not too surprising, then, that the answers you get via different methods don’t agree very well. We do still need a rough way to say “benefit X to N people is better/worse than benefit Y to M people”, but trying to do this in the general case doesn’t seem to have worked out very well.
These disability weights are only one step in estimating $/DALY for various interventions, and the messiness here is in some ways much less than the messiness in the other steps. After reading about how these estimates came to be, I’m pretty glad GiveWell doesn’t put much trust in them in figuring out which charities to recommend:
The resources that have already been invested in these cost-effectiveness estimates are significant. Yet in our view, the estimates are still far too simplified, sensitive, and esoteric to be relied upon. If such a high level of financial and (especially) human-capital investment leaves us this far from having reliable estimates, it may be time to rethink the goal.
All that said—if this sort of analysis were the only way to figure out how to allocate resources for maximal impact, we’d be advocating for more investment in cost-effectiveness analysis and we’d be determined to “get it right”. But in our view, there are other ways of maximizing cost-effectiveness that can work better in this domain—in particular, making limited use of cost-effectiveness estimates while focusing on finding high-quality evidence.
This isn’t to say we should never use disability weights; even if they were just made up by one guy on the spot (and they’re better than that) this would probably still be better in some cases than refusing to make quantitative comparisons at all. Getting rough numbers like this is especially useful for avoiding scope insensitivity problems, where you might be comparing a large number of people with something minor against a small number with something major.
(I’d really like to look into the QALY numbers people use and how they get them. I believe the process is similar, but I’m not too sure.)
For curiosity, however, and with all that in mind, what are the actual numbers they found? Here are the 2010 weights:
0.756
Schizophrenia: acute state
0.707
Multiple sclerosis: severe
0.673
Spinal cord lesion at neck: untreated
0.657
Epilepsy: severe
0.655
Major depressive disorder: severe episode
0.641
Heroin and other opioid dependence
0.625
Traumatic brain injury: long-term consequences, severe, with or without treatment
0.606
Musculoskeletal problems: generalised, severe
0.576
Schizophrenia: residual state
0.573
End-stage renal disease: on dialysis
0.567
Stroke: long-term consequences, severe plus cognition problems
0.562
Disfigurement: level 3, with itch or pain
0.549
Parkinson’s disease: severe
0.549
Alcohol use disorder: severe
0.547
AIDS: not receiving antiretroviral treatment
0.539
Stroke: long-term consequences, severe
0.523
Anxiety disorders: severe
0.519
Terminal phase: without medication (for cancers, end-stage kidney or liver disease)
0.508
Terminal phase: with medication (for cancers, end-stage kidney or liver disease)
0.494
Amputation of both legs: long term, without treatment
0.492
Rectovaginal fistula
0.480
Bipolar disorder: manic episode
0.484
Cancer: metastatic
0.440
Spinal cord lesion below neck: untreated
0.445
Multiple sclerosis: moderate
0.438
Dementia: severe
0.438
Burns of >=20% total surface area or >=10% total surface area if head or neck, or hands or wrist involved: long term, without treatment
0.433
Headache: migraine
0.420
Epilepsy: untreated
0.425
Motor plus cognitive impairments: severe
0.422
Acute myocardial infarction: days 1-2
0.406
Major depressive disorder: moderate episode
0.390
Fracture of pelvis: short term
0.399
Tuberculosis: with HIV infection
0.398
Disfigurement: level 3
0.388
Fracture of neck of femur: long term, without treatment
0.388
Alcohol use disorder: moderate
0.383
COPD and other chronic respiratory diseases: severe
0.377
Motor impairment: severe
0.376
Cocaine dependence
0.374
Low back pain: chronic, with leg pain
0.373
Lower airway burns: with or without treatment
0.369
Spinal cord lesion at neck: treated
0.366
Low back pain: chronic, without leg pain
0.359
Amputation of both arms: long term, without treatment
0.353
Amphetamine dependence
0.352
Severe chest injury: short term, with or without treatment
0.346
Dementia: moderate
0.338
Vesicovaginal fistula
0.333
Burns of >=20% total surface area: short term, with or without treatment
0.331
Tuberculosis: without HIV infection
0.329
Cannabis dependence
0.326
Abdominopelvic problem: severe
0.323
Gastric bleeding
0.322
Low back pain: acute, with leg pain
0.319
Epilepsy: treated, with recent seizures
0.312
Stroke: long-term consequences, moderate plus cognition problems
0.308
Fracture of neck of femur: short term, with or without treatment
0.200
Iodine-deficiency goitre
0.294
Cancer: diagnosis and primary therapy
0.293
Gout: acute
0.292
Musculoskeletal problems: generalised, moderate
0.288
Drowning and non-fatal submersion: short or long term, with or without treatment
Severe traumatic brain injury: short term, with or without treatment
0.225
Crohn’s disease or ulcerative colitis
0.224
Traumatic brain injury: long-term consequences, moderate, with or without treatment
0.223
Bulimia nervosa
0.223
Anorexia nervosa
0.221
Neck pain: acute, severe
0.221
Motor plus cognitive impairments: moderate
0.221
HIV: symptomatic, pre-AIDS
0.210
Infectious disease: acute episode, severe
0.202
Diarrhoea: moderate
0.198
Multiple sclerosis: mild
0.195
Distance vision blindness
0.194
Fracture of pelvis: long term
0.194
Decompensated cirrhosis of the liver
0.192
Fracture other than neck of femur: short term, with or without treatment
0.192
COPD and other chronic respiratory diseases: moderate
0.191
Distance vision: severe impairment
0.187
Disfigurement: level 2, with itch or pain
0.186
Heart failure: severe
0.177
Fetal alcohol syndrome: severe
0.173
Fracture of face bone: short or long term, with or without treatment
0.171
Poisoning: short term, with or without treatment
0.171
Musculoskeletal problems: legs, severe
0.167
Angina pectoris: severe
0.164
Anaemia: severe
0.164
Amputation of one leg: long term, without treatment
0.150
Fracture of sternum or fracture of one or two ribs: short term, with or without treatment
0.159
Major depressive disorder: mild episode
0.157
Intellectual disability: profound
0.149
Anxiety disorders: moderate
0.145
Crush injury: short or long term, with or without treatment
0.145
Cardiac conduction disorders and cardiac dysrhythmias
0.142
Urinary incontinence
0.130
Amputation of one arm: long term, with or without treatment
0.136
Injured nerves: long term
0.132
Fracture of vertebral column: short or long term, with or without treatment
0.132
Asthma: uncontrolled
0.129
Dislocation of knee: long term, with or without treatment
0.127
Severe wasting
0.127
Burns of >=20% total surface area or >=10% total surface area if head or neck, or hands or wrist involved: long term, with treatment
0.126
Intellectual disability: severe
0.123
Abdominopelvic problem: moderate
0.110
Lymphatic filariasis: symptomatic
0.110
Asperger’s syndrome
0.114
Musculoskeletal problems: arms, moderate
0.106
Traumatic brain injury: long-term consequences, minor, with or without treatment
0.105
Chronic kidney disease (stageIV)
0.101
Neck pain: chronic, mild
0.099
Diabetic neuropathy
0.097
Epididymo-orchitis
0.096
Burns of <20% total surface area without lower airway burns: short term, with or without treatment
0.092
Hearing loss: complete, with ringing
0.080
Intellectual disability: moderate
0.080
Dislocation of shoulder: long term, with or without treatment
0.088
Hearing loss: profound, with ringing
0.087
Fracture of patella, tibia or fibula, or ankle: short term,with or without treatment
0.086
Stoma
0.082
Dementia: mild
0.070
Heart failure: moderate
0.070
Fracture of patella, tibia or fibula, or ankle: long term, with or without treatment
0.070
Benign prostatic hypertrophy: symptomatic
0.079
Musculoskeletal problems: legs, moderate
0.079
Injury to eyes: short term
0.076
Stroke: long-term consequences, moderate
0.076
Motor impairment: moderate
0.073
Fracture of skull: short or long term, with or without treatment
0.072
Severe toothloss
0.072
Fracture of neck of femur: long term, with treatment
0.072
Epilepsy: treated, seizure free
0.072
Disfigurement: level 2
0.066
Angina pectoris: moderate
0.065
Injured nerves: short term
0.065
Hearing loss: severe, with ringing
0.065
Fracture of radius or ulna: short term, with or without treatment
0.061
Herpes zoster
0.061
Diarrhoea: mild
0.050
Fracture of radius or ulna: long term, without treatment
0.058
Hearing loss: moderate, with ringing
0.058
Anaemia: moderate
0.057
Fetal alcohol syndrome: moderate
0.056
Severe chest injury: long term, with or without treatment
0.056
Acute myocardial infarction: days 3-28
0.055
Kwashiorkor
0.054
Speech problems
0.054
Motor plus cognitive impairments: mild
0.054
Generic uncomplicated disease: anxiety about diagnosis
0.053
Infectious disease: acute episode, moderate
0.053
HIV/AIDS: receiving antiretroviral treatment
0.053
Fracture other than neck of femur: long term, without treatment
0.053
Fracture of clavicle, scapula, or humerus: short or long term, with or without treatment
0.051
Amputation of both legs: long term, with treatment
0.040
Neck pain: acute, mild
0.040
Headache: tension-type
0.049
Attention-deficit hyperactivity disorder
0.047
Spinal cord lesion below neck: treated
0.044
Amputation of both arms: long term, with treatment
0.030
Intestinal nematode infections: symptomatic
0.030
Anxiety disorders: mild
0.030
Amputation of finger(s), excluding thumb: long term, with treatment
0.038
Mastectomy
0.038
Hearing loss: mild, with ringing
0.037
Heart failure: mild
0.037
Angina pectoris: mild
0.035
Bipolar disorder: residual state
0.033
Hearing loss: complete
0.033
Fracture of foot bones: short term, with or without treatment
0.033
Fracture of foot bones: long term, without treatment
0.033
Distance vision: moderate impairment
0.032
Hearing loss: severe
0.031
Intellectual disability: mild
0.031
Hearing loss: profound
0.031
Generic uncomplicated disease: lly controlled
0.025
Fracture of hand: short term, with or without treatment
0.024
Musculoskeletal problems: arms, mild
0.023
Musculoskeletal problems: legs, mild
0.023
Hearing loss: moderate
0.023
Diabetic foot
0.021
Stroke: long-term consequences, mild
0.021
Amputation of one leg: long term, with treatment
0.019
Impotence
0.018
Ear pain
0.018
Burns of <20% total surface area or <10% total surface area if head or neck, or hands or wrist involved: long term, with or without treatment
0.017
Fetal alcohol syndrome: mild
0.017
Dislocation of hip: long term, with or without treatment
0.016
Fracture of hand: long term, without treatment
0.016
Claudication
0.015
COPD and other chronic respiratory diseases: mild
0.013
Near vision impairment
0.013
Disfigurement: level 1
0.013
Amputation of thumb: long term
0.012
Motor impairment: mild
0.012
Dental caries:symptomatic
0.012
Abdominopelvic problem: mild
0.011
Parkinson’s disease: mild
0.011
Infertility: primary
0.009
Other injuries of muscle and tendon (includes sprains, strains, and dislocations other than shoulder, knee, or hip)
0.009
Asthma: controlled
0.008
Periodontitis
0.008
Amputation of toe
0.006
Infertility: secondary
0.005
Open wound: short term, with or without treatment
0.005
Infectious disease: acute episode, mild
0.005
Hearing loss: mild
0.005
Anaemia: mild
0.004
Distance vision: mild impairment
0.003
Fractures: treated, long term
[1] Technically these are the results from the 2004 update to the 1990 version, but when you look at where their estimates come from (pdf you see that most just say they’re kept unchanged from the 1990 version.
Disability Weights
If you’re talking to two people, one with a small cut and another with multiple sclerosis, everyone present will agree that having multiple sclerosis is much worse. If you offered the two of them some magic option that would restore exactly one of them to full health, they would probably be able to agree on who should get it. But in general, how should we compare across people to figure out whose situation is worse, who would benefit more from treatment and who, everything else being equal, should be treated first? This example was easy because the difference was nice and large, but what do we do in harder cases?
One is to ask people questions like “if there were a surgery that could restore you to full health (without improving your lifespan) but had a 20% chance of killing you, would you take it?” If they say “yes” then this indicates that this disability, for this person, is more than 20% as bad as being dead. Ask these “standard gamble” questions to a lot of people with a lot of disabilities, varying the percentages, and you could build up a list of how bad different ones are, all on a common scale.
This would useful for balancing projects against each other, figuring out what to focus on, and generally setting funding priorities. Unfortunately people are really bad at answering questions like this. Mostly we’re just bad at thinking about percentages and chances of bad things happening, but you also wonder about the problems of asking someone with, say, “Schizophrenia: acute state” to answer this sort of question.
You could fix this by asking people about “time tradeoffs”. For example, you could ask someone with a disability about whether they would take a medicine that would restore them to full health for a year even if it took two years off their life. Alternatively you can ask people, generally public health professionals, if given the choice between curing 1000 people with disability X and 2000 people with disability Y which one they would choose. These “person tradeoffs” get us out of needing to ask about probabilities, which means we can probably trust the numbers more, but we’re stuck either with collecting data from people with the disabilities in question (hard work, maybe the disability affects mental function) or trusting that public health experts fully understand what it’s like to have different disabilities (seems unlikely). And even if we did decide that we were only going to collect data from people actually affected by a disability, remember that in many cases they can’t actually give the comparison we want because they haven’t experienced both having and not having the disability (ex: blindness from birth).
The first Global Burden of Disease Report (pdf) attempted to collect these weights for a large number of different diseases. They got a panel of public health experts to come to Geneva and through discussion around “person tradeoffs” came to consensus first on weights for 22 “indicator diseases”. Then they agreed on weights for the several hundred remaining diseases by comparing them to these anchor conditions.
You might worry that the final weights would be really strongly affected by the particular consensus the Geneva group happened to get for the 22 anchors, but they ran nine other attempts with different experts and the average of those attempts correlates pretty well with the Geneva results:
For the 2010 update to the Global Burden of Disease weights (pdf, also see the appendix pdf) they decided to take an entirely different approach. Instead of asking experts to figure out tradeoffs they asked lots of people in several countries (Indonesia, Peru, USA, Bangladesh, Tanzania, plus a ‘global’ internet survey) to do lots of comparisons where given two people they would say which one was healthier. This makes a lot of sense, asking lots of regular people, and the question is much simpler to answer. On the other hand, while I’m not sure experts are all that good at estimating how bad it is to have various disabilities I would expect regular people to be even worse at it. Still, there was at least pretty good correlation between countries:
But hold on: how did they turn a large number of responses where people said one disability was more or less healthy than another into weightings on a 0-1 scale where 0 is full health and 1 is death? It turns out that a quarter (n=4000) of the people who took the survey on the internet were also asked the “person tradeoff” style questions used in the 1990 version, which they called “population health equivalence questions”. So they first determined an ordering from most to least healthy using their large quantity of comparison data, and then used the tradeoff data to map this ordering onto the “0=healthy 1=death” line.
This means that when we look at the cross-country correlations above we’re only seeing their agreement on the relative ordering of conditions, not on the absolute differences. If people in Indonesia on average think that the worst disabilities are only 10% as bad as being dead while people in Peru think they’re 90% as bad, this wouldn’t keep them from having perfect correlation on a chart like this. Which is kind of a problem, because we need more than an ordering for prioritization.
It turns out that this method of estimation actually gives pretty different results from the one used in the earlier version: [1]
Yes, there’s a correlation, but it’s pretty weak. And it’s probably not just about the fifteen years between when most of the first estimates were made and when most of the second were; these aren’t disabilities that are quickly changing. This indicates that what we’re trying to measure is just not that well captured by the measurements we’re making.
So, a summary. Getting good answers means asking people questions they’re not good at thinking about, or that they are good at thinking about but don’t have the right experience to be able to answer. It’s not too surprising, then, that the answers you get via different methods don’t agree very well. We do still need a rough way to say “benefit X to N people is better/worse than benefit Y to M people”, but trying to do this in the general case doesn’t seem to have worked out very well.
These disability weights are only one step in estimating $/DALY for various interventions, and the messiness here is in some ways much less than the messiness in the other steps. After reading about how these estimates came to be, I’m pretty glad GiveWell doesn’t put much trust in them in figuring out which charities to recommend:
This isn’t to say we should never use disability weights; even if they were just made up by one guy on the spot (and they’re better than that) this would probably still be better in some cases than refusing to make quantitative comparisons at all. Getting rough numbers like this is especially useful for avoiding scope insensitivity problems, where you might be comparing a large number of people with something minor against a small number with something major.
(I’d really like to look into the QALY numbers people use and how they get them. I believe the process is similar, but I’m not too sure.)
For curiosity, however, and with all that in mind, what are the actual numbers they found? Here are the 2010 weights:
[1] Technically these are the results from the 2004 update to the 1990 version, but when you look at where their estimates come from (pdf you see that most just say they’re kept unchanged from the 1990 version.