Disability Weights

If you’re talk­ing to two peo­ple, one with a small cut and an­other with mul­ti­ple scle­ro­sis, ev­ery­one pre­sent will agree that hav­ing mul­ti­ple scle­ro­sis is much worse. If you offered the two of them some magic op­tion that would re­store ex­actly one of them to full health, they would prob­a­bly be able to agree on who should get it. But in gen­eral, how should we com­pare across peo­ple to figure out whose situ­a­tion is worse, who would benefit more from treat­ment and who, ev­ery­thing else be­ing equal, should be treated first? This ex­am­ple was easy be­cause the differ­ence was nice and large, but what do we do in harder cases?

One is to ask peo­ple ques­tions like “if there were a surgery that could re­store you to full health (with­out im­prov­ing your lifes­pan) but had a 20% chance of kil­ling you, would you take it?” If they say “yes” then this in­di­cates that this dis­abil­ity, for this per­son, is more than 20% as bad as be­ing dead. Ask these “stan­dard gam­ble” ques­tions to a lot of peo­ple with a lot of dis­abil­ities, vary­ing the per­centages, and you could build up a list of how bad differ­ent ones are, all on a com­mon scale.

This would use­ful for bal­anc­ing pro­jects against each other, figur­ing out what to fo­cus on, and gen­er­ally set­ting fund­ing pri­ori­ties. Un­for­tu­nately peo­ple are re­ally bad at an­swer­ing ques­tions like this. Mostly we’re just bad at think­ing about per­centages and chances of bad things hap­pen­ing, but you also won­der about the prob­lems of ask­ing some­one with, say, “Schizophre­nia: acute state” to an­swer this sort of ques­tion.

You could fix this by ask­ing peo­ple about “time trade­offs”. For ex­am­ple, you could ask some­one with a dis­abil­ity about whether they would take a medicine that would re­store them to full health for a year even if it took two years off their life. Alter­na­tively you can ask peo­ple, gen­er­ally pub­lic health pro­fes­sion­als, if given the choice be­tween cur­ing 1000 peo­ple with dis­abil­ity X and 2000 peo­ple with dis­abil­ity Y which one they would choose. Th­ese “per­son trade­offs” get us out of need­ing to ask about prob­a­bil­ities, which means we can prob­a­bly trust the num­bers more, but we’re stuck ei­ther with col­lect­ing data from peo­ple with the dis­abil­ities in ques­tion (hard work, maybe the dis­abil­ity af­fects men­tal func­tion) or trust­ing that pub­lic health ex­perts fully un­der­stand what it’s like to have differ­ent dis­abil­ities (seems un­likely). And even if we did de­cide that we were only go­ing to col­lect data from peo­ple ac­tu­ally af­fected by a dis­abil­ity, re­mem­ber that in many cases they can’t ac­tu­ally give the com­par­i­son we want be­cause they haven’t ex­pe­rienced both hav­ing and not hav­ing the dis­abil­ity (ex: blind­ness from birth).

The first Global Bur­den of Disease Re­port (pdf) at­tempted to col­lect these weights for a large num­ber of differ­ent dis­eases. They got a panel of pub­lic health ex­perts to come to Geneva and through dis­cus­sion around “per­son trade­offs” came to con­sen­sus first on weights for 22 “in­di­ca­tor dis­eases”. Then they agreed on weights for the sev­eral hun­dred re­main­ing dis­eases by com­par­ing them to these an­chor con­di­tions.

You might worry that the fi­nal weights would be re­ally strongly af­fected by the par­tic­u­lar con­sen­sus the Geneva group hap­pened to get for the 22 an­chors, but they ran nine other at­tempts with differ­ent ex­perts and the av­er­age of those at­tempts cor­re­lates pretty well with the Geneva re­sults:

For the 2010 up­date to the Global Bur­den of Disease weights (pdf, also see the ap­pendix pdf) they de­cided to take an en­tirely differ­ent ap­proach. In­stead of ask­ing ex­perts to figure out trade­offs they asked lots of peo­ple in sev­eral coun­tries (In­done­sia, Peru, USA, Bangladesh, Tan­za­nia, plus a ‘global’ in­ter­net sur­vey) to do lots of com­par­i­sons where given two peo­ple they would say which one was healthier. This makes a lot of sense, ask­ing lots of reg­u­lar peo­ple, and the ques­tion is much sim­pler to an­swer. On the other hand, while I’m not sure ex­perts are all that good at es­ti­mat­ing how bad it is to have var­i­ous dis­abil­ities I would ex­pect reg­u­lar peo­ple to be even worse at it. Still, there was at least pretty good cor­re­la­tion be­tween coun­tries:

But hold on: how did they turn a large num­ber of re­sponses where peo­ple said one dis­abil­ity was more or less healthy than an­other into weight­ings on a 0-1 scale where 0 is full health and 1 is death? It turns out that a quar­ter (n=4000) of the peo­ple who took the sur­vey on the in­ter­net were also asked the “per­son trade­off” style ques­tions used in the 1990 ver­sion, which they called “pop­u­la­tion health equiv­alence ques­tions”. So they first de­ter­mined an or­der­ing from most to least healthy us­ing their large quan­tity of com­par­i­son data, and then used the trade­off data to map this or­der­ing onto the “0=healthy 1=death” line.

This means that when we look at the cross-coun­try cor­re­la­tions above we’re only see­ing their agree­ment on the rel­a­tive or­der­ing of con­di­tions, not on the ab­solute differ­ences. If peo­ple in In­done­sia on av­er­age think that the worst dis­abil­ities are only 10% as bad as be­ing dead while peo­ple in Peru think they’re 90% as bad, this wouldn’t keep them from hav­ing perfect cor­re­la­tion on a chart like this. Which is kind of a prob­lem, be­cause we need more than an or­der­ing for pri­ori­ti­za­tion.

It turns out that this method of es­ti­ma­tion ac­tu­ally gives pretty differ­ent re­sults from the one used in the ear­lier ver­sion: [1]

Yes, there’s a cor­re­la­tion, but it’s pretty weak. And it’s prob­a­bly not just about the fif­teen years be­tween when most of the first es­ti­mates were made and when most of the sec­ond were; these aren’t dis­abil­ities that are quickly chang­ing. This in­di­cates that what we’re try­ing to mea­sure is just not that well cap­tured by the mea­sure­ments we’re mak­ing.

So, a sum­mary. Get­ting good an­swers means ask­ing peo­ple ques­tions they’re not good at think­ing about, or that they are good at think­ing about but don’t have the right ex­pe­rience to be able to an­swer. It’s not too sur­pris­ing, then, that the an­swers you get via differ­ent meth­ods don’t agree very well. We do still need a rough way to say “benefit X to N peo­ple is bet­ter/​worse than benefit Y to M peo­ple”, but try­ing to do this in the gen­eral case doesn’t seem to have worked out very well.

Th­ese dis­abil­ity weights are only one step in es­ti­mat­ing $/​DALY for var­i­ous in­ter­ven­tions, and the messi­ness here is in some ways much less than the messi­ness in the other steps. After read­ing about how these es­ti­mates came to be, I’m pretty glad GiveWell doesn’t put much trust in them in figur­ing out which char­i­ties to recom­mend:

The re­sources that have already been in­vested in these cost-effec­tive­ness es­ti­mates are sig­nifi­cant. Yet in our view, the es­ti­mates are still far too sim­plified, sen­si­tive, and es­o­teric to be re­lied upon. If such a high level of fi­nan­cial and (es­pe­cially) hu­man-cap­i­tal in­vest­ment leaves us this far from hav­ing re­li­able es­ti­mates, it may be time to re­think the goal.

All that said—if this sort of anal­y­sis were the only way to figure out how to al­lo­cate re­sources for max­i­mal im­pact, we’d be ad­vo­cat­ing for more in­vest­ment in cost-effec­tive­ness anal­y­sis and we’d be de­ter­mined to “get it right”. But in our view, there are other ways of max­i­miz­ing cost-effec­tive­ness that can work bet­ter in this do­main—in par­tic­u­lar, mak­ing limited use of cost-effec­tive­ness es­ti­mates while fo­cus­ing on find­ing high-qual­ity ev­i­dence.

This isn’t to say we should never use dis­abil­ity weights; even if they were just made up by one guy on the spot (and they’re bet­ter than that) this would prob­a­bly still be bet­ter in some cases than re­fus­ing to make quan­ti­ta­tive com­par­i­sons at all. Get­ting rough num­bers like this is es­pe­cially use­ful for avoid­ing scope in­sen­si­tivity prob­lems, where you might be com­par­ing a large num­ber of peo­ple with some­thing minor against a small num­ber with some­thing ma­jor.

(I’d re­ally like to look into the QALY num­bers peo­ple use and how they get them. I be­lieve the pro­cess is similar, but I’m not too sure.)

For cu­ri­os­ity, how­ever, and with all that in mind, what are the ac­tual num­bers they found? Here are the 2010 weights:

0.756 Schizophre­nia: acute state
0.707 Mul­ti­ple scle­ro­sis: se­vere
0.673 Spinal cord le­sion at neck: un­treated
0.657 Epilepsy: se­vere
0.655 Ma­jor de­pres­sive di­s­or­der: se­vere epi­sode
0.641 Heroin and other opi­oid de­pen­dence
0.625 Trau­matic brain in­jury: long-term con­se­quences, se­vere, with or with­out treat­ment
0.606 Mus­cu­loskele­tal prob­lems: gen­er­al­ised, se­vere
0.576 Schizophre­nia: resi­d­ual state
0.573 End-stage re­nal dis­ease: on dial­y­sis
0.567 Stroke: long-term con­se­quences, se­vere plus cog­ni­tion prob­lems
0.562 Dis­figure­ment: level 3, with itch or pain
0.549 Park­in­son’s dis­ease: se­vere
0.549 Al­co­hol use di­s­or­der: se­vere
0.547 AIDS: not re­ceiv­ing an­tiretro­viral treat­ment
0.539 Stroke: long-term con­se­quences, se­vere
0.523 Anx­iety di­s­or­ders: se­vere
0.519 Ter­mi­nal phase: with­out med­i­ca­tion (for can­cers, end-stage kid­ney or liver dis­ease)
0.508 Ter­mi­nal phase: with med­i­ca­tion (for can­cers, end-stage kid­ney or liver dis­ease)
0.494 Am­pu­ta­tion of both legs: long term, with­out treat­ment
0.492 Rec­to­vagi­nal fis­tula
0.480 Bipo­lar di­s­or­der: manic epi­sode
0.484 Cancer: metastatic
0.440 Spinal cord le­sion be­low neck: un­treated
0.445 Mul­ti­ple scle­ro­sis: mod­er­ate
0.438 De­men­tia: se­vere
0.438 Burns of >=20% to­tal sur­face area or >=10% to­tal sur­face area if head or neck, or hands or wrist in­volved: long term, with­out treat­ment
0.433 Headache: mi­graine
0.420 Epilepsy: un­treated
0.425 Mo­tor plus cog­ni­tive im­pair­ments: se­vere
0.422 Acute my­ocar­dial in­farc­tion: days 1-2
0.406 Ma­jor de­pres­sive di­s­or­der: mod­er­ate epi­sode
0.390 Frac­ture of pelvis: short term
0.399 Tu­ber­cu­lo­sis: with HIV in­fec­tion
0.398 Dis­figure­ment: level 3
0.388 Frac­ture of neck of fe­mur: long term, with­out treat­ment
0.388 Al­co­hol use di­s­or­der: mod­er­ate
0.383 COPD and other chronic res­pi­ra­tory dis­eases: se­vere
0.377 Mo­tor im­pair­ment: se­vere
0.376 Co­caine de­pen­dence
0.374 Low back pain: chronic, with leg pain
0.373 Lower air­way burns: with or with­out treat­ment
0.369 Spinal cord le­sion at neck: treated
0.366 Low back pain: chronic, with­out leg pain
0.359 Am­pu­ta­tion of both arms: long term, with­out treat­ment
0.353 Am­phetamine de­pen­dence
0.352 Se­vere chest in­jury: short term, with or with­out treat­ment
0.346 De­men­tia: mod­er­ate
0.338 Vesi­co­v­agi­nal fis­tula
0.333 Burns of >=20% to­tal sur­face area: short term, with or with­out treat­ment
0.331 Tu­ber­cu­lo­sis: with­out HIV in­fec­tion
0.329 Cannabis de­pen­dence
0.326 Ab­dom­inopelvic prob­lem: se­vere
0.323 Gas­tric bleed­ing
0.322 Low back pain: acute, with leg pain
0.319 Epilepsy: treated, with re­cent seizures
0.312 Stroke: long-term con­se­quences, mod­er­ate plus cog­ni­tion prob­lems
0.308 Frac­ture of neck of fe­mur: short term, with or with­out treat­ment
0.200 Io­dine-defi­ciency goitre
0.294 Cancer: di­ag­no­sis and pri­mary ther­apy
0.293 Gout: acute
0.292 Mus­cu­loskele­tal prob­lems: gen­er­al­ised, mod­er­ate
0.288 Drown­ing and non-fatal sub­mer­sion: short or long term, with or with­out treat­ment
0.286 Neck pain: chronic, se­vere
0.281 Diar­rhoea: se­vere
0.269 Low back pain: acute, with­out leg pain
0.263 Park­in­son’s dis­ease: mod­er­ate
0.259 Autism
0.259 Al­co­hol use di­s­or­der: mild
0.254 In­fec­tious dis­ease: post-acute con­se­quences (fa­tigue, emo­tional la­bil­ity, in­som­nia)
0.236 Con­duct di­s­or­der
0.235 Se­vere trau­matic brain in­jury: short term, with or with­out treat­ment
0.225 Crohn’s dis­ease or ul­cer­a­tive col­i­tis
0.224 Trau­matic brain in­jury: long-term con­se­quences, mod­er­ate, with or with­out treat­ment
0.223 Bulimia ner­vosa
0.223 Anorexia ner­vosa
0.221 Neck pain: acute, se­vere
0.221 Mo­tor plus cog­ni­tive im­pair­ments: mod­er­ate
0.221 HIV: symp­tomatic, pre-AIDS
0.210 In­fec­tious dis­ease: acute epi­sode, se­vere
0.202 Diar­rhoea: mod­er­ate
0.198 Mul­ti­ple scle­ro­sis: mild
0.195 Dis­tance vi­sion blind­ness
0.194 Frac­ture of pelvis: long term
0.194 Decom­pen­sated cir­rho­sis of the liver
0.192 Frac­ture other than neck of fe­mur: short term, with or with­out treat­ment
0.192 COPD and other chronic res­pi­ra­tory dis­eases: mod­er­ate
0.191 Dis­tance vi­sion: se­vere im­pair­ment
0.187 Dis­figure­ment: level 2, with itch or pain
0.186 Heart failure: se­vere
0.177 Fe­tal al­co­hol syn­drome: se­vere
0.173 Frac­ture of face bone: short or long term, with or with­out treat­ment
0.171 Poi­son­ing: short term, with or with­out treat­ment
0.171 Mus­cu­loskele­tal prob­lems: legs, se­vere
0.167 Ang­ina pec­toris: se­vere
0.164 Anaemia: se­vere
0.164 Am­pu­ta­tion of one leg: long term, with­out treat­ment
0.150 Frac­ture of ster­num or frac­ture of one or two ribs: short term, with or with­out treat­ment
0.159 Ma­jor de­pres­sive di­s­or­der: mild epi­sode
0.157 In­tel­lec­tual dis­abil­ity: profound
0.149 Anx­iety di­s­or­ders: mod­er­ate
0.145 Crush in­jury: short or long term, with or with­out treat­ment
0.145 Car­diac con­duc­tion di­s­or­ders and car­diac dys­rhyth­mias
0.142 Uri­nary in­con­ti­nence
0.130 Am­pu­ta­tion of one arm: long term, with or with­out treat­ment
0.136 In­jured nerves: long term
0.132 Frac­ture of ver­te­bral column: short or long term, with or with­out treat­ment
0.132 Asthma: un­con­trol­led
0.129 Dis­lo­ca­tion of knee: long term, with or with­out treat­ment
0.127 Se­vere wast­ing
0.127 Burns of >=20% to­tal sur­face area or >=10% to­tal sur­face area if head or neck, or hands or wrist in­volved: long term, with treat­ment
0.126 In­tel­lec­tual dis­abil­ity: se­vere
0.123 Ab­dom­inopelvic prob­lem: mod­er­ate
0.110 Lym­phatic filari­a­sis: symp­tomatic
0.110 Asperger’s syn­drome
0.114 Mus­cu­loskele­tal prob­lems: arms, mod­er­ate
0.106 Trau­matic brain in­jury: long-term con­se­quences, minor, with or with­out treat­ment
0.105 Chronic kid­ney dis­ease (stageIV)
0.101 Neck pain: chronic, mild
0.099 Di­a­betic neu­ropa­thy
0.097 Epi­didymo-or­chitis
0.096 Burns of <20% to­tal sur­face area with­out lower air­way burns: short term, with or with­out treat­ment
0.092 Hear­ing loss: com­plete, with ring­ing
0.080 In­tel­lec­tual dis­abil­ity: mod­er­ate
0.080 Dis­lo­ca­tion of shoulder: long term, with or with­out treat­ment
0.088 Hear­ing loss: profound, with ring­ing
0.087 Frac­ture of patella, tibia or fibula, or an­kle: short term,with or with­out treat­ment
0.086 Stoma
0.082 De­men­tia: mild
0.070 Heart failure: mod­er­ate
0.070 Frac­ture of patella, tibia or fibula, or an­kle: long term, with or with­out treat­ment
0.070 Benign pro­static hy­per­tro­phy: symp­tomatic
0.079 Mus­cu­loskele­tal prob­lems: legs, mod­er­ate
0.079 In­jury to eyes: short term
0.076 Stroke: long-term con­se­quences, mod­er­ate
0.076 Mo­tor im­pair­ment: mod­er­ate
0.073 Frac­ture of skull: short or long term, with or with­out treat­ment
0.072 Se­vere toothloss
0.072 Frac­ture of neck of fe­mur: long term, with treat­ment
0.072 Epilepsy: treated, seizure free
0.072 Dis­figure­ment: level 2
0.066 Ang­ina pec­toris: mod­er­ate
0.065 In­jured nerves: short term
0.065 Hear­ing loss: se­vere, with ring­ing
0.065 Frac­ture of ra­dius or ulna: short term, with or with­out treat­ment
0.061 Her­pes zoster
0.061 Diar­rhoea: mild
0.050 Frac­ture of ra­dius or ulna: long term, with­out treat­ment
0.058 Hear­ing loss: mod­er­ate, with ring­ing
0.058 Anaemia: mod­er­ate
0.057 Fe­tal al­co­hol syn­drome: mod­er­ate
0.056 Se­vere chest in­jury: long term, with or with­out treat­ment
0.056 Acute my­ocar­dial in­farc­tion: days 3-28
0.055 Kwash­iorkor
0.054 Speech prob­lems
0.054 Mo­tor plus cog­ni­tive im­pair­ments: mild
0.054 Generic un­com­pli­cated dis­ease: anx­iety about di­ag­no­sis
0.053 In­fec­tious dis­ease: acute epi­sode, mod­er­ate
0.053 HIV/​AIDS: re­ceiv­ing an­tiretro­viral treat­ment
0.053 Frac­ture other than neck of fe­mur: long term, with­out treat­ment
0.053 Frac­ture of clav­i­cle, scapula, or humerus: short or long term, with or with­out treat­ment
0.051 Am­pu­ta­tion of both legs: long term, with treat­ment
0.040 Neck pain: acute, mild
0.040 Headache: ten­sion-type
0.049 At­ten­tion-deficit hy­per­ac­tivity di­s­or­der
0.047 Spinal cord le­sion be­low neck: treated
0.044 Am­pu­ta­tion of both arms: long term, with treat­ment
0.030 In­testi­nal ne­ma­tode in­fec­tions: symp­tomatic
0.030 Anx­iety di­s­or­ders: mild
0.030 Am­pu­ta­tion of finger(s), ex­clud­ing thumb: long term, with treat­ment
0.038 Mas­tec­tomy
0.038 Hear­ing loss: mild, with ring­ing
0.037 Heart failure: mild
0.037 Ang­ina pec­toris: mild
0.035 Bipo­lar di­s­or­der: resi­d­ual state
0.033 Hear­ing loss: com­plete
0.033 Frac­ture of foot bones: short term, with or with­out treat­ment
0.033 Frac­ture of foot bones: long term, with­out treat­ment
0.033 Dis­tance vi­sion: mod­er­ate im­pair­ment
0.032 Hear­ing loss: se­vere
0.031 In­tel­lec­tual dis­abil­ity: mild
0.031 Hear­ing loss: profound
0.031 Generic un­com­pli­cated dis­ease: lly con­trol­led
0.025 Frac­ture of hand: short term, with or with­out treat­ment
0.024 Mus­cu­loskele­tal prob­lems: arms, mild
0.023 Mus­cu­loskele­tal prob­lems: legs, mild
0.023 Hear­ing loss: mod­er­ate
0.023 Di­a­betic foot
0.021 Stroke: long-term con­se­quences, mild
0.021 Am­pu­ta­tion of one leg: long term, with treat­ment
0.019 Im­po­tence
0.018 Ear pain
0.018 Burns of <20% to­tal sur­face area or <10% to­tal sur­face area if head or neck, or hands or wrist in­volved: long term, with or with­out treat­ment
0.017 Fe­tal al­co­hol syn­drome: mild
0.017 Dis­lo­ca­tion of hip: long term, with or with­out treat­ment
0.016 Frac­ture of hand: long term, with­out treat­ment
0.016 Clau­di­ca­tion
0.015 COPD and other chronic res­pi­ra­tory dis­eases: mild
0.013 Near vi­sion im­pair­ment
0.013 Dis­figure­ment: level 1
0.013 Am­pu­ta­tion of thumb: long term
0.012 Mo­tor im­pair­ment: mild
0.012 Den­tal caries:symp­tomatic
0.012 Ab­dom­inopelvic prob­lem: mild
0.011 Park­in­son’s dis­ease: mild
0.011 In­fer­til­ity: pri­mary
0.009 Other in­juries of mus­cle and ten­don (in­cludes sprains, strains, and dis­lo­ca­tions other than shoulder, knee, or hip)
0.009 Asthma: con­trol­led
0.008 Pe­ri­odon­titis
0.008 Am­pu­ta­tion of toe
0.006 In­fer­til­ity: sec­ondary
0.005 Open wound: short term, with or with­out treat­ment
0.005 In­fec­tious dis­ease: acute epi­sode, mild
0.005 Hear­ing loss: mild
0.005 Anaemia: mild
0.004 Dis­tance vi­sion: mild im­pair­ment
0.003 Frac­tures: treated, long term

[1] Tech­ni­cally these are the re­sults from the 2004 up­date to the 1990 ver­sion, but when you look at where their es­ti­mates come from (pdf you see that most just say they’re kept un­changed from the 1990 ver­sion.