The DALY framework has undergone substantial revision since its initial creation, with the GBD 2010 removing many of the most harshly criticized methodological aspects of previous editions. As a result, the disability weight of blindness for example was reduced from 0.57 in the GBD 2004 to 0.195 in the GBD 2010
I don’t think it is so easy to avoid looking bad to disability rights activists by changing the weights. Either disabilities get a high weighting, in which case it makes sense to put a lot of emphasis on preventing and treating disabilities, but less emphasis on saving the lives of people with disabilities, or they get a low weighting, in which case saving the lives of people with disabilities is measured as similarly valuable to saving the lives of able-bodied people, but also there is little emphasis on preventing or treating disabilities. Both options seem potentially offensive to some people, and any solution seems like it is fundamentally shifting between these two options—although some ‘solutions’ obfuscate this fundamental tradeoff.
(Also, perhaps more importantly, I think we should design our evaluation system to be true / good / consistent with our moral intuitions, rather than prioritizing PR. If disability activists have good arguments we should heed them, but the mere fact that they find some conclusion offensive is not a strong argument. )
I wonder if there might also be some background level of skepticism in the disabilty community about explicit cost-benefit analysis, because of how often “it’s too expensive / we can use the money better somewhere else” is used in other contexts as a justification not to provide supports for people with disabilities. I’m not saying that is the case, only that I wouldn’t be surprised if it were.
I wouldn’t really expect the disability advocacy community to be united on which tradeoff should be preferred.
(But I also wouldn’t expect mechanisms of PR-risk aversion to highlight the variance or disagreement within the disability advocacy community, because those mechanisms tend to favor the more articulate/charismatic side at the expense of others).
This is a good point, and it’s likely that for many people there will be quite a wide range in the variance of how they experience a disability. If so, then you’d expect most people with a given disability to disagree with the GBD weight, simply because they would personally rate it somewhat higher or lower than the average value.
EDIT to add: In fact it seems the 2010 GBD weights were obtained by surveying members of the public, so it could be the case that the weights are either higher or lower than most individuals with a given disability would have indicated if they had been asked.
Thanks for your comment! I would basically distinguish three questions:
Can we address critiques of the DALY framework by turning a big dial that says “disability weight” on it and constantly looking back at the audience for approval like a contestant on The Price is Right? - Probably not.
Can we address critiques of the DALY framework by selecting moral weighting frameworks that are appropriate for our particular applications, addressing methodological critiques when they get raised, and taking care to contextualize our usage of a particular framework? - Maybe.
Is the disability advocacy community a monolith that will be 100% united in approval or disapproval for a particular methodology? - Definitely not.
In general, I agree that we should be trying to have true/good evaluation systems, not systems that are optimized for PR. I just think that right now we’re not on the Pareto frontier of that tradeoff.
Can we address critiques of the DALY framework by selecting moral weighting frameworks that are appropriate for our particular applications, addressing methodological critiques when they get raised, and taking care to contextualize our usage of a particular framework? - Maybe.
I’m pretty sure the answer is “No, we can’t”. The whole point of DALY is that it lets us compare completely different interventions. If you replace it with something that is different in each context, you have not replaced it.
I think the best we can do is to calibrate it better, buy asking actual disabled people about their life quality. I think the answer will be very different depending on the disability, and also surrounding support and culture. This can be baked in, but you can’t change the waits around for different interventions.
Sorry if my comment was unclear. I don’t mean that we should use a different set of weights when looking at different interventions, I mean that we should use different weighting frameworks depending on the types of questions we are trying to ask. If we’re trying to quantify the impacts of different interventions on health outcomes, the post-2010 DALY scale might be reasonable. If we’re trying to quantify the impacts of different interventions on wellbeing, then WELLBYs might be reasonable. If we value improvements in health outcomes independent of their impact on subjective wellbeing, then some type of blended framework (e.g. GiveWell’s moral weighting scheme) might make sense.
I’ll return to the RP Moral Weights Project as an example of what I’m critiquing (the Moral Weight Project is fantastic in lots of ways, I don’t mean to say the whole project is bad because of this one critique). For the project, the authors are trying to develop weights that express animals’ changes in hedonic wellbeing in terms of human DALYs. But it’s not clear that DALYs are a coherent unit for what they’re trying to measure. The give trying to “estimate the welfare gain from, say, moving layer hens from cages to a cage-free system” as an example of the kind of application they’re looking at. But locking a human in a cage wouldn’t obviously change the number of DALYs gained in the world, at least under the post-2010 definition. For that application, a unit that included subjective wellbeing would make a lot more sense. That’s the kind of thing I’m trying to get at.
But I do agree with you that asking disabled people about their experiences and incorporating those results into whatever weighting scale we use is a very valuable step!
I don’t think it is so easy to avoid looking bad to disability rights activists by changing the weights. Either disabilities get a high weighting, in which case it makes sense to put a lot of emphasis on preventing and treating disabilities, but less emphasis on saving the lives of people with disabilities, or they get a low weighting, in which case saving the lives of people with disabilities is measured as similarly valuable to saving the lives of able-bodied people, but also there is little emphasis on preventing or treating disabilities. Both options seem potentially offensive to some people, and any solution seems like it is fundamentally shifting between these two options—although some ‘solutions’ obfuscate this fundamental tradeoff.
(Also, perhaps more importantly, I think we should design our evaluation system to be true / good / consistent with our moral intuitions, rather than prioritizing PR. If disability activists have good arguments we should heed them, but the mere fact that they find some conclusion offensive is not a strong argument. )
I wonder if there might also be some background level of skepticism in the disabilty community about explicit cost-benefit analysis, because of how often “it’s too expensive / we can use the money better somewhere else” is used in other contexts as a justification not to provide supports for people with disabilities. I’m not saying that is the case, only that I wouldn’t be surprised if it were.
I wouldn’t really expect the disability advocacy community to be united on which tradeoff should be preferred.
(But I also wouldn’t expect mechanisms of PR-risk aversion to highlight the variance or disagreement within the disability advocacy community, because those mechanisms tend to favor the more articulate/charismatic side at the expense of others).
This is a good point, and it’s likely that for many people there will be quite a wide range in the variance of how they experience a disability. If so, then you’d expect most people with a given disability to disagree with the GBD weight, simply because they would personally rate it somewhat higher or lower than the average value.
EDIT to add: In fact it seems the 2010 GBD weights were obtained by surveying members of the public, so it could be the case that the weights are either higher or lower than most individuals with a given disability would have indicated if they had been asked.
Thanks for your comment! I would basically distinguish three questions:
Can we address critiques of the DALY framework by turning a big dial that says “disability weight” on it and constantly looking back at the audience for approval like a contestant on The Price is Right? - Probably not.
Can we address critiques of the DALY framework by selecting moral weighting frameworks that are appropriate for our particular applications, addressing methodological critiques when they get raised, and taking care to contextualize our usage of a particular framework? - Maybe.
Is the disability advocacy community a monolith that will be 100% united in approval or disapproval for a particular methodology? - Definitely not.
In general, I agree that we should be trying to have true/good evaluation systems, not systems that are optimized for PR. I just think that right now we’re not on the Pareto frontier of that tradeoff.
I’m pretty sure the answer is “No, we can’t”. The whole point of DALY is that it lets us compare completely different interventions. If you replace it with something that is different in each context, you have not replaced it.
I think the best we can do is to calibrate it better, buy asking actual disabled people about their life quality. I think the answer will be very different depending on the disability, and also surrounding support and culture. This can be baked in, but you can’t change the waits around for different interventions.
Sorry if my comment was unclear. I don’t mean that we should use a different set of weights when looking at different interventions, I mean that we should use different weighting frameworks depending on the types of questions we are trying to ask. If we’re trying to quantify the impacts of different interventions on health outcomes, the post-2010 DALY scale might be reasonable. If we’re trying to quantify the impacts of different interventions on wellbeing, then WELLBYs might be reasonable. If we value improvements in health outcomes independent of their impact on subjective wellbeing, then some type of blended framework (e.g. GiveWell’s moral weighting scheme) might make sense.
I’ll return to the RP Moral Weights Project as an example of what I’m critiquing (the Moral Weight Project is fantastic in lots of ways, I don’t mean to say the whole project is bad because of this one critique). For the project, the authors are trying to develop weights that express animals’ changes in hedonic wellbeing in terms of human DALYs. But it’s not clear that DALYs are a coherent unit for what they’re trying to measure. The give trying to “estimate the welfare gain from, say, moving layer hens from cages to a cage-free system” as an example of the kind of application they’re looking at. But locking a human in a cage wouldn’t obviously change the number of DALYs gained in the world, at least under the post-2010 definition. For that application, a unit that included subjective wellbeing would make a lot more sense. That’s the kind of thing I’m trying to get at.
But I do agree with you that asking disabled people about their experiences and incorporating those results into whatever weighting scale we use is a very valuable step!