Fantastic work. very excited to see Rethink Priorities branch out into more meta questions on how to measure what value is and so on. Excited to read the next few posts when I have time
A few thoughts:
1. Have you done much stakeholder engagement? One thing that was not here (although maybe I have to wait for post 9 on this) that I would love to see is some idea of how this work feeds through to change. Have you met with staff at NICE or Gates or DCP other policy professionals and talked to them about why they are not improving these metrics and how excited they would be to have someone work on improving these metrics. (This feels like the kind of step that should be taken before the project goes too far).
2. Problem 4 - neglect of spillover affects – probably cannot be solved by changing the metric. It feels more like an issue with the way the metric is used. You sort of cover this when you say “The appropriate response is unclear.” I expect making the metric include all spillover affects is the wrong approach as the spillover effects are often quite uncertain and quantifying the high uncertainty effects and within the main metric seems problematic. That said I am not sure about this so just chipping in my two cents.
(For example when I worked at Treasury we refused to consider spillover effects at all, I think because there was a view that any policy could be justified by someone claiming it had spillover effects. Then again the National Audit Office did say our own spending measures were not leading to long-term value for money so maybe that was the wrong approach.)
3. Who would you recommend to fund if I want to see more work like this? Who do you recommend funding if I want to see more work like this or a project to improve and change these metrics. You personally? Rethink Priorities? Happier Lives Institute? Someone else? Nobody at present?
1. Have you done much stakeholder engagement?
No. I discuss this a little bit in this section of Part 2, but I basically just suggest that people look into this and come up with a strategy before spending a huge amount of time on the research. I do know of academics who would may be able to advise on this, e.g. people who have developed previous metrics in consultation with NICE etc, but they’re busy and I suspect they wouldn’t want to invest a lot of time into efforts outside academia.
I think they’d reject the assumption that they are “not improving these metrics” and would point to considerable quantities of research in this area. The main issue, I think, is that they want a different kind of metric that what I’m proposing, e.g. they think it’s important that they are based on public preferences and are focused on health rather than wellbeing. A lot of resources are going into what I see (perhaps unfairly) as “tinkering around the edges,” e.g. testing variations of the time tradeoff/DCE and different versions of the EQ-5D, rather than addressing the fundamental problems.
As I say in Part 3 with respect to the sHALY (SWB-based HALY):
In my view, the strongest reason not to do this project is the apparent lack of interest among key stakeholders. Clinicians, patients, and major HALY “consumers” such as NICE and IHME seem strongly opposed to a pure SWB measure, even if focused on dimensions of health, and to the use of patient-reported values more broadly. As discussed in previous posts, this is due to a combination of normative concerns, such as the belief that those who pay for healthcare have the right to determine its distribution or that disability has disvalue beyond its effect on wellbeing, and doubts about the practicality of SWB measures in these domains.
So this project may only be worth considering if the sHALY would be useful for non-governmental purposes (e.g., within effective altruism), or in “supplementary” analyses alongside more standard methods (e.g., to highlight how QALYs neglect mental health). Either that, or changing the minds of large numbers of influential stakeholders will have to be a major part of the project—which may not be entirely unrealistic, given the increasing prominence of wellbeing in the public sector. We should also consider the possibility that projects such as this, which offer a viable alternative to the status quo, would themselves help to shift opinion.
That said, there is increasing increasing interest in hybrid health/wellbeing measures like the E-QALY, and scope for incremental improvement of current HALYs (see Part 2), and in the use of wellbeing for cross-sector prioritisation. In at least the latter case, you are likely to know more than me about how to effect policy change within governments.
2. Problem 4 - neglect of spillover affects – probably cannot be solved by changing the metric.
I discuss spillovers a little in Part 2 and plan to have a separate post on it in Part 6 (but it might be a while before that’s out, and it’s likely to focus on raising questions rather than providing solutions). I’m still unsure what to do about them and would like to see more research on this. I agree changing the metric alone won’t solve the issue, but it may help—knowing the extent to which the metric captures spillovers seems like an important starting point.
3. Who would you recommend to fund if I want to see more work like this?
It probably depends what your aims are. If it’s to influence NICE, IHME, etc, it probably has to go via academia or those institutions. If you want to develop a metric for use in EA, funding individual EAs or EA orgs may work—but even then, it’s probably wise to work closely with relevant academics to avoid reinventing the wheel. So I guess if you have a lot of money to throw at this, funding academics or PhD students may be a good bet; there is already some funding available (I’m applying for PhD scholarships in this area at the moment), but it may be hard to get funding for ideas that depart radically from existing approaches. I list some relevant institutions and individuals in Part 2.
4. How is the E-QALY project going?
It got very delayed due to COVID-19. I’m not sure what the new timeline is.
FYI the E-QALY work has been progressing quite well since you asked that question; I’ve just come out of a webinar on it. Let me know if you want me to send you notes/slides.
A few key points:
The measure has been named the EuroQol Health and Wellbeing (EQ-HWB); E-QALY seems to be what they are calling the broader project of extending the scope of the QALY.
Psychometric work and stakeholder consultation resulting in a 25-item ‘long’ measure, then further consultation resulted in a 9-item EQ-HWB-S (Short Form) covering 9 domains: Mobility, Daily activitie, Pain, Fatigue, Loneliness, Concentration & thinking clearly, Depression, Anxiety, Control.
A feasibility valuation study in 521 members of the UK public uses the time tradeoff (TTO, EQ-VT protocol) and discrete choice experiments (DCE). Due to covid this was done using video conferencing.
There was also a deliberative exercise with a 12-member panel of experts at NICE which reviewed the valuation results.
Based on the size of the utility decrement associated with the most severe level of each dimension, the order of importance is: Pain (by a long way); Mobility; Daily activities; Depression; Loneliness; Anxiety; Fatigue; Control; Concentration. (To me, the weight given to Mobility in particular might indicate that this measure does not overcome some of the biggest problems with earlier measures like the EQ-5D, though it seems to be much better overall.)
Other valuation studies, using different methodologies, are underway or planned. As far as I know, these don’t include ones that obtain weights based on SWB, but I think they will be looking at own-state utilities (i.e. weights derived from preferences of people with the relevant conditions).
Several papers are being published on it this year in a special edition of the journal Value in Health.
It started with a grant of 850,000 GBP; more has been spent since, but I’m not sure how much.
NICE still seems wedded to the EQ-5D for the foreseeable future, at least in standard health technology assessments, but they may use/accept the EQ-HWB in cases where broader effects are particularly important, e.g. impacts on carers.
Mukuria, C et al. “Qualitative Review on Domains of Quality of Life Important for Patients, Social Care Users, and Informal Carers to Inform the Development of the EQ Health and Wellbeing.” Value in Health (2022).
Monteiro AL, et al. A Comparison of a Preliminary Version of the EQ Health and Wellbeing Short and the 5-Level Version EQ-5D. Value Health. 2022 Mar 8:S1098-3015(22)00051-1. doi: 10.1016/j.jval.2022.01.003. Epub ahead of print. PMID: 35279371.
Hi Derek.
Fantastic work. very excited to see Rethink Priorities branch out into more meta questions on how to measure what value is and so on. Excited to read the next few posts when I have time
A few thoughts:
1. Have you done much stakeholder engagement? One thing that was not here (although maybe I have to wait for post 9 on this) that I would love to see is some idea of how this work feeds through to change. Have you met with staff at NICE or Gates or DCP other policy professionals and talked to them about why they are not improving these metrics and how excited they would be to have someone work on improving these metrics. (This feels like the kind of step that should be taken before the project goes too far).
2. Problem 4 - neglect of spillover affects – probably cannot be solved by changing the metric. It feels more like an issue with the way the metric is used. You sort of cover this when you say “The appropriate response is unclear.” I expect making the metric include all spillover affects is the wrong approach as the spillover effects are often quite uncertain and quantifying the high uncertainty effects and within the main metric seems problematic. That said I am not sure about this so just chipping in my two cents.
(For example when I worked at Treasury we refused to consider spillover effects at all, I think because there was a view that any policy could be justified by someone claiming it had spillover effects. Then again the National Audit Office did say our own spending measures were not leading to long-term value for money so maybe that was the wrong approach.)
3. Who would you recommend to fund if I want to see more work like this? Who do you recommend funding if I want to see more work like this or a project to improve and change these metrics. You personally? Rethink Priorities? Happier Lives Institute? Someone else? Nobody at present?
4. How is the E-QALY project going? I clicked the link for the E-QALY project (https://scharr.dept.shef.ac.uk/e-qaly/about-the-project/) It says it finishes in 2019. Any idea what happened to it?
Best of luck with the rest of the project.
Hi Sam,
Thanks for the comments.
1. Have you done much stakeholder engagement? No. I discuss this a little bit in this section of Part 2, but I basically just suggest that people look into this and come up with a strategy before spending a huge amount of time on the research. I do know of academics who would may be able to advise on this, e.g. people who have developed previous metrics in consultation with NICE etc, but they’re busy and I suspect they wouldn’t want to invest a lot of time into efforts outside academia.
I think they’d reject the assumption that they are “not improving these metrics” and would point to considerable quantities of research in this area. The main issue, I think, is that they want a different kind of metric that what I’m proposing, e.g. they think it’s important that they are based on public preferences and are focused on health rather than wellbeing. A lot of resources are going into what I see (perhaps unfairly) as “tinkering around the edges,” e.g. testing variations of the time tradeoff/DCE and different versions of the EQ-5D, rather than addressing the fundamental problems.
As I say in Part 3 with respect to the sHALY (SWB-based HALY):
That said, there is increasing increasing interest in hybrid health/wellbeing measures like the E-QALY, and scope for incremental improvement of current HALYs (see Part 2), and in the use of wellbeing for cross-sector prioritisation. In at least the latter case, you are likely to know more than me about how to effect policy change within governments.
2. Problem 4 - neglect of spillover affects – probably cannot be solved by changing the metric. I discuss spillovers a little in Part 2 and plan to have a separate post on it in Part 6 (but it might be a while before that’s out, and it’s likely to focus on raising questions rather than providing solutions). I’m still unsure what to do about them and would like to see more research on this. I agree changing the metric alone won’t solve the issue, but it may help—knowing the extent to which the metric captures spillovers seems like an important starting point.
3. Who would you recommend to fund if I want to see more work like this? It probably depends what your aims are. If it’s to influence NICE, IHME, etc, it probably has to go via academia or those institutions. If you want to develop a metric for use in EA, funding individual EAs or EA orgs may work—but even then, it’s probably wise to work closely with relevant academics to avoid reinventing the wheel. So I guess if you have a lot of money to throw at this, funding academics or PhD students may be a good bet; there is already some funding available (I’m applying for PhD scholarships in this area at the moment), but it may be hard to get funding for ideas that depart radically from existing approaches. I list some relevant institutions and individuals in Part 2.
4. How is the E-QALY project going? It got very delayed due to COVID-19. I’m not sure what the new timeline is.
FYI the E-QALY work has been progressing quite well since you asked that question; I’ve just come out of a webinar on it. Let me know if you want me to send you notes/slides.
A few key points:
The measure has been named the EuroQol Health and Wellbeing (EQ-HWB); E-QALY seems to be what they are calling the broader project of extending the scope of the QALY.
Psychometric work and stakeholder consultation resulting in a 25-item ‘long’ measure, then further consultation resulted in a 9-item EQ-HWB-S (Short Form) covering 9 domains: Mobility, Daily activitie, Pain, Fatigue, Loneliness, Concentration & thinking clearly, Depression, Anxiety, Control.
A feasibility valuation study in 521 members of the UK public uses the time tradeoff (TTO, EQ-VT protocol) and discrete choice experiments (DCE). Due to covid this was done using video conferencing.
There was also a deliberative exercise with a 12-member panel of experts at NICE which reviewed the valuation results.
Based on the size of the utility decrement associated with the most severe level of each dimension, the order of importance is: Pain (by a long way); Mobility; Daily activities; Depression; Loneliness; Anxiety; Fatigue; Control; Concentration. (To me, the weight given to Mobility in particular might indicate that this measure does not overcome some of the biggest problems with earlier measures like the EQ-5D, though it seems to be much better overall.)
Other valuation studies, using different methodologies, are underway or planned. As far as I know, these don’t include ones that obtain weights based on SWB, but I think they will be looking at own-state utilities (i.e. weights derived from preferences of people with the relevant conditions).
Several papers are being published on it this year in a special edition of the journal Value in Health.
It started with a grant of 850,000 GBP; more has been spent since, but I’m not sure how much.
NICE still seems wedded to the EQ-5D for the foreseeable future, at least in standard health technology assessments, but they may use/accept the EQ-HWB in cases where broader effects are particularly important, e.g. impacts on carers.
[Recording of the talk and related papers]
You can now view the recording of the talk from Professor John Brazier—Extending the QALY beyond health—the EQ HWB (Health and Wellbeing)
Kaltura
https://digitalmedia.sheffield.ac.uk/media/t/1_8k5slrc4
YouTube
https://www.youtube.com/watch?v=KTlsIvqyhNI
Papers associated with this talk
Special issue of Value in Health Development papers:
Brazier, J et al. ‘The EQ-HWB: overview of the development of a measure of health and well-being and key results’. Value in Health. https://www.sciencedirect.com/science/article/pii/S1098301522000833
Mukuria, C et al. “Qualitative Review on Domains of Quality of Life Important for Patients, Social Care Users, and Informal Carers to Inform the Development of the EQ Health and Wellbeing.” Value in Health (2022).
https://www.sciencedirect.com/science/article/pii/S1098301521032277
Carlton, J et al. “Generation, Selection, and Face Validation of Items for a New Generic Measure of Quality of Life: The EQ Health and Wellbeing.” Value in Health (2022). https://www.sciencedirect.com/science/article/pii/S1098301522000109
Peasgood, T et al. “Developing a New Generic Health and Wellbeing Measure: Psychometric Survey Results for the EQ Health and Wellbeing.” Value in Health (2022). https://www.sciencedirect.com/science/article/pii/S1098301521031922
International papers:
Monteiro AL, et al. A Comparison of a Preliminary Version of the EQ Health and Wellbeing Short and the 5-Level Version EQ-5D. Value Health. 2022 Mar 8:S1098-3015(22)00051-1. doi: 10.1016/j.jval.2022.01.003. Epub ahead of print. PMID: 35279371.
Augustovski F, Argento F, Rocío R, Luz G, Mukuria C, Belizán M. The Development of a New International Generic Measure (EQ Health and Wellbeing): Face Validity And Psychometric Stages In Argentina. https://www.sciencedirect.com/science/article/abs/pii/S1098301522000134