Global Burden of Disease (GBD) is okay, it depends a lot on what disease & metric you’re looking at, and how aware you are of the caveats around it. Some of these:
A lot of the data is estimated, rather than real measurements of prevalence
I think most people understand this, but it’s always worth a reminder
The GBD provides credible intervals for all major statistics and these should definitely be used!
The moral weights for estimating the years lived with disability for a given disease are compiled from a wide survey of the general public
This means they’re based on people’s general belief of what it would be like to have that condition, even if they haven’t experienced it or know anyone who has
Older GBDs included expert opinion in their moral weights, but to remove biases they don’t do this anymore (IMHO I think this is the right call)
The estimates for prevalence are compiled differently per condition by experts in that condition
There is some overall standardisation, but equally, there’s some wiggle room for a motivated researcher to inflate their prevalence estimates. I assume the thinking is that these biases cancel out in some overall sense.
Overall, I think the GBD is very robust and an extremely useful tool, especially for (a) making direct comparisons between countries or diseases and (b) where no direct, trustworthy, country-specific data is available. But you should be able to improve on its accuracy if you have an inside view on a particular situation. I don’t think it’s subject to the incentives you mention above in quite the same way.
Chiming in to note a tangentially related experience that somewhat lowered my opinion of IHME/GBD, though I’m not a health economist or anything. I interacted with several analysts after requesting information related to IHME’s estimates for global hepatitis C burden (which differed substantially from the WHO’s). After a meeting and some emails promising to followup, we were ghosted. I have heard from one other organization that they’ve had a really hard time getting similar information out of IHME as well. This may be more of an organizational/operational problem rather than a methodological one, but it wasn’t very confidence-inspiring.
Whenever I do a sanity checks of GBD it usually make sense for UgAnda here where I live, with the possible exception of diarrhoea which I think is overrated (with moderate confidence).
I’m not sure exactly how GBD would “exaggerate” overall, because the contribution of every condition to the disease burden has to add up to the actual burden—if you were to exaggerate the effect of one condition you would have to intentionally downplay another to compensate, which seems unlikely. I would imagine mistakes on GBD are usually good faith mistakes rather than motivated exaggerations.
Nitpicky reply, but reflecting an attitude that I think has some value to emphasize:
Based on what you wrote, I think it would be far more accurate to describe GBD as ‘robust enough to be an useful tool for specific purposes’, rather than ‘very robust’.
Global Burden of Disease (GBD) is okay, it depends a lot on what disease & metric you’re looking at, and how aware you are of the caveats around it. Some of these:
A lot of the data is estimated, rather than real measurements of prevalence
I think most people understand this, but it’s always worth a reminder
The GBD provides credible intervals for all major statistics and these should definitely be used!
This paper on the Major Depressive Disorder estimates is a good overview for a specific disease
The moral weights for estimating the years lived with disability for a given disease are compiled from a wide survey of the general public
This means they’re based on people’s general belief of what it would be like to have that condition, even if they haven’t experienced it or know anyone who has
Older GBDs included expert opinion in their moral weights, but to remove biases they don’t do this anymore (IMHO I think this is the right call)
The estimates for prevalence are compiled differently per condition by experts in that condition
There is some overall standardisation, but equally, there’s some wiggle room for a motivated researcher to inflate their prevalence estimates. I assume the thinking is that these biases cancel out in some overall sense.
Overall, I think the GBD is very robust and an extremely useful tool, especially for (a) making direct comparisons between countries or diseases and (b) where no direct, trustworthy, country-specific data is available. But you should be able to improve on its accuracy if you have an inside view on a particular situation. I don’t think it’s subject to the incentives you mention above in quite the same way.
Chiming in to note a tangentially related experience that somewhat lowered my opinion of IHME/GBD, though I’m not a health economist or anything. I interacted with several analysts after requesting information related to IHME’s estimates for global hepatitis C burden (which differed substantially from the WHO’s). After a meeting and some emails promising to followup, we were ghosted. I have heard from one other organization that they’ve had a really hard time getting similar information out of IHME as well. This may be more of an organizational/operational problem rather than a methodological one, but it wasn’t very confidence-inspiring.
Whenever I do a sanity checks of GBD it usually make sense for UgAnda here where I live, with the possible exception of diarrhoea which I think is overrated (with moderate confidence).
I’m not sure exactly how GBD would “exaggerate” overall, because the contribution of every condition to the disease burden has to add up to the actual burden—if you were to exaggerate the effect of one condition you would have to intentionally downplay another to compensate, which seems unlikely. I would imagine mistakes on GBD are usually good faith mistakes rather than motivated exaggerations.
Nitpicky reply, but reflecting an attitude that I think has some value to emphasize:
Based on what you wrote, I think it would be far more accurate to describe GBD as ‘robust enough to be an useful tool for specific purposes’, rather than ‘very robust’.