I spent most of my early career as a data analyst in industry, which engendered in me a deep wariness of quantitative data sources and plumbing, and a neverending discomfort at how often others tended to just take them as given for input into consequential decision-making, even if at an intellectual level I knew their constraints and other priorities justified it and they were doing the best they could. …and then I moved to global health applied research and realised that the data trustworthiness situation was so much worse I had to recalibrate a lot of expectations / intuitions.
Disease burden estimates, such as child mortality rates, are a key input in our cost-effectiveness analyses. Historically, for consistency and convenience, we’ve primarily relied on a single source for these estimates.
Going forward, we plan to consider multiple sources for burden estimates, apply a higher level of scrutiny to these estimates, and adjust for potential biases or inaccuracies, like we do when estimating other parameters in our models.
This change has already led to us making over $25m in additional grants we would not have otherwise. (Footnote: Our updated estimates of malaria burden in Chad have led us to allocate $3.3 million in grantmaking for seasonal malaria chemoprevention (more), and $25.9m for insecticide-treated nets (not yet published).) We expect to consider additional research to improve estimates of burden of disease in the future.
The rest of the note was cathartic to skim-read. For instance, when I looked into the idea of distributing low-cost glasses to correct presbyopia in low-income countries awhile back (a problem that afflicts over 1.8 billion people globally with >$50 billion in annual lost potential productivity annually in LMICs alone), the industry data analyst in me was dismayed to learn that the WHO didn’t even collect data on how many people needed glasses prior to 2008, so governments and associated stakeholders understandably prioritised allocation of resources towards surgical and medical interventions instead. I think the existence of orgs like IHME and OWID greatly improve the GHD data situation nowadays, but there are many “pockets” where it remains a far cry from what it could be, so I appreciated that GiveWell said they’re considering
Fund data collection. This includes potentially funding additional nationally representative surveys (DHS/MIS/MICS) or additional modules to these surveys, or supporting more autopsy data collection to better understand cause-specific mortality, particularly for malaria in sub-Saharan Africa. Our guess is that part of the reason different models disagree is that the data underlying these models is limited. We may look for cases where we could fund additional data collection to improve burden of disease estimates.
Another example: a fair bit of my earlier analyst work involved either reconciling discrepant figures for ostensibly similar metrics (e.g. campaign revenue breakdowns etc) or root-cause analysing-via-data-plumbing whether a flagged metric needed to be acted on or was a false positive, which made me appreciate this section:
Key uncertainties: …
There are likely technical nuances we haven’t captured. We’ve found that comparisons between sources are more complex than they first appear. For example, we recently learned that IGME and IHME define diarrheal diseases differently. Similar technical differences likely exist elsewhere.
Possible next steps:
Get a better understanding of what’s driving differences in models. This may come from bringing together modeling groups in regions with high disagreement to understand methodological differences.
Look for ways to improve model transparency. We’ve found it difficult to engage with burden of disease models, and think that finding ways to see inside the black box of how they produce estimates may make it easier to understand which estimates to rely on and how to improve them.
This is fantastic to hear! The Global burden of disease process (while the best and most reputable we have) is surprisingly opaque and hard to follow in many cases. I haven’t been able to find the spreadsheets with their calculations.
Their numbers are usually reasonable but bewildering in some cases and obviously wrong in others. GiveWell moving towards combining GBD with other sensible models is a great way forward.
Its a bit unfortunate that the best burden of disease models we have aren’t more understandable.
I spent most of my early career as a data analyst in industry, which engendered in me a deep wariness of quantitative data sources and plumbing, and a neverending discomfort at how often others tended to just take them as given for input into consequential decision-making, even if at an intellectual level I knew their constraints and other priorities justified it and they were doing the best they could. …and then I moved to global health applied research and realised that the data trustworthiness situation was so much worse I had to recalibrate a lot of expectations / intuitions.
In that regard I appreciate GiveWell’s new guidance on burden note:
The rest of the note was cathartic to skim-read. For instance, when I looked into the idea of distributing low-cost glasses to correct presbyopia in low-income countries awhile back (a problem that afflicts over 1.8 billion people globally with >$50 billion in annual lost potential productivity annually in LMICs alone), the industry data analyst in me was dismayed to learn that the WHO didn’t even collect data on how many people needed glasses prior to 2008, so governments and associated stakeholders understandably prioritised allocation of resources towards surgical and medical interventions instead. I think the existence of orgs like IHME and OWID greatly improve the GHD data situation nowadays, but there are many “pockets” where it remains a far cry from what it could be, so I appreciated that GiveWell said they’re considering
Another example: a fair bit of my earlier analyst work involved either reconciling discrepant figures for ostensibly similar metrics (e.g. campaign revenue breakdowns etc) or root-cause analysing-via-data-plumbing whether a flagged metric needed to be acted on or was a false positive, which made me appreciate this section:
This is fantastic to hear! The Global burden of disease process (while the best and most reputable we have) is surprisingly opaque and hard to follow in many cases. I haven’t been able to find the spreadsheets with their calculations.
Their numbers are usually reasonable but bewildering in some cases and obviously wrong in others. GiveWell moving towards combining GBD with other sensible models is a great way forward.
Its a bit unfortunate that the best burden of disease models we have aren’t more understandable.