Epistemic status: public attempt at self-deconfusion & not just stopping at knee-jerk skepticism
The recently published Cost-effectiveness of interventions for HIV/AIDS, malaria, syphilis, and tuberculosis in 128 countries: a meta-regression analysis (so recent it’s listed as being published next month), in my understanding, aims to fill country-specific gaps in CEAs for all interventions in all countries for HIV/AIDS, malaria, syphilis, and tuberculosis, to help national decision-makers allocate resources effectively – to a first approximation I think of it as “like the DCP3 but at country granularity and for Global Fund-focused programs”. They do this by predicting ICERs, IQRs, and 95% UIs in US$/DALY using the meta-regression parameters obtained from analysing ICERs published for these interventions (more here).
AFAICT their methodology and execution seem superb, so I was keen to see their results:
Antenatal syphilis screening ranks as the lowest median ICER in 81 (63%) of 128 countries, with median ICERs ranging from $3 (IQR 2–4) per DALY averted in Equatorial Guinea to $3473 (2244–5222) in Ukraine.
At risk of being overly skeptical: $3 per DALY averted is >30x better than Open Phil’s 1,000x bar of $100 per DALY which is roughly around GW top charity level which OP have said are hard to beat, especially for a direct intervention like antenatal syphilis screening. It makes me wonder how much credence to put in the study’s findings for actual resource allocation decisions (esp. Figure 4 ranking top interventions at country granularity). Also:
Specifically re: antenatal syphilis screening, CE/AIM’s report on screening + treating antenatal syphilis estimates $81 per DALY; I’m hard-pressed to believe that removing treatment improves cost-eff >1 OOM
I’m reminded of the time GW found 5 separate spreadsheet errors in a DCP2 estimate of soil-transmitted-helminth (STH) treatment that together misleadingly ‘improved’ its cost-effectiveness ~100-fold from $326.43 per DALY (correct output) to just $3.41 (wrong, and coincidentally in the ballpark of the estimate above that triggered my skepticism)
So how should I think about and use their findings given what seems like reasonable grounds for skepticism, if I’m primarily interested in helping decision-makers help people better? Scattered thoughts to defend the study / push back on my nitpicking above:
even if imperfect – and I’m not confident in my skepticism above – they clearly improve substantially upon the previous state of affairs (CEA gaps everywhere at country-disease-intervention level granularity; expert opinion not lending itself to country-specific predictions; case-by-case methods often being unsuccessful)
their recommendations seem reasonably hedged, not naively maximalist: they include 95% uncertainty intervals; they clearly say “cost-effectiveness… should not be the only criterion… [consider also] enhancing equity and providing financial risk protection”
even a naively maximalist recommendation (“first fund lowest-ICER intervention, then 2nd-lowest, … until funds run out”) doesn’t seem unreasonable in this context – essentially countries would end up funding more antenatal syphilis screening, intermittent preventive treatment of malaria in pregnant women and infants, and chemotherapy for drug-susceptible TB (just from eyeballing Figure 4)
I interpret what they’re trying to do as not so much “here are the ICER league tables, use them”, but shifting decision-makers’ approach to resource allocation from needing a single threshold for all healthcare funding decisions to (quoting them) “ICERs ranked in country-specific league tables”, and in the long run this perspective shift seems useful to “bake into” decision-making processes, even if the specific figures in this specific study aren’t necessarily the most accurate and shouldn’t be taken at face value
That said, I do wonder if the authors could have done a bit better, like
cautioning against naively taking the best cost-eff estimates at face value, instead of suggesting “Funds could be first spent on the intervention that has the lowest ICER. Following that, other interventions could be funded in order of their ICER rankings, as long as there are available funds”
spot-checking some of (not all) the top cost-eff ICERs that went into their meta-regression analysis to get a sense of their credibility, especially those which feed into their main recommendations, like GW did above with the DCP2 estimate for STH treatment
extracting qualitative proxies for decision-maker guidance from an analysis of the main drivers behind the substantial ranking differences in intervention ICERs across economic and epidemiological contexts (eg “we should expect antenatal syphilis screening to be substantially less cost-effective in our context due to factors XYZ, let’s look at other interventions instead” – what would a short useful list of XYZ look like?), instead of just saying “we found the rankings differ substantially”
The positive spin is that someone got funded to do this kind of big-picture analysis and got it published in The Lancet.
There were 1,792 potential country-intervention pairs (although it is not immediately clear if they did all 1,792 pairs). So I don’t think most reasonable readers would view these findings as substitutes for a more in-depth, country-specific analysis on the potentially promising intervention. They did publish at least some data for each intervention, although maybe it isn’t enough to poke at each of the country-intervention pairs.
Epistemic status: public attempt at self-deconfusion & not just stopping at knee-jerk skepticism
The recently published Cost-effectiveness of interventions for HIV/AIDS, malaria, syphilis, and tuberculosis in 128 countries: a meta-regression analysis (so recent it’s listed as being published next month), in my understanding, aims to fill country-specific gaps in CEAs for all interventions in all countries for HIV/AIDS, malaria, syphilis, and tuberculosis, to help national decision-makers allocate resources effectively – to a first approximation I think of it as “like the DCP3 but at country granularity and for Global Fund-focused programs”. They do this by predicting ICERs, IQRs, and 95% UIs in US$/DALY using the meta-regression parameters obtained from analysing ICERs published for these interventions (more here).
AFAICT their methodology and execution seem superb, so I was keen to see their results:
At risk of being overly skeptical: $3 per DALY averted is >30x better than Open Phil’s 1,000x bar of $100 per DALY which is roughly around GW top charity level which OP have said are hard to beat, especially for a direct intervention like antenatal syphilis screening. It makes me wonder how much credence to put in the study’s findings for actual resource allocation decisions (esp. Figure 4 ranking top interventions at country granularity). Also:
Specifically re: antenatal syphilis screening, CE/AIM’s report on screening + treating antenatal syphilis estimates $81 per DALY; I’m hard-pressed to believe that removing treatment improves cost-eff >1 OOM
I’m reminded of the time GW found 5 separate spreadsheet errors in a DCP2 estimate of soil-transmitted-helminth (STH) treatment that together misleadingly ‘improved’ its cost-effectiveness ~100-fold from $326.43 per DALY (correct output) to just $3.41 (wrong, and coincidentally in the ballpark of the estimate above that triggered my skepticism)
So how should I think about and use their findings given what seems like reasonable grounds for skepticism, if I’m primarily interested in helping decision-makers help people better? Scattered thoughts to defend the study / push back on my nitpicking above:
even if imperfect – and I’m not confident in my skepticism above – they clearly improve substantially upon the previous state of affairs (CEA gaps everywhere at country-disease-intervention level granularity; expert opinion not lending itself to country-specific predictions; case-by-case methods often being unsuccessful)
their recommendations seem reasonably hedged, not naively maximalist: they include 95% uncertainty intervals; they clearly say “cost-effectiveness… should not be the only criterion… [consider also] enhancing equity and providing financial risk protection”
even a naively maximalist recommendation (“first fund lowest-ICER intervention, then 2nd-lowest, … until funds run out”) doesn’t seem unreasonable in this context – essentially countries would end up funding more antenatal syphilis screening, intermittent preventive treatment of malaria in pregnant women and infants, and chemotherapy for drug-susceptible TB (just from eyeballing Figure 4)
I interpret what they’re trying to do as not so much “here are the ICER league tables, use them”, but shifting decision-makers’ approach to resource allocation from needing a single threshold for all healthcare funding decisions to (quoting them) “ICERs ranked in country-specific league tables”, and in the long run this perspective shift seems useful to “bake into” decision-making processes, even if the specific figures in this specific study aren’t necessarily the most accurate and shouldn’t be taken at face value
That said, I do wonder if the authors could have done a bit better, like
cautioning against naively taking the best cost-eff estimates at face value, instead of suggesting “Funds could be first spent on the intervention that has the lowest ICER. Following that, other interventions could be funded in order of their ICER rankings, as long as there are available funds”
spot-checking some of (not all) the top cost-eff ICERs that went into their meta-regression analysis to get a sense of their credibility, especially those which feed into their main recommendations, like GW did above with the DCP2 estimate for STH treatment
extracting qualitative proxies for decision-maker guidance from an analysis of the main drivers behind the substantial ranking differences in intervention ICERs across economic and epidemiological contexts (eg “we should expect antenatal syphilis screening to be substantially less cost-effective in our context due to factors XYZ, let’s look at other interventions instead” – what would a short useful list of XYZ look like?), instead of just saying “we found the rankings differ substantially”
The positive spin is that someone got funded to do this kind of big-picture analysis and got it published in The Lancet.
There were 1,792 potential country-intervention pairs (although it is not immediately clear if they did all 1,792 pairs). So I don’t think most reasonable readers would view these findings as substitutes for a more in-depth, country-specific analysis on the potentially promising intervention. They did publish at least some data for each intervention, although maybe it isn’t enough to poke at each of the country-intervention pairs.