Great to see analysis like this on the forum, and I would love for more charities to try to lay out their impact like this.
I’m struggling a bit to get my head around this bit and wonder whether an alternative approach might work better (or maybe I’m just misunderstanding it): ”We used the average Global burden of disease DALY burden per patient in Uganda to estimate the DALY benefit of treating individual patients. This average includes everyone who suffered from each disease in Uganda, whether they were treated correctly, poorly or not at all. This accounts somewhat for what might have happened if we hadn’t treated the patient and avoids the counterfactual of assuming that patients would have not been treated without us.”
I think the main pathways for impact from your model are (please add if I’ve missed something): 1) reaching patients who otherwise would have missed out on care 2) improved timeliness of care (probably quite a big deal for malaria, maybe less so for family planning) 3) improved quality of care vs. alternative (unclear whether you are claiming this or not, I could easily believe this is a big factor if the alternative is faith healers or traditional medicine but less so if you think govt clinics are similar standard)
Estimating the proportion of 1) is crucial I think.
One way of generating more evidence would be a baseline of careseeking frequency from health facilities before you establish your centre. If it is 2 visits/year/family before and 4 visits/year/family after—that gives you a reasonable sense of how much additional access you are providing. It sounds like you might already have some of that data too.
So (made up numbers) if we were just thinking about malaria patients… say there are 10 per month, we could assume that 5 of those are ‘additional’ ones vs. counterfactual with no facility-> 100% of your treatment benefit is counted. The other 5 would have gotten slower care/poorer quality care → 50% of your treatment benefit is counted.
Patients will also benefit from reduced travel cost. I think you could model that as just the equivalent to a givedirectly donation with no overhead probably. Time savings for patients could be substantial also, I imagine for rural people who need to plant crops/harvest this could be a big factor.
Thanks Ray. I think it’s really valuable for smaller orgs like us to try and calculate our potential impact even with all the flaws!
Yes you’re exactly right with those 3 points driving our impact. I think improved quality (which includes common complete misdiagnosis in many settings) and timeliness might be nearly as important in driving impact as serving those who would have missed out on care completely. Its not like those who don’t get treated are likely to die, the human body is an incredible thing - without treatment we heal ourselves most of the time for most diseases, even malaria. Treatment Quality, prompt treatment and getting any treatment at all are all important impact factors
Sorry about the poor explanation—that’s my bad I should have done better. The average DALYs incurred by a Ugandan with any given disease seemed the best measure available at this time, as it takes the average DALYs per person of whole spectrum of people who get that disease. From those who got no treatment at all to the majority who would get treatment. It’s one of the few ways I could think of to get I’m very open to other ways of calculating DALYs averted per individual patient . At the time neither PSI or myself could think of a better one.
Measuring careseeking behaviour is a good thought, we have considered measuring this (we don’t right now). One of the issues is that variability of malaria prevalence is so high that it can confound the data. For example let’s say the first year there’s a high malaria season and they visit healthcare 8x a year, then the second year is low malaria and they only visit 4x. It looks like careseeking behaviour is worsening but it’s just that there’s ess malaria. Obviously we could try and control for this using regional malaria data but it ain’t easy. Also how do we account for going to a drug shop and buying a few pills? Does that count as accessing healthcare? There is much depth to these things.
Also decreased careseeking behaviour can even be the opposite, a sign of better health in the community. If an ODH health center had been doing good work treating patients well and the community is getting generally healthier, they will need to visit the facility less often. If people are treated poorly on the other hand the could end up coming back 5 times for the same condition. It’s complicated that’s for sure but I still think looking at healthseeking beahviour could have value!
I like your idea of 100%, 50% benefit etc and I might hit you up about that for futre analysis. We stlil run into that same problem though that we still need to decide what 100% benefit actually means in DALYs. We still need to pull that from somewhere—the problem described above that we currently use the GBD DALYs per person as a proxy for. Our current approach kind of does take this into account in a blunt and flawed way, as the average patent in Uganda takes into account whole spectrum of patient treatment (High quality, late, not at all)
Yes saving money a big factor and I like your idea of modelling it perhaps using a givedirectly model. I even thought about trying to include those benefits in the analysis, but it seemed like a lot both to do and present all at once. WE should definitely do this soon!
Thanks for the explanation, definitely agree that there are some big limitations on looking at careseeking behaviour in that way. No perfect solution but possibly excluding malaria cases as they are so seasonal would be appropriate, or if you can collect baseline data for a year then you can compare month to month.
suggests that in their intervention, treating an additional 124 cases of diarrhoea = saving almost 5 DALYs (if my quick skim of table 3 is right). That’s modelled I think, but might be a good additional datapoint.
One of the things I like about modeling direct/indirect patient financial benefits is that it should allow a high-confidence lower bound for ODH’s effectiveness. Let’s unrealistically and uncharitably assume that all of ODH’s patients would have counterfactually sought care at a more distant facility, and would have received as-timely and as-good care. In that case, the net benefit to patients includes avoided travel costs + loss-of-time costs + other-care costs (the other care is supposed to be free, but often isn’t in practice for various reasons), less fees paid to ODH.
In that pessimistic scenario, we should also include estimated (but difficult to calculate!) benefits to those served by the public health care system, because ODH’s existence diverted tens of thousands of patients a year from that underresourced and underfunded system. (Unless ODH scaled to multiple times its size, I doubt its existence would funge overall public health-care spending in Uganda. It’s just too small to affect a nationwide budget given how political processes generally work.)
A somewhat more realistic, but still probably awfully conservative, model would credit ODH (say) 25% of the DALYs in Nick’s model and 75% of the income effects in the model described above. Because the DALY source includes both treated and untreated cases, such a mixed model would assume more than 75% of patients would be counterfactually treated.
Doing a really quick BOTEC, that would yield 3.425 DALYs plus the [75% of net economic effects from the financial-only model, less $87.50[1]] for a donor/grantor spend of $137.50. If we assume $3 savings per patient in the financial-only model (based on Nick’s travel estimate of $3 + time value of money and other-care costs equal to the ODH fee), that yields us a patient savings of 112.5 patients (75% of average volume) * $3, or $337.50. Subtracting the $87.50 yields $250. If you think those savings are equal in value to GiveDirectly cash transfers, you could model that at 1.81X GiveDirectly + getting the DALYs (and benefits to non-ODH patients from relieving the burden on nearby government health centers) for “free.” (I note that this is primarily intended to demonstrate the idea of a mixed-benefits model only, not to make assertion about a reasonable lower-bound estimate for ODH.)
I’m probably doing this wrong, but GiveWell’s moral weights seem to suggest a DALY is morally “worth” about doubling the consumption of a person for two years. Given the median income in Uganda is $804, averting a DALY would be roughly equal to increasing consumption by $1600, which would make averting 3.425 DALYs roughly as effective as increasing consumption by $5480. I don’t mean to assert that ODH is 41.67X GiveDirectly on this conservative model ($5480 from DALYs + $250 from economic savings / $137.50 donor spend), but this quick analysis does suggest that the bulk of the value in the mixed-model described above comes from the “free” components and not the hypothesized 1.81X GiveDirectly economic effect.
Great to see analysis like this on the forum, and I would love for more charities to try to lay out their impact like this.
I’m struggling a bit to get my head around this bit and wonder whether an alternative approach might work better (or maybe I’m just misunderstanding it):
”We used the average Global burden of disease DALY burden per patient in Uganda to estimate the DALY benefit of treating individual patients. This average includes everyone who suffered from each disease in Uganda, whether they were treated correctly, poorly or not at all. This accounts somewhat for what might have happened if we hadn’t treated the patient and avoids the counterfactual of assuming that patients would have not been treated without us.”
I think the main pathways for impact from your model are (please add if I’ve missed something):
1) reaching patients who otherwise would have missed out on care
2) improved timeliness of care (probably quite a big deal for malaria, maybe less so for family planning)
3) improved quality of care vs. alternative (unclear whether you are claiming this or not, I could easily believe this is a big factor if the alternative is faith healers or traditional medicine but less so if you think govt clinics are similar standard)
Estimating the proportion of 1) is crucial I think.
One way of generating more evidence would be a baseline of careseeking frequency from health facilities before you establish your centre. If it is 2 visits/year/family before and 4 visits/year/family after—that gives you a reasonable sense of how much additional access you are providing. It sounds like you might already have some of that data too.
So (made up numbers) if we were just thinking about malaria patients… say there are 10 per month, we could assume that 5 of those are ‘additional’ ones vs. counterfactual with no facility-> 100% of your treatment benefit is counted. The other 5 would have gotten slower care/poorer quality care → 50% of your treatment benefit is counted.
Patients will also benefit from reduced travel cost. I think you could model that as just the equivalent to a givedirectly donation with no overhead probably. Time savings for patients could be substantial also, I imagine for rural people who need to plant crops/harvest this could be a big factor.
Thanks for the writeup!
Thanks Ray. I think it’s really valuable for smaller orgs like us to try and calculate our potential impact even with all the flaws!
Yes you’re exactly right with those 3 points driving our impact. I think improved quality (which includes common complete misdiagnosis in many settings) and timeliness might be nearly as important in driving impact as serving those who would have missed out on care completely. Its not like those who don’t get treated are likely to die, the human body is an incredible thing - without treatment we heal ourselves most of the time for most diseases, even malaria. Treatment Quality, prompt treatment and getting any treatment at all are all important impact factors
Sorry about the poor explanation—that’s my bad I should have done better. The average DALYs incurred by a Ugandan with any given disease seemed the best measure available at this time, as it takes the average DALYs per person of whole spectrum of people who get that disease. From those who got no treatment at all to the majority who would get treatment. It’s one of the few ways I could think of to get I’m very open to other ways of calculating DALYs averted per individual patient . At the time neither PSI or myself could think of a better one.
Measuring careseeking behaviour is a good thought, we have considered measuring this (we don’t right now). One of the issues is that variability of malaria prevalence is so high that it can confound the data. For example let’s say the first year there’s a high malaria season and they visit healthcare 8x a year, then the second year is low malaria and they only visit 4x. It looks like careseeking behaviour is worsening but it’s just that there’s ess malaria. Obviously we could try and control for this using regional malaria data but it ain’t easy. Also how do we account for going to a drug shop and buying a few pills? Does that count as accessing healthcare? There is much depth to these things.
Also decreased careseeking behaviour can even be the opposite, a sign of better health in the community. If an ODH health center had been doing good work treating patients well and the community is getting generally healthier, they will need to visit the facility less often. If people are treated poorly on the other hand the could end up coming back 5 times for the same condition. It’s complicated that’s for sure but I still think looking at healthseeking beahviour could have value!
I like your idea of 100%, 50% benefit etc and I might hit you up about that for futre analysis. We stlil run into that same problem though that we still need to decide what 100% benefit actually means in DALYs. We still need to pull that from somewhere—the problem described above that we currently use the GBD DALYs per person as a proxy for. Our current approach kind of does take this into account in a blunt and flawed way, as the average patent in Uganda takes into account whole spectrum of patient treatment (High quality, late, not at all)
Yes saving money a big factor and I like your idea of modelling it perhaps using a givedirectly model. I even thought about trying to include those benefits in the analysis, but it seemed like a lot both to do and present all at once. WE should definitely do this soon!
Thanks for the explanation, definitely agree that there are some big limitations on looking at careseeking behaviour in that way. No perfect solution but possibly excluding malaria cases as they are so seasonal would be appropriate, or if you can collect baseline data for a year then you can compare month to month.
I think existing cost-effectiveness studies might be something you can mine to get to DALY/case… for instance, this study here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8757489/#!po=51.5625
suggests that in their intervention, treating an additional 124 cases of diarrhoea = saving almost 5 DALYs (if my quick skim of table 3 is right). That’s modelled I think, but might be a good additional datapoint.
One of the things I like about modeling direct/indirect patient financial benefits is that it should allow a high-confidence lower bound for ODH’s effectiveness. Let’s unrealistically and uncharitably assume that all of ODH’s patients would have counterfactually sought care at a more distant facility, and would have received as-timely and as-good care. In that case, the net benefit to patients includes avoided travel costs + loss-of-time costs + other-care costs (the other care is supposed to be free, but often isn’t in practice for various reasons), less fees paid to ODH.
In that pessimistic scenario, we should also include estimated (but difficult to calculate!) benefits to those served by the public health care system, because ODH’s existence diverted tens of thousands of patients a year from that underresourced and underfunded system. (Unless ODH scaled to multiple times its size, I doubt its existence would funge overall public health-care spending in Uganda. It’s just too small to affect a nationwide budget given how political processes generally work.)
A somewhat more realistic, but still probably awfully conservative, model would credit ODH (say) 25% of the DALYs in Nick’s model and 75% of the income effects in the model described above. Because the DALY source includes both treated and untreated cases, such a mixed model would assume more than 75% of patients would be counterfactually treated.
Doing a really quick BOTEC, that would yield 3.425 DALYs plus the [75% of net economic effects from the financial-only model, less $87.50[1]] for a donor/grantor spend of $137.50. If we assume $3 savings per patient in the financial-only model (based on Nick’s travel estimate of $3 + time value of money and other-care costs equal to the ODH fee), that yields us a patient savings of 112.5 patients (75% of average volume) * $3, or $337.50. Subtracting the $87.50 yields $250. If you think those savings are equal in value to GiveDirectly cash transfers, you could model that at 1.81X GiveDirectly + getting the DALYs (and benefits to non-ODH patients from relieving the burden on nearby government health centers) for “free.” (I note that this is primarily intended to demonstrate the idea of a mixed-benefits model only, not to make assertion about a reasonable lower-bound estimate for ODH.)
I’m probably doing this wrong, but GiveWell’s moral weights seem to suggest a DALY is morally “worth” about doubling the consumption of a person for two years. Given the median income in Uganda is $804, averting a DALY would be roughly equal to increasing consumption by $1600, which would make averting 3.425 DALYs roughly as effective as increasing consumption by $5480. I don’t mean to assert that ODH is 41.67X GiveDirectly on this conservative model ($5480 from DALYs + $250 from economic savings / $137.50 donor spend), but this quick analysis does suggest that the bulk of the value in the mixed-model described above comes from the “free” components and not the hypothesized 1.81X GiveDirectly economic effect.
If 25% of visits are counterfactual, then 25% of patient expenses would never have occured.