Thanks so much for the comment and the questions. I really appreciate you reading this and thinking about it, especially given you are so engaged in longtermism stuff. Loved the skepitical braindump on existential risk from AI.
Not Nitpicks at all
1) This is the biggest weakness in our calculation. We use the global burden of disease data, because it estimates the average DALY burden of everyone who gets a disease (say malaria) in Uganda whether they were treated early lateor not at all. OneDay Health centers treatthe people who previously had the worst healthcare access in Uganda, so we assume that our treatment could remove at least that nationwide average DALY burden. You couldargue that this still is likely to overestimate the benefit of treating patients, but you could argue the opposite direction too. Without RCT data I think this is the best we can do at the moment—I couldn’t think of a better way to estimate the DALYs we might avert for each disease and also others in big orgs like PSI have used this approach before which is reassuring.
5km can seem like a short distance, but when you or your child are sick it can be very difficult to walk that far. For what it’s worth, the WHO uses this 5km threshold as a key accessibility indicator in countries like Uganda where walking is the only option for the vast majority of rural subsistance farmers. Also for context only 5-10% of Ugandans live in these healthcare black holes so the majority of people are in a better situation.
5km is only an indicator too—most of our patients live further than this from a government health center as well. It’s also a very complicated ecosystem as many government health centers don’t have enough medications so people have to buy the medication anyway even when you reach there. Distance isn’t the whole story, I simplified somewhat for this post
2) Subsidising transport would be (close to) impossible for a variety of reasons, mostly the first 2 I thnk
- How would you decide who to give the money to? Over the phone it would be impossible to know who was really sick and needed money. The system would likely fall down pretty quickly as everyone claimed to be sick. I might be strawmanning your proposed system though if so let me know! Give directly only works because they give unconditional money to most people in a given area.
- Many people in “Healthcare black holes” (only the most remote 5-10% of Uganda) don’t have good access to phones or cellphone reception which would be necessary for any kind of transport program.
- Motorbike transport means are not easily available in some of these areas (not the most important barrier, and motorbike accessibility is improving all the time).
Moreover, a transit-subsidy model would overload the already under-resourced government health centers with additional patients compared to what they would have if ODH did not exist at all. So you’d need to account for the likely increased costs (due to, e.g., medications that patients had to purchase from private sources) and worsened care quality for all patients in those facilities.
In contrast, the existence of an ODH health center should reduce the demand on government health centers, which may well create some benefits for other users of those facilities. Nick’s model does not attempt to capture those benefits.
Some more information about accessibility in Uganda is available here although the most important map is hard to read. ODH works in Northern Uganda, where the access to healthcare is significantly lower than near Kampala.
1) Ok, so let me try to rephrase and then you can tell me whether this makes sense
Per here, for an individual malaria patient, you are calculating the impact as:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working[1].
And then we are both thinking, well, this should really be something like:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working × Chance that the treatment was counterfactual.
But then you are pointing out that your term for “Badness of malaria” is actually the “Burden of disease of malaria”, which is actually how bad malaria is, but given that some of the patients are already receiving treatment. So in the original estimate, you didn’t have the counterfactual adjustment, but in exchange your “Badness of malaria factor” was too low.
So that’s my paraphrasing this so far. Do you think it’s mostly correct?
Then, what my intuition tells me one could do would be to try to:
a) Make the “Badness of malaria” factor be “Badness of untreated malaria” and model the “Chance that the treatment was counterfactual” factor explicitly. But then this wouldn’t capture all the benefits, so then maybe have some upwards adjustment for quality of care?
b) If gettting estimates for the badness of malaria is somehow too difficult, keep the burden of disease number, but add the counterfactual adjustment, and note that this is explicitly an underestimate.
At this point though, the counterfactual adjustment might get really gnarly:
perhaps the most severe cases would have counterfactually been treated more often? Unclear how much this could reduce your estimate of impact.
One could also model downstream effects, e.g., maybe making it less urgent for the Ugandan gvmt to have more hospitals? But then freeing its capacity to do other things, etc.
The more factors you start to consider, the less this would be an apples-to-apples comparison against, say, GiveWell’s estimate of AMF. Like, they have some adjustment for what would happen if AMF didn’t exist, but do I buy those estimates? Not sure.
One could also just have the non-counterfactually adjusted numbers, compare to the non-counterfactually adjusted AMF numbers, and leave it as an exercise to the reader to input the relative counterfactual adjustment[2]. Idk, maybe this is too cute.
My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment]. Anyways, hope that these thoughts are midly useful, though obviously I’m missing lots of context and looking at this from a really abstracted perspective.
E.g., if the non-counterfactually-adjusted impact for ODH is X, and the non-counterfactually adjusted impact for AMF is Y, and you are choosing between the two, you don’t actually have to calculate the counterfactual adjustments for both, you can just estimate the ratio R of “how much more counterfactual is ODH than AMF”, and then see if X * R > Y. As I said, maybe too cute.
Your paraphrasing is amazing (probably better than my original post). I just fear you know my brain a bit better than I do. Are you the first GAI? I also don’t feel like your analysis is that abstracted at all—your points seem quite concrete actually.
One small correction I might make is that most Ugandans who get malaria would get treatment, not just a few. We target the 5-10% of places which are really remote and getting treatment is difficult—that’s what where here for.
It’s an interesting idea to do “Badness of untreated malaria” x “Chance the treatment was counterfactual”. This is a cleaner method than what I did that’s for sure. The first issue with this is that I’m not sure we have a clear data point for badness of untreated malaria (although I can look into this more). Obviously impossible to study now and we need to rely on older data.
The chance of treatment (yes or no) is counterfactual would be more realistic to find, but is very black and white when really there’s a lot more too it than getting treatment or not. Quality of care is important—but perhaps even more important like @Ray_Kennedypointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.
On your counterfactual adjustments (love it)
The most severe cases would be more likely to get treated yes (interesting point never thought of ths) - but as a counterpoint as already discussed early treatment is really important. Often (not always) severe malaria is a direct result of inadequate, late or non-treatment.
Yes there’s definitely the counterfactual of the government doing less because NGOs are doing the work. A given thing which should probably warrant a dscount with basically any NGO program! As a caveat here, the Ugandan government made the active decision to spend their resources upgrading current facilities rather than building more in remote places. In the last 10 years the government has opened 0 (or maybe 1) new health center in Northern Uganda. Also of course like many other NGOs we have the dream that perhaps someday a government could see the value in our model and potentially take it over or roll out a version of their own. So this could be a tiny point in our favour (see my bias kicking in again...)
(Side note) Givewell seems quite rough on AMF on this front, positing a 50% chance that other funders would come in if they weren’t there in some countries. This feels overconservative to me—what might well be happening is that Global Fund (or others) instead fund more nets in other places instead of working where AMF are working
The only comment I didn’t really understand was the difference between where the vale comes from.
“My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment].”
I might well be missing something, but better quality of care and more “convenient” treatment (meaning people get earlier treatment) both avert deaths and save DALYs, just like getting treatment vs. not getting at all does. So doesn’t it all play into the same value proposition?
I might well be missing something, but better quality of care and more “convenient” treatment (meaning people get earlier treatment) both avert deaths and save DALYs, just like getting treatment vs. not getting at all does. So doesn’t it all play into the same value proposition?
See, me missing context matters here. I was imagining that the most pessimistic scenario would be that:
ODH provides treatment for malaria which is faster, nearer & more convenient
but patients would have otherwise gotten the same treatment, just later, further away and paying more costs to get it
So the value of ODH wouldn’t be the value of the treatment, it would be the value of making it more convenient
But as you point out (“Quality of care is important—but perhaps even more important like @Ray_Kennedypointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.”) you can’t just neatly separate getting faster care from getting better care. There are some fun things you could do with distributions, i.e., explicitly model the benefit as a function of how fast you get treatment, and then estimate the counterfactual value as
∫∫ (Value of getting treatment in h hours—Chance of having otherwise gotten treatment in (h + x) hours instead × Value of getting treatment in (h + x) hours) dx dh
(where the double integral just means that you are explicitly estimating the value of each possible pair of x and h and then weighing them according to how likely they are)
But I think this would be overkill, and only worth coming back to do explicitly if/when ODH is spending a few million a year. Still they might add some clarity if we don’t do the calculations. Anyways, best of luck.
On the counterfactual of the government potentially doing less, I speculate that it would be politically difficult for the government to copy ODH’s business model under which 2⁄3 of total costs are covered by patient fees. Specifically, my understanding is that user fees for public healthcare were dropped in the early 2000s, although as a practical matter the public system isn’t always free. Reinstating official fees only in certain areas probably wouldn’t fly well politically. So the government would likely have to spend several times what ODH does to set up the same health centers, and that is probably relevant to assessing the odds that it might counterfactually do so.
Thanks so much for the comment and the questions. I really appreciate you reading this and thinking about it, especially given you are so engaged in longtermism stuff. Loved the skepitical braindump on existential risk from AI.
Not Nitpicks at all
1) This is the biggest weakness in our calculation. We use the global burden of disease data, because it estimates the average DALY burden of everyone who gets a disease (say malaria) in Uganda whether they were treated early lateor not at all. OneDay Health centers treatthe people who previously had the worst healthcare access in Uganda, so we assume that our treatment could remove at least that nationwide average DALY burden. You couldargue that this still is likely to overestimate the benefit of treating patients, but you could argue the opposite direction too. Without RCT data I think this is the best we can do at the moment—I couldn’t think of a better way to estimate the DALYs we might avert for each disease and also others in big orgs like PSI have used this approach before which is reassuring.
5km can seem like a short distance, but when you or your child are sick it can be very difficult to walk that far. For what it’s worth, the WHO uses this 5km threshold as a key accessibility indicator in countries like Uganda where walking is the only option for the vast majority of rural subsistance farmers. Also for context only 5-10% of Ugandans live in these healthcare black holes so the majority of people are in a better situation.
5km is only an indicator too—most of our patients live further than this from a government health center as well. It’s also a very complicated ecosystem as many government health centers don’t have enough medications so people have to buy the medication anyway even when you reach there. Distance isn’t the whole story, I simplified somewhat for this post
2) Subsidising transport would be (close to) impossible for a variety of reasons, mostly the first 2 I thnk
- How would you decide who to give the money to? Over the phone it would be impossible to know who was really sick and needed money. The system would likely fall down pretty quickly as everyone claimed to be sick. I might be strawmanning your proposed system though if so let me know! Give directly only works because they give unconditional money to most people in a given area.
- Many people in “Healthcare black holes” (only the most remote 5-10% of Uganda) don’t have good access to phones or cellphone reception which would be necessary for any kind of transport program.
- Motorbike transport means are not easily available in some of these areas (not the most important barrier, and motorbike accessibility is improving all the time).
Moreover, a transit-subsidy model would overload the already under-resourced government health centers with additional patients compared to what they would have if ODH did not exist at all. So you’d need to account for the likely increased costs (due to, e.g., medications that patients had to purchase from private sources) and worsened care quality for all patients in those facilities.
In contrast, the existence of an ODH health center should reduce the demand on government health centers, which may well create some benefits for other users of those facilities. Nick’s model does not attempt to capture those benefits.
Some more information about accessibility in Uganda is available here although the most important map is hard to read. ODH works in Northern Uganda, where the access to healthcare is significantly lower than near Kampala.
1) Ok, so let me try to rephrase and then you can tell me whether this makes sense
Per here, for an individual malaria patient, you are calculating the impact as:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working[1].
And then we are both thinking, well, this should really be something like:
Badness of malaria × Chance of correct diagnosis × Chance of treatment regime working × Chance that the treatment was counterfactual.
But then you are pointing out that your term for “Badness of malaria” is actually the “Burden of disease of malaria”, which is actually how bad malaria is, but given that some of the patients are already receiving treatment. So in the original estimate, you didn’t have the counterfactual adjustment, but in exchange your “Badness of malaria factor” was too low.
So that’s my paraphrasing this so far. Do you think it’s mostly correct?
Then, what my intuition tells me one could do would be to try to:
a) Make the “Badness of malaria” factor be “Badness of untreated malaria” and model the “Chance that the treatment was counterfactual” factor explicitly. But then this wouldn’t capture all the benefits, so then maybe have some upwards adjustment for quality of care?
b) If gettting estimates for the badness of malaria is somehow too difficult, keep the burden of disease number, but add the counterfactual adjustment, and note that this is explicitly an underestimate.
At this point though, the counterfactual adjustment might get really gnarly:
perhaps the most severe cases would have counterfactually been treated more often? Unclear how much this could reduce your estimate of impact.
One could also model downstream effects, e.g., maybe making it less urgent for the Ugandan gvmt to have more hospitals? But then freeing its capacity to do other things, etc.
The more factors you start to consider, the less this would be an apples-to-apples comparison against, say, GiveWell’s estimate of AMF. Like, they have some adjustment for what would happen if AMF didn’t exist, but do I buy those estimates? Not sure.
One could also just have the non-counterfactually adjusted numbers, compare to the non-counterfactually adjusted AMF numbers, and leave it as an exercise to the reader to input the relative counterfactual adjustment[2]. Idk, maybe this is too cute.
My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment]. Anyways, hope that these thoughts are midly useful, though obviously I’m missing lots of context and looking at this from a really abstracted perspective.
Or, chance that the treatment works × magnitude of the improvement, if we are being punctillious.
E.g., if the non-counterfactually-adjusted impact for ODH is X, and the non-counterfactually adjusted impact for AMF is Y, and you are choosing between the two, you don’t actually have to calculate the counterfactual adjustments for both, you can just estimate the ratio R of “how much more counterfactual is ODH than AMF”, and then see if X * R > Y. As I said, maybe too cute.
Your paraphrasing is amazing (probably better than my original post). I just fear you know my brain a bit better than I do. Are you the first GAI? I also don’t feel like your analysis is that abstracted at all—your points seem quite concrete actually.
One small correction I might make is that most Ugandans who get malaria would get treatment, not just a few. We target the 5-10% of places which are really remote and getting treatment is difficult—that’s what where here for.
It’s an interesting idea to do “Badness of untreated malaria” x “Chance the treatment was counterfactual”. This is a cleaner method than what I did that’s for sure. The first issue with this is that I’m not sure we have a clear data point for badness of untreated malaria (although I can look into this more). Obviously impossible to study now and we need to rely on older data.
The chance of treatment (yes or no) is counterfactual would be more realistic to find, but is very black and white when really there’s a lot more too it than getting treatment or not. Quality of care is important—but perhaps even more important like @Ray_Kennedy pointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.
On your counterfactual adjustments (love it)
The most severe cases would be more likely to get treated yes (interesting point never thought of ths) - but as a counterpoint as already discussed early treatment is really important. Often (not always) severe malaria is a direct result of inadequate, late or non-treatment.
Yes there’s definitely the counterfactual of the government doing less because NGOs are doing the work. A given thing which should probably warrant a dscount with basically any NGO program! As a caveat here, the Ugandan government made the active decision to spend their resources upgrading current facilities rather than building more in remote places. In the last 10 years the government has opened 0 (or maybe 1) new health center in Northern Uganda. Also of course like many other NGOs we have the dream that perhaps someday a government could see the value in our model and potentially take it over or roll out a version of their own. So this could be a tiny point in our favour (see my bias kicking in again...)
(Side note) Givewell seems quite rough on AMF on this front, positing a 50% chance that other funders would come in if they weren’t there in some countries. This feels overconservative to me—what might well be happening is that Global Fund (or others) instead fund more nets in other places instead of working where AMF are working
The only comment I didn’t really understand was the difference between where the vale comes from.
“My biggest uncertainty would be to what extent the value of ODH comes from [counterfactually averting deaths/saving lots of DALYs], vs [providing better quality of care, or more convenient treatment].”
I might well be missing something, but better quality of care and more “convenient” treatment (meaning people get earlier treatment) both avert deaths and save DALYs, just like getting treatment vs. not getting at all does. So doesn’t it all play into the same value proposition?
See, me missing context matters here. I was imagining that the most pessimistic scenario would be that:
ODH provides treatment for malaria which is faster, nearer & more convenient
but patients would have otherwise gotten the same treatment, just later, further away and paying more costs to get it
So the value of ODH wouldn’t be the value of the treatment, it would be the value of making it more convenient
But as you point out (“Quality of care is important—but perhaps even more important like @Ray_Kennedy pointed out is how quickly people get the treatment. Malaria is an exponentially replicating parasite, and hours can make a differece.”) you can’t just neatly separate getting faster care from getting better care. There are some fun things you could do with distributions, i.e., explicitly model the benefit as a function of how fast you get treatment, and then estimate the counterfactual value as
∫∫ (Value of getting treatment in h hours—Chance of having otherwise gotten treatment in (h + x) hours instead × Value of getting treatment in (h + x) hours) dx dh
(where the double integral just means that you are explicitly estimating the value of each possible pair of x and h and then weighing them according to how likely they are)
But I think this would be overkill, and only worth coming back to do explicitly if/when ODH is spending a few million a year. Still they might add some clarity if we don’t do the calculations. Anyways, best of luck.
On the counterfactual of the government potentially doing less, I speculate that it would be politically difficult for the government to copy ODH’s business model under which 2⁄3 of total costs are covered by patient fees. Specifically, my understanding is that user fees for public healthcare were dropped in the early 2000s, although as a practical matter the public system isn’t always free. Reinstating official fees only in certain areas probably wouldn’t fly well politically. So the government would likely have to spend several times what ODH does to set up the same health centers, and that is probably relevant to assessing the odds that it might counterfactually do so.