Interesting, I got the opposite impression from their about page (“4,000+ hospitals and health centers served, 51% fewer deaths from postpartum hemorrhaging in hospitals Zipline serves, 96% of providers report increased access to vaccinations in their area” which I assume means they’re already targeting those hard-to-access areas), but of course they’d want to paint themselves in a good light and I’d be inclined to trust your in the field experience far more (plus general skepticism just being a sensible starting point).
Actually your point about a cheap bike being able to carry a lot more stuff makes obvious sense, and so me wonder how Zipline’s modelling study in Ghana can claim that their cost per incremental fully immunised child was cheaper than “monthly immunization by mobile teams” which I assume includes dirt bikes.
Don’t be inclined to trust my in-the-field experience, Zipline has plenty of that too!
I just had a read of their study but couldn’t see how they calculated costing (the most important thing).
One thing to note is that vaccine supply chains currently often unnecessarily use trucks and cars rather than motorcycles because, well, GAVI has funded them so they may well be fairly comparing to status quo rather than other more efficient methods. For the life of me I don’t know why so many NGOs use cars for si many things that public transport and motorcycles could do sometimes orders of magnitude cheaper. Comparing to status quo is a fair enough thing to do (probably what I would do) but might not be investigating the most cost effective way of doing things.
Also I doubt they are including R and D and the real drone costs in the costs in of that study, but I’ll try and dig and get more detail.
It annoys me that most modeling studies focus so hard on their math method, rather than explaining now about how they estimate their cost input data—which is really what defines the model itself.
The modelling study has a “costs” section (quoted below), but for what it’s worth GiveWell said they “were unable to quickly assess how key parameters like program costs… were being estimated” so I don’t think this quote will satisfy you:
Given the Ghana Health Service (GHS)‘s dominant role, the government perspective in this analysis included healthcare treatment costs and incremental last mile delivery (LMD) costs. The societal perspective also accounted for externalities such as caregivers’ wage loss and transport costs.
To calculate the total cost for aerial LMD of vaccines, we analyzed Zipline’s monthly operational costs and the depreciation of capital expenditures for the GH4 distribution center in the Western North Region. These were adjusted to 2023 US dollar values, and the corresponding portion attributed to vaccine delivery was determined, resulting in a cost per dose of $0.27.
To estimate the incremental cost of the intervention, we took into account that the impact of aerial logistics on vaccination rates can be explained through either a pure expansion of access (ie, health facilities receiving vaccine doses that they otherwise would not have) or more efficient access (ie, health facilities receiving the same number of vaccine doses they would have otherwise received but in a more timely manner, leading to fewer missed opportunities of vaccination). Anecdotal evidence suggests that the impact is likely a combination of both factors. The distinction is significant when computing costs in an ICER: in the former, aerial logistics LMD cost is an additional expense to the existing supply chain cost for the government, whereas, in the latter, aerial logistics LMD replaces the traditional supply chain cost for transporting those vaccines. …
Due to the absence of detailed data on traditional LMD, we were unable to differentiate between incremental and replaced doses within the number of doses delivered with aerial logistics during the intervention period. To mitigate the impact of this uncertainty on our estimations, for our primary ICER calculation, we proceeded with the conservative assumption that all doses delivered by aerial logistics during this period were incremental. This approach may inflate our incremental cost estimates but ensures the solidity of our findings amid the well-known ambiguous quality and high variance of the traditional LMD data that were used for illustrative purposes in the sensitivity analysis.
But no input numbers, just methods and a dash of conservatism.
I share your annoyance re: modelling studies. Garbage in garbage out as they say (not accusing Zipline of putting garbage data into their model of course!)
Re: NGOs using trucks and cars unnecessarily, I’m just speculating here but I wonder if it’s got a bit to do with the NGOs wanting to attract “top talent” (salary difference being the main attractor but also “you get to ride in a car instead of on a bike” being implicitly part of the “comp package”, sort of like how top talent in higher-income countries are lured to prestigious industries by not just pay but “comped stays in nice hotels” or whatever). This paper I read awhile back made me think of that: The unintended consequences of NGO-provided aid on government services in Uganda. It argues that NGOs sometimes “poach” scarce local skilled government workers via higher pay, resulting in various adverse effects, although I guess it’s a bit different in this case because the adverse effects happen as a result of the pay structure (NGO workers who would’ve otherwise distributed health products instead sell household products like soap and fortified oil because they get paid on a per-piece basis).
Interesting, I got the opposite impression from their about page (“4,000+ hospitals and health centers served, 51% fewer deaths from postpartum hemorrhaging in hospitals Zipline serves, 96% of providers report increased access to vaccinations in their area” which I assume means they’re already targeting those hard-to-access areas), but of course they’d want to paint themselves in a good light and I’d be inclined to trust your in the field experience far more (plus general skepticism just being a sensible starting point).
Actually your point about a cheap bike being able to carry a lot more stuff makes obvious sense, and so me wonder how Zipline’s modelling study in Ghana can claim that their cost per incremental fully immunised child was cheaper than “monthly immunization by mobile teams” which I assume includes dirt bikes.
Don’t be inclined to trust my in-the-field experience, Zipline has plenty of that too!
I just had a read of their study but couldn’t see how they calculated costing (the most important thing).
One thing to note is that vaccine supply chains currently often unnecessarily use trucks and cars rather than motorcycles because, well, GAVI has funded them so they may well be fairly comparing to status quo rather than other more efficient methods. For the life of me I don’t know why so many NGOs use cars for si many things that public transport and motorcycles could do sometimes orders of magnitude cheaper. Comparing to status quo is a fair enough thing to do (probably what I would do) but might not be investigating the most cost effective way of doing things.
Also I doubt they are including R and D and the real drone costs in the costs in of that study, but I’ll try and dig and get more detail.
It annoys me that most modeling studies focus so hard on their math method, rather than explaining now about how they estimate their cost input data—which is really what defines the model itself.
The modelling study has a “costs” section (quoted below), but for what it’s worth GiveWell said they “were unable to quickly assess how key parameters like program costs… were being estimated” so I don’t think this quote will satisfy you:
But no input numbers, just methods and a dash of conservatism.
I share your annoyance re: modelling studies. Garbage in garbage out as they say (not accusing Zipline of putting garbage data into their model of course!)
Re: NGOs using trucks and cars unnecessarily, I’m just speculating here but I wonder if it’s got a bit to do with the NGOs wanting to attract “top talent” (salary difference being the main attractor but also “you get to ride in a car instead of on a bike” being implicitly part of the “comp package”, sort of like how top talent in higher-income countries are lured to prestigious industries by not just pay but “comped stays in nice hotels” or whatever). This paper I read awhile back made me think of that: The unintended consequences of NGO-provided aid on government services in Uganda. It argues that NGOs sometimes “poach” scarce local skilled government workers via higher pay, resulting in various adverse effects, although I guess it’s a bit different in this case because the adverse effects happen as a result of the pay structure (NGO workers who would’ve otherwise distributed health products instead sell household products like soap and fortified oil because they get paid on a per-piece basis).