COVID: How did we do? How can we know?

Ghost_of_Li_Wenliang30 Jun 2021 12:29 UTC
170 points
47 comments9 min readEA link

Obviously it was a triumph: the fastest vaccine development, approval, and rollout in history (by a factor of 5). We’re up to 2.5bn doses in arms, out of say 12bn. And we got several good ones! Huzzah!

Obviously it was a catastrophe: through dumb inaction and a comedy of errors, we squandered the chance to suppress the virus. 4 million people are confirmed to have died of (or with) COVID—and given unbelievable underreporting that might be actually 12 million—and given Delta’s momentum 20 million by the end is not unlikely.* This is despite this virus being easy mode: unlike 1918, very few of the deaths were among the young frontline workers keeping the healthcare and delivery systems working; unlike SARS 1, post-viral disability is relatively rare. This is just the present pandemic and doesn’t count future deaths from letting the thing become a permanent fixture.

How did we do? How can we even answer that question?

When I talk about whether a given country’s response to COVID was a success or a failure, smart friends reply “governments had to balance the tradeoffs, so what looks like failure is really just compromise between multiple objectives (like economic activity)”, “it’s easy to say the optimal response in hindsight”, that “it’s difficult to compare different countries because of the different distances from China, wealth, state capacity”.

For instance, they think the UK did ok. They can think this because they choose to compare to the average actual response (never mind that the UK is top 20 in deaths per capita). But what would the best possible response look like? What did our institutions stop us from getting?

Vax

Any self-respecting COVID rant must foreground vaccination. It is the solution, where other policies just buy time, or else consume old or disabled people.

We underinvested, and prevented market investment.

The EU paid $14 per Pfizer dose. What was it really worth? The current black market price for Pfizer is about $500. But that’s a gross underestimate of the shadow price, since you get almost zero quality assurance or liability from darknet dealers. (You might still get the travel passport, depending on how weak your country’s infosec is.) One proper estimate of the per-vaccine social benefit is $6000.

So we should have spent trillions in massive pre-purchase, on every credible vaccine. (The much-praised Operation Warp Speed and its equivalents elsewhere only pre-purchased about 2bn out of the necessary 12-14bn, and did so shockingly late, in August 2020.)

$3000 x 14 bn doses pays for a lot of overtime on microlipid machine assembly (which was the bottleneck on mRNA vaccine supply last year).

(That’s assuming that you continue to ban vaccine markets, believing, as you apparently do, that fairness is worth the early death of millions. Another way to fund supply expansion for the global south is to just not get in our own way.)

The rich world defected, duh

16% of the world bought 70% of the vaccines. What force on earth could stop them? None, so we needed the massive supply increases, which were effectively banned.

This was not even good selfishness: it guaranteed the emergence of new strains in the global south.

This is the real evil of the EU procurement. They want to harm their own by delaying 4 weeks, to look strong? Well, that’s one thing. But had they done a pre-purchase in March 2020, then global supply could have scaled up, so that the inevitable snatch away from the global south was completely balanced out by expanded supply

What did we do? Overall, about 2bn doses ordered by August 2020, i.e. 5 times too little, 5 months too late.

The strange death of human challenge trials

Probably the biggest mistake was not intentionally infecting vaccinated volunteers. This could be done in 1 month, vs 6.5 months for the ecological trials that the entire world did out of misguided PR ethics. (2.5 is probably more realistic given signups, approvals, and big pharma’s slow data analysis and reporting. That’s still hundreds of thousands of lives.)

1DaySooner wrote a letter. The world’s foremost consequentialist signed. The world’s foremost deontologist signed. Two of the most prominent bioethicists in the world signed. 15 Nobellists signed. Dozens of philosophers who otherwise agree on extremely little signed. But they’re unethical.

Rarely do I so strongly feel the boot of others on my neck, and humanity’s neck.

The one distinctively courageous thing about the UK—the human challenge trials which got 40,000 volunteers—actually eventually started!.. In January 2021, with n=90.

I am extremely puzzled why China or one of the other ahem non-individualist governments didn’t do these.

What fraction of all vaccines were wasted?

  • by excessive dosing. No sign of the dosing regimes changing despite strong evidence. 50-75%?**

  • by bad needle design. 10%-30% more.

  • by rigid queueing leading to needless expiry of doses. 1%?

  • by not checking anyone for prior antibodies before using a dose on them. Maybe 1.1 bn people have had COVID (confirmed case count * 6.2 factor undercount). So something like a 14% risk of wasting any given dose. Actually not clear if the overhead of testing and delaying vaccine appointments would swamp the saved doses from this. Also supposedly 9% of natural COVID patients don’t have antibodies afterwards, and 5% lose them within 6 months(!)

So maybe most of em.

Stuff I’m not counting as waste

  • Logistics failures (cold chain faults, delivery errors, fill-to-finish spills, breakages). It’s a hard problem! Say 5%.

  • The experimental dermal injections which might quintuple supply.

  • I won’t talk about single-dosing the two-dose vaccines because although it looks like a win (e.g. Pfizer first dose: 94% effect against hospitalization from Delta, second dose 96%) there were credibly thought to be possible tradeoffs re: selection pressure. (No longer?)

The happy timeline: what to compare to

Taking the Moderna vaccine as a model:

  • Jan 2020: Beijing doesn’t lie and cover-up for 2 weeks. They don’t arrest or rectify. The genome goes out Jan 6th 2020 at the latest, 5-10 days early.

  • Moderna vaccine (etc) still designed in 2 days, prototyped in a month.

  • Feb 2020: Massive pre-purchase of all credible candidates ex ante, before the effectiveness results. Ideally billions each. New factories minted, Huoshenshan-style. Lead time on dozens of new microfluidics machines starts February, conceivably ready by August but maybe not til 2021. Whatever the next limiting factor is, we also identify it and start the lead time in February. Tap the existing spare capacity for 3bn doses.

  • I’m going to assume Phase I was already done as quick as possible, in 63 days.

  • April 2020: Challenge trial instead of ecological trials for Phase II and III. This would plausibly have taken 2.5 months instead of 6.5 months.

  • July 2020: Approval ready to go as soon as Phase III results out. (Unlike say the 3 week FDA delay.)

  • Approval sharing between UK /​ EU /​ US. Saves the EU three months on AZ.

  • New microfluidics machines spun up and debugged at some unknown point between August 2020 and August 2021. (The UK’s brand new vaccine factory will end up taking 16 months to build and ramp.)

What does that get us, and at what cost?

We approve in August 2020 instead of December. There were 700,000 confirmed deaths then. In December, when the earliest vaccinations began in our dark timeline, there were 1.4 million. (Again, maybe more like 3m.)

We needed 5 times more vaccines, preferably within 18 months. Was that even possible? One good simulation study looks at what it would take to scale mRNA production from 2bn to 8bn, and says $17bn for the chunky-dose vaccines. This decent study says $36bn and 2 years to scale up. (But that’s just the mRNA—they don’t seem to include the new microfluidics machines! Not to mention procurement and logistics, i.e. 90% of the total cost.) But add in the dose stretching and zero-dead-volume syringes, and we’re done in a year. I am confused by how few experts have produced estimates for this cost—one guy worked on both of the extant estimates I quote above—but then this whole pandemic has been an abyss for cost-benefit arguments.

Here’s a terrible guess:

  • The UK spent $16.3bn on doses, facilities, R&D and logistics so far. £1.3bn of that was the sticker price of the doses. That got them 267m doses. So naively, 12⁄0.267 * 16.3 = $732bn for 12bn doses. That would imply totally subsidising 12⁄0.267 = 44 different vaccines though, which is a bit much even for me. This also includes converting a plant into a vaccine factory, but let’s say 44 of those is good. Subtract the $860m R&D subsidy first then, then add 5 lots on: 12⁄0.267 * (16.3 − 0.86) = $693bn + 5 x 0.86 = $698bn.

  • IQVIA estimate a world vaccine spend of $53bn in 2021. That clearly can’t be the all-inclusive price. If 90% of costs are not doses but logistics, vaccinators, and procurement then it roughly accords with my guess.

  • It’s hard to see total charitable spending on R&D and purchase. For once it’s probably not too bad to ignore it.

  • Again, there were bottlenecks in the manufacturing which would not have allowed for 5 times the money to get 5 times the juice (within a year anyway). But $0.7tn is a decent guess.

  • If that looks like a lot, consider what the ongoing stimulus took.

The executive bodies of the world failed to pre-purchase enough, early enough to overcome an unknown number of bottlenecks. The health regulators of the world vetoed challenge trials and in some cases added months of pennywise bickering and font size rejections.

I’m not saying that the above justifies FDA abolition. I’m not even saying that we’d have been better off last year if the FDA, MHRA, SAMR, PMDA, and EMA simply did not exist. (We got lucky that the first vaccines were safe and so effective.) But it does permanently alter the balance of trust for me.

A mandate to cry wolf?

An obvious objection is that we didn’t know COVID was going to be COVID in February. Would it really have been optimal during swine flu (2009)?

First off, who’s “we”?

Secondly: fine, start it in March and scale down the savings by a quarter.

Third: we certainly knew that we were underprepared for a pandemic. That corrective investment had to happen at some point. Physical plant does depreciate, but over a decade, so helping us deal with the 10-30% risk of another pandemic.

Finally, building 10 more mRNA vaccine factories would be amortized pretty quickly even if no other pandemic happens. Thus, the above is not just hindsight bias or absurd overreaction.

Shut up and calculate

Probably best to follow the script here if you can.

  • Average 5 QALYs lost per COVID death, supposedly (seems low).

  • We’re not getting away without 2 million confirmed deaths (~5 million excess). So say 2 million (~5 million) saved as of now. Then project forward to overall pandemic deaths, which Metaculus has annoyingly stopped predicting so fill in your own number. Mine is 20 million, or 15m saved. So 25 mega QALYs on deaths up to June 2021, or conceivably 75 mega QALYs including all future COVID deaths.

    +

  • QALYs per averted infection: 1.51?? Way way too high; it counts secondary infections. Table 2 suggest that COVID is 0.43 QALY loss for symptomatic, and 0.5 for the hospitalized. Maybe 1bn averted by the happy path. Median duration of symptoms 27 days. 83% symptomatic supposedly. So 11.5 mega QALYs from symptoms averted.

    +

  • Long COVID burden. Very unclear. Martin et al use ARDS disutility as their estimate for long COVID: 0.32 QALY loss. This seems like too much but I don’t have anything better. 0.14% to 2% risk of permanent damage. So maybe 1.4 mega QALYs per year for the next 5?? years = 7??.

    +

  • QALY loss from fear, uncertainty, and lockdown. Surprisingly few estimates of how much last year sucked for uninfected people, maybe because it’s a tasteless question. I seriously think I have to go with Bryan Caplan’s twitter poll(!) At least 2 months less for say half of the world population, so ~151 mega QALYs. (I’m surprised too, but note that this isn’t about just not being able to go on holiday: it’s about the whole car crash, including the frustration at the response! And deaths would have been far higher without NPIs, so this isn’t to say “lockdown was worse than COVID”.)

Extremely rough cost per QALY: $2850. Treat this as you would treat working by a very drunk Enrico Fermi. (Run the script and change whatever seems off to you; do parameter sweeps of the dodgy bits.)

Another smart friend dismissed the above argument. He just couldn’t believe that politicians would leave such a deal on the table—and so really it wasn’t possible to speed it up so much. Sadly his cynicism is pointing in the wrong direction: the last year has to make you realise how incompetent our rulers are. (Not malign.)

Would such people really have been willing to buy a high-variance portfolio of hits-based vaccines, ex ante? to look like they wasted half of the public spend? To risk something for the public good? Yeah, some of them!

Bonfire of the Authorities

Is there any public organisation which can be proud of last year?

Several times (fomites, masks, asymptomatic spread, long COVID, schools, lab-leak) we saw dramatic reversals of global policy, based not on dramatic new evidence, but on shifts in how elite factions decided to interpret the evidence.

I’m not going to enumerate all of the errors, lies, feet dragging, or negative-sum bullshit. For that I can’t recommend this Twitter account enough, whose supply of accountability and context is what people think journalists do. Here’s a sprinkling:

  • WHO and all of public health on masks

  • The FDA refused to license any rapid home tests until November because their processes were designed for diagnostic tests (rather than the intended monitoring tests) and had no exception clauses.

  • “FDA insiders say the agency and its approximately 17,000 employees were dark for the four-day Thanksgiving holiday, including those working on the vaccine approval.”

  • The EU’s sluggishness, bullying, pandering to risk aversion, and total lack of (short-term) accountability for its own decisions. AZ approval came three months after UK approval, Pfizer three weeks after. Supposedly this was mostly taken up with haggling prices down from crazy low to crazy low. (EDIT: no, that was the procurement process, not the approval process.)

  • Behavioural science shown to be unfit for purpose. Systematic and fatal overestimate of risk compensation and “behavioural fatigue”. This continues, and will continue.

Many of these processes were run at normal speed; COVID cases weren’t reported over the weekend; in some places vaccinations paused for the weekend; pausing research grants; pausing vaccine approval!

It’s enough to make a ghost want to hoist the black flag and join the revolt.

The above requires no new science, no new social structures, no public executions. It just needed some courage, needed a small number of people to act as if they were at war, as if trillions of dollars were at stake, as if millions of people were.

This greatly upgrades my estimate of institutional decision-making as a cause, and in particular getting people willing to take (personal reputation) risks as high up as possible, everywhere.

* What about the proper EA units though?: One estimate says 32 million DALYs lost (Jan 2020 - Apr 2021), about the same as malaria. This seems like a serious underestimate to me, probably because they use confirmed deaths (understandable), lowball long COVID (eh), and say nothing about the QALY hit of COVID restrictions or long-term economic scarring(!).

** You might not believe that these things are arbitrary and then locked-in by regulators. Here’s another instance: the prescribed “3 week” gap between doses is not even close to optimal. This is a subtler kind of waste: wasting immunity rather than inoculum. Luckily the shortages forced us to fix this one.

EDIT: Did the QALY estimate as a Python script. Not much change.

Note: I’m not a engineer, I’m not an economist, I’m not a pharmacologist, I’m not an epidemiologist. There will be something wrong with the above, no doubt in misunderstanding the nature of vaccine engineering bottlenecks.

I hope it’s clear just how many of these numbers are guesses, as opposed to the sanctified guesses of peer-reviewed economics. (Martin et al: “Given the longer timeframe of our model… we felt 0.10 is appropriate”.) The QALY estimate could be off by a factor of 10. The key parameter, the microfluidics machine lead time, is based on how long the Braintree facility is taking and little else.

Ghost_of_Li_Wenliang30 Jun 2021 12:29 UTC
170 points
47 comments9 min readEA link