[Linkpost] Claim that US cancer research is comparable to Givewell top charities in DALYs per $

[Jalen_Lyle-Holmes]

Link post

I just read this in the Astral Codex Ten post of links for February. I haven’t looked into it at all myself or even read the report itself. (And Scott doesn’t explicitly say ‘Givewell’, I’m just assuming that’s what he means by “effective altruists have tentatively found one or two opportunities in the poorest parts of Africa to save lives at $100/​DALY”). From Astral Codex Ten:

Study: federal cancer funding is extraordinarily effective. Cancer research produces so many valuable treatments that it saves one DALY per $326 spent. For comparison, health systems usually consider an intervention good value-for-money if it saves at least one DALY per $50,000. By combing the Earth far and wide, effective altruists have tentatively found one or two opportunities in the poorest parts of Africa to save lives at $100/​DALY, but these are extremely rare exceptions and I wouldn’t have expected anything in the US to be within an order of magnitude of that. Either this finding is fake, or we should all be donating to federal cancer research instead of whatever else we’re doing.

(I should also note Scott’s disclaimer at the top of his post: I haven’t independently verified each link. On average, commenters will end up spotting evidence that around two or three of the links in each links post are wrong or misleading. I correct these as I see them, and will highlight important corrections later, but I can’t guarantee I will have caught them all by the time you read this)

The report: Population, Clinical, and Scientific Impact of National Cancer Institute’s National Clinical Trials Network Treatment Studies

[Jeff Kaufman]

Zvi’s objections seem pretty reasonable:

I’ll say in advance this is based on a quick read, so it’s plausible some of these issues are my misunderstanding of what was going on, if only because it seems so stupid. If I did misunderstand one or more points here, I apologize, but the whole thing seems pretty terrible.

1. The average dollar spent is going to be vastly more effective than the marginal dollar spent going forward, since the low hanging fruit will already be gone.

2. Cancer treatments are neither cheap nor pleasant. The calculation here counts lives saved without counting the time and money spent on treatment. Yes, you won’t be paying that bill, but the bill is real and represents the loss of real resources that would have otherwise likely gone to other ‘life saving’ spending.

3. Crowding out effects are real. At least a large portion of these findings would have happened anyway, and the Feds are taking credit for the full effect of the treatments studied.

4. That’s right, this essentially counts any life saved through cancer treatment, ever.

5. Study results are being translated into estimated gains at the population level, at face value. Which we all know won’t hold up in practice, it never does – they won’t work as well in the field, and also won’t be implemented where you’d want.

6. Also I don’t see anything in the analysis of impact on how often the treatments actually got administered, at all? As in, maybe I’m missing something, but I can’t find the part where they check how many people actually got cancer treatments in order to estimate how many lives got saved.

7. Instead, they seem to be using the formula: On the basis of a previously published method, for each trial-proven new treatment for a given type of cancer, life-years gained (LYG) at the population (Pop) level was calculated as the product of model-estimated additional life accrued to the average patient (Pt) and multiplied by the number of patients in the cancer population (NCaPop) who would benefit from the new treatment (ie, LYGPop = LYGPt × NCaPop).9.

8. And then: To derive the number of patients in the cancer population to whom the new treatment would apply (NCaPop), we matched the major cancer type, stage, tumor characteristic, prior cancer, surgery, sex, and age (ie, ≥ 18 years) eligibility criteria from the clinical trial to corresponding cancer population data using the Surveillance, Epidemiology, and End Results (SEER) program.

9. That’s not how many people did benefit. That’s how many people they say in theory would benefit if we gave everything to everyone.

[Burner1989]

It’s a very interesting study and a compelling idea. I think the big issue is that we need to look at the marginal impact of extra dollars on cancer research, this is looking at the average impact of money spent on cancer drug trials. Expected effectiveness of more money should be lower, as the most promising drugs are more likely to already have funding.