Marshall

HealthLearn: Im­pact Eval­u­a­tion and Next Steps

How to Catch a ChatGPT Cheat: 7 Prac­ti­cal Tips

• MarshallDec 23, 2022, 11:55 AM

  18 points

  3 ∶ 0

  on: New in­ter­ven­tion: pay­ing farm­ers to not burn crops

  It’s great to see more thoughtful posts proposing new global health and development interventions. Upvoted.

  Just sharing as an FYI—one of the key papers you cite for the cost-effectiveness analysis issued a correction, and the revised estimate of DALYs averted is 149 thousand rather than 14.9 million. It looks like there was an order of magnitude error somewhere along the way in their calculations.

  What links here?

  • New in­ter­ven­tion: pay­ing farm­ers to not burn crops by Karthik Tadepalli (Dec 22, 2022, 5:18 PM; 88 points)

• MarshallNov 22, 2022, 12:24 PM

  3 points

  0 ∶ 0

  on: Shal­low Re­port on Hypertension

  Thanks—this is super interesting and I agree that hypertension is a promising cause area—and that taxes on unhealthy foods may be a promising intervention.

  The estimate of the degree to which an average 1 mg reduction in sodium consumption reduces incidence of high systolic blood pressure in a single country and hence the global disease burden of hypertension, relies on an extremely long and complicated chain of calculations – there is hence a high degree of uncertainty here.

  I agree with your assessment that this is an area of uncertainty. In particular, I think we need to be careful about assuming linear effects on BP and burden of disease based on a very small change in daily sodium intake. You’ve thought deeply about this problem, so would be keen to hear your thoughts on what I’ve sketched out below.

  You estimated that a tax would reduce sodium consumption by 67 mg per day; a meta-analyses suggests that reducing dietary salt by a mean of 4.4 g per day leads to a mean reduction of systolic BP of 4.18 mm Hg. There was a bigger drop (5.39 mm Hg) in hypertensive people. [Note: my best guess is that your estimate of sodium reduction is for molecular sodium whereas these numbers are for salt—NaCl - 4.4 g of NaCl would be about 1.7 g of Na.] Based on this, I would say it’s very generous to assume any more than 0.5 mm Hg average reduction in systolic BP from 67 mg less sodium per day. 0.5 mm Hg is well within measurement error. This may be a case where a very small effect multiplied by a huge number of people still has a huge effect, but I think more evidence would be helpful. In particular, can we find evidence that:

  • the relationship between sodium intake and BP is linear and holds true for milligram quantities of change? BP is a homeostatic system where I don’t think we can always assume linear effects.

  • very small reductions in BP over time lead to meaningful changes in health outcomes?

  Maybe deriving additional estimates of the health impacts of −67 mg/​day salt intake, using different reference classes, could reduce this uncertainty.

• MarshallSep 13, 2022, 10:14 PM

  10 points

  0 ∶ 0

  in reply to: Vaidehi Agarwalla 🔸’s comment on: Re­quest­ing feed­back: pro­posal for a new non­profit with po­ten­tial to be­come highly effective

  Thanks for this feedback! This is exactly why I posted, so before I provide any specific responses to your points, please know that I appreciate all of the questions and suggestions and I’m already thinking of how they could be addressed in a future version of this proposal.

  1. I appreciate your point that the key step in the theory of change is not clear—and I think this is not due to a gap in the data itself but instead due to a gap in my presentation of the evidence. The key supporting evidence is linked out from this statement:

  My analysis of existing research studies shows that training HWs to properly care for newborn babies is likely to be highly cost-effective, with an average cost of $59 per DALY averted ($100 per DALY averted is sometimes cited as a benchmark for highly effective interventions)....

  The linked post cites six studies that show reductions in mortality due to HW training. While there are remaining reasons for skepticism, I think these six studies support this key step in the theory of change, at least for some types of training. Regarding your sub-points on point (1), I accept the feedback that we can and should provide more detail on the evaluation in a future version of this. The six studies provide pretty clear guidance on the type of data we would collect.

  2. I agree that a roadmap of regions /​ countries /​ priority courses would be helpful to include and can add this to a future version. Thanks for the suggestion. We’d want to start with topics that have the strongest existing evidence base (such as neonatal care and management of childhood illness).

  3. The dollar amount may seem high, but this is a technology development project. I think it will be very difficult to build a truly excellent learning platform that is tailored to this target audience without attracting top engineering talent, and that gets expensive. As I mentioned in the post, we’ve already done substantial piloting on a shoestring and I plan to continue to do that! I’ll think further about whether we can present a tiered approach, with additional pilots done with an MVP.

• MarshallSep 13, 2022, 6:35 PM

  2 points

  0 ∶ 0

  in reply to: SiebeRozendal’s comment on: Re­quest­ing feed­back: pro­posal for a new non­profit with po­ten­tial to be­come highly effective

  Many thanks for reading and for your suggestions, which I’ve acted on! The title is now updated :).

• MarshallSep 13, 2022, 6:29 PM

  1 point

  0 ∶ 0

  in reply to: Felix Wolf 🔸’s comment on: Re­quest­ing feed­back: pro­posal for a new non­profit with po­ten­tial to be­come highly effective

  Thanks! I have a few possible names but haven’t picked one (and the associated website domain name) yet. The pilots described here recently wrapped up but I’d be happy to share a demo module hosted on our MVP that’s focused on neonatal /​ child health. Please DM me if you’re interested.

Re­quest­ing feed­back: pro­posal for a new non­profit with po­ten­tial to be­come highly effective

• MarshallAug 26, 2022, 11:18 AM

  13 points

  0 ∶ 0

  on: A crit­i­cal re­view of GiveWell’s 2022 cost-effec­tive­ness model

  Thanks so much. This is a tour de force! I have one more suggestion about this model. I know that GiveWell has strong reasons to use its own metrics rather than DALYs or QALYs. The problem is that DALYs and QALYs are much more widely used in the academic literature.

  My suggestion is the model should report estimated $/​DALY averted in addition to (not instead of) the preferred units of cost-effectiveness as a multiple of cash transfers. This would:

  • Provide up-to-date benchmarks to catalyze shallow investigations of new cause areas. There’s no need to do a lot of modeling to quickly evaluate a new intervention that reports in units of $/​DALY averted.

  • Allow for a quick “reality check” of these models. Do they lead to results that are in the same ballpark as published estimates from the literature?

  Even if the EA cost-effectiveness units are indisputably better, the benefits of being able to engage more directly with the research community seem to outweigh the costs of adding a few rows to the model.

• MarshallAug 17, 2022, 12:39 PM

  2 points

  0 ∶ 0

  in reply to: Falk Lieder’s comment on: Find­ing be­fore fund­ing: Why EA should prob­a­bly in­vest more in research

  Thanks, your points make a lot of sense to me! The case does seem to be stronger for R&D generally and it’s helpful to know that you’re not arguing for investment in a specific stage of research. I also agree that targeting existing interventions for improvement could be very high yield :).

• MarshallAug 17, 2022, 11:53 AM

  8 points

  0 ∶ 0

  on: Find­ing be­fore fund­ing: Why EA should prob­a­bly in­vest more in research

  This is very neat, thanks for sharing! Some comments.

  First, the term “academic research” is used a lot in the text. Does this modeling speak to a need for more academic research or more research and development more generally?

  • It’s not clear to me that academia (as opposed to say, the nonprofit sector or even for-profit business) can claim the best track record of creating effective interventions. Academia might be involved at early stages but interventions often transition out of the academy when they scale up.

  • In other sectors, development (after the basic science research is done) costs the most and makes a product commercially viable. Maybe a majority of R&D spending should go towards development of interventions after initial proof-of-concept research is done?

  Secondly, this modeling seems sensitive to assumptions about the efficacy of new interventions.

  ″...we found that, on average, investing at least 50% of the initial annual budget in scientific research is optimal even if the new interventions are only about half as cost-effective as the best existing intervention, on average…”

  One could argue that new interventions are unlikely to be, on average, even half as effective as the best existing interventions, given that the best current interventions are recognized to be outliers (maybe even extreme outliers). Could you use some historical data to model average effectiveness of new interventions? There is a lot of cost-effectiveness data out there for public health interventions.

• MarshallAug 13, 2022, 2:55 PM

  9 points

  0 ∶ 0

  on: Struc­tur­ing new char­i­ties (even more) like tech star­tups

  Thanks for the great post!

  I completely agree and have been thinking about many of the same things. Many of the properties that make for-profit startups such a source of innovation (razor focus, fast decision-making, competition, rapid iteration) could also apply to nonprofit startups aiming to become highly effective. Here’s a bit of a challenge that I see in this:

  • The EA community has high standards of evidence for effective interventions.

  • Interventions that now have the highly effective label probably had high R&D costs to begin with, but now their unit costs are low.

  • It’s going to be very hard to achieve the highly effective benchmark when you’re just getting started (building an intervention, piloting it, evaluating it).

  • Central to the startup approach is the idea that you continuously and rapidly improve your intervention—iterating towards becoming highly effective.

  • Therefore, funders need to accept a high level of initial risk and be prepared to fund for some time before the highly effective label can be achieved.

  To extend your analogy to for-profit startups further, it seems there’s a need for a VC-like ecosystem, with multiple rounds of funding in escalating amounts and milestones based on progress towards highly cost-effective benchmarks.

  Also agreed that Charity Entrepreneurship is leading in this space. :)

• MarshallAug 13, 2022, 2:03 PM

  12 points

  0 ∶ 0

  on: We’re re­ally bad at guess­ing the future

  Thanks for the post. Some quick comments!

  I think that people now matter more than people in the future.

  This could be interpreted as a moral claim that, on an individual basis, a current person matters more than a counterfactual future person. Based on the rest of your post, I don’t think you’re claiming that at all. Instead you’re making arguments about the uncertainty of the future.

  I think a lot has been written about these claims around future uncertainty and those well-versed in longtermism have some compelling counterarguments. It would be nice to see a concise summary of the arguments for and against written in a way that’s really accessible to an EA newcomer.

• MarshallAug 6, 2022, 12:59 PM

  5 points

  0 ∶ 0

  on: Oral health­care in LMIC is a promis­ing new cause area: Founders Pledge (Medium In­ves­ti­ga­tion)

  Thanks for this great analysis. This definitely speaks to some limitations of the most widely-used global health and well-being metrics. It seems to me that sugar taxes could be particularly promising due to multiple public health benefits (not only improvements in dental health). This seems like a worthy contender for the Cause Exploration Prizes; you might consider submitting it if you haven’t already!

• MarshallJul 20, 2022, 11:16 AM

  7 points

  0 ∶ 0

  in reply to: Eli Rose’s comment on: Why EAs are skep­ti­cal about AI Safety

  Thanks! I agree—AI risk is at a much earlier stage of development as a field. Even as the field develops and experts can be identified, I would not expect a very high degree of consensus. Expert consensus is more achievable for existential risks such as climate science and asteroid impacts that can be mathematically modeled with high historical accuracy—there’s less to dispute on empirical /​ logical grounds.

  A campaign to educate skeptics seems appropriate for a mature field with high consensus, whereas constructively engaging skeptics supports the advancement of a nascent field with low consensus.

• MarshallJul 19, 2022, 10:57 AM

  9 points

  0 ∶ 0

  on: Why EAs are skep­ti­cal about AI Safety

  Thanks! I thought this was great. I really like the goals of fostering a more in-depth discussion and understanding skeptics’ viewpoints.

  I’m not sure about modeling a follow-up project on Skeptical Science, which is intended (in large part) to rebut misinformation about climate change. There’s essentially consensus in the scientific community that human beings are causing climate change, so such a project seems appropriate.

  • Is there an equally high level of expert consensus on the existential risks posed by AI?

  • Have all of the strongest of the AI safety skeptics’ arguments been thoroughly debunked using evidence, logic, and reason?

  If the answer to either of these questions is “no,” then maybe more foundational work (in the vein of this interview project) should be done first. I like your idea of using double crux interviews to determine which arguments are the most important.

  One other idea would be to invite some prominent skeptics and proponents to synthesize the best of their arguments and debate them, live or in writing, with an emphasis on clear, jargon-free language (maybe such a project already exists?).

• MarshallJul 15, 2022, 11:04 AM

  4 points

  0 ∶ 0

  on: Cause Ex­plo­ra­tion Prizes: Could New Tech­nol­ogy Help Solve the Glasses Prob­lem?

  Thanks to the authors for an excellent post! This is clearly an important problem. I wonder if it would be possible to estimate cost-effectiveness ($ per DALY averted) of some existing programs to norm this against global health and development causes currently favored by the EA community, such as GiveWell’s top charities?

• MarshallJul 6, 2022, 12:59 PM

  3 points

  0 ∶ 0

  in reply to: Richard R’s comment on: New cause area: train­ing health work­ers to pre­vent new­born deaths

  Thanks for the comment! I agree with you—ensuring that the training works truly is the key. There are multiple lines of evidence showing that it’s entirely possible to create effective online training for health workers—all of the technology exists. There’s more to be said on this than can be covered well in one comment, but here are some thoughts.

  • It’s critical to have an understanding of why most online learning doesn’t work well and deliberately design better solutions based upon that understanding.

  • There’s plenty of research demonstrating that online learning can increase knowledge and clinical skills of HWs living in LMICs (and other research demonstrating the same in high income countries).

  • Keep in mind that in-service HWs are working adults—they have greater motivations and capacities for self-regulated learning than they did when they were students, particularly when that learning is directly applicable to their work.

  All that said, there’s much work to do in terms of developing better trainings, evaluating them, and measuring their impacts on clinical practices and public health outcomes!

• MarshallJun 20, 2022, 7:22 PM

  2 points

  0 ∶ 0

  in reply to: Karthik Tadepalli’s comment on: New cause area: train­ing health work­ers to pre­vent new­born deaths

  Thanks for the comment and for reading! I agree with your interpretation of neglectedness: even when we account for both government and philanthropic aid, neonatal health is underfunded. My main point there is that philanthropic interventions will need to complement national priorities given the huge role that governments play in this space.

  You’re also correct that delivering trainings is complex, but I think it’s surmountable. As far as I know, many of the cost-effectiveness studies cited were done in government-supported health facilities, so there’s some existing proof of concept. As mentioned in the next steps, NGOs need to align carefully with governments to ensure that trainings reach HWs. One step in the right direction would be to get courses accredited for continuing professional development, which provides an important incentive.

New cause area: train­ing health work­ers to pre­vent new­born deaths