Samuel Dupret

Pain re­lief: a shal­low cause exploration

Vida Plena Pre­dic­tive Cost-Effec­tive­ness Analysis

A can of worms: the non-sig­nifi­cant effect of de­worm­ing on hap­piness in the KLPS

• Samuel Dupret 24 Mar 2023 14:26 UTC

  41 points

  6 ∶ 0

  on: Assess­ment of Hap­pier Lives In­sti­tute’s Cost-Effec­tive­ness Anal­y­sis of StrongMinds

  As my colleagues have mentioned in their responses (Michael’s general response, Joel’s technical response), the WELLBYs per $1000 that GiveWell put forward for AMF are dependent on philosophical choices about the badness of death and the neutral point. There are a range of plausible possible choices and these can affect the results. HLI does not hold a view.
  
  We’ve whipped up an R Shiny app so that you, the reader, can play around with these choices and see how your views affect the comparison between StrongMinds and AMF.
  
  

  

  Please note that this is a work in progress and was done very quickly. Also, I’m using the free plan for hosting the app so it might be a bit slow/​limited in monthly bandwidth.

  What links here?

  • JoelMcGuire's comment on Assess­ment of Hap­pier Lives In­sti­tute’s Cost-Effec­tive­ness Anal­y­sis of StrongMinds by GiveWell (22 Mar 2023 21:59 UTC; 29 points)

• Samuel Dupret 20 Mar 2023 11:09 UTC

  17 points

  3 ∶ 0

  on: Why I’m suss on wellbe­ing surveys

  Hello Henry,

  Thank you for presenting this thought experiment.

  The core here is about whether groups like the Sentinelese who do not have the same levels of development as others would give similar levels of SWB. I think the other comments here have done a great job at pointing out possible explanations.

  • if Sentinelese have ⇒ wellbeing

    • maybe their lifestyle works really well for their wellbeing (as Charlie mentions, we might not want to be too quick to dismiss this possibility). It would be a cool area to research.

    • maybe there are issues of interpersonal comparability in scale use, which is what we are exploring with our pilot.

    • maybe they have access to factors (e.g., social ties) that improve wellbeing without being affected as much by reference frames (e.g., the benefit I get from a higher income is relative to other people) or hedonic adaptation (as pointed out by Alex).

  • if Sentinelese have < wellbeing

    • As Nick pointed out, countries with lower levels of development tend to have lower levels of LS. [edit: adding this figure here because it represents this well and supports the validity of SWB scales]

      

    • It might be well worth the investment (especially as Vasco pointed out, it can increase the number of humans and life expectancy)

    • It is possible that whilst our development reached a point where we have higher wellbeing, it did so through inefficient periods and our task now is to think more carefully about how to do (and whether there are adverse consequences like climate change, x-risks, etc.)

  Some briefs answers /​ pointers. Many of these things have been discussed in more details elsewhere.

  • The estimate for grief is shallow [edit: I want to make this point a bit stronger—this was a quick estimate and it is a tad unfair to compare it to the SM estimate which represents hundreds of hours of work and meta-analyses] but important in making the difficult work of live-saving vs life-improving accurate. You can see some discussion about it from my colleague. I, personally, wouldn’t be too surprised if we found a higher estimate in the future but there is some reasoning as to why this might not be as big.

  • I don’t think there are ‘objective’ measures of health widely used. DALYs and QALYs are not objective, they rely on humans reports—and contrary to SWB, these are reports about things humans are know to struggle to report on. In DALYs it is people (who do not have the conditions!) making binary judgements about which conditions are more or less healthy. In QALYs it generally involves people forecasting how their life will be in the future. See To WELLBY or not to WELLBY where we discuss the strengths and weaknesses of SWB.

  • health, wealth, education, are all instrumental. Why do we want these? Because they contribute to what we believe is ultimately good for humans. Most charity evaluators give there answer in some form of wellbeing /​ good, either by measuring SWB as directly as possible (HLI) or converting lives and income into the moral weights of the evaluators (GW).

  • There is extensive debate about the relationship between SWB and growth through the lens of the Easterlin Paradox

• Samuel Dupret 23 Mar 2023 11:53 UTC

  11 points

  3 ∶ 0

  in reply to: NickLaing’s comment on: Can we trust wellbe­ing sur­veys? A pi­lot study of com­pa­ra­bil­ity, lin­ear­ity, and neutrality

  Hi Nick,

  Thanks for pointing out both kinds of biases. These biases can cause a failure of comparability. Concretely, if an intervention causes you to give counterfactually higher scores as a matter of ‘courtesy’ to the researcher, then the intervention changed the meaning of each given response category.

  I therefore take it that you don’t think that our particular tests of comparability will cover the two biases you mention. If so, I agree. However, my colleague has given reasons for why we might not be as worried about these sorts of biases.

  I don’t think this can be tested in our current survey format, but it might be testable in a different design. We are open to suggestions!

• Samuel Dupret 23 Dec 2022 22:20 UTC

  9 points

  1 ∶ 0

  on: A can of worms: the non-sig­nifi­cant effect of de­worm­ing on hap­piness in the KLPS

  Thank you for your engagement with our research. Our research team is going on winter break so we might not respond to additional comments until after the holidays.

  The point is to form a view on whether we recommend deworming. What matters to us are effects on subjective wellbeing. We would like to restate that this is the only SWB data on deworming that we could find, and this the only SWB analysis that we know of. The effects are non-significant, so this data does not give us good grounds to recommend deworming (see Section 4). The broader literature on outcomes other than SWB (reviewed in Section 1.3) is so mixed that our results do not strike us as particularly surprising – it’s just one null result amongst many.

  We could just stop there, but because this is a novel analysis and dealing with non-significant findings isn’t easy, we aired on the side of being overly-thorough. For completeness, we discuss many other considerations in Section 2.3 (alternative analyses, Bayes factors, cost-effectiveness, converting GiveWell’s analysis to WELLBYs, etc. - cost-effectiveness being one of them). These converging lines of evidence all fail to provide good grounds to recommend deworming. Of course, non-significant findings do not prove there is no effect, and we discuss limitations of the data in section 5 that highlight key uncertainties about the results. In section 6, we discuss future research that could help address these limitations.

  If strong evidence is produced showing that deworming is more cost-effective than StrongMinds for SWB, then we would change our recommendation. Collecting this data could be expensive, but given the insufficient evidence we think it is necessary for proponents to clarify the effect of deworming on SWB (which we think is what ultimately matters).

  Have a nice end of the year and keep doing good folks.

• Samuel Dupret 12 Oct 2022 9:46 UTC

  9 points

  0 ∶ 0

  in reply to: Karthik Tadepalli’s comment on: To WELLBY or not to WELLBY? Mea­sur­ing non-health, non-pe­cu­niary benefits us­ing sub­jec­tive wellbeing

  Hello Karthik. Thank you for your comment. Apologies, it seems that we missed your comment at the time of posting so we’re providing our responses now.

  I worry that they over-weight the immediate effects of interventions, and underweight the long-term effects.

  This is not an issue with the measures, but rather how much data we can collect for them.

  The problem is that SWB data have to be collected with a much higher frequency than income/​health data. By their nature, SWB data are reliable when reporting on current state: all the studies of SWB validity I’ve seen are showing the validity when people introspect on their state of life now, not their state of life a year ago. I think it’s very likely that people recalling SWB in the past would be highly biased by their current SWB. In contrast, income/​health are more objective for people to recall, and they can also be collected from administrative data. So I don’t think WELLBYs in practice could adequately measure effects with primarily long-term benefits and little to no short-term benefits.

  If you measure someone’s life satisfaction at point t, it is just like measuring someone’s income at point t, in that both are at a single point in time. If you want to analyse the effects of an intervention overtime, it doesn’t matter if it is income or life satisfaction, you need to measure the effect across time.

  The advantage of income is that you’re more likely to have written records of it (bank statements, etc.) compared to reports of your subjective wellbeing. However, if a researcher didn’t record income/​health/​etc. (e.g. they failed to record it at a certain point in their intervention), then they have the same issue in that they would have to rely on people’s memory (for past information) or predictions (for future information).

  Health outcomes are not ‘objective’ when it comes to measures of quality of health. You can remember having a disease and use the DALY score for said disease, but then you rely on the survey of people that were asked (without having the disease themselves) how ‘healthy’ or not it is to have that disease. Note that this is potentially ‘easy’ to recall not so much because of ‘objectivity’ but likely because of ‘granularity of detail’: the disease is likely a ‘binary’ state—you have covid or you don’t—and not a numerical score out of 10. Either way, this question about memory is the realm of empirical psychological work, and my point is that even if it is easier to recall it is still not a great measure.

  Countries like the UK collect wellbeing measures as part of their administrative data.

  Just so I answer your examples, quoted below. The general answer is “we need to measure the outcome in the long run”.

  Alice is a teen targeted by an education intervention that increases her test scores dramatically but also requires her to put in more effort. Alice likes getting good grades, but it’s a very small part of her subjective wellbeing as a teenager, and it’s also offset by the annoyance of having to spend more time on schoolwork, so she reports essentially the same SWB on her survey. Did the education intervention have zero value?

  The education intervention might lead to better wellbeing in the future and wellbeing measures would capture all the potential impacts of the intervention. If you collect income or health at this very moment, you also get no difference. Why is increasing test scores good? Because it increases x or y later. Why is x or y good? Ultimately, because it increases wellbeing.

  Bob is a farm laborer who gets a free bus ticket to migrate to the city and work there. He earns higher income in the city and sends much of it back to his family. But being alone in the city is lonely and difficult. He is happy that he can provide for his family, but they are far away, and the difficulty of being a migrant is much more salient to him on any given day. He reports a reduced SWB on the survey. Was migration a harmful intervention?

  You would need to measure the effect on the SWB of the family and take everything into account. Just because the intervention increased income (but potentially affected social relationships) does not mean it was a good intervention.

  Chris lives in a generally polluted city. He dislikes pollution, but it’s usually not so bad that he notices it very saliently on a day-to-day basis. Unbeknownst to him, an air-quality intervention reduces pollution by 10%, reducing his risk of respiratory disease over twenty years. But he wasn’t aware of it, or even if he was, he wasn’t thinking about risks twenty years from now, so he reports the same SWB as before. Did the air-pollution intervention have zero value?

  Counterfactually, 20 years from now he would rate his SWB higher. Same with income or health, the effect only occurs 20 years from now (in this scenario). With health measures one could use previous data and make a prediction that “respiratory diseases cause X DALYs”. But here we could also look at data that relates SWB and respiratory diseases and see that “respiratory diseases decrease life satisfaction by X”. Same principle with income.

• Samuel Dupret 24 May 2023 13:11 UTC

  7 points

  1 ∶ 0

  on: Top Idea Re­ports from the EA Philip­pines Men­tal Health Char­ity Ideas Re­search Project

  Excited to see more work on mental health charities! Thank you for this. I will need a bit of time to read before I comment I could comment in more detail.
  
  What’s stopping me from have a good overview of your results is that the cost-effectiveness of each proposed intervention is on a different mental-health outcome. If I am not mistaken, these have different scale sizes. Do you have results converted in effect sizes (Cohen’s d)? This would mean all the outcomes are converted to the same unit, standard deviations. This makes it easier to compare and allows you to compare them to other interventions that are also evaluated in affect/​wellbeing (e.g., McGuire et al., 2022).

• Samuel Dupret 18 Aug 2023 9:03 UTC

  6 points

  3 ∶ 0

  on: How ex­pen­sive is leav­ing your org? Squig­gle Model

  This is a cool tool!

  Maybe it is my french/​worker rights bias, but I do feel weird about the framing towards the workers. Shouldn’t this be more for bosses to be incentivised to retain their workforce? “If you don’t treat your employees well enough and they leave, it will cost you”.

• Samuel Dupret 18 Jul 2023 9:33 UTC

  5 points

  0 ∶ 0

  on: Min­i­mal­ist views of wellbeing

  Thank you for presenting these views! This was very interesting.

  I have some questions of interest to me—apologies if I’ve missed something and for these being slightly outside of scope.

  1. Can one use these views in a quantification exercise (e.g., cost-effectiveness of interventions in terms of wellbeing), and if so, how would this work?

     1. How distinct would minimalism be from prioritarianism /​ views focused on reducing suffering?

     2. Couldn’t someone still be an offsetter but dislike the roller-coaster life compared to a more neutral life because they add a discount for variance?

  2. To me, antifrustrationism doesn’t sound too different from non-minimalist preferentialism. Can you help me understand, please. All the desires I have can create a frustration if not satisfied, whether that desire is to have more board games or to have less pain. Say I have both of these desires satisfied, I’ve removed 2 units of frustration from the leger /​ satisfied 2 desires, right? I’ve resolved the same % of preferences on both views. Or is that me already thinking in minimalist terms that preferentialism is a % of things to resolve?
     

• Samuel Dupret 23 Dec 2022 7:35 UTC

  4 points

  2 ∶ 1

  in reply to: NickLaing’s comment on: A can of worms: the non-sig­nifi­cant effect of de­worm­ing on hap­piness in the KLPS

  Hi Nick,
  Thank you for your comment.
  
  About your major point first.
  If it was up to us (we didn’t collect this data), we would use a nicer 0-10 scale. However, this is the only SWB data we are aware of. There are other measures of wellbeing in the data (including a 1-10 scale, some 1-6 frequency scales, and some binary scales) but the 3-point scale is the only measure that was collected across all three KLPS rounds. None of the other measures are significant. Some are negative, some are positive. In Appendix A3.1 we conduct an analysis where we use the effect sizes of all the other measures, and we obtain very similar results, which gives us more confidence about this measure.

  Imagine that deworming really does increase 1000 recipient’s happiness by 2% each. This won’t tip them over the line from “happy” to “very happy” so people will report the same level of happiness

  I’m not sure this is completely true. Some people will be tipped over the line; namely, all the people that are 2% away from changing between categories (the people who answer ‘not happy’ but will be close to answering ‘happy’).

  Where data is as bad as this, it’s better to say that there is not enough meaningful data to draw a conclusion, rather than saying “with the best evidence we have (which is bad), conclusion is X”. It’s better to not use bad evidence at all, and say there is no meaningful evidence available, than to try and draw weak conclusions from it like you do here.

  Both the literature (that does not contain SWB data; Section 1.3) and this SWB data (the only we could find) do not give us good grounds to recommend deworming. For completeness, we go through many considerations that we present in Section 2.3 (alternative analyses, Bayes factors, cost-effectiveness, converting GiveWell’s analysis to WELLBYs, etc.). As we note in the report, this evidence does not prove the effect is zero, but these converging lines of evidence support the conclusion that the effect is zero or very small. If strong SWB evidence that deworming is more cost-effective than StrongMinds is produced, then we would change our minds. Yes, collecting this data will be expensive, but we’d prefer some of the money going to deworming serves to nail down the effect of deworming (on SWB because we think this is what ultimately matters).

  On the bigger WELLBY picture, the more I think about it, I think the Happier Lives institute should prioritise funding a WELLBY measured RCT of StrongMinds vs. cash transfers A.S.A.P performed by an independent organisation, to answer 2 questions.

  1. If StrongMinds really are superior to cash on the WEllBY front?

  2. If before and after wellbeing scores can be valid, and not spoiled by the effects I’ve discussed earlier

  Just to clarify for people reading this (I know you are not saying this, it is just that I know that people too often misunderstand this): StrongMinds gives depression treatment for people in LMIC (who happen to also be poor). GiveDirectly transfers cash to people in poverty (some of which might be depressed). If you want to increase WELLBYs, StrongMinds will produce more WELLBYs per dollar than GiveDirectly. We are not suggesting that psychotherapy treats poverty nor that psychotherapy should be given to non-depressed poor people. The reasons why psychotherapy can help in LMICs are (1) the lower costs and (2) the counterfactual that there are many people there who need help with mental health and that LMICs rarely have any infrastructure to help (and sometimes, when they do have some, it involves actively hurting people—like chaining people up). There is a bidirectional relationship between poverty and mental health that is complex and fascinating (Ridley et al., 2020).

  With that out of the way (sorry, I know this wasn’t your point), the RCT you propose would be testing whether giving the StrongMinds sessions helps their depressed patients more than giving them the cash equivalent of that session. This is, admittedly, very interesting and can provide extra data about the effect of cash on people with depression and using a very special kind of active control. We do not have the funding to do something like that for now. Additionally, I think there might be other areas that are more in need of investigation (e.g., more research on household spillovers). Resources permitting we would be interested in conducting more RCTs with partners

  Thank you for engaging with our work, I hope our answers help.

• Samuel Dupret 22 Dec 2022 9:05 UTC

  4 points

  1 ∶ 0

  in reply to: MichaelStJules’s comment on: A can of worms: the non-sig­nifi­cant effect of de­worm­ing on hap­piness in the KLPS

  Hi Michael, Thank you for trying to connect the non-sig wellbeing findings and the non-sig wealth findings of the KLPS. Indeed, Baird et al. (2016) found an increase in people from the treatment condition having manufacturing jobs (see their Table III). However, they only used the KLPS 2 round. Hamory et al. (2021) use KLPS 2, 3, and 4 and find a tiny, negative, non-significant decrease in manufacturing jobs for the treatment condition. What they do find is fewer hours worked in agriculture and more hours worked in non-agriculture. I don’t read much into any of these.

• Samuel Dupret 7 Oct 2022 14:06 UTC

  4 points

  0 ∶ 0

  on: New cause area: Im­prov­ing di­ag­no­sis and treat­ment of bipo­lar spec­trum disorders

  A very interesting read. I didn’t know much about BSD.

  I was wondering if you had a spreadsheet or something equivalent where you put all the numbers you mention together? Prevalence, loss due to misdiagnoses, false positives, costs, etc. It could help with understanding the predicted cost-effectiveness of the different actions you mention. Apologies if I missed it in the core text.

• Samuel Dupret 28 Jul 2022 20:05 UTC

  4 points

  0 ∶ 0

  in reply to: Lucas Lewit-Mendes’s comment on: De­worm­ing and de­cay: repli­cat­ing GiveWell’s cost-effec­tive­ness anal­y­sis

  Thank you for your comment!

  Indeed, we did take the average of the logs instead of the log of the averages. This doesn’t change the end and start point, so it wouldn’t change the overall decay rate we estimate. We could do more complex modelling where effects between KLPS2 and KLPS3 see small growth and KLPS3 and KLPS4 see large decay. I think this shows that the overall results are sensitive to how we model effect across time.

  See Figure 4 of the appendix, which shows, whether in earnings or in consumption, that the relative gains, as shown by the log difference, decrease over time.

  We used the pooled data because it is what GiveWell does. In the appendix we note that the consumption and earnings data look different. So, perhaps a more principle way would be to look at the decay within earnings and within consumption. The decay within earnings (84%) and the decay within consumption (81%) are both stronger (i.e., would lead to smaller effects) than the 88% pooled decay.

• Samuel Dupret 20 Jul 2023 7:19 UTC

  3 points

  0 ∶ 0

  in reply to: Teo Ajantaival’s comment on: Min­i­mal­ist views of wellbeing

  Thank you for taking the time to reply. Your responses to 1a and 1b make sense to me. 2 I’m still exploring and turning these ideas around in my mind—thank you for the paper. I wonder if some of this can be tested by asking people about their number of desires, general life satisfaction, % life satisfaction/​desires fulfilled.

  If I may, I’d like to expand a bit on number 1.

  • It seems like in terms of extending lives minimalist views have an Epicurean view of the badness of death /​ value of life? The good of saving a life is only the spillovers (what the person would do to the wellbeing of others, the prevented grief, etc).

  • If we narrow the scope to improving existing lives, is the general conclusion of minimalist wellbeing theories that we should deliver interventions that prevent/​reduce suffering rather than add wellbeing?
    

• Samuel Dupret 27 Mar 2023 15:59 UTC

  3 points

  0 ∶ 0

  in reply to: Jason’s comment on: Assess­ment of Hap­pier Lives In­sti­tute’s Cost-Effec­tive­ness Anal­y­sis of StrongMinds

  Thank you for the feedback! We are keen on that feature too, distributing credences between the views is the next step I am working on.

• Samuel Dupret 25 Nov 2022 9:54 UTC

  3 points

  0 ∶ 0

  in reply to: JackM’s comment on: Don’t just give well, give WELLBYs: HLI’s 2022 char­ity recommendation

  Hi Jack,

  And read this as you planning to continue evaluating everything in WELLBYs, which in turn I thought meant ruling out evaluating research—because it isn’t clear to me how you evaluate something like psychedelics research using WELLBYs.

  I’ll try and give a general answer to your concern. I hope this helps?

  Whilst there might be some aspects of research that can be evaluated without looking at WELLBYs (e.g., how costly is psychedelics treatment), the core point is still that wellbeing is what matters. More research will tell us that something is worth it if it does ‘good’; namely, it increases WELLBYs (cost-effectively).

  We hope to obtain wellbeing data (life satisfaction, affect, etc.) for each area we evaluate. If it is lacking, then we hope to encourage more data collection and /​ or to evaluate pathways (e.g., if psychedelics affect variable X, do we know the relationship between X and wellbeing).

• Samuel Dupret 20 Mar 2023 10:24 UTC

  2 points

  0 ∶ 0

  in reply to: Nathan Young’s comment on: Why I’m suss on wellbe­ing surveys

  To caveat this:
  If we have higher wellbeing, and they are trying to maximise wellbeing, then all these things align.
  But people might not be maximising wellbeing or might not be good at it.

• Samuel Dupret 31 Jan 2023 11:00 UTC

  2 points

  0 ∶ 0

  in reply to: NickLaing’s comment on: Pain re­lief: a shal­low cause exploration

  Hi Nick, A quick comment to thank you for engaging with our work and for your insights. This is super interesting.

  Arthritis—same as treating most other pain - large amounts of paractamol, ibuprofen (and other nsaids) and diclofenac gel is what we do for arthritis.

  This suggests that this could be really cost-effective, considering the price of NSAIDs! However, wouldn’t issues of side effects also occur here? Or is this less of an issue because the gains would be higher?