Extremely skeptical of this. Reading through Strong Mindsâ evidence base it looks like youâre leaning very heavily on 2 RCTs of a total of about 600 people in Uganda, in which the outcomes were a slight reduction in a questionable mental health metric that relies on self-reporting.
Youâre making big claims about your intervention including that it is a better use of money than saving the lives of children. I hope youâre right, otherwise you might be doing a lot of harm.
I weak downvoted this comment because of the following sentence:
âI hope youâre right, otherwise you might be doing a lot of harm.â
I agree that technically, a person whoâs very successful at advocating for a low impact intervention might do harm. But I think we should assume that most contributions to public discourse about cost-effectiveness are beneficial even if they are false.
Accusing people of doing harm makes it more difficult to discuss the issue in a distanced and calm way. It also makes it more difficult to get people contribute to those discussions.
Epistemic adjectives such as âunfoundedâ, âweakâ, âfalseâ, âmisleadingâ are still available, and these are less likely to provoke guilt and stifle discussion. Unless there are egregious violations of honesty, I donât think people should be accused of doing harm for defending their beliefs on cause-prioritisation.
Separately, I disagree voted because it seems wrong about the facts. HLIâs analysis relies on many different studies with a sample size >30K, as Michael explains below.
Hello Henry. It may look like weâre just leaning on 2 RCTs, but weâre not! If you read further down in the âcash transfers vs treating depressionâ section, we mention that we compared cash transfers to talk therapy on the basis of a meta-analysis of each.
The evidence base for therapy is explained in full in Section 4 of our StrongMinds cost-effectiveness analysis. We use four direct studies and a meta-analysis of 39 indirect studies (n > 38,000). You can see how much weight we give to each source of evidence in Table 2, reproduced below. To be clear, we donât take the results from StrongMindsâ own trials at face value. We basically use an average figure for their effect size, even though they find a high figure themselves.
Also, whatâs wrong with the self-reports? People are self-reporting how they feel. How else should we determine how people feel? Should we just ignore them and assume that we know best? Also, weâre comparing self-reports to other self-reports, so itâs unclear what bias we need to worry about.
Regarding the issues of comparing saving lives to improving lives, weâve just written a whole report on how to think about that. Weâre hoping that, by bringing these difficult issues to the surfaceârather than glossing over them, which is what normally happensâpeople can make better-informed decisions. Weâre very much on your side: we think people should be thinking harder about which does more good.
I havenât looked in detail at how Give Well evaluates evidence, so maybe youâre no worse here, but I donât think âweighted average of published evidenceâ is appropriate when one has concerns about the quality of published evidence. Furthermore, I think some level of concern about the quality of published evidence should be oneâs baseline positionâI.e. a weighted average is only appropriate when there are unusually strong reasons to think the published evidence is good.
Iâm broadly supportive of the project of evaluating impacts on happiness.
Youâre right that we should be concerned with the quality of published evidence. I discounted psychotherapyâs effect by 17% for having a higher risk of effect inflation than cash transfers, see Appendix C of McGuire & Plant (2021). However, this was the first pass at a fundamental problem in science, and I recognize we could do better here.
Weâre planning on revisiting this analysis and improving our methods â but weâre currently prioritizing finding new interventions more than improving our analyses of old ones. Unfortunately, we currently donât have the research capacity to do both well!
Extremely skeptical of this. Reading through Strong Mindsâ evidence base it looks like youâre leaning very heavily on 2 RCTs of a total of about 600 people in Uganda, in which the outcomes were a slight reduction in a questionable mental health metric that relies on self-reporting.
Youâre making big claims about your intervention including that it is a better use of money than saving the lives of children. I hope youâre right, otherwise you might be doing a lot of harm.
I weak downvoted this comment because of the following sentence:
âI hope youâre right, otherwise you might be doing a lot of harm.â
I agree that technically, a person whoâs very successful at advocating for a low impact intervention might do harm. But I think we should assume that most contributions to public discourse about cost-effectiveness are beneficial even if they are false.
Accusing people of doing harm makes it more difficult to discuss the issue in a distanced and calm way. It also makes it more difficult to get people contribute to those discussions.
Epistemic adjectives such as âunfoundedâ, âweakâ, âfalseâ, âmisleadingâ are still available, and these are less likely to provoke guilt and stifle discussion. Unless there are egregious violations of honesty, I donât think people should be accused of doing harm for defending their beliefs on cause-prioritisation.
Separately, I disagree voted because it seems wrong about the facts. HLIâs analysis relies on many different studies with a sample size >30K, as Michael explains below.
Hello Henry. It may look like weâre just leaning on 2 RCTs, but weâre not! If you read further down in the âcash transfers vs treating depressionâ section, we mention that we compared cash transfers to talk therapy on the basis of a meta-analysis of each.
The evidence base for therapy is explained in full in Section 4 of our StrongMinds cost-effectiveness analysis. We use four direct studies and a meta-analysis of 39 indirect studies (n > 38,000). You can see how much weight we give to each source of evidence in Table 2, reproduced below. To be clear, we donât take the results from StrongMindsâ own trials at face value. We basically use an average figure for their effect size, even though they find a high figure themselves.
Also, whatâs wrong with the self-reports? People are self-reporting how they feel. How else should we determine how people feel? Should we just ignore them and assume that we know best? Also, weâre comparing self-reports to other self-reports, so itâs unclear what bias we need to worry about.
Regarding the issues of comparing saving lives to improving lives, weâve just written a whole report on how to think about that. Weâre hoping that, by bringing these difficult issues to the surfaceârather than glossing over them, which is what normally happensâpeople can make better-informed decisions. Weâre very much on your side: we think people should be thinking harder about which does more good.
I havenât looked in detail at how Give Well evaluates evidence, so maybe youâre no worse here, but I donât think âweighted average of published evidenceâ is appropriate when one has concerns about the quality of published evidence. Furthermore, I think some level of concern about the quality of published evidence should be oneâs baseline positionâI.e. a weighted average is only appropriate when there are unusually strong reasons to think the published evidence is good.
Iâm broadly supportive of the project of evaluating impacts on happiness.
Hi David,
Youâre right that we should be concerned with the quality of published evidence. I discounted psychotherapyâs effect by 17% for having a higher risk of effect inflation than cash transfers, see Appendix C of McGuire & Plant (2021). However, this was the first pass at a fundamental problem in science, and I recognize we could do better here.
Weâre planning on revisiting this analysis and improving our methods â but weâre currently prioritizing finding new interventions more than improving our analyses of old ones. Unfortunately, we currently donât have the research capacity to do both well!