Reading comments from Matt (FP) and Sjir (GWWC), it sounds like the situation is:
FP performed a detailed public evaluation of SM, which they published in 2019.
This was sufficient for FP to recommend giving to SM.
Because FP is one of GWWCâs trusted evaluators, this was sufficient for GWWC to designate SM as top rated.
The public FP evaluation is now stale, though FP has additional unpublished information that is sufficient for them to still recommend SM. Due to resource constraints they havenât been able to update their public evaluation.
Itâs not clear to me what FP should have done differently: resource constraints are hard. The note at the top of the evaluation (which predates this post) is a good start, though it would be better if it included something like âAs of fall 2022, we have continued to follow StrongMinds and still recommend them. We are planning a full update before the 2023 giving season.â
In the case of GWWC, I think one of the requirements they should have for endorsing recommendations from their trusted evaluators is that they be supported by public evaluations, and that those evaluations be current. I think in this case GWWC would ideally have moved SM from âtop ratedâ to âlistedâ sometime in the past ~18m.
(As a practical matter, one way this could be implemented is that any time GWWC imports a charity recommendation from a trusted evaluator it includes the date of evaluation, and the import is only valid for, say, 2.5y from that date.)
Thanks Jeff, I think your summary is helpful and broadly correct, except for two (somewhat relevant) details:
GWWC didnât recommend SM based on FPâs recommendation in 2019 but based on FPâs decision to still recommend SM as of this giving season (which is based on FPâs internal re-evaluation of SM).
I donât expect there to be any new, decision-relevant information in FPâs recent internal re-evaluation that isnât captured by the 2019 report + the recent HLI analysis (but Iâm not sure about thisâMatt can correct me if Iâm wrong, though also see his comment here). Obviously the internal re-evaluation has extra âinformationâ in the sense that FP has reviewed the HLI analysis, converted metrics to their new system, and run some extra tests, as Matt has explained, so maybe this is just semantics, but I think itâs relevant to the extent that a crux would be âFP is still recommending SM because of something only they knowâ.
I understand the reasons for your suggestion w.r.t. GWWCâs inclusion criteriaâweâve seriously considered doing this beforeâbut I explain at length why I still think we shouldnât under (4) here. Would welcome any further comments if you disagree!
I agree strongly here re: GWWC. I think it is very odd that they endorse a charity without a clear public explanation of why the charity is effective which could satisfy a mildly skeptical outsider. This is a bar that this clearly does not reach in my opinion. They donât need to have the same evidential requirements as Givewell, but the list of charities they recommend is sufficiently long that they should prefer to have a moderately high bar for charities to make that list.
To admit my priors here: I am very skeptical of Strong Minds effectiveness given the flimsiness of the public evidence, and Peterâs general skeptical prior about cool sounding interventions described below. I think people really want there to be a good âEA approvedâ mental health charity and this means evaluations are frequently much less cautious and careful than they should be.
In the case of GWWC, I think one of the requirements they should have for endorsing recommendations from their trusted evaluators is that they be supported by public evaluations, and that those evaluations be current.
Thank you for taking the time to write this. This is (almost) exactly how I feel.
(I personally do not agree that FP can reasonably still have the view that they do about StrongMinds based on what theyâve said about their reasoning here, but I accept that I will have to wait until Iâve published my thoughts on the HLI analyses before I can expect people who havenât looked at the HLI work to agree with me)
Hey Simon, I remain slightly confused about this element of the conversation. I take you to mean that, since we base our assessment mostly on HLIâs work, and since we draw different conclusions from HLIâs work than you think are reasonable, we should reassess StrongMinds on that basis. Is that right?
If so, I do look forward to your thoughts on the HLI analysis, but in the meantime Iâd be curious to get a sense of your personal levels of confidence here â what does a distribution of your beliefs over cost-effectiveness for StrongMinds look like?
since we base our assessment mostly on HLIâs work, and since we draw different conclusions from HLIâs work than you think are reasonable, we should reassess StrongMinds on that basis. Is that right
Iâm not sure exactly what youâve done, so itâs hard for me to comment precisely. Iâm just struggling to see how you can be confident in a â6x as effective as GDâ conclusion.
what does a distribution of your beliefs over cost-effectiveness for StrongMinds look like?
So there are two sides to this:
Is my confidence in HLIâs philisophical views. I have both spoken to Joel and read all their materials several times and thinkI understand their views. I am sure I do not fully agree with them and Iâm not sure how much I believe them. Iâd put myself at roughly 30% that I agree with their general philosophy. This is important because how cost-effective you believe StrongMinds are is quite sensitive to philisophical assumptions. (I plan on expanding upon this when discussing HLI)
Under HLIâs philosophical assumptions, I think Iâm roughly speaking:
10% SM is 4-8x as good at GiveDirectly 25% SM is 1-4x as good as GiveDirectly 35% SM is 0.5-1x as good as GiveDirectly 30% SM not effective at all
So roughly speaking under HLIâs assumptions I think StrongMinds is roughly as good as GiveDirectly.
I think you will probably say on this basis that youâd still be recommending StrongMinds based on your risk-neutral principle but I think this underestimates quite how uncertain I would expect people to be in the HLI worldview. (I also disagree with being risk-neutral, but I suspect thatâs a discussion for another day!)
Reading comments from Matt (FP) and Sjir (GWWC), it sounds like the situation is:
FP performed a detailed public evaluation of SM, which they published in 2019.
This was sufficient for FP to recommend giving to SM.
Because FP is one of GWWCâs trusted evaluators, this was sufficient for GWWC to designate SM as top rated.
The public FP evaluation is now stale, though FP has additional unpublished information that is sufficient for them to still recommend SM. Due to resource constraints they havenât been able to update their public evaluation.
Itâs not clear to me what FP should have done differently: resource constraints are hard. The note at the top of the evaluation (which predates this post) is a good start, though it would be better if it included something like âAs of fall 2022, we have continued to follow StrongMinds and still recommend them. We are planning a full update before the 2023 giving season.â
In the case of GWWC, I think one of the requirements they should have for endorsing recommendations from their trusted evaluators is that they be supported by public evaluations, and that those evaluations be current. I think in this case GWWC would ideally have moved SM from âtop ratedâ to âlistedâ sometime in the past ~18m.
(As a practical matter, one way this could be implemented is that any time GWWC imports a charity recommendation from a trusted evaluator it includes the date of evaluation, and the import is only valid for, say, 2.5y from that date.)
Thanks Jeff, I think your summary is helpful and broadly correct, except for two (somewhat relevant) details:
GWWC didnât recommend SM based on FPâs recommendation in 2019 but based on FPâs decision to still recommend SM as of this giving season (which is based on FPâs internal re-evaluation of SM).
I donât expect there to be any new, decision-relevant information in FPâs recent internal re-evaluation that isnât captured by the 2019 report + the recent HLI analysis (but Iâm not sure about thisâMatt can correct me if Iâm wrong, though also see his comment here). Obviously the internal re-evaluation has extra âinformationâ in the sense that FP has reviewed the HLI analysis, converted metrics to their new system, and run some extra tests, as Matt has explained, so maybe this is just semantics, but I think itâs relevant to the extent that a crux would be âFP is still recommending SM because of something only they knowâ.
I understand the reasons for your suggestion w.r.t. GWWCâs inclusion criteriaâweâve seriously considered doing this beforeâbut I explain at length why I still think we shouldnât under (4) here. Would welcome any further comments if you disagree!
Responded above, thanks!
I agree strongly here re: GWWC. I think it is very odd that they endorse a charity without a clear public explanation of why the charity is effective which could satisfy a mildly skeptical outsider. This is a bar that this clearly does not reach in my opinion. They donât need to have the same evidential requirements as Givewell, but the list of charities they recommend is sufficiently long that they should prefer to have a moderately high bar for charities to make that list.
To admit my priors here: I am very skeptical of Strong Minds effectiveness given the flimsiness of the public evidence, and Peterâs general skeptical prior about cool sounding interventions described below. I think people really want there to be a good âEA approvedâ mental health charity and this means evaluations are frequently much less cautious and careful than they should be.
I think this is a good idea.
Thank you for taking the time to write this. This is (almost) exactly how I feel.
(I personally do not agree that FP can reasonably still have the view that they do about StrongMinds based on what theyâve said about their reasoning here, but I accept that I will have to wait until Iâve published my thoughts on the HLI analyses before I can expect people who havenât looked at the HLI work to agree with me)
Hey Simon, I remain slightly confused about this element of the conversation. I take you to mean that, since we base our assessment mostly on HLIâs work, and since we draw different conclusions from HLIâs work than you think are reasonable, we should reassess StrongMinds on that basis. Is that right?
If so, I do look forward to your thoughts on the HLI analysis, but in the meantime Iâd be curious to get a sense of your personal levels of confidence here â what does a distribution of your beliefs over cost-effectiveness for StrongMinds look like?
Iâm not sure exactly what youâve done, so itâs hard for me to comment precisely. Iâm just struggling to see how you can be confident in a â6x as effective as GDâ conclusion.
So there are two sides to this:
Is my confidence in HLIâs philisophical views. I have both spoken to Joel and read all their materials several times and thinkI understand their views. I am sure I do not fully agree with them and Iâm not sure how much I believe them. Iâd put myself at roughly 30% that I agree with their general philosophy. This is important because how cost-effective you believe StrongMinds are is quite sensitive to philisophical assumptions. (I plan on expanding upon this when discussing HLI)
Under HLIâs philosophical assumptions, I think Iâm roughly speaking:
10% SM is 4-8x as good at GiveDirectly
25% SM is 1-4x as good as GiveDirectly
35% SM is 0.5-1x as good as GiveDirectly
30% SM not effective at all
So roughly speaking under HLIâs assumptions I think StrongMinds is roughly as good as GiveDirectly.
I think you will probably say on this basis that youâd still be recommending StrongMinds based on your risk-neutral principle but I think this underestimates quite how uncertain I would expect people to be in the HLI worldview. (I also disagree with being risk-neutral, but I suspect thatâs a discussion for another day!)