Reading comments from Matt (FP) and Sjir (GWWC), it sounds like the situation is:
FP performed a detailed public evaluation of SM, which they published in 2019.
This was sufficient for FP to recommend giving to SM.
Because FP is one of GWWC’s trusted evaluators, this was sufficient for GWWC to designate SM as top rated.
The public FP evaluation is now stale, though FP has additional unpublished information that is sufficient for them to still recommend SM. Due to resource constraints they haven’t been able to update their public evaluation.
It’s not clear to me what FP should have done differently: resource constraints are hard. The note at the top of the evaluation (which predates this post) is a good start, though it would be better if it included something like “As of fall 2022, we have continued to follow StrongMinds and still recommend them. We are planning a full update before the 2023 giving season.”
In the case of GWWC, I think one of the requirements they should have for endorsing recommendations from their trusted evaluators is that they be supported by public evaluations, and that those evaluations be current. I think in this case GWWC would ideally have moved SM from “top rated” to “listed” sometime in the past ~18m.
(As a practical matter, one way this could be implemented is that any time GWWC imports a charity recommendation from a trusted evaluator it includes the date of evaluation, and the import is only valid for, say, 2.5y from that date.)
Thanks Jeff, I think your summary is helpful and broadly correct, except for two (somewhat relevant) details:
GWWC didn’t recommend SM based on FP’s recommendation in 2019 but based on FP’s decision to still recommend SM as of this giving season (which is based on FP’s internal re-evaluation of SM).
I don’t expect there to be any new, decision-relevant information in FP’s recent internal re-evaluation that isn’t captured by the 2019 report + the recent HLI analysis (but I’m not sure about this—Matt can correct me if I’m wrong, though also see his comment here). Obviously the internal re-evaluation has extra “information” in the sense that FP has reviewed the HLI analysis, converted metrics to their new system, and run some extra tests, as Matt has explained, so maybe this is just semantics, but I think it’s relevant to the extent that a crux would be “FP is still recommending SM because of something only they know”.
I understand the reasons for your suggestion w.r.t. GWWC’s inclusion criteria—we’ve seriously considered doing this before—but I explain at length why I still think we shouldn’t under (4) here. Would welcome any further comments if you disagree!
I agree strongly here re: GWWC. I think it is very odd that they endorse a charity without a clear public explanation of why the charity is effective which could satisfy a mildly skeptical outsider. This is a bar that this clearly does not reach in my opinion. They don’t need to have the same evidential requirements as Givewell, but the list of charities they recommend is sufficiently long that they should prefer to have a moderately high bar for charities to make that list.
To admit my priors here: I am very skeptical of Strong Minds effectiveness given the flimsiness of the public evidence, and Peter’s general skeptical prior about cool sounding interventions described below. I think people really want there to be a good “EA approved” mental health charity and this means evaluations are frequently much less cautious and careful than they should be.
In the case of GWWC, I think one of the requirements they should have for endorsing recommendations from their trusted evaluators is that they be supported by public evaluations, and that those evaluations be current.
Thank you for taking the time to write this. This is (almost) exactly how I feel.
(I personally do not agree that FP can reasonably still have the view that they do about StrongMinds based on what they’ve said about their reasoning here, but I accept that I will have to wait until I’ve published my thoughts on the HLI analyses before I can expect people who haven’t looked at the HLI work to agree with me)
Hey Simon, I remain slightly confused about this element of the conversation. I take you to mean that, since we base our assessment mostly on HLI’s work, and since we draw different conclusions from HLI’s work than you think are reasonable, we should reassess StrongMinds on that basis. Is that right?
If so, I do look forward to your thoughts on the HLI analysis, but in the meantime I’d be curious to get a sense of your personal levels of confidence here — what does a distribution of your beliefs over cost-effectiveness for StrongMinds look like?
since we base our assessment mostly on HLI’s work, and since we draw different conclusions from HLI’s work than you think are reasonable, we should reassess StrongMinds on that basis. Is that right
I’m not sure exactly what you’ve done, so it’s hard for me to comment precisely. I’m just struggling to see how you can be confident in a “6x as effective as GD” conclusion.
what does a distribution of your beliefs over cost-effectiveness for StrongMinds look like?
So there are two sides to this:
Is my confidence in HLI’s philisophical views. I have both spoken to Joel and read all their materials several times and thinkI understand their views. I am sure I do not fully agree with them and I’m not sure how much I believe them. I’d put myself at roughly 30% that I agree with their general philosophy. This is important because how cost-effective you believe StrongMinds are is quite sensitive to philisophical assumptions. (I plan on expanding upon this when discussing HLI)
Under HLI’s philosophical assumptions, I think I’m roughly speaking:
10% SM is 4-8x as good at GiveDirectly 25% SM is 1-4x as good as GiveDirectly 35% SM is 0.5-1x as good as GiveDirectly 30% SM not effective at all
So roughly speaking under HLI’s assumptions I think StrongMinds is roughly as good as GiveDirectly.
I think you will probably say on this basis that you’d still be recommending StrongMinds based on your risk-neutral principle but I think this underestimates quite how uncertain I would expect people to be in the HLI worldview. (I also disagree with being risk-neutral, but I suspect that’s a discussion for another day!)
Reading comments from Matt (FP) and Sjir (GWWC), it sounds like the situation is:
FP performed a detailed public evaluation of SM, which they published in 2019.
This was sufficient for FP to recommend giving to SM.
Because FP is one of GWWC’s trusted evaluators, this was sufficient for GWWC to designate SM as top rated.
The public FP evaluation is now stale, though FP has additional unpublished information that is sufficient for them to still recommend SM. Due to resource constraints they haven’t been able to update their public evaluation.
It’s not clear to me what FP should have done differently: resource constraints are hard. The note at the top of the evaluation (which predates this post) is a good start, though it would be better if it included something like “As of fall 2022, we have continued to follow StrongMinds and still recommend them. We are planning a full update before the 2023 giving season.”
In the case of GWWC, I think one of the requirements they should have for endorsing recommendations from their trusted evaluators is that they be supported by public evaluations, and that those evaluations be current. I think in this case GWWC would ideally have moved SM from “top rated” to “listed” sometime in the past ~18m.
(As a practical matter, one way this could be implemented is that any time GWWC imports a charity recommendation from a trusted evaluator it includes the date of evaluation, and the import is only valid for, say, 2.5y from that date.)
Thanks Jeff, I think your summary is helpful and broadly correct, except for two (somewhat relevant) details:
GWWC didn’t recommend SM based on FP’s recommendation in 2019 but based on FP’s decision to still recommend SM as of this giving season (which is based on FP’s internal re-evaluation of SM).
I don’t expect there to be any new, decision-relevant information in FP’s recent internal re-evaluation that isn’t captured by the 2019 report + the recent HLI analysis (but I’m not sure about this—Matt can correct me if I’m wrong, though also see his comment here). Obviously the internal re-evaluation has extra “information” in the sense that FP has reviewed the HLI analysis, converted metrics to their new system, and run some extra tests, as Matt has explained, so maybe this is just semantics, but I think it’s relevant to the extent that a crux would be “FP is still recommending SM because of something only they know”.
I understand the reasons for your suggestion w.r.t. GWWC’s inclusion criteria—we’ve seriously considered doing this before—but I explain at length why I still think we shouldn’t under (4) here. Would welcome any further comments if you disagree!
Responded above, thanks!
I agree strongly here re: GWWC. I think it is very odd that they endorse a charity without a clear public explanation of why the charity is effective which could satisfy a mildly skeptical outsider. This is a bar that this clearly does not reach in my opinion. They don’t need to have the same evidential requirements as Givewell, but the list of charities they recommend is sufficiently long that they should prefer to have a moderately high bar for charities to make that list.
To admit my priors here: I am very skeptical of Strong Minds effectiveness given the flimsiness of the public evidence, and Peter’s general skeptical prior about cool sounding interventions described below. I think people really want there to be a good “EA approved” mental health charity and this means evaluations are frequently much less cautious and careful than they should be.
I think this is a good idea.
Thank you for taking the time to write this. This is (almost) exactly how I feel.
(I personally do not agree that FP can reasonably still have the view that they do about StrongMinds based on what they’ve said about their reasoning here, but I accept that I will have to wait until I’ve published my thoughts on the HLI analyses before I can expect people who haven’t looked at the HLI work to agree with me)
Hey Simon, I remain slightly confused about this element of the conversation. I take you to mean that, since we base our assessment mostly on HLI’s work, and since we draw different conclusions from HLI’s work than you think are reasonable, we should reassess StrongMinds on that basis. Is that right?
If so, I do look forward to your thoughts on the HLI analysis, but in the meantime I’d be curious to get a sense of your personal levels of confidence here — what does a distribution of your beliefs over cost-effectiveness for StrongMinds look like?
I’m not sure exactly what you’ve done, so it’s hard for me to comment precisely. I’m just struggling to see how you can be confident in a “6x as effective as GD” conclusion.
So there are two sides to this:
Is my confidence in HLI’s philisophical views. I have both spoken to Joel and read all their materials several times and thinkI understand their views. I am sure I do not fully agree with them and I’m not sure how much I believe them. I’d put myself at roughly 30% that I agree with their general philosophy. This is important because how cost-effective you believe StrongMinds are is quite sensitive to philisophical assumptions. (I plan on expanding upon this when discussing HLI)
Under HLI’s philosophical assumptions, I think I’m roughly speaking:
10% SM is 4-8x as good at GiveDirectly
25% SM is 1-4x as good as GiveDirectly
35% SM is 0.5-1x as good as GiveDirectly
30% SM not effective at all
So roughly speaking under HLI’s assumptions I think StrongMinds is roughly as good as GiveDirectly.
I think you will probably say on this basis that you’d still be recommending StrongMinds based on your risk-neutral principle but I think this underestimates quite how uncertain I would expect people to be in the HLI worldview. (I also disagree with being risk-neutral, but I suspect that’s a discussion for another day!)