Edit 03-01-23: I have now replied more elaborately here
Hi Simon, thanks for this post! Iâm research director at GWWC, and we really appreciate people engaging with our work like this and scrutinising it.
Iâm on holiday currently and wonât be able to reply much more in the coming few days, but will check this page again next Tuesday at the latest to see if thereâs anything more I/âthe GWWC team need to get back on.
For now, Iâll just very quickly address your two key claims that GWWC shouldnât have recommended StrongMinds as a top-rated charity and that we should remove it now, both of which I disagree with.
Our process and criteria for making charity recommendations are outlined here. Crucially, note that we generally donât do (and donât have capacity to do) individual charity research: we almost entirely rely on our trusted evaluatorsâincluding Founders Pledgeâfor our recommendations. As a research team, we plan to specialize in providing guidance on which evaluators to rely rather than in doing individual charity evaluation research.
In the case of StrongMinds, they are a top-rated charity primarily because Founders Pledge recommended them to us, as you highlight. There were no reasons for us to doubt the quality of FPâs research supporting this recommendation at the time, and though you make a few good points on StrongMinds that showcase some of the uncertainties/âcaveats/âcounterarguments involved in this recommendation, I think Matt Lerner adequately addresses these in his reply above, so I donât see reason for us to consider diverging from FPâs recommendation now either.
I do have a takeaway from your post on our communication. On our website we currently state
âStrongMinds meets our criteria to be a top-rated charity because one of our trusted evaluators, Founders Pledge, has conducted an extensive evaluation highlighting its cost-effectiveness as part of its report on mental health.â
We then also highlight HLIâs recent report as evidence supporting this recommendation. However, I think we should have been/âbe clearer on the fact that itâs not only FPâs 2019 report that supports this recommendation, but their ongoing research, some of which isnât public yet (even though the report still represents their overall view, as Matt Lerner mentions above). Iâll make sure to edit this.
Thanks again for your post, and I look forward to engaging more next week at the latest if there are any other comments/âquestions you or anyone else have.
I just want to add my support for GWWC here. I strongly support the way they have made decisions on what to list to date:
As a GWWC member who often donates through the GWWC platform I think it is great that they take a very broad brush and have lots of charities that people might see as top on the platform. I think if their list got to small they would not be able to usefully serve the GWWC donor community (or other donors) as well.
I would note that (contrary to what some of the comments suggest) that GWWC recommend giving to Funds and do recommend giving to these charities (so they do not explicitly recommend Strong Minds). In this light I see the listing of these charities not as recommendations but as convenience for donors who are going to be giving there.
I find GWWC very transparent. Simon says ideally âGWWC would clarify what their threshold is for Top Charityâ. On that specific point I donât see how GWWC could be any more clear. Every page explains that a top charity is one that has been listed as top by an evaluator GWWC trust. Although I do agree with Simon more description of how GWWC choose certain evaluators could be helpful.
That said I would love it if going forwards GWWC could find the time. To evaluate the evaluators and the Funds and their recommendations (for example I have some concerns about the LTFF and know others do too, I know there have been concerns about ACE in the past etc).
I would not want GWWC to unlist Strong Minds from their website but I could imagine them adding a section on the Strong Minds page saying âThe GWWC team viewsâ that says: âthis is listed as it is a FP top but our personal views are that âŠ. meaning this might or might not be a good place to give especially if you care about ⊠etcâ
(Conflict of interest note: I donât work at GWWC or FP but I do work at a FP recommended charity and at a charity whoâs recommendations make it into the GWWC criteria so I might be bias).
As a GWWC member who often donates through the GWWC platform I think it is great that they take a very broad brush and have lots of charities that people might see as top on the platform. I think if their list got to small they would not be able to usefully serve the GWWC donor community (or other donors) as well.
I agree, and Iâm not advocating removing StrongMinds from the platform, just removing the label âTop-ratedâ. Some examples of charities on the platform which are not top-rated include: GiveDirectly, SCI, Deworm the World, Happier Lives Institute, Fish Welfare Initiative, Rethink Priorities, Clean Air Task Force...
I would note that (contrary to what some of the comments suggest) that GWWC recommend giving to Funds and do recommend giving to these charities (so they do not explicitly recommend Strong Minds). In this light I see the listing of these charities not as recommendations but as convenience for donors who are going to be giving there.
Iâm afraid to say I believe you are mistaken here, as I explained in my other comment. The recommendations section clearly includes top-charities recommended by trusted evaluators and explicitly includes StrongMinds. There is also a two-tier labelling of âTop-ratedâ and not-top-rated and StrongMinds is included in the former. Both of these are explicit recommendations afaic.
I find GWWC very transparent. Simon says ideally âGWWC would clarify what their threshold is for Top Charityâ. On that specific point I donât see how GWWC could be any more clear. Every page explains that a top charity is one that has been listed as top by an evaluator GWWC trust. Although I do agree with Simon more description of how GWWC choose certain evaluators could be helpful.
Iâm not really complaining about transparency as much as I would like the threshold to include requirements for trusted evaluators to have public reasoning and a time-threshold on their recommendations.
I would not want GWWC to unlist Strong Minds from their website but I could imagine them adding a section on the Strong Minds page saying âThe GWWC team viewsâ that says: âthis is listed as it is a FP top but our personal views are that âŠ. meaning this might or might not be a good place to give especially if you care about ⊠etcâ
Again, to repeat myself. Me either! I just want them to remove the âTop-ratedâ label.
Recognizing GWWCâs limited bandwidth for individual charity research, what would you think of the following policy: When GWWC learns of a charity recommendation from a trusted recommender, it will post a thread on this forum and invite comments about whether the candidate is in the same ballpark as the median top-rated organization in that cause area (as defined by GWWC, so âImproving Human Well-Beingâ). Although GWWC will still show significant deference to its trusted evaluators in deciding how to list organizations, it will include one sentence on the organizationâs description linking to the forum notice-and-comment discussion. It will post a new thread on each listed organization at 2-3 year intervals, or when there is reason to believe that new information may materially affect the charityâs evaluation.
Given GWWCâs role and the length of its writeups, I donât think it is necessary for GWWC to directly state reasons why a donor might reasonably choose not to donate to the charity in question. However, there does need to be an accessible way for potential donors to discover if those reasons might exist. While I donât disagree with using FP as a trusted evaluator, its mission is not primarily directed toward producing public materials written with GWWC-type donors in mind. Its materials do not meet the bar I suggested in another comment for advisory organizations to GWWC-type donors: âAfter engaging with the recommenderâs donor-facing materials about the recommended charity for 7-10 minutes, most potential donors should have a solid understanding of the quality of evidence and degree of uncertainty behind the recommendation; this will often include at least a brief mention of any major technical issues that might significantly alter the decision of a significant number of donors.â That is not a criticism of FP because itâs not trying to make recommendations to GWWC-type donors.
So giving the community an opportunity to state concerns/âreservations (if any) and link to the community discussion seems potentially valuable as a way to meet this need without consuming much in the way of limited GWWC research resources.
Iâd like to flag that I think itâs bad that my friend (yes Iâm biased) has done a lot of work to criticise something (and I havenât read pushback against that work) but wonât affect the outcome because of work that he and we cannot see.
Is there a way that we can do a little better than this?
Some thoughts:
Could he be allowed to sign an NDA to read Founderâs pledgeâs work?
Would you be interested in forecasts that Stronger Minds wont be a GWWC top charity by say 2025?
Could I add this criticism and a summary of your response to Stronger Minds EA wiki page so that others can see this criticism and it doesnât get lost?
Can anyone come up with other suggestions?
edits:
Changed âdisregardedâ the sentence with âwonât affect the outcomeâ
Tbh I think this is a bit unfair: his criticism isnât being disregarded at all. He received a substantial reply from FPâs research director Matt Lernerâeven while heâs on holidayâwithin a day, and Matt seems very happy to discuss this further when heâs back to work.
I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLIâs analysis this year. I donât think what FP has internally is crucial to interpreting Mattâs responses.
I am sure there is a better word than âdisregardedâ. Apologies for being grumpy, have edited.
This seems like legitimate criticism. Matt says so. But currently, it feels like nothing might happen as a result. You have secret info, end of discussion. This is a common problem within charity evaluation, I thinkâsomeone makes some criticism, someone disagrees and so it gets lost to the sands of time.
I guess my question is, how can this work better? How can this criticism be stored and how can your response of âwe have secret info, trust usâ be a bit more costly for you now (with appropriate rewards later).
If you are interested in forecasting, would you prefer a metaculus or manifold market?
Eg if you like manifold, you can bet here (there is a lot of liquidity and the market currently heavily thinks GWWC will revoke its recommendation. If you disagree you can win money that can be donated to GWWC and status. This is one way to tax and reward you for your secret info)
Is this form of the market the correct wording? If so Iâll write a metaculus version.
As I tried to clarify above, this is not a case of secret info having muchâif anyâbearing on a recommendation. As far as Iâm aware, nearly all decision-relevant information is and has been available publicly, and where it isnât Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly canât think of any secret info that is influencing GWWCâs decision here.
FWIW my personal forecast wouldnât be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMindsâ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.
Iâm fine with the wording: itâs technically âtop-rated charityâ currently but both naming and system may change over the coming years, as weâll hopefully be ramping up research efforts.
Hmmmm this still feels like a bit of a dodge. If the work is all public, what specific thing has Simon missed or misunderstood or what are you going to change? Letâs give it two weeks, but if there is no secret info there ought to be an answer to that question.
Also, what do you expect the results of the RCT to be? And if you think they will be negative shouldnât you remove the recommendation now?
Hi Nathan, I donât think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though Iâd admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isnât immediately clear; depends on the interactions with the other reasons).
And I hope the more elaborate reply I just wrote to Simon answers your remaining question.
meta-comment: If youâre going to edit a comment, it would be useful to be specific and say how you edited the comment e.g. in this case, I think you changed the word âdisregardedâ to something weaker on further reflection.
Could he be allowed to sign an NDA to read Founderâs pledgeâs work?
Unfortunately that wouldnât help, because the part of the point of looking at FPâs work would be to evaluate it. Another person saying âI looked at some work privately and I agree/âdisagree with itâ doesnât seem helpful to people trying to evaluate StrongMinds.
GWWC would clarify what their threshold is for Top Charity
GWWC would explain how they decide what is a Trusted Evaluator and when their evaluations count to be a Top Charity (this decision process would include evaluators publishing their reasoning)
Edit 03-01-23: I have now replied more elaborately here
Hi Simon, thanks for this post! Iâm research director at GWWC, and we really appreciate people engaging with our work like this and scrutinising it.
Iâm on holiday currently and wonât be able to reply much more in the coming few days, but will check this page again next Tuesday at the latest to see if thereâs anything more I/âthe GWWC team need to get back on.
For now, Iâll just very quickly address your two key claims that GWWC shouldnât have recommended StrongMinds as a top-rated charity and that we should remove it now, both of which I disagree with.
Our process and criteria for making charity recommendations are outlined here. Crucially, note that we generally donât do (and donât have capacity to do) individual charity research: we almost entirely rely on our trusted evaluatorsâincluding Founders Pledgeâfor our recommendations. As a research team, we plan to specialize in providing guidance on which evaluators to rely rather than in doing individual charity evaluation research.
In the case of StrongMinds, they are a top-rated charity primarily because Founders Pledge recommended them to us, as you highlight. There were no reasons for us to doubt the quality of FPâs research supporting this recommendation at the time, and though you make a few good points on StrongMinds that showcase some of the uncertainties/âcaveats/âcounterarguments involved in this recommendation, I think Matt Lerner adequately addresses these in his reply above, so I donât see reason for us to consider diverging from FPâs recommendation now either.
I do have a takeaway from your post on our communication. On our website we currently state
âStrongMinds meets our criteria to be a top-rated charity because one of our trusted evaluators, Founders Pledge, has conducted an extensive evaluation highlighting its cost-effectiveness as part of its report on mental health.â
We then also highlight HLIâs recent report as evidence supporting this recommendation. However, I think we should have been/âbe clearer on the fact that itâs not only FPâs 2019 report that supports this recommendation, but their ongoing research, some of which isnât public yet (even though the report still represents their overall view, as Matt Lerner mentions above). Iâll make sure to edit this.
Thanks again for your post, and I look forward to engaging more next week at the latest if there are any other comments/âquestions you or anyone else have.
I just want to add my support for GWWC here. I strongly support the way they have made decisions on what to list to date:
As a GWWC member who often donates through the GWWC platform I think it is great that they take a very broad brush and have lots of charities that people might see as top on the platform. I think if their list got to small they would not be able to usefully serve the GWWC donor community (or other donors) as well.
I would note that (contrary to what some of the comments suggest) that GWWC recommend giving to Funds and do recommend giving to these charities (so they do not explicitly recommend Strong Minds). In this light I see the listing of these charities not as recommendations but as convenience for donors who are going to be giving there.
I find GWWC very transparent. Simon says ideally âGWWC would clarify what their threshold is for Top Charityâ. On that specific point I donât see how GWWC could be any more clear. Every page explains that a top charity is one that has been listed as top by an evaluator GWWC trust. Although I do agree with Simon more description of how GWWC choose certain evaluators could be helpful.
That said I would love it if going forwards GWWC could find the time. To evaluate the evaluators and the Funds and their recommendations (for example I have some concerns about the LTFF and know others do too, I know there have been concerns about ACE in the past etc).
I would not want GWWC to unlist Strong Minds from their website but I could imagine them adding a section on the Strong Minds page saying âThe GWWC team viewsâ that says: âthis is listed as it is a FP top but our personal views are that âŠ. meaning this might or might not be a good place to give especially if you care about ⊠etcâ
(Conflict of interest note: I donât work at GWWC or FP but I do work at a FP recommended charity and at a charity whoâs recommendations make it into the GWWC criteria so I might be bias).
I agree, and Iâm not advocating removing StrongMinds from the platform, just removing the label âTop-ratedâ. Some examples of charities on the platform which are not top-rated include: GiveDirectly, SCI, Deworm the World, Happier Lives Institute, Fish Welfare Initiative, Rethink Priorities, Clean Air Task Force...
Iâm afraid to say I believe you are mistaken here, as I explained in my other comment. The recommendations section clearly includes top-charities recommended by trusted evaluators and explicitly includes StrongMinds. There is also a two-tier labelling of âTop-ratedâ and not-top-rated and StrongMinds is included in the former. Both of these are explicit recommendations afaic.
Iâm not really complaining about transparency as much as I would like the threshold to include requirements for trusted evaluators to have public reasoning and a time-threshold on their recommendations.
Again, to repeat myself. Me either! I just want them to remove the âTop-ratedâ label.
Ah. Good point. Replied to the other thread here: https://ââforum.effectivealtruism.org/ââposts/ââffmbLCzJctLac3rDu/ââstrongminds-should-not-be-a-top-rated-charity-yet?commentId=TMbymn5Cyqdpv5diQ .
Recognizing GWWCâs limited bandwidth for individual charity research, what would you think of the following policy: When GWWC learns of a charity recommendation from a trusted recommender, it will post a thread on this forum and invite comments about whether the candidate is in the same ballpark as the median top-rated organization in that cause area (as defined by GWWC, so âImproving Human Well-Beingâ). Although GWWC will still show significant deference to its trusted evaluators in deciding how to list organizations, it will include one sentence on the organizationâs description linking to the forum notice-and-comment discussion. It will post a new thread on each listed organization at 2-3 year intervals, or when there is reason to believe that new information may materially affect the charityâs evaluation.
Given GWWCâs role and the length of its writeups, I donât think it is necessary for GWWC to directly state reasons why a donor might reasonably choose not to donate to the charity in question. However, there does need to be an accessible way for potential donors to discover if those reasons might exist. While I donât disagree with using FP as a trusted evaluator, its mission is not primarily directed toward producing public materials written with GWWC-type donors in mind. Its materials do not meet the bar I suggested in another comment for advisory organizations to GWWC-type donors: âAfter engaging with the recommenderâs donor-facing materials about the recommended charity for 7-10 minutes, most potential donors should have a solid understanding of the quality of evidence and degree of uncertainty behind the recommendation; this will often include at least a brief mention of any major technical issues that might significantly alter the decision of a significant number of donors.â That is not a criticism of FP because itâs not trying to make recommendations to GWWC-type donors.
So giving the community an opportunity to state concerns/âreservations (if any) and link to the community discussion seems potentially valuable as a way to meet this need without consuming much in the way of limited GWWC research resources.
Thanks for the suggestion Jason, though I hope the longer comment I just posted will clarify why I think this wouldnât be worth doing.
edited (see bottom)
Iâd like to flag that I think itâs bad that my friend (yes Iâm biased) has done a lot of work to criticise something (and I havenât read pushback against that work) but wonât affect the outcome because of work that he and we cannot see.
Is there a way that we can do a little better than this?
Some thoughts:
Could he be allowed to sign an NDA to read Founderâs pledgeâs work?
Would you be interested in forecasts that Stronger Minds wont be a GWWC top charity by say 2025?
Could I add this criticism and a summary of your response to Stronger Minds EA wiki page so that others can see this criticism and it doesnât get lost?
Can anyone come up with other suggestions?
edits:
Changed âdisregardedâ the sentence with âwonât affect the outcomeâ
Tbh I think this is a bit unfair: his criticism isnât being disregarded at all. He received a substantial reply from FPâs research director Matt Lernerâeven while heâs on holidayâwithin a day, and Matt seems very happy to discuss this further when heâs back to work.
I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLIâs analysis this year. I donât think what FP has internally is crucial to interpreting Mattâs responses.
I do like the forecasting idea though :).
I am sure there is a better word than âdisregardedâ. Apologies for being grumpy, have edited.
This seems like legitimate criticism. Matt says so. But currently, it feels like nothing might happen as a result. You have secret info, end of discussion. This is a common problem within charity evaluation, I thinkâsomeone makes some criticism, someone disagrees and so it gets lost to the sands of time.
I guess my question is, how can this work better? How can this criticism be stored and how can your response of âwe have secret info, trust usâ be a bit more costly for you now (with appropriate rewards later).
If you are interested in forecasting, would you prefer a metaculus or manifold market?
Eg if you like manifold, you can bet here (there is a lot of liquidity and the market currently heavily thinks GWWC will revoke its recommendation. If you disagree you can win money that can be donated to GWWC and status. This is one way to tax and reward you for your secret info)
Is this form of the market the correct wording? If so Iâll write a metaculus version.
As I tried to clarify above, this is not a case of secret info having muchâif anyâbearing on a recommendation. As far as Iâm aware, nearly all decision-relevant information is and has been available publicly, and where it isnât Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly canât think of any secret info that is influencing GWWCâs decision here.
FWIW my personal forecast wouldnât be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMindsâ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.
Iâm fine with the wording: itâs technically âtop-rated charityâ currently but both naming and system may change over the coming years, as weâll hopefully be ramping up research efforts.
Hmmmm this still feels like a bit of a dodge. If the work is all public, what specific thing has Simon missed or misunderstood or what are you going to change? Letâs give it two weeks, but if there is no secret info there ought to be an answer to that question.
Also, what do you expect the results of the RCT to be? And if you think they will be negative shouldnât you remove the recommendation now?
Props for engaging here.
Hi Nathan, I donât think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though Iâd admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isnât immediately clear; depends on the interactions with the other reasons).
And I hope the more elaborate reply I just wrote to Simon answers your remaining question.
meta-comment: If youâre going to edit a comment, it would be useful to be specific and say how you edited the comment e.g. in this case, I think you changed the word âdisregardedâ to something weaker on further reflection.
Unfortunately that wouldnât help, because the part of the point of looking at FPâs work would be to evaluate it. Another person saying âI looked at some work privately and I agree/âdisagree with itâ doesnât seem helpful to people trying to evaluate StrongMinds.
I sense it would be better than the status quo.
What do you think would be better outcome here?
Ideally from my point of view:
GWWC would clarify what their threshold is for Top Charity
GWWC would explain how they decide what is a Trusted Evaluator and when their evaluations count to be a Top Charity (this decision process would include evaluators publishing their reasoning)
FP would publish their reasoning