Edit 03-01-23: I have now replied more elaborately here
Hi Simon, thanks for this post! I’m research director at GWWC, and we really appreciate people engaging with our work like this and scrutinising it.
I’m on holiday currently and won’t be able to reply much more in the coming few days, but will check this page again next Tuesday at the latest to see if there’s anything more I/the GWWC team need to get back on.
For now, I’ll just very quickly address your two key claims that GWWC shouldn’t have recommended StrongMinds as a top-rated charity and that we should remove it now, both of which I disagree with.
Our process and criteria for making charity recommendations are outlined here. Crucially, note that we generally don’t do (and don’t have capacity to do) individual charity research: we almost entirely rely on our trusted evaluators—including Founders Pledge—for our recommendations. As a research team, we plan to specialize in providing guidance on which evaluators to rely rather than in doing individual charity evaluation research.
In the case of StrongMinds, they are a top-rated charity primarily because Founders Pledge recommended them to us, as you highlight. There were no reasons for us to doubt the quality of FP’s research supporting this recommendation at the time, and though you make a few good points on StrongMinds that showcase some of the uncertainties/caveats/counterarguments involved in this recommendation, I think Matt Lerner adequately addresses these in his reply above, so I don’t see reason for us to consider diverging from FP’s recommendation now either.
I do have a takeaway from your post on our communication. On our website we currently state
“StrongMinds meets our criteria to be a top-rated charity because one of our trusted evaluators, Founders Pledge, has conducted an extensive evaluation highlighting its cost-effectiveness as part of its report on mental health.”
We then also highlight HLI’s recent report as evidence supporting this recommendation. However, I think we should have been/be clearer on the fact that it’s not only FP’s 2019 report that supports this recommendation, but their ongoing research, some of which isn’t public yet (even though the report still represents their overall view, as Matt Lerner mentions above). I’ll make sure to edit this.
Thanks again for your post, and I look forward to engaging more next week at the latest if there are any other comments/questions you or anyone else have.
I just want to add my support for GWWC here. I strongly support the way they have made decisions on what to list to date:
As a GWWC member who often donates through the GWWC platform I think it is great that they take a very broad brush and have lots of charities that people might see as top on the platform. I think if their list got to small they would not be able to usefully serve the GWWC donor community (or other donors) as well.
I would note that (contrary to what some of the comments suggest) that GWWC recommend giving to Funds and do recommend giving to these charities (so they do not explicitly recommend Strong Minds). In this light I see the listing of these charities not as recommendations but as convenience for donors who are going to be giving there.
I find GWWC very transparent. Simon says ideally “GWWC would clarify what their threshold is for Top Charity”. On that specific point I don’t see how GWWC could be any more clear. Every page explains that a top charity is one that has been listed as top by an evaluator GWWC trust. Although I do agree with Simon more description of how GWWC choose certain evaluators could be helpful.
That said I would love it if going forwards GWWC could find the time. To evaluate the evaluators and the Funds and their recommendations (for example I have some concerns about the LTFF and know others do too, I know there have been concerns about ACE in the past etc).
I would not want GWWC to unlist Strong Minds from their website but I could imagine them adding a section on the Strong Minds page saying “The GWWC team views” that says: “this is listed as it is a FP top but our personal views are that …. meaning this might or might not be a good place to give especially if you care about … etc”
(Conflict of interest note: I don’t work at GWWC or FP but I do work at a FP recommended charity and at a charity who’s recommendations make it into the GWWC criteria so I might be bias).
As a GWWC member who often donates through the GWWC platform I think it is great that they take a very broad brush and have lots of charities that people might see as top on the platform. I think if their list got to small they would not be able to usefully serve the GWWC donor community (or other donors) as well.
I agree, and I’m not advocating removing StrongMinds from the platform, just removing the label “Top-rated”. Some examples of charities on the platform which are not top-rated include: GiveDirectly, SCI, Deworm the World, Happier Lives Institute, Fish Welfare Initiative, Rethink Priorities, Clean Air Task Force...
I would note that (contrary to what some of the comments suggest) that GWWC recommend giving to Funds and do recommend giving to these charities (so they do not explicitly recommend Strong Minds). In this light I see the listing of these charities not as recommendations but as convenience for donors who are going to be giving there.
I’m afraid to say I believe you are mistaken here, as I explained in my other comment. The recommendations section clearly includes top-charities recommended by trusted evaluators and explicitly includes StrongMinds. There is also a two-tier labelling of “Top-rated” and not-top-rated and StrongMinds is included in the former. Both of these are explicit recommendations afaic.
I find GWWC very transparent. Simon says ideally “GWWC would clarify what their threshold is for Top Charity”. On that specific point I don’t see how GWWC could be any more clear. Every page explains that a top charity is one that has been listed as top by an evaluator GWWC trust. Although I do agree with Simon more description of how GWWC choose certain evaluators could be helpful.
I’m not really complaining about transparency as much as I would like the threshold to include requirements for trusted evaluators to have public reasoning and a time-threshold on their recommendations.
I would not want GWWC to unlist Strong Minds from their website but I could imagine them adding a section on the Strong Minds page saying “The GWWC team views” that says: “this is listed as it is a FP top but our personal views are that …. meaning this might or might not be a good place to give especially if you care about … etc”
Again, to repeat myself. Me either! I just want them to remove the “Top-rated” label.
Recognizing GWWC’s limited bandwidth for individual charity research, what would you think of the following policy: When GWWC learns of a charity recommendation from a trusted recommender, it will post a thread on this forum and invite comments about whether the candidate is in the same ballpark as the median top-rated organization in that cause area (as defined by GWWC, so “Improving Human Well-Being”). Although GWWC will still show significant deference to its trusted evaluators in deciding how to list organizations, it will include one sentence on the organization’s description linking to the forum notice-and-comment discussion. It will post a new thread on each listed organization at 2-3 year intervals, or when there is reason to believe that new information may materially affect the charity’s evaluation.
Given GWWC’s role and the length of its writeups, I don’t think it is necessary for GWWC to directly state reasons why a donor might reasonably choose not to donate to the charity in question. However, there does need to be an accessible way for potential donors to discover if those reasons might exist. While I don’t disagree with using FP as a trusted evaluator, its mission is not primarily directed toward producing public materials written with GWWC-type donors in mind. Its materials do not meet the bar I suggested in another comment for advisory organizations to GWWC-type donors: “After engaging with the recommender’s donor-facing materials about the recommended charity for 7-10 minutes, most potential donors should have a solid understanding of the quality of evidence and degree of uncertainty behind the recommendation; this will often include at least a brief mention of any major technical issues that might significantly alter the decision of a significant number of donors.” That is not a criticism of FP because it’s not trying to make recommendations to GWWC-type donors.
So giving the community an opportunity to state concerns/reservations (if any) and link to the community discussion seems potentially valuable as a way to meet this need without consuming much in the way of limited GWWC research resources.
I’d like to flag that I think it’s bad that my friend (yes I’m biased) has done a lot of work to criticise something (and I haven’t read pushback against that work) but won’t affect the outcome because of work that he and we cannot see.
Is there a way that we can do a little better than this?
Some thoughts:
Could he be allowed to sign an NDA to read Founder’s pledge’s work?
Would you be interested in forecasts that Stronger Minds wont be a GWWC top charity by say 2025?
Could I add this criticism and a summary of your response to Stronger Minds EA wiki page so that others can see this criticism and it doesn’t get lost?
Can anyone come up with other suggestions?
edits:
Changed “disregarded” the sentence with “won’t affect the outcome”
Tbh I think this is a bit unfair: his criticism isn’t being disregarded at all. He received a substantial reply from FP’s research director Matt Lerner—even while he’s on holiday—within a day, and Matt seems very happy to discuss this further when he’s back to work.
I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLI’s analysis this year. I don’t think what FP has internally is crucial to interpreting Matt’s responses.
I am sure there is a better word than “disregarded”. Apologies for being grumpy, have edited.
This seems like legitimate criticism. Matt says so. But currently, it feels like nothing might happen as a result. You have secret info, end of discussion. This is a common problem within charity evaluation, I think—someone makes some criticism, someone disagrees and so it gets lost to the sands of time.
I guess my question is, how can this work better? How can this criticism be stored and how can your response of “we have secret info, trust us” be a bit more costly for you now (with appropriate rewards later).
If you are interested in forecasting, would you prefer a metaculus or manifold market?
Eg if you like manifold, you can bet here (there is a lot of liquidity and the market currently heavily thinks GWWC will revoke its recommendation. If you disagree you can win money that can be donated to GWWC and status. This is one way to tax and reward you for your secret info)
Is this form of the market the correct wording? If so I’ll write a metaculus version.
As I tried to clarify above, this is not a case of secret info having much—if any—bearing on a recommendation. As far as I’m aware, nearly all decision-relevant information is and has been available publicly, and where it isn’t Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly can’t think of any secret info that is influencing GWWC’s decision here.
FWIW my personal forecast wouldn’t be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMinds’ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.
I’m fine with the wording: it’s technically “top-rated charity” currently but both naming and system may change over the coming years, as we’ll hopefully be ramping up research efforts.
Hmmmm this still feels like a bit of a dodge. If the work is all public, what specific thing has Simon missed or misunderstood or what are you going to change? Let’s give it two weeks, but if there is no secret info there ought to be an answer to that question.
Also, what do you expect the results of the RCT to be? And if you think they will be negative shouldn’t you remove the recommendation now?
Hi Nathan, I don’t think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though I’d admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isn’t immediately clear; depends on the interactions with the other reasons).
And I hope the more elaborate reply I just wrote to Simon answers your remaining question.
meta-comment: If you’re going to edit a comment, it would be useful to be specific and say how you edited the comment e.g. in this case, I think you changed the word “disregarded” to something weaker on further reflection.
Could he be allowed to sign an NDA to read Founder’s pledge’s work?
Unfortunately that wouldn’t help, because the part of the point of looking at FP’s work would be to evaluate it. Another person saying “I looked at some work privately and I agree/disagree with it” doesn’t seem helpful to people trying to evaluate StrongMinds.
GWWC would clarify what their threshold is for Top Charity
GWWC would explain how they decide what is a Trusted Evaluator and when their evaluations count to be a Top Charity (this decision process would include evaluators publishing their reasoning)
Edit 03-01-23: I have now replied more elaborately here
Hi Simon, thanks for this post! I’m research director at GWWC, and we really appreciate people engaging with our work like this and scrutinising it.
I’m on holiday currently and won’t be able to reply much more in the coming few days, but will check this page again next Tuesday at the latest to see if there’s anything more I/the GWWC team need to get back on.
For now, I’ll just very quickly address your two key claims that GWWC shouldn’t have recommended StrongMinds as a top-rated charity and that we should remove it now, both of which I disagree with.
Our process and criteria for making charity recommendations are outlined here. Crucially, note that we generally don’t do (and don’t have capacity to do) individual charity research: we almost entirely rely on our trusted evaluators—including Founders Pledge—for our recommendations. As a research team, we plan to specialize in providing guidance on which evaluators to rely rather than in doing individual charity evaluation research.
In the case of StrongMinds, they are a top-rated charity primarily because Founders Pledge recommended them to us, as you highlight. There were no reasons for us to doubt the quality of FP’s research supporting this recommendation at the time, and though you make a few good points on StrongMinds that showcase some of the uncertainties/caveats/counterarguments involved in this recommendation, I think Matt Lerner adequately addresses these in his reply above, so I don’t see reason for us to consider diverging from FP’s recommendation now either.
I do have a takeaway from your post on our communication. On our website we currently state
“StrongMinds meets our criteria to be a top-rated charity because one of our trusted evaluators, Founders Pledge, has conducted an extensive evaluation highlighting its cost-effectiveness as part of its report on mental health.”
We then also highlight HLI’s recent report as evidence supporting this recommendation. However, I think we should have been/be clearer on the fact that it’s not only FP’s 2019 report that supports this recommendation, but their ongoing research, some of which isn’t public yet (even though the report still represents their overall view, as Matt Lerner mentions above). I’ll make sure to edit this.
Thanks again for your post, and I look forward to engaging more next week at the latest if there are any other comments/questions you or anyone else have.
I just want to add my support for GWWC here. I strongly support the way they have made decisions on what to list to date:
As a GWWC member who often donates through the GWWC platform I think it is great that they take a very broad brush and have lots of charities that people might see as top on the platform. I think if their list got to small they would not be able to usefully serve the GWWC donor community (or other donors) as well.
I would note that (contrary to what some of the comments suggest) that GWWC recommend giving to Funds and do recommend giving to these charities (so they do not explicitly recommend Strong Minds). In this light I see the listing of these charities not as recommendations but as convenience for donors who are going to be giving there.
I find GWWC very transparent. Simon says ideally “GWWC would clarify what their threshold is for Top Charity”. On that specific point I don’t see how GWWC could be any more clear. Every page explains that a top charity is one that has been listed as top by an evaluator GWWC trust. Although I do agree with Simon more description of how GWWC choose certain evaluators could be helpful.
That said I would love it if going forwards GWWC could find the time. To evaluate the evaluators and the Funds and their recommendations (for example I have some concerns about the LTFF and know others do too, I know there have been concerns about ACE in the past etc).
I would not want GWWC to unlist Strong Minds from their website but I could imagine them adding a section on the Strong Minds page saying “The GWWC team views” that says: “this is listed as it is a FP top but our personal views are that …. meaning this might or might not be a good place to give especially if you care about … etc”
(Conflict of interest note: I don’t work at GWWC or FP but I do work at a FP recommended charity and at a charity who’s recommendations make it into the GWWC criteria so I might be bias).
I agree, and I’m not advocating removing StrongMinds from the platform, just removing the label “Top-rated”. Some examples of charities on the platform which are not top-rated include: GiveDirectly, SCI, Deworm the World, Happier Lives Institute, Fish Welfare Initiative, Rethink Priorities, Clean Air Task Force...
I’m afraid to say I believe you are mistaken here, as I explained in my other comment. The recommendations section clearly includes top-charities recommended by trusted evaluators and explicitly includes StrongMinds. There is also a two-tier labelling of “Top-rated” and not-top-rated and StrongMinds is included in the former. Both of these are explicit recommendations afaic.
I’m not really complaining about transparency as much as I would like the threshold to include requirements for trusted evaluators to have public reasoning and a time-threshold on their recommendations.
Again, to repeat myself. Me either! I just want them to remove the “Top-rated” label.
Ah. Good point. Replied to the other thread here: https://forum.effectivealtruism.org/posts/ffmbLCzJctLac3rDu/strongminds-should-not-be-a-top-rated-charity-yet?commentId=TMbymn5Cyqdpv5diQ .
Recognizing GWWC’s limited bandwidth for individual charity research, what would you think of the following policy: When GWWC learns of a charity recommendation from a trusted recommender, it will post a thread on this forum and invite comments about whether the candidate is in the same ballpark as the median top-rated organization in that cause area (as defined by GWWC, so “Improving Human Well-Being”). Although GWWC will still show significant deference to its trusted evaluators in deciding how to list organizations, it will include one sentence on the organization’s description linking to the forum notice-and-comment discussion. It will post a new thread on each listed organization at 2-3 year intervals, or when there is reason to believe that new information may materially affect the charity’s evaluation.
Given GWWC’s role and the length of its writeups, I don’t think it is necessary for GWWC to directly state reasons why a donor might reasonably choose not to donate to the charity in question. However, there does need to be an accessible way for potential donors to discover if those reasons might exist. While I don’t disagree with using FP as a trusted evaluator, its mission is not primarily directed toward producing public materials written with GWWC-type donors in mind. Its materials do not meet the bar I suggested in another comment for advisory organizations to GWWC-type donors: “After engaging with the recommender’s donor-facing materials about the recommended charity for 7-10 minutes, most potential donors should have a solid understanding of the quality of evidence and degree of uncertainty behind the recommendation; this will often include at least a brief mention of any major technical issues that might significantly alter the decision of a significant number of donors.” That is not a criticism of FP because it’s not trying to make recommendations to GWWC-type donors.
So giving the community an opportunity to state concerns/reservations (if any) and link to the community discussion seems potentially valuable as a way to meet this need without consuming much in the way of limited GWWC research resources.
Thanks for the suggestion Jason, though I hope the longer comment I just posted will clarify why I think this wouldn’t be worth doing.
edited (see bottom)
I’d like to flag that I think it’s bad that my friend (yes I’m biased) has done a lot of work to criticise something (and I haven’t read pushback against that work) but won’t affect the outcome because of work that he and we cannot see.
Is there a way that we can do a little better than this?
Some thoughts:
Could he be allowed to sign an NDA to read Founder’s pledge’s work?
Would you be interested in forecasts that Stronger Minds wont be a GWWC top charity by say 2025?
Could I add this criticism and a summary of your response to Stronger Minds EA wiki page so that others can see this criticism and it doesn’t get lost?
Can anyone come up with other suggestions?
edits:
Changed “disregarded” the sentence with “won’t affect the outcome”
Tbh I think this is a bit unfair: his criticism isn’t being disregarded at all. He received a substantial reply from FP’s research director Matt Lerner—even while he’s on holiday—within a day, and Matt seems very happy to discuss this further when he’s back to work.
I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLI’s analysis this year. I don’t think what FP has internally is crucial to interpreting Matt’s responses.
I do like the forecasting idea though :).
I am sure there is a better word than “disregarded”. Apologies for being grumpy, have edited.
This seems like legitimate criticism. Matt says so. But currently, it feels like nothing might happen as a result. You have secret info, end of discussion. This is a common problem within charity evaluation, I think—someone makes some criticism, someone disagrees and so it gets lost to the sands of time.
I guess my question is, how can this work better? How can this criticism be stored and how can your response of “we have secret info, trust us” be a bit more costly for you now (with appropriate rewards later).
If you are interested in forecasting, would you prefer a metaculus or manifold market?
Eg if you like manifold, you can bet here (there is a lot of liquidity and the market currently heavily thinks GWWC will revoke its recommendation. If you disagree you can win money that can be donated to GWWC and status. This is one way to tax and reward you for your secret info)
Is this form of the market the correct wording? If so I’ll write a metaculus version.
As I tried to clarify above, this is not a case of secret info having much—if any—bearing on a recommendation. As far as I’m aware, nearly all decision-relevant information is and has been available publicly, and where it isn’t Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly can’t think of any secret info that is influencing GWWC’s decision here.
FWIW my personal forecast wouldn’t be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMinds’ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.
I’m fine with the wording: it’s technically “top-rated charity” currently but both naming and system may change over the coming years, as we’ll hopefully be ramping up research efforts.
Hmmmm this still feels like a bit of a dodge. If the work is all public, what specific thing has Simon missed or misunderstood or what are you going to change? Let’s give it two weeks, but if there is no secret info there ought to be an answer to that question.
Also, what do you expect the results of the RCT to be? And if you think they will be negative shouldn’t you remove the recommendation now?
Props for engaging here.
Hi Nathan, I don’t think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though I’d admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isn’t immediately clear; depends on the interactions with the other reasons).
And I hope the more elaborate reply I just wrote to Simon answers your remaining question.
meta-comment: If you’re going to edit a comment, it would be useful to be specific and say how you edited the comment e.g. in this case, I think you changed the word “disregarded” to something weaker on further reflection.
Unfortunately that wouldn’t help, because the part of the point of looking at FP’s work would be to evaluate it. Another person saying “I looked at some work privately and I agree/disagree with it” doesn’t seem helpful to people trying to evaluate StrongMinds.
I sense it would be better than the status quo.
What do you think would be better outcome here?
Ideally from my point of view:
GWWC would clarify what their threshold is for Top Charity
GWWC would explain how they decide what is a Trusted Evaluator and when their evaluations count to be a Top Charity (this decision process would include evaluators publishing their reasoning)
FP would publish their reasoning