I’d like to flag that I think it’s bad that my friend (yes I’m biased) has done a lot of work to criticise something (and I haven’t read pushback against that work) but won’t affect the outcome because of work that he and we cannot see.
Is there a way that we can do a little better than this?
Some thoughts:
Could he be allowed to sign an NDA to read Founder’s pledge’s work?
Would you be interested in forecasts that Stronger Minds wont be a GWWC top charity by say 2025?
Could I add this criticism and a summary of your response to Stronger Minds EA wiki page so that others can see this criticism and it doesn’t get lost?
Can anyone come up with other suggestions?
edits:
Changed “disregarded” the sentence with “won’t affect the outcome”
Tbh I think this is a bit unfair: his criticism isn’t being disregarded at all. He received a substantial reply from FP’s research director Matt Lerner—even while he’s on holiday—within a day, and Matt seems very happy to discuss this further when he’s back to work.
I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLI’s analysis this year. I don’t think what FP has internally is crucial to interpreting Matt’s responses.
I am sure there is a better word than “disregarded”. Apologies for being grumpy, have edited.
This seems like legitimate criticism. Matt says so. But currently, it feels like nothing might happen as a result. You have secret info, end of discussion. This is a common problem within charity evaluation, I think—someone makes some criticism, someone disagrees and so it gets lost to the sands of time.
I guess my question is, how can this work better? How can this criticism be stored and how can your response of “we have secret info, trust us” be a bit more costly for you now (with appropriate rewards later).
If you are interested in forecasting, would you prefer a metaculus or manifold market?
Eg if you like manifold, you can bet here (there is a lot of liquidity and the market currently heavily thinks GWWC will revoke its recommendation. If you disagree you can win money that can be donated to GWWC and status. This is one way to tax and reward you for your secret info)
Is this form of the market the correct wording? If so I’ll write a metaculus version.
As I tried to clarify above, this is not a case of secret info having much—if any—bearing on a recommendation. As far as I’m aware, nearly all decision-relevant information is and has been available publicly, and where it isn’t Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly can’t think of any secret info that is influencing GWWC’s decision here.
FWIW my personal forecast wouldn’t be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMinds’ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.
I’m fine with the wording: it’s technically “top-rated charity” currently but both naming and system may change over the coming years, as we’ll hopefully be ramping up research efforts.
Hmmmm this still feels like a bit of a dodge. If the work is all public, what specific thing has Simon missed or misunderstood or what are you going to change? Let’s give it two weeks, but if there is no secret info there ought to be an answer to that question.
Also, what do you expect the results of the RCT to be? And if you think they will be negative shouldn’t you remove the recommendation now?
Hi Nathan, I don’t think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though I’d admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isn’t immediately clear; depends on the interactions with the other reasons).
And I hope the more elaborate reply I just wrote to Simon answers your remaining question.
meta-comment: If you’re going to edit a comment, it would be useful to be specific and say how you edited the comment e.g. in this case, I think you changed the word “disregarded” to something weaker on further reflection.
Could he be allowed to sign an NDA to read Founder’s pledge’s work?
Unfortunately that wouldn’t help, because the part of the point of looking at FP’s work would be to evaluate it. Another person saying “I looked at some work privately and I agree/disagree with it” doesn’t seem helpful to people trying to evaluate StrongMinds.
GWWC would clarify what their threshold is for Top Charity
GWWC would explain how they decide what is a Trusted Evaluator and when their evaluations count to be a Top Charity (this decision process would include evaluators publishing their reasoning)
edited (see bottom)
I’d like to flag that I think it’s bad that my friend (yes I’m biased) has done a lot of work to criticise something (and I haven’t read pushback against that work) but won’t affect the outcome because of work that he and we cannot see.
Is there a way that we can do a little better than this?
Some thoughts:
Could he be allowed to sign an NDA to read Founder’s pledge’s work?
Would you be interested in forecasts that Stronger Minds wont be a GWWC top charity by say 2025?
Could I add this criticism and a summary of your response to Stronger Minds EA wiki page so that others can see this criticism and it doesn’t get lost?
Can anyone come up with other suggestions?
edits:
Changed “disregarded” the sentence with “won’t affect the outcome”
Tbh I think this is a bit unfair: his criticism isn’t being disregarded at all. He received a substantial reply from FP’s research director Matt Lerner—even while he’s on holiday—within a day, and Matt seems very happy to discuss this further when he’s back to work.
I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLI’s analysis this year. I don’t think what FP has internally is crucial to interpreting Matt’s responses.
I do like the forecasting idea though :).
I am sure there is a better word than “disregarded”. Apologies for being grumpy, have edited.
This seems like legitimate criticism. Matt says so. But currently, it feels like nothing might happen as a result. You have secret info, end of discussion. This is a common problem within charity evaluation, I think—someone makes some criticism, someone disagrees and so it gets lost to the sands of time.
I guess my question is, how can this work better? How can this criticism be stored and how can your response of “we have secret info, trust us” be a bit more costly for you now (with appropriate rewards later).
If you are interested in forecasting, would you prefer a metaculus or manifold market?
Eg if you like manifold, you can bet here (there is a lot of liquidity and the market currently heavily thinks GWWC will revoke its recommendation. If you disagree you can win money that can be donated to GWWC and status. This is one way to tax and reward you for your secret info)
Is this form of the market the correct wording? If so I’ll write a metaculus version.
As I tried to clarify above, this is not a case of secret info having much—if any—bearing on a recommendation. As far as I’m aware, nearly all decision-relevant information is and has been available publicly, and where it isn’t Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly can’t think of any secret info that is influencing GWWC’s decision here.
FWIW my personal forecast wouldn’t be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMinds’ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.
I’m fine with the wording: it’s technically “top-rated charity” currently but both naming and system may change over the coming years, as we’ll hopefully be ramping up research efforts.
Hmmmm this still feels like a bit of a dodge. If the work is all public, what specific thing has Simon missed or misunderstood or what are you going to change? Let’s give it two weeks, but if there is no secret info there ought to be an answer to that question.
Also, what do you expect the results of the RCT to be? And if you think they will be negative shouldn’t you remove the recommendation now?
Props for engaging here.
Hi Nathan, I don’t think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though I’d admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isn’t immediately clear; depends on the interactions with the other reasons).
And I hope the more elaborate reply I just wrote to Simon answers your remaining question.
meta-comment: If you’re going to edit a comment, it would be useful to be specific and say how you edited the comment e.g. in this case, I think you changed the word “disregarded” to something weaker on further reflection.
Unfortunately that wouldn’t help, because the part of the point of looking at FP’s work would be to evaluate it. Another person saying “I looked at some work privately and I agree/disagree with it” doesn’t seem helpful to people trying to evaluate StrongMinds.
I sense it would be better than the status quo.
What do you think would be better outcome here?
Ideally from my point of view:
GWWC would clarify what their threshold is for Top Charity
GWWC would explain how they decide what is a Trusted Evaluator and when their evaluations count to be a Top Charity (this decision process would include evaluators publishing their reasoning)
FP would publish their reasoning