edited to that I only had a couple of comments rather than 4
I am confident those involved really care about doing good and work really hard. And i don’t want that to be lost in this confusion. Something is going on here, but I think “it is confusing” is better than “HLI are baddies”.
For clarity being 2x better than cash transfers would still provide it with good reason to be on GWWC’s top charity list, right? Since GiveDirectly is?
I guess the most damning claim seems to be about dishonesty, which I find hard to square with the caliber of the team. So, what’s going on here? If, as seems likely the forthcoming RCT downgrades SM a lot and the HLI team should have seen this coming, why didn’t they act? Or do they still believe that they RCT will return very positive results. What happens when as seems likely, they are very wrong?
Among other things, this would confirm a) SimonM produced a more accurate and trustworthy assessment of Strongminds in their spare time as a non-subject matter expert than HLI managed as the centrepiece of their activity
Note that SimonM is a quant by day and for a time top on metaculus, so I am less surprised that he can produce such high caliber work in his spare time[1].
I don’t know how to say this but it doesn’t surprise me that top individuals are able to do work comparable with research teams. In fact I think it’s one of the best cases for the forum. Sometimes talented generalists compete toe to toe with experts.
Finally it seems possible to me that criticisms can be true but HLI can still have done work we want to fund. The world is ugly and complicated like this. I think we should aim to make the right call in this case. For me the key question is, why haven’t they updated in light of StrongMinds likely being worse than they thought.
Simon worked as a crypto quant and has since lost his job (cos of the crash caused by FTX) so is looking for work including EA work. You can message him if interested.
+1 Regarding extending the principle of charity towards HLI. Anecdotally it seems very common for initial CEA estimates to be revised down as the analysis is critiqued. I think HLI has done an exceptional job at being transparent and open regarding their methodology and the source of disagreements e.g. see Joel’s comment outlining the sources of disagreement between HLI and GiveWell, which I thought were really exceptional (https://forum.effectivealtruism.org/posts/h5sJepiwGZLbK476N/assessment-of-happier-lives-institute-s-cost-effectiveness?commentId=LqFS5yHdRcfYmX9jw). Obviously I haven’t spent as much time digging into the results as Gregory has made, but the mistakes he points to don’t seem like the kind that should be treated too harshly.
As a separate point, I think it’s generally a lot easier to critique and build upon an analysis after the initial work has been done. E.g. even if it is the case that SimonM’s assessment of Strong Minds is more reliable than HLI’s (HLI seem to dispute that the critique he levies are all that important as they only assign a 13% weight to that RCT), this isn’t necessarily evidence that SimonM is more competent than the HLI team. When the heavy lifting has been done, it’s easier to focus in on particular mistakes (and of course valuable to do so!).
For clarity being 2x better than cash transfers would still provide it with good reason to be on GWWC’s top charity list, right? Since GiveDirectly is?
I think GiveDirectly gets special privilege because “just give the money to the poorest people” is such a safe bet for how to spend money altruistically.
Like if a billionaire wanted to spend a million dollars making your life better, they could either:
just give you the million dollars directly, or
spend the money on something that they personally think would be best for you
You’d want them to set a pretty high bar of “I have high confidence that the thing I chose to spend the money on will be much better than whatever you would spend the money on yourself.”
GiveDirectly does not have the “top-rated” label on GWWC’s list, while SM does as of this morning.
I can’t find the discussion, but my understanding is that “top-rated” means that an evaluator GWWC trusts—in SM’s case, that was Founder’s Pledge—thinks that a charity is at a certain multiple (was it like 4x?) over GiveDirectly.
However, on this post, Matt Lerner @ FP wrote that “We disagree with HLI about SM’s rating — we use HLI’s work as a starting point and arrive at an undiscounted rating of 5-6x; subjective discounts place it between 1-2x, which squares with GiveWell’s analysis.”
So it seems that GWWC should withdraw the “top-rated” flag because none of its trusted evaluation partners currently rate SM at better than 2.3X cash. It should not, however, remove SM from the GWWC platform as it meets the criteria for inclusion.
Hmm this feels a bit off. I don’t think GiveDirectly should get special privelege. Though I agree the out of model factors seem to go well for GD than others, so I would kind of bump it up.
Hello Nathan. Thanks for the comment. I think the only key place where I would disagree with you is what you said here
If, as seems likely the forthcoming RCT downgrades SM a lot and the HLI team should have seen this coming, why didn’t they act?
As I said in response to Greg (to which I see you’ve replied) we use the conventional scientific approach of relying on the sweep of existing data—rather than on our predictions of what future evidence (from a single study) will show. Indeed, I’m not sure how easily these would come apart: I would base my predictions substantially on the existing data, which we’ve already gathered in our meta-analysis (obviously, it’s a matter of debate as to how to synthesise data from different sources and opinions will differ). I don’t have any reason to assume the new RCT will show effects substantially lower than the existing evidence, but perhaps others are aware of something we’re not.
edited to that I only had a couple of comments rather than 4
I am confident those involved really care about doing good and work really hard. And i don’t want that to be lost in this confusion. Something is going on here, but I think “it is confusing” is better than “HLI are baddies”.
For clarity being 2x better than cash transfers would still provide it with good reason to be on GWWC’s top charity list, right? Since GiveDirectly is?
I guess the most damning claim seems to be about dishonesty, which I find hard to square with the caliber of the team. So, what’s going on here? If, as seems likely the forthcoming RCT downgrades SM a lot and the HLI team should have seen this coming, why didn’t they act? Or do they still believe that they RCT will return very positive results. What happens when as seems likely, they are very wrong?
Note that SimonM is a quant by day and for a time top on metaculus, so I am less surprised that he can produce such high caliber work in his spare time[1].
I don’t know how to say this but it doesn’t surprise me that top individuals are able to do work comparable with research teams. In fact I think it’s one of the best cases for the forum. Sometimes talented generalists compete toe to toe with experts.
Finally it seems possible to me that criticisms can be true but HLI can still have done work we want to fund. The world is ugly and complicated like this. I think we should aim to make the right call in this case. For me the key question is, why haven’t they updated in light of StrongMinds likely being worse than they thought.
I’d be curious Gregory on your thoughts on this comment by Matt Lerner that responds to yours. https://forum.effectivealtruism.org/posts/g4QWGj3JFLiKRyxZe/the-happier-lives-institute-is-funding-constrained-and-needs?commentId=Bd9jqxAR6zfg8z4Wy
Simon worked as a crypto quant and has since lost his job (cos of the crash caused by FTX) so is looking for work including EA work. You can message him if interested.
+1 Regarding extending the principle of charity towards HLI. Anecdotally it seems very common for initial CEA estimates to be revised down as the analysis is critiqued. I think HLI has done an exceptional job at being transparent and open regarding their methodology and the source of disagreements e.g. see Joel’s comment outlining the sources of disagreement between HLI and GiveWell, which I thought were really exceptional (https://forum.effectivealtruism.org/posts/h5sJepiwGZLbK476N/assessment-of-happier-lives-institute-s-cost-effectiveness?commentId=LqFS5yHdRcfYmX9jw). Obviously I haven’t spent as much time digging into the results as Gregory has made, but the mistakes he points to don’t seem like the kind that should be treated too harshly.
As a separate point, I think it’s generally a lot easier to critique and build upon an analysis after the initial work has been done. E.g. even if it is the case that SimonM’s assessment of Strong Minds is more reliable than HLI’s (HLI seem to dispute that the critique he levies are all that important as they only assign a 13% weight to that RCT), this isn’t necessarily evidence that SimonM is more competent than the HLI team. When the heavy lifting has been done, it’s easier to focus in on particular mistakes (and of course valuable to do so!).
I think GiveDirectly gets special privilege because “just give the money to the poorest people” is such a safe bet for how to spend money altruistically.
Like if a billionaire wanted to spend a million dollars making your life better, they could either:
just give you the million dollars directly, or
spend the money on something that they personally think would be best for you
You’d want them to set a pretty high bar of “I have high confidence that the thing I chose to spend the money on will be much better than whatever you would spend the money on yourself.”
GiveDirectly does not have the “top-rated” label on GWWC’s list, while SM does as of this morning.
I can’t find the discussion, but my understanding is that “top-rated” means that an evaluator GWWC trusts—in SM’s case, that was Founder’s Pledge—thinks that a charity is at a certain multiple (was it like 4x?) over GiveDirectly.
However, on this post, Matt Lerner @ FP wrote that “We disagree with HLI about SM’s rating — we use HLI’s work as a starting point and arrive at an undiscounted rating of 5-6x; subjective discounts place it between 1-2x, which squares with GiveWell’s analysis.”
So it seems that GWWC should withdraw the “top-rated” flag because none of its trusted evaluation partners currently rate SM at better than 2.3X cash. It should not, however, remove SM from the GWWC platform as it meets the criteria for inclusion.
Hmm this feels a bit off. I don’t think GiveDirectly should get special privelege. Though I agree the out of model factors seem to go well for GD than others, so I would kind of bump it up.
Hello Nathan. Thanks for the comment. I think the only key place where I would disagree with you is what you said here
As I said in response to Greg (to which I see you’ve replied) we use the conventional scientific approach of relying on the sweep of existing data—rather than on our predictions of what future evidence (from a single study) will show. Indeed, I’m not sure how easily these would come apart: I would base my predictions substantially on the existing data, which we’ve already gathered in our meta-analysis (obviously, it’s a matter of debate as to how to synthesise data from different sources and opinions will differ). I don’t have any reason to assume the new RCT will show effects substantially lower than the existing evidence, but perhaps others are aware of something we’re not.
Yeah for what it’s worth it wasn’t clear to me until later that this was only like 10% of the weighting on your analysis.
Man, why don’t images resize properly. I’ve deleted it because it was too obnoxious when huge.