The HLI discussion on the Forum recently felt off to me, bad vibes all around. It seems very heated, not a lot of scout mindset, and reading the various back-and-forth chains I felt like I was âgetting Euleredâ as Scott once described.
Iâm not an expert on evaluating charities, but I followed a lot of links to previous discussions and found this discussion involving one of the people running an RCT on Strongminds (which a lot of people are waiting for the final results of) who was highly sceptical of SM efficacy. But the person offering counterarguments in the thread seems to be just as valid to me? My current position, for what itâs worth,[1] is:
the initial Strongminds results of 10x cash transfer should raise a sceptical response. most things arenât that effective
itâs worth there being exploration of what the SWB approach would recommend as the top charities (think of this as trying other bandits in a multi-armed bandit charity evaluation problem)
itâs very difficult to do good social science, and the RCT wonât give us dispositive evidence about the effectiveness of Strongminds (especially at scale), but it may help us update. In general we should be mindful of how far we can make rigorous empirical claims in the social sciences
HLI has used language too loosely in the past and overclaimed/âbeen overconfident, which Michael has apologised for, though perhaps some critics would like a stronger signal of neutrality (this links to the âepistemic probationâ comments)
GiveWellâs own âbest guessâ analysis seems to be that Strongminds is 2.3x that of GiveDirectly.[2] Iâm generally a big fan of the GiveDirectly approach for reasons of autonomyâeven if Strongminds got reduced in efficacy to around ~1x GD, itâd still be a good intervention? Iâm much more concerned with what this number is than the tone of HLIs or Michaelâs claims tbh (though not at the expense of epistemic rigour).
The world is rife with actively wasted or even negative action, spending, and charity. The integrity of EA research, and holding charity evaluators to account is important to both the EA mission and EAs identity. But HLI seems to have been singled out for very harsh criticism,[3] but so much of the world is worse.
Iâm also quite unsettled by a lot of what I call âdrive-by downvotingâ. While writing a comment is a lot more effort than clicking to vote on a comment/âpost, I think the signal is a lot higher, and would help those involved in debates reach consensus better. Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
So Iâm very unsure how to feel. Itâs an important issue, but Iâm not sure the Forum has shown itself in a good light in this instance.
Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
I think this is a significant datum in favor of being able to see the strong up/âup/âdown/âstrong down spread for each post/âcomment. If it appeared that much of the karma activity was the result of a handful of people strongvoting each comment in a directional activity, that would influence how I read the karma count as evidence in trying to discern the communityâs viewpoint. More importantly, it would probably inform HLIâs takeawaysâin its shoes, I would treat evidence of a broad consensus of support for certain negative statements much, much more seriously than evidence of carpet-bomb voting by a small group on those statements.
Indeed our new reacts system separates them. But our new reacts system also doesnât have strong votes. A problem with displaying the number of types of votes when strong votes are involved is that it much more easily allows for deanonymization if there are only a few people in the thread.
That makes sense. On the karma side, I think some of my discomfort comes from the underlying operationalization of post/âcomment karma as merely additive of individual karma weights.
True opinion of the value of the bulk of posts/âcomments probably lies on a bell curve, so I would expect most posts/âcomments to have significantly more upvotes than strong upvotes if voters are âhonestlyâ conveying preferences and those preferences are fairly representative of the user base. Where the karma is coming predominately from strongvotes, the odds that the displayed total reflects the opinion of a smallish minority that feels passionately is much higher. That can be problematic if it gives the impression of community consensus where no such consensus exists.
If it were up to me, I would probably favor a rule along the lines of: a post/âcomment canât get more than X% of its net positive karma from strongvotes, to ensure that a high karma count reflects some degree of breadth of community support rather than voting by a small handful of people with powerful strongvotes. Downvotes are a bit trickier, because the strong downvote hammer is an effective way of quickly pushing down norm-breaking and otherwise problematic content, and I think putting posts into deep negative territory is generally used for that purpose.
EA is just a few months out from a massive scandal caused in part by socially enforced artificial consensus (FTX), but judging by this post nothing has been learned and the âshut up and just be nice to everyone else on the teamâ culture is back again, even when truth gets sacrificed on the process. No thinks HLI is stealing billions of dollars of course, but the charge that they keep quasi-deliberately stacking the deck in StrongMindsâ favour is far from outrageous and should be discussed honestly and straightforwardly.
JWSâ quick take has often been in negative agreevote territory and is +3 at this writing. Meanwhile, the comments of the lead HLI critic suggesting potential bad faith have seen consistent patterns of high upvote /â agreevote. I donât see much evidence of âshut up and just be nice to everyone else on the teamâ culture here.
I donât think the Forumâs reaction to the HLI post has been âshut up and just be nice to everyone else on the teamâ, as Jasonâs response suggested.
I donât think mine suggests that either! In fact, my first bullet point has a similar sceptical prior to what you express in this comment[1] I also literally say âholding charity evaluators to account is important to both the EA mission and EAs identityâ, and point that I donât want to sacrifice epistemic rigour. In fact, one of my main points is that peopleâeven those disagreeing with HLI, are shutting up too much! I think disagreement without explanation is bad, and I salute the thorough critics on that post who have made their reasoning for putting HLI in âepistemic probationâ clear.
I donât suggest âsacrificing the truthâ. My position is that the truth on StrongMindâs efficacy is hard to get a strong signal on, and therefore HLI should have been more modest early on their history, instead of framing it as the most effective way to donate.
As for the question of whether HLI were âquasi-deliberately stacking the deckâ, well I was quite open that I think I am confused on where the truth is, and find it difficult to adjudicate what the correct takeway should be.
I donât think we really disagree that much, and I definitely agree that the HLI discussion should proceed transparently and EA has a lot to learn from the last year, including FTX. I think if you maybe re-read my Quick Take, Iâm not taking the position you think I am.
The HLI discussion on the Forum recently felt off to me, bad vibes all around. It seems very heated, not a lot of scout mindset, and reading the various back-and-forth chains I felt like I was âgetting Euleredâ as Scott once described.
Iâm not an expert on evaluating charities, but I followed a lot of links to previous discussions and found this discussion involving one of the people running an RCT on Strongminds (which a lot of people are waiting for the final results of) who was highly sceptical of SM efficacy. But the person offering counterarguments in the thread seems to be just as valid to me? My current position, for what itâs worth,[1] is:
the initial Strongminds results of 10x cash transfer should raise a sceptical response. most things arenât that effective
itâs worth there being exploration of what the SWB approach would recommend as the top charities (think of this as trying other bandits in a multi-armed bandit charity evaluation problem)
itâs very difficult to do good social science, and the RCT wonât give us dispositive evidence about the effectiveness of Strongminds (especially at scale), but it may help us update. In general we should be mindful of how far we can make rigorous empirical claims in the social sciences
HLI has used language too loosely in the past and overclaimed/âbeen overconfident, which Michael has apologised for, though perhaps some critics would like a stronger signal of neutrality (this links to the âepistemic probationâ comments)
GiveWellâs own âbest guessâ analysis seems to be that Strongminds is 2.3x that of GiveDirectly.[2] Iâm generally a big fan of the GiveDirectly approach for reasons of autonomyâeven if Strongminds got reduced in efficacy to around ~1x GD, itâd still be a good intervention? Iâm much more concerned with what this number is than the tone of HLIs or Michaelâs claims tbh (though not at the expense of epistemic rigour).
The world is rife with actively wasted or even negative action, spending, and charity. The integrity of EA research, and holding charity evaluators to account is important to both the EA mission and EAs identity. But HLI seems to have been singled out for very harsh criticism,[3] but so much of the world is worse.
Iâm also quite unsettled by a lot of what I call âdrive-by downvotingâ. While writing a comment is a lot more effort than clicking to vote on a comment/âpost, I think the signal is a lot higher, and would help those involved in debates reach consensus better. Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
So Iâm very unsure how to feel. Itâs an important issue, but Iâm not sure the Forum has shown itself in a good light in this instance.
And I stress this isnât much in this area, I generally defer to evaluators
On the table at the top of the link, go to the column âGiveWell best guessâ and the row âCost-effectiveness, relative to cashâ
Again, I donât think I have the ability to adjudicate here, which is part of why Iâm so confused.
I think this is a significant datum in favor of being able to see the strong up/âup/âdown/âstrong down spread for each post/âcomment. If it appeared that much of the karma activity was the result of a handful of people strongvoting each comment in a directional activity, that would influence how I read the karma count as evidence in trying to discern the communityâs viewpoint. More importantly, it would probably inform HLIâs takeawaysâin its shoes, I would treat evidence of a broad consensus of support for certain negative statements much, much more seriously than evidence of carpet-bomb voting by a small group on those statements.
Indeed our new reacts system separates them. But our new reacts system also doesnât have strong votes. A problem with displaying the number of types of votes when strong votes are involved is that it much more easily allows for deanonymization if there are only a few people in the thread.
That makes sense. On the karma side, I think some of my discomfort comes from the underlying operationalization of post/âcomment karma as merely additive of individual karma weights.
True opinion of the value of the bulk of posts/âcomments probably lies on a bell curve, so I would expect most posts/âcomments to have significantly more upvotes than strong upvotes if voters are âhonestlyâ conveying preferences and those preferences are fairly representative of the user base. Where the karma is coming predominately from strongvotes, the odds that the displayed total reflects the opinion of a smallish minority that feels passionately is much higher. That can be problematic if it gives the impression of community consensus where no such consensus exists.
If it were up to me, I would probably favor a rule along the lines of: a post/âcomment canât get more than X% of its net positive karma from strongvotes, to ensure that a high karma count reflects some degree of breadth of community support rather than voting by a small handful of people with powerful strongvotes. Downvotes are a bit trickier, because the strong downvote hammer is an effective way of quickly pushing down norm-breaking and otherwise problematic content, and I think putting posts into deep negative territory is generally used for that purpose.
Looks like this feature is being rolled out on new posts. Or at least one post: https://ââforum.effectivealtruism.org/ââposts/ââgEmkxFuMck8SHC55w/ââintroducing-the-effective-altruism-addiction-recovery-group
EA is just a few months out from a massive scandal caused in part by socially enforced artificial consensus (FTX), but judging by this post nothing has been learned and the âshut up and just be nice to everyone else on the teamâ culture is back again, even when truth gets sacrificed on the process. No thinks HLI is stealing billions of dollars of course, but the charge that they keep quasi-deliberately stacking the deck in StrongMindsâ favour is far from outrageous and should be discussed honestly and straightforwardly.
JWSâ quick take has often been in negative agreevote territory and is +3 at this writing. Meanwhile, the comments of the lead HLI critic suggesting potential bad faith have seen consistent patterns of high upvote /â agreevote. I donât see much evidence of âshut up and just be nice to everyone else on the teamâ culture here.
Hey Sol, some thoughts on this comment:
I donât think the Forumâs reaction to the HLI post has been âshut up and just be nice to everyone else on the teamâ, as Jasonâs response suggested.
I donât think mine suggests that either! In fact, my first bullet point has a similar sceptical prior to what you express in this comment[1] I also literally say âholding charity evaluators to account is important to both the EA mission and EAs identityâ, and point that I donât want to sacrifice epistemic rigour. In fact, one of my main points is that peopleâeven those disagreeing with HLI, are shutting up too much! I think disagreement without explanation is bad, and I salute the thorough critics on that post who have made their reasoning for putting HLI in âepistemic probationâ clear.
I donât suggest âsacrificing the truthâ. My position is that the truth on StrongMindâs efficacy is hard to get a strong signal on, and therefore HLI should have been more modest early on their history, instead of framing it as the most effective way to donate.
As for the question of whether HLI were âquasi-deliberately stacking the deckâ, well I was quite open that I think I am confused on where the truth is, and find it difficult to adjudicate what the correct takeway should be.
I donât think we really disagree that much, and I definitely agree that the HLI discussion should proceed transparently and EA has a lot to learn from the last year, including FTX. I think if you maybe re-read my Quick Take, Iâm not taking the position you think I am.
Thatâs my interpretation of course, please correct me if Iâve misunderstood