I think the harm from posting things with flimsy methodology and get a lot of upvotes/uncritical comments is something like “lower epistemic rigour on the forum in general”, rather than this article in particular causing a great deal of harm. I think the impact of this article whether positive or negative is likely to be small.
I think Nuno acted in a permissable way. I only mentioned the expected EV likely being negative because of how much attention the post was getting + how straightforward Nuno generally is.
For what it’s worth, Nuno and I were both expecting this post to get a lot less attention. Maybe 30 karma or so (for myself). I think a lot of the interest is mainly due to the topic.
Seems like a signal that much more rigorous work here would be read.
Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
I commented on the blog, but will reproduce my comment here too since maybe not everyone will click the link.
I suspect that some of the negative reaction was from a combination of (1) the methodology being flimsy and (2) the subject matter being somewhat controversial. Out there in the non-EA world, billionaires are extremely controversial. I suspect some of the people who read your post thought there’s a cost in seeming to approve of billionaires and as a consequence that, if we do want to state our approval of them, we should better be sure that we’re right.
Thanks! Your explanation makes a lot of sense. I think ~20h of work is reasonable for an exploratory Forum post (the majority of my posts have taken less time[1]). Re lots of uncritical comments, I do think my comment (right now the top comment) is fairly critical: the implication of “I think this ranking neglects to track the most important factor” is pretty damning in terms of decision-relevancy. But it’s possible my words were too sugarcoated.
Thanks Linch, I think the amount of time /effort probably depends on the topic, and is unfortunately very difficult to make much progress on “ranking ten people’s entire life impact” with 20 hours. For many posts, 2 hours of effort would suffice, it just depends on the topic! But I’m certain reasonable people can disagree here
I also think the tone of the comments has shifted since I posted this, and ironically the number of people who agree with my comments might suggest that I was wrong to be concerned.
I think the harm from posting things with flimsy methodology and get a lot of upvotes/uncritical comments is something like “lower epistemic rigour on the forum in general”, rather than this article in particular causing a great deal of harm. I think the impact of this article whether positive or negative is likely to be small.
I think Nuno acted in a permissable way. I only mentioned the expected EV likely being negative because of how much attention the post was getting + how straightforward Nuno generally is.
For what it’s worth, Nuno and I were both expecting this post to get a lot less attention. Maybe 30 karma or so (for myself). I think a lot of the interest is mainly due to the topic.
Seems like a signal that much more rigorous work here would be read.
Hey Kirsten (& others), I’ve briefly written my thoughts about when flimsier evaluations are worth it. I would be curious to get your thoughts <https://nunosempere.com/blog/2022/10/27/are-flimsy-evaluations-worth-it/> before I either post it to or reference in the EA forum.
Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
Thanks Kirsten, these are good points.
I commented on the blog, but will reproduce my comment here too since maybe not everyone will click the link.
I suspect that some of the negative reaction was from a combination of (1) the methodology being flimsy and (2) the subject matter being somewhat controversial. Out there in the non-EA world, billionaires are extremely controversial. I suspect some of the people who read your post thought there’s a cost in seeming to approve of billionaires and as a consequence that, if we do want to state our approval of them, we should better be sure that we’re right.
Thanks! Your explanation makes a lot of sense. I think ~20h of work is reasonable for an exploratory Forum post (the majority of my posts have taken less time[1]). Re lots of uncritical comments, I do think my comment (right now the top comment) is fairly critical: the implication of “I think this ranking neglects to track the most important factor” is pretty damning in terms of decision-relevancy. But it’s possible my words were too sugarcoated.
I would guess this post took 15-20h, and this post took ~5-8, including reading the primary source
Thanks Linch, I think the amount of time /effort probably depends on the topic, and is unfortunately very difficult to make much progress on “ranking ten people’s entire life impact” with 20 hours. For many posts, 2 hours of effort would suffice, it just depends on the topic! But I’m certain reasonable people can disagree here
I also think the tone of the comments has shifted since I posted this, and ironically the number of people who agree with my comments might suggest that I was wrong to be concerned.