Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
I commented on the blog, but will reproduce my comment here too since maybe not everyone will click the link.
I suspect that some of the negative reaction was from a combination of (1) the methodology being flimsy and (2) the subject matter being somewhat controversial. Out there in the non-EA world, billionaires are extremely controversial. I suspect some of the people who read your post thought there’s a cost in seeming to approve of billionaires and as a consequence that, if we do want to state our approval of them, we should better be sure that we’re right.
Hey Kirsten (& others), I’ve briefly written my thoughts about when flimsier evaluations are worth it. I would be curious to get your thoughts <https://nunosempere.com/blog/2022/10/27/are-flimsy-evaluations-worth-it/> before I either post it to or reference in the EA forum.
Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
Thanks Kirsten, these are good points.
I commented on the blog, but will reproduce my comment here too since maybe not everyone will click the link.
I suspect that some of the negative reaction was from a combination of (1) the methodology being flimsy and (2) the subject matter being somewhat controversial. Out there in the non-EA world, billionaires are extremely controversial. I suspect some of the people who read your post thought there’s a cost in seeming to approve of billionaires and as a consequence that, if we do want to state our approval of them, we should better be sure that we’re right.