Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
Thanks Kirsten, these are good points.