I don’t see where the harm is coming from to be honest.
I doubt someone’s going to be like “I was originally hesitant to work at OpenAI, but after seeing Nuño’s “B” rating for Elon Musk, I now think working for him is a good idea.” Or “I was unsure of where to donate to, but Warren Buffet’s “A” rating means I should probably donate to him instead.”
I think the harm from posting things with flimsy methodology and get a lot of upvotes/uncritical comments is something like “lower epistemic rigour on the forum in general”, rather than this article in particular causing a great deal of harm. I think the impact of this article whether positive or negative is likely to be small.
I think Nuno acted in a permissable way. I only mentioned the expected EV likely being negative because of how much attention the post was getting + how straightforward Nuno generally is.
For what it’s worth, Nuno and I were both expecting this post to get a lot less attention. Maybe 30 karma or so (for myself). I think a lot of the interest is mainly due to the topic.
Seems like a signal that much more rigorous work here would be read.
Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
I commented on the blog, but will reproduce my comment here too since maybe not everyone will click the link.
I suspect that some of the negative reaction was from a combination of (1) the methodology being flimsy and (2) the subject matter being somewhat controversial. Out there in the non-EA world, billionaires are extremely controversial. I suspect some of the people who read your post thought there’s a cost in seeming to approve of billionaires and as a consequence that, if we do want to state our approval of them, we should better be sure that we’re right.
Thanks! Your explanation makes a lot of sense. I think ~20h of work is reasonable for an exploratory Forum post (the majority of my posts have taken less time[1]). Re lots of uncritical comments, I do think my comment (right now the top comment) is fairly critical: the implication of “I think this ranking neglects to track the most important factor” is pretty damning in terms of decision-relevancy. But it’s possible my words were too sugarcoated.
Thanks Linch, I think the amount of time /effort probably depends on the topic, and is unfortunately very difficult to make much progress on “ranking ten people’s entire life impact” with 20 hours. For many posts, 2 hours of effort would suffice, it just depends on the topic! But I’m certain reasonable people can disagree here
I also think the tone of the comments has shifted since I posted this, and ironically the number of people who agree with my comments might suggest that I was wrong to be concerned.
I don’t think (1) is a problem, if anything I think the world is far too skewed in its tendency to judge powerful people based on personal virtues and foibles, and not enough on judging them based on their impact (in either direction).
E.g. all the comments about Zuckerberg sounding robotic and very few about him donating money to random charities, far more comments about Elon’s various Twitter spats and few about either space or OpenAI, various public figures lambasted for breaking lockdowns (rather than utterly poor pandemic preparedness and response), the whole Andrew Cuomo thing, etc.
Well, I disagree on that one. In my view it’s harmful to “give a free pass” to obviously bad behaviours by powerful people because of other things they do, both in terms of justice in general and of sending a message to society that morality is optional.
E.g. many people fetishizing Musk over his space ideas or Twitter personae, and ignoring his awful labour practices or his sexual harassment and what appears to me to be predatory behaviour.
Edit to add: I wouldn’t put “Zuckerberg sounding robotic” in this category, but I would “various public figures breaking lockdowns”. I think you greatly underestimate the importance of leading by example.
I don’t see where the harm is coming from to be honest.
I doubt someone’s going to be like “I was originally hesitant to work at OpenAI, but after seeing Nuño’s “B” rating for Elon Musk, I now think working for him is a good idea.” Or “I was unsure of where to donate to, but Warren Buffet’s “A” rating means I should probably donate to him instead.”
I think the harm from posting things with flimsy methodology and get a lot of upvotes/uncritical comments is something like “lower epistemic rigour on the forum in general”, rather than this article in particular causing a great deal of harm. I think the impact of this article whether positive or negative is likely to be small.
I think Nuno acted in a permissable way. I only mentioned the expected EV likely being negative because of how much attention the post was getting + how straightforward Nuno generally is.
For what it’s worth, Nuno and I were both expecting this post to get a lot less attention. Maybe 30 karma or so (for myself). I think a lot of the interest is mainly due to the topic.
Seems like a signal that much more rigorous work here would be read.
Hey Kirsten (& others), I’ve briefly written my thoughts about when flimsier evaluations are worth it. I would be curious to get your thoughts <https://nunosempere.com/blog/2022/10/27/are-flimsy-evaluations-worth-it/> before I either post it to or reference in the EA forum.
Thanks for linking! I agree with your points. In some situations my evaluation is pretty flimsy but I have to make a decision anyways, so the evaluation still seems worth doing and using.
I might distinguish between doing an evaluation and publishing the full evaluation. If you’re testing a new evaluation method and you notice it’s giving bad results, maybe you want to just post “I tried this evaluation method and it gave bad results,” or post your evaluation but with a disclaimer that the results are clearly wrong and you hope it will help other people improve their methods.
I think you might be more optimistic than me about other people’s ability to update away from an incorrect evaluation. I’ve found it very difficult for me to update away from the first thing I read on a topic even if it’s later shown to be clearly wrong. I subconsciously have a much higher bar for later evaluations than the first one I read. That’s part of why I try to point out when evaluations aren’t very rigorous—I need to remind myself when I shouldn’t update much on something and when I should.
Thanks Kirsten, these are good points.
I commented on the blog, but will reproduce my comment here too since maybe not everyone will click the link.
I suspect that some of the negative reaction was from a combination of (1) the methodology being flimsy and (2) the subject matter being somewhat controversial. Out there in the non-EA world, billionaires are extremely controversial. I suspect some of the people who read your post thought there’s a cost in seeming to approve of billionaires and as a consequence that, if we do want to state our approval of them, we should better be sure that we’re right.
Thanks! Your explanation makes a lot of sense. I think ~20h of work is reasonable for an exploratory Forum post (the majority of my posts have taken less time[1]). Re lots of uncritical comments, I do think my comment (right now the top comment) is fairly critical: the implication of “I think this ranking neglects to track the most important factor” is pretty damning in terms of decision-relevancy. But it’s possible my words were too sugarcoated.
I would guess this post took 15-20h, and this post took ~5-8, including reading the primary source
Thanks Linch, I think the amount of time /effort probably depends on the topic, and is unfortunately very difficult to make much progress on “ranking ten people’s entire life impact” with 20 hours. For many posts, 2 hours of effort would suffice, it just depends on the topic! But I’m certain reasonable people can disagree here
I also think the tone of the comments has shifted since I posted this, and ironically the number of people who agree with my comments might suggest that I was wrong to be concerned.
I can see some, although I admit I too am skeptical of the harm here:
People become aware of good things about [powerful person] without realising that person is also a douchebag in many ways
People want to bring about a world without billionaires slightly less (or more, depending on which of these you think is bad)
I don’t think (1) is a problem, if anything I think the world is far too skewed in its tendency to judge powerful people based on personal virtues and foibles, and not enough on judging them based on their impact (in either direction).
E.g. all the comments about Zuckerberg sounding robotic and very few about him donating money to random charities, far more comments about Elon’s various Twitter spats and few about either space or OpenAI, various public figures lambasted for breaking lockdowns (rather than utterly poor pandemic preparedness and response), the whole Andrew Cuomo thing, etc.
Well, I disagree on that one. In my view it’s harmful to “give a free pass” to obviously bad behaviours by powerful people because of other things they do, both in terms of justice in general and of sending a message to society that morality is optional.
E.g. many people fetishizing Musk over his space ideas or Twitter personae, and ignoring his awful labour practices or his sexual harassment and what appears to me to be predatory behaviour.
Edit to add: I wouldn’t put “Zuckerberg sounding robotic” in this category, but I would “various public figures breaking lockdowns”. I think you greatly underestimate the importance of leading by example.
I don’t understand why “we should hold powerful figures accountable for the impact of their actions” translates to “morality is optional.”