Doing so, especially as a “trusted community member”, would hammer home the danger of well intentioned unilateralists in the way an essay can’t, and I think that idea is important.
It seems to me that either the decision to push the button is net negative, and you shouldn’t do it, or it isn’t, and if you do it people should learn the lesson “people in my community will do helpful net-positive things”. There’s something strange about the reasoning of “if I do X, people will realize that people do things like X for reasons like Y, even tho I would not be doing it for reasons like Y” (compare e.g. “I will lie about Santa to my child because that will teach them that other people in the world aren’t careful about only communicating true things”, which I am similarly suspicious of).
Yeah I guess you could read what I’m saying as that I actually think I should have pressed it for these reasons, but my moral conviction is not strong enough to have borne the social cost of doing so.
One read of that is that the community is strong enough in its social pressure to quiet bad actors like me from doing stupid harmful stuff we think is right.
Another is that social pressure is often enough to stop people from doing the right thing, and that we should be extra grateful to Petrov, and others in similar situations, because of this.
But if you actually should press the button, and do so because you correctly understand why you should, then people shouldn’t learn the lesson “people will do wild crazy stuff out of misunderstandings or malice”, because that won’t be what happened.
Perhaps the idea is that it should be a symbolic reminder that trusted community members could do bad things, rather than evidence for that proposition?
This is closer, I think the framing I might have had in mind is closer to:
people underestimate the probability of tail risks.
I think one of the reasons why is that they don’t appreciate the size of the space of unknown unknowns (which in this case includes people pushing the button for reasons like this).
causing them to see something from the unknown unknown space is therefore useful.
I think last year’s phishing incident was actually a reasonable example of this. I don’t think many people would have put sufficiently high probability on it happening, even given the button getting pressed.
It seems to me that either the decision to push the button is net negative, and you shouldn’t do it, or it isn’t, and if you do it people should learn the lesson “people in my community will do helpful net-positive things”. There’s something strange about the reasoning of “if I do X, people will realize that people do things like X for reasons like Y, even tho I would not be doing it for reasons like Y” (compare e.g. “I will lie about Santa to my child because that will teach them that other people in the world aren’t careful about only communicating true things”, which I am similarly suspicious of).
Yeah I guess you could read what I’m saying as that I actually think I should have pressed it for these reasons, but my moral conviction is not strong enough to have borne the social cost of doing so.
One read of that is that the community is strong enough in its social pressure to quiet bad actors like me from doing stupid harmful stuff we think is right.
Another is that social pressure is often enough to stop people from doing the right thing, and that we should be extra grateful to Petrov, and others in similar situations, because of this.
Either reading seems reasonable to discuss today.
But if you actually should press the button, and do so because you correctly understand why you should, then people shouldn’t learn the lesson “people will do wild crazy stuff out of misunderstandings or malice”, because that won’t be what happened.
The lesson people I would want people to learn is “I might not have considered all the reasons people might do stuff”. See comment below.
Perhaps the idea is that it should be a symbolic reminder that trusted community members could do bad things, rather than evidence for that proposition?
This is closer, I think the framing I might have had in mind is closer to:
people underestimate the probability of tail risks.
I think one of the reasons why is that they don’t appreciate the size of the space of unknown unknowns (which in this case includes people pushing the button for reasons like this).
causing them to see something from the unknown unknown space is therefore useful.
I think last year’s phishing incident was actually a reasonable example of this. I don’t think many people would have put sufficiently high probability on it happening, even given the button getting pressed.