In this FLI podcast episode, Andrew Critch suggested handling a potentially dangerous idea like a software update rollout procedure, in which the update is distributed gradually rather than to all customers at once:
… I would tell you the same thing I would tell anyone who discovers a potentially dangerous idea, which is not to write a blog post about it right away.
I would say, find three close, trusted individuals that you think reason well about human extinction risk, and ask them to think about the consequences and who to tell next. Make sure you’re fair-minded about it. Make sure that you don’t underestimate the intelligence of other people and assume that they’ll never make this prediction
...
Then do a rollout procedure. In software engineering, you developed a new feature for your software, but it could crash the whole network. It could wreck a bunch of user experiences, so you just give it to a few users and see what they think, and you slowly roll it out. I think a slow rollout procedure is the same thing you should do with any dangerous idea, any potentially dangerous idea. You might not even know the idea is dangerous. You may have developed something that only seems plausibly likely to be a civilizational scale threat, but if you zoom out and look at the world, and you imagine all the humans coming up with ideas that could be civilizational scale threats.
...
If you just think you’ve got a small chance of causing human extinction, go ahead, be a little bit worried. Tell your friends to be a little bit worried with you for like a day or three. Then expand your circle a little bit. See if they can see problems with the idea, see dangers with the idea, and slowly expand, roll out the idea into an expanding circle of responsible people until such time as it becomes clear that the idea is not dangerous, or you manage to figure out in what way it’s dangerous and what to do about it, because it’s quite hard to figure out something as complicated as how to manage a human extinction risk all by yourself or even by a team of three or maybe even ten people. You have to expand your circle of trust, but, at the same time, you can do it methodically like a software rollout, until you come up with a good plan for managing it. As for what the plan will be, I don’t know. That’s why I need you guys to do your slow rollout and figure it out.
That is absolutely right, and I am always discussing ideas with friends and advanced specialist before discussing them publicly. But doing this, I discovered two obstacles:
1) If the idea is really simple, it is likely not new, but in case of a complex idea not much people are able to properly evaluate it. Maybe if Bostrom will spend a few days analysing it, he will say “yes” or “no”, but typically best thinkers are very busy with their own deadlines, and will not have time to evaluate the ideas of random people. So you are limited to your closer friends, who could be biased in favour of you and ignore the info-hazard.
2) “False negatives”. This is the situation when a person thinks that the idea X is not an informational hazard because it is false. However, the reasons why he thinks that the idea X is false are wrong. In that situation, the info hazard assessment is not happening.
In this FLI podcast episode, Andrew Critch suggested handling a potentially dangerous idea like a software update rollout procedure, in which the update is distributed gradually rather than to all customers at once:
That is absolutely right, and I am always discussing ideas with friends and advanced specialist before discussing them publicly. But doing this, I discovered two obstacles:
1) If the idea is really simple, it is likely not new, but in case of a complex idea not much people are able to properly evaluate it. Maybe if Bostrom will spend a few days analysing it, he will say “yes” or “no”, but typically best thinkers are very busy with their own deadlines, and will not have time to evaluate the ideas of random people. So you are limited to your closer friends, who could be biased in favour of you and ignore the info-hazard.
2) “False negatives”. This is the situation when a person thinks that the idea X is not an informational hazard because it is false. However, the reasons why he thinks that the idea X is false are wrong. In that situation, the info hazard assessment is not happening.