Are there examples of standards in other industries where people were quite confused about what “safety” would require?
Yes, medical robotics is one I was involved in. Though there, the answer is often just wait for the first product to hit the market (there is nothing quite there yet, doing full autonomous surgery), and then copy their approach.
As is, the medical standards don’t cover much ML, and so the companies have to come up with the reasoning themselves for convincing the FDA in the audit. Which in practice means many companies just don’t risk it, and do something robotic, but surgeon controled, or use classical algorithms instead of deep learning.
Yes, medical robotics is one I was involved in. Though there, the answer is often just wait for the first product to hit the market (there is nothing quite there yet, doing full autonomous surgery), and then copy their approach. As is, the medical standards don’t cover much ML, and so the companies have to come up with the reasoning themselves for convincing the FDA in the audit. Which in practice means many companies just don’t risk it, and do something robotic, but surgeon controled, or use classical algorithms instead of deep learning.