Iâm curious how and why safety cases got so popular in AI safety. There are so many other risk assessment techniques out there, for reference ISO31010 lists 30 of them (see here) and theyâre far from exhaustive. My instinct is that itâs because safety cases are purely text-based, easily understandable, and does not require proper risk management concepts (e.g. hazards, events, consequences) for it to work; so at some point perhaps someone suggested to use it and the rest of the field just went with it.
(also I donât know how common safety cases are in safety engineering, in my decade in the oil and gas industry I have never heard of it before, though it may be more common in other industries)
There are so many other risk assessment techniques out there, for reference ISO31010 lists 30 of them (see here) and theyâre far from exhaustive.
Almost nothing on the list youâve linked is an alternative approach to the same problem safety cases try to solve. E.g. âbrainstormingâ is obviously not a competitor to safety cases. And safety cases are not even an item in that list!
Of course. Many of the these techniques are specific to certain parts of the risk assessment process. The document is unfortunately paywalled, but risk assessment can be said to have these three parts:
Risk identification
Risk analysis
Consequence
Likelihood
Level of risk
Risk evaluation
Risk treatment is missing here becacuse itâs sort of a separate process outside of risk assessment (but is tied to risk evaluation), and ISO 31010 specifically addresses the risk assessement phase. âBrainstormingâ as a class of techniques and can take different shapes and form (in my previous work we used to have structured sessions to generate âwhat-if scenariosâ), and they specifically address mainly 1 and some of 2a above.
But as Jan pointed out in his comment, perhaps safety cases are a meta-framework and not the technique itself, so the quality of a safety case depends on the quality of the evidence put forth alongside the arguments, and this quality may be related to the suitability and implementation of the specific techniques used to generate the evidence.
Itâs a great question. I see Safety Cases more as a meta-framework in which you can use different kinds of evidence. Other risk management techniques can be used as evidence in a Safety Case (eg this paper uses a delphi method).
Also I think Safety Cases are attractive to people in AI Safety because: 1) They offer flexibility for the kind of evidence and reasoning that is allowed. From skimming it seems to me that many of the other risk management practices you linked are more strict about the kind of arguments or the kind of evidence that can be brought. 2) They strive to comprehensively prove that overall risk is low. I think most of the other techniques donât let you make claims such as âoverall risk from a system is <x%â (which AI Safety people want). 3) (I might be wrong here), but it seems to me that many other risk management techniques require you to understand the system and itâs environment decently well, whereas this is very difficult for AI Safety.
Overall, you might well be right that other risk management techniques have been overlooked and we shouldnât just focus on Safety Cases.
On 2, I think there are quite a number of techniques that give you quantitative risk estimates, and itâs quite routine in safety engineering and often required (e.g. to demonstrate that you have achieved 1e-4 fatality threshold and any further risk reduction is impractical). I donât fully understand most of the techniques listed in ISO31010, but it seems that a number of them do give quantitative risk estimates as a result the risk evaluation process, e.g. monte carlo, bayesian networks, F/âN diagrams, VaR, toxicological risk assessment, etc.
If you havenât already seen this paper on risk modelling, they use FTA and bayesian networks to estimate risks quantitatively.
Iâm curious how and why safety cases got so popular in AI safety. There are so many other risk assessment techniques out there, for reference ISO31010 lists 30 of them (see here) and theyâre far from exhaustive. My instinct is that itâs because safety cases are purely text-based, easily understandable, and does not require proper risk management concepts (e.g. hazards, events, consequences) for it to work; so at some point perhaps someone suggested to use it and the rest of the field just went with it.
(also I donât know how common safety cases are in safety engineering, in my decade in the oil and gas industry I have never heard of it before, though it may be more common in other industries)
Almost nothing on the list youâve linked is an alternative approach to the same problem safety cases try to solve. E.g. âbrainstormingâ is obviously not a competitor to safety cases. And safety cases are not even an item in that list!
Of course. Many of the these techniques are specific to certain parts of the risk assessment process. The document is unfortunately paywalled, but risk assessment can be said to have these three parts:
Risk identification
Risk analysis
Consequence
Likelihood
Level of risk
Risk evaluation
Risk treatment is missing here becacuse itâs sort of a separate process outside of risk assessment (but is tied to risk evaluation), and ISO 31010 specifically addresses the risk assessement phase. âBrainstormingâ as a class of techniques and can take different shapes and form (in my previous work we used to have structured sessions to generate âwhat-if scenariosâ), and they specifically address mainly 1 and some of 2a above.
But as Jan pointed out in his comment, perhaps safety cases are a meta-framework and not the technique itself, so the quality of a safety case depends on the quality of the evidence put forth alongside the arguments, and this quality may be related to the suitability and implementation of the specific techniques used to generate the evidence.
Itâs a great question. I see Safety Cases more as a meta-framework in which you can use different kinds of evidence. Other risk management techniques can be used as evidence in a Safety Case (eg this paper uses a delphi method).
Also I think Safety Cases are attractive to people in AI Safety because:
1) They offer flexibility for the kind of evidence and reasoning that is allowed. From skimming it seems to me that many of the other risk management practices you linked are more strict about the kind of arguments or the kind of evidence that can be brought.
2) They strive to comprehensively prove that overall risk is low. I think most of the other techniques donât let you make claims such as âoverall risk from a system is <x%â (which AI Safety people want).
3) (I might be wrong here), but it seems to me that many other risk management techniques require you to understand the system and itâs environment decently well, whereas this is very difficult for AI Safety.
Overall, you might well be right that other risk management techniques have been overlooked and we shouldnât just focus on Safety Cases.
Yeah 1 and 3 seems right to me, thanks.
On 2, I think there are quite a number of techniques that give you quantitative risk estimates, and itâs quite routine in safety engineering and often required (e.g. to demonstrate that you have achieved 1e-4 fatality threshold and any further risk reduction is impractical). I donât fully understand most of the techniques listed in ISO31010, but it seems that a number of them do give quantitative risk estimates as a result the risk evaluation process, e.g. monte carlo, bayesian networks, F/âN diagrams, VaR, toxicological risk assessment, etc.
If you havenât already seen this paper on risk modelling, they use FTA and bayesian networks to estimate risks quantitatively.