Another related concept I just stumbled upon is the “Swiss cheese model of accident causation”. According to Wikipedia, this is:
a model used in risk analysis and risk management, including aviation safety, engineering, healthcare, emergency service organizations, and as the principle behind layered security, as used in computer security and defense in depth. It likens human systems to multiple slices of swiss cheese, stacked side by side, in which the risk of a threat becoming a reality is mitigated by the differing layers and types of defenses which are “layered” behind each other. Therefore, in theory, lapses and weaknesses in one defense do not allow a risk to materialize, since other defenses also exist, to prevent a single point of failure.
[...] The Swiss cheese model of accident causation illustrates that, although many layers of defense lie between hazards and accidents, there are flaws in each layer that, if aligned, can allow the accident to occur.
(Interestingly, I don’t recall the paper mentioning getting the term from computer security, and, skimming it again now, I indeed can’t see them mention that. In fact, they only seem to say “defence in depth” once in the paper.
I wonder if they got the term from computer security and forgot they’d done so, if they got it from computer security but thought it wasn’t worth mentioning, or if the term has now become fairly common outside of computer security, but with the same basic meaning, rather than the somewhat different military meaning. Not really an important question, though.)
I’ve noticed something similar around “security mindset”: Eliezer and MIRI have used the phrase to talk about a specific version of it in relation to AI safety, but the term, as far as I know, originates with Bruce Schneier and computer security, although I can’t recall MIRI publications mentioning that much, possibly because they didn’t even realize that’s where the term came from. Hard to know, a probably not very relevant other than to weirdos like us. ;-)
The initial post by Eliezer on security mindset explicitly cites Bruce Schneier as the source of the term, and quotes extensively from this piece by Schneier.
A related notion from computer security, defense in depth.
Another related concept I just stumbled upon is the “Swiss cheese model of accident causation”. According to Wikipedia, this is:
This is referenced in a viral image about preventing covid-19 infections.
Also related is the recent (very interesting) paper using that same term (linkpost).
(Interestingly, I don’t recall the paper mentioning getting the term from computer security, and, skimming it again now, I indeed can’t see them mention that. In fact, they only seem to say “defence in depth” once in the paper.
I wonder if they got the term from computer security and forgot they’d done so, if they got it from computer security but thought it wasn’t worth mentioning, or if the term has now become fairly common outside of computer security, but with the same basic meaning, rather than the somewhat different military meaning. Not really an important question, though.)
I’ve noticed something similar around “security mindset”: Eliezer and MIRI have used the phrase to talk about a specific version of it in relation to AI safety, but the term, as far as I know, originates with Bruce Schneier and computer security, although I can’t recall MIRI publications mentioning that much, possibly because they didn’t even realize that’s where the term came from. Hard to know, a probably not very relevant other than to weirdos like us. ;-)
The initial post by Eliezer on security mindset explicitly cites Bruce Schneier as the source of the term, and quotes extensively from this piece by Schneier.