Thanks for the article! I agree. Like it or not standards are going to be created, and regulators (FTC, FDA etc.) will likely rely on them.
One tangible area to work on: publicizing well-researched best practice ‘safe/ aligned’ implementations of LLMs. Given the resource challenges that organizations like NIST has, they will likely put a lot of weight behind such research.
I’m working with NIST as part of my masters dissertation to ‘operationalize the risk management framework’. If you’d like to discuss please reach out to samyoon@hks.harvard.edu
It’s great to see this renewed call for safety standardization! A few years after my initial report, I continue to view standardization of safety processes as an important way to spread beneficial practices and as a precursor to regulation, as you describe. A few reactions to forward the conversation:
1. It’s worth underlining a key limitation to standards in my view: it’s difficult for them to influence the vanguard. Standards are most useful in disseminating best practices (from the vanguard where they’re developed to everyone else) and thus raising the safety floor. This poses obvious challenges for standards’ use in alignment. Though not insurmountable, effective use of standards here would be a deviation from the common path in standardization.
2. Following from 1, a dedicated SSO for AI safety that draws from actors concerned about alignment could well make sense. One possible vehicle could be the Joint Development Foundation.
3. I appreciate the list of best practices worth considering for standardization. These are promising directions, though it would be helpful to understand if there is much buy-in from safety experts. A useful intervention: create a (recurring) expert survey that measures perceived maturity of candidate best practices and their priority for standardization. This would be a good intervention in the short-term.
4. I agree that AI safety expertise should be brought to existing standardization venues and also with your footnote 14 caveat that the opportunity cost of researchers time should not be treated lightly. In practice, leading AI labs would benefit from emulating large companies’ approaches: dedicated staff (or even teams) to monitor developments at SSOs and to channel expertise (whether inviting an expert researcher to one SSO meeting or by circulating SSO submission internally for AI safety researcher feedback) in a way that does not overburden researchers. At the community level, individuals may be able to fill this role, as Tony Barrett has with NIST (as Evan Murphy linked, his submission is worth a close read).
5. I appreciate your identification of transparency as a pitfall of many SSOs and a point to improve. Open availability of standards should be encouraged. I’d go further to encourage actors to be transparent about their engagement in standardization: publish blogs/specifications for wider scrutiny. Transparency can also increase data for researchers trying to measure the efficacy of standards engagement (itself a challenging question).
6. It’s worth underlining the importance of standards to implementing the EU AI Act as currently envisioned. Even if the incentives are not such that we see a Brussels Effect for AI, the standards themselves may be expected to be used beyond the single market. This would mean prioritizing engagement in CEN-CENELEC to inform standards that will support conformity assessment.
Thank you for sharing! Great post and I’m glad there’s more attention going towards standard-setting activities. Some misc. ‘off the top of my head’ thoughts:
You’re right to highlight that standards are not a panacea and can be difficult in practice:
Looking into when and why companies deviate from standards would be a useful area of study. What would the Volkswagen emissions scandal look like for TAI?
As you mention, it might be difficult to find consensus where other stakeholders are not necessarily aligned or have the same incentives. A potentially useful thought experiment could be “if we were negotiating standards with Yann LeCun, where/why would we disagree?”
Trade secrets and intellectual property considerations are also important in this process: this piece on 5G standards and Huawei is quite illustrative. This could be either a blocker or an opportunity depending on how you see it.
Geopolitical challenges (see this and this) might make things a bit more complicated in practice.
Some SSOs work closely with others: e.g. CEN/CENELEC and ISO. I’m not familiar with their work but orgs like OCEANIS might be worth looking into as well.
Companies tend to value harmonisation, so avoiding fragmentation should be a key aim too.
In order to do standards well, a lot of that work on measurement and assessment is needed first—though a lot of this work is ongoing .
From a US policy POV, this bill might be of interest. Worth thinking about the impact of subsidizing or incentivising the involvement of more small and medium-sized companies.
I think an interesting project might be developing three ‘real’ AI use cases and assessing throughout what best practice / desirable standards might look like. It’s a complex area for AI systems in particular, so a demonstration would be very persuasive. Definitely an area where AI capabilities and AI safety people could work together on. And perhaps policymakers and regulators (like the ICO in the UK) could facilitate this with sandboxes.
Just my 2c. Very supportive otherwise, as this is definitely an under-explored area: haven’t seen much on standards in EA world since Peter Cihon’s excellent paper. Thanks afor sharing :)
This is a bit of a hot-take, but I’m somewhat skeptical of the ability of standards to effectively regulate TAI. I suspect that in order to be safe, an actor will have to be willing to take measures beyond any standards, in which case implementing paragraph 23 subsection d will only be a distraction. On the other hand, standards could very easily slow the most responsible actors and cause one of the least responsible actors who doesn’t care about them at all to win the AGI race.
I can respond to your message right now via a myriad of potential software because of the establishment of a technical standard, HTTP. Additionally, all major web browsers run and interpret Javascript, in large part due to SSOs like IETF and W3C. By contrast, on mobile, we have two languages for the duopoly, and a myriad of issues I won’t go into, but suffice to say there has been a failure of SSOs in the space to replicate what happened with web browsing and early internet. It may be that TAI present novel and harder challenges, but in some of the hardest such technical coordination challenges to date, SSOs have been very useful. I’m not as worried about defection as you if we get something good going—the leaders will likely have significant resources, and therefore be under bigger public scrutiny and will want to show they are also leading on participating in standard setting. I am hopeful that there will be significant innovation in this area in the next few years. [Disclaimer, I work in this area, so naturally biased]
I guess the success of those standards for the web doesn’t feel very relevant to the problem of aligning AI. For a start, the design of the protocols has led to countless security flaws, hardly seems robust?
In addition, the technology has often evolved by messing up and then being patched later.
AI doesn’t exist in a vacuum, and TAI won’t either. AI has messed up, is messing up and will mess up bigger as it gets more advanced. Security will never be a 100% solved problem, and aiming for zero breaches of all AI systems is unrealistic. I think we’re more likely to have better AI security with standards—do you disagree with that? I’m not a security expert, but here some relevant considerations of one applied to TAI. See in particular the section “Assurance Requires Formal Proofs, Which Are Provably Impossible”. Given the probably impossible nature of having formal guarantees (not to say we shouldn’t try to get as close as possible), it really does seem that leveraging whatever institutional and coordination mechanisms have worked in the past is a worthwhile idea. I consider SSOs to be one set of these, all things considered.
Here is a section from an article written by someone who has worked in SSOs and security for decades: > Most modern encryption is based on standardised algorithms and protocols; the use of open, well-tested and thoroughly analysed encryption standards is generally recommended. WhatsApp, Facebook Messenger, Skype, and Google Messages now all use the same encryption standard (the Signal protocol) because it has proven to be secure and reliable. Even if weaknesses are found in such encryption standards, solutions are often quickly made available thanks to the sheer number of adopters.
I think that’s a valid worry and I also don’t expect the standards to end up specifying how to solve the alignment problem. :P I’d still be pretty happy about the proposed efforts on standard setting because I also expect standards to have massive effects that can be more or less useful for a) directing research in directions that reduce longterm risks (e.g. pushing for more mechanistic interpretability), b) limiting how quickly an agentic AI can escape our control (e.g. via regulating internet access, making manipulation harder), c) enabling strong(er) international agreements (e.g. shared standards could become basis for international monitoring efforts of AI development and deployment).
Lack of access to the incorporated standards, since the standards often cost hundreds of dollars each to access.
Not only are many standards expensive, but they often include digital rights management that make them cumbersome to access and open.
In Australia, access to standards is controlled by private companies that can charge whatever they like. There’s currently a petition to the Australian parliament with 22,526 signatures requesting free or affordable access to Australian Standards, including standards mandated by legislation. Across the ditch, the New Zealand government has set a great example by funding free access to building standards.
It’s important for AI safety standards to be open access from the start.
Great post! I agree that standard setting could be useful. I think it could be especially important to set standards on how AI systems interact with animals and the natural environment, in addition to humans.
Thanks for the article! I agree. Like it or not standards are going to be created, and regulators (FTC, FDA etc.) will likely rely on them.
One tangible area to work on: publicizing well-researched best practice ‘safe/ aligned’ implementations of LLMs. Given the resource challenges that organizations like NIST has, they will likely put a lot of weight behind such research.
I’m working with NIST as part of my masters dissertation to ‘operationalize the risk management framework’. If you’d like to discuss please reach out to samyoon@hks.harvard.edu
Just a sidenote for anyone interested in this. There is an existing effort from some folks in the AI safety community to influence the development of this framework in a positive direction. See Actionable Guidance for High-Consequence AI Risk Management (Barett et al. 2022).
It’s great to see this renewed call for safety standardization! A few years after my initial report, I continue to view standardization of safety processes as an important way to spread beneficial practices and as a precursor to regulation, as you describe. A few reactions to forward the conversation:
1. It’s worth underlining a key limitation to standards in my view: it’s difficult for them to influence the vanguard. Standards are most useful in disseminating best practices (from the vanguard where they’re developed to everyone else) and thus raising the safety floor. This poses obvious challenges for standards’ use in alignment. Though not insurmountable, effective use of standards here would be a deviation from the common path in standardization.
2. Following from 1, a dedicated SSO for AI safety that draws from actors concerned about alignment could well make sense. One possible vehicle could be the Joint Development Foundation.
3. I appreciate the list of best practices worth considering for standardization. These are promising directions, though it would be helpful to understand if there is much buy-in from safety experts. A useful intervention: create a (recurring) expert survey that measures perceived maturity of candidate best practices and their priority for standardization. This would be a good intervention in the short-term.
4. I agree that AI safety expertise should be brought to existing standardization venues and also with your footnote 14 caveat that the opportunity cost of researchers time should not be treated lightly. In practice, leading AI labs would benefit from emulating large companies’ approaches: dedicated staff (or even teams) to monitor developments at SSOs and to channel expertise (whether inviting an expert researcher to one SSO meeting or by circulating SSO submission internally for AI safety researcher feedback) in a way that does not overburden researchers. At the community level, individuals may be able to fill this role, as Tony Barrett has with NIST (as Evan Murphy linked, his submission is worth a close read).
5. I appreciate your identification of transparency as a pitfall of many SSOs and a point to improve. Open availability of standards should be encouraged. I’d go further to encourage actors to be transparent about their engagement in standardization: publish blogs/specifications for wider scrutiny. Transparency can also increase data for researchers trying to measure the efficacy of standards engagement (itself a challenging question).
6. It’s worth underlining the importance of standards to implementing the EU AI Act as currently envisioned. Even if the incentives are not such that we see a Brussels Effect for AI, the standards themselves may be expected to be used beyond the single market. This would mean prioritizing engagement in CEN-CENELEC to inform standards that will support conformity assessment.
Thank you for sharing! Great post and I’m glad there’s more attention going towards standard-setting activities. Some misc. ‘off the top of my head’ thoughts:
You’re right to highlight that standards are not a panacea and can be difficult in practice:
Looking into when and why companies deviate from standards would be a useful area of study. What would the Volkswagen emissions scandal look like for TAI?
As you mention, it might be difficult to find consensus where other stakeholders are not necessarily aligned or have the same incentives. A potentially useful thought experiment could be “if we were negotiating standards with Yann LeCun, where/why would we disagree?”
Trade secrets and intellectual property considerations are also important in this process: this piece on 5G standards and Huawei is quite illustrative. This could be either a blocker or an opportunity depending on how you see it.
Geopolitical challenges (see this and this) might make things a bit more complicated in practice.
Some SSOs work closely with others: e.g. CEN/CENELEC and ISO. I’m not familiar with their work but orgs like OCEANIS might be worth looking into as well.
Companies tend to value harmonisation, so avoiding fragmentation should be a key aim too.
In order to do standards well, a lot of that work on measurement and assessment is needed first—though a lot of this work is ongoing .
From a US policy POV, this bill might be of interest. Worth thinking about the impact of subsidizing or incentivising the involvement of more small and medium-sized companies.
I think an interesting project might be developing three ‘real’ AI use cases and assessing throughout what best practice / desirable standards might look like. It’s a complex area for AI systems in particular, so a demonstration would be very persuasive. Definitely an area where AI capabilities and AI safety people could work together on. And perhaps policymakers and regulators (like the ICO in the UK) could facilitate this with sandboxes.
Just my 2c. Very supportive otherwise, as this is definitely an under-explored area: haven’t seen much on standards in EA world since Peter Cihon’s excellent paper. Thanks afor sharing :)
Just to add to UK regulator stuff in the space: the DRCF has a stream on algorithm auditing. Here is a paper with a short section on standards. Obviously it’s early days, and focused on current AI systems, but it’s a start: https://www.gov.uk/government/publications/findings-from-the-drcf-algorithmic-processing-workstream-spring-2022/auditing-algorithms-the-existing-landscape-role-of-regulators-and-future-outlook
This is a bit of a hot-take, but I’m somewhat skeptical of the ability of standards to effectively regulate TAI. I suspect that in order to be safe, an actor will have to be willing to take measures beyond any standards, in which case implementing paragraph 23 subsection d will only be a distraction. On the other hand, standards could very easily slow the most responsible actors and cause one of the least responsible actors who doesn’t care about them at all to win the AGI race.
I can respond to your message right now via a myriad of potential software because of the establishment of a technical standard, HTTP. Additionally, all major web browsers run and interpret Javascript, in large part due to SSOs like IETF and W3C. By contrast, on mobile, we have two languages for the duopoly, and a myriad of issues I won’t go into, but suffice to say there has been a failure of SSOs in the space to replicate what happened with web browsing and early internet. It may be that TAI present novel and harder challenges, but in some of the hardest such technical coordination challenges to date, SSOs have been very useful. I’m not as worried about defection as you if we get something good going—the leaders will likely have significant resources, and therefore be under bigger public scrutiny and will want to show they are also leading on participating in standard setting. I am hopeful that there will be significant innovation in this area in the next few years. [Disclaimer, I work in this area, so naturally biased]
I guess the success of those standards for the web doesn’t feel very relevant to the problem of aligning AI. For a start, the design of the protocols has led to countless security flaws, hardly seems robust?
In addition, the technology has often evolved by messing up and then being patched later.
AI doesn’t exist in a vacuum, and TAI won’t either. AI has messed up, is messing up and will mess up bigger as it gets more advanced. Security will never be a 100% solved problem, and aiming for zero breaches of all AI systems is unrealistic. I think we’re more likely to have better AI security with standards—do you disagree with that? I’m not a security expert, but here some relevant considerations of one applied to TAI. See in particular the section “Assurance Requires Formal Proofs, Which Are Provably Impossible”. Given the probably impossible nature of having formal guarantees (not to say we shouldn’t try to get as close as possible), it really does seem that leveraging whatever institutional and coordination mechanisms have worked in the past is a worthwhile idea. I consider SSOs to be one set of these, all things considered.
Here is a section from an article written by someone who has worked in SSOs and security for decades:
> Most modern encryption is based on standardised algorithms and protocols; the use of open, well-tested and thoroughly analysed encryption standards is generally recommended. WhatsApp, Facebook Messenger, Skype, and Google Messages now all use the same encryption standard (the Signal protocol) because it has proven to be secure and reliable. Even if weaknesses are found in such encryption standards, solutions are often quickly made available thanks to the sheer number of adopters.
Standards can help with security b/c that’s more of a standard problem, but I suspect it’ll be a distraction for aligning AGI.
Well I disagree but there’s no need to agree—diverse approaches to a hard problem sounds good to me.
I think that’s a valid worry and I also don’t expect the standards to end up specifying how to solve the alignment problem. :P I’d still be pretty happy about the proposed efforts on standard setting because I also expect standards to have massive effects that can be more or less useful for
a) directing research in directions that reduce longterm risks (e.g. pushing for more mechanistic interpretability),
b) limiting how quickly an agentic AI can escape our control (e.g. via regulating internet access, making manipulation harder),
c) enabling strong(er) international agreements (e.g. shared standards could become basis for international monitoring efforts of AI development and deployment).
Not only are many standards expensive, but they often include digital rights management that make them cumbersome to access and open.
In Australia, access to standards is controlled by private companies that can charge whatever they like. There’s currently a petition to the Australian parliament with 22,526 signatures requesting free or affordable access to Australian Standards, including standards mandated by legislation. Across the ditch, the New Zealand government has set a great example by funding free access to building standards.
It’s important for AI safety standards to be open access from the start.
Great post! I agree that standard setting could be useful. I think it could be especially important to set standards on how AI systems interact with animals and the natural environment, in addition to humans.
Great initiative! Slight nuance to Chris Leong ’s earlier comment. Though I’m not an expert, I would just caution for standards-setting bodies hastingly standardizing a losing standard, see https://en.wikipedia.org/wiki/Protocol_Wars
Your encryption standards examples feel like a great comparison of the way to go