The following is a midterm assignment I submitted for my Cyber Operations class at Georgetown, regarding the risk of large AI model theft. I figured I would just publish this since it’s fairly relevant to recent discussions and events around AI model theft. (I also am posting this so I have a non-Google-Doc link to share with people)
Note: In this assignment I had a 500-word limit and was only tasked to describe a problem’s relevance to my client/audience while briefly mentioning policy options. In an upcoming memo assignment I will need to actually go into more detail on the policy recommendations (and I’d be happy to receive suggestions for what CISA should do if you have any).
(I also acknowledge that the recommendations I lay out here are a bit milquetoast, but I genuinely just didn’t know what else to say...)
-------------
Memorandum for the Cybersecurity and Infrastructure Security Agency (CISA)
SUBJECT: Securing Large AI Models Against Theft
Large artificial intelligence (AI) models such as ChatGPT have increasingly demonstrated AI’s potential. However, as proprietary models become more powerful it is increasingly important to protect them against theft. CISA should work to facilitate information sharing that supports public policy and private responses. The following four sections will discuss some of the threat motivations/trends, potential consequences, market failures, and policy recommendations for CISA.
Motivations and Relevant Trends Regarding AI Model Theft
There are many reasons to expect that hackers will attempt to exfiltrate proprietary AI models:
China and other actors have repeatedly stolen sensitive data and intellectual property (IP).[1]
Future models may prove to have such significant economic or military value that state actors are willing to expend substantial effort/assets to steal them.
Current large models have high up-front development (“training”) costs/requirements but comparatively low operational costs/requirements after training.[2] This makes theft of models attractive even for non-state actors. Additionally, recent export controls on semiconductors to China could undermine China’s ability to train future large models,[3] which would further increase Beijing’s incentive to support model theft.
Someone reportedly leaked Meta’s new large language model (LLaMA) within days of Meta providing model access to researchers.[4]
Potential Consequences of AI Model Theft
Theft of powerful AI models—or the threat thereof—could have significant negative consequences beyond straightforward economic losses:
Many powerful AI models could be abused:
Content generation models could enhance disinformation and spear phishing campaigns.[5]
Image recognition models could empower semi-autonomous weapons or authoritarian surveillance.[6]
Simulation models could facilitate the design of novel pathogens.[7]
The mere threat of theft/leaks may discourage efforts to improve AI safety and interpretability that involve providing more access to powerful models.[8]
Enhanced Chinese AI research could intensify AI racing dynamics that prove catastrophic if “very powerful systems”[9] are attainable over the next 15 years.[10]
Why Traditional Market Incentives May Fail to Mitigate These Risks
Many companies will have some incentives to protect their models, but there are some reasons to expect their efforts will be suboptimal relative to the risks:
The risks described in the previous section are largely externalities and companies that do not appropriately guard against these risks may out-compete companies that do.
Unauthorized use of models may be limited to foreign jurisdictions where the companies did not expect to make substantial profits (e.g., an off-limits Chinese economy).
Market dynamics may disincentivize some prosocial actions such as cybersecurity incident disclosures.[11]
Suggestions for CISA
CISA should explore some options to inform and facilitate public policy and private responses to these threats:
Map relevant actors and stakeholders.
Evaluate and/or propose platforms and frameworks for information sharing.
Assess the presence and impact of market failures.
Collect research relevant to actions that other actors could take (e.g., programs at DARPA/IARPA,[12] mandatory incident disclosure legislation).
Begin drafting a report which incorporates the previous suggestions and elicits input from relevant actors.
Longpre, Shayne, Marcus Storm, and Rishi Shah. 2022. “Lethal Autonomous Weapons Systems & Artificial Intelligence: Trends, Challenges, and Policies.” Edited by Kevin McDermott. MIT Science Policy Review 3 (August): 47–56. https://doi.org/10.38105/spr.360apm5typ.
Roser, Max. 2023. “AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?” Our World in Data. February 7, 2023. https://ourworldindata.org/ai-timelines.
Additionally, one report suggested that by 2030, state-of-the-art models may cost hundreds of millions or even >$1B dollars to train (although the report highlights that these estimates could significantly change). Source: Cottier, Ben. 2023. “Trends in the Dollar Training Cost of Machine Learning Systems.” Epoch. January 31, 2023. https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.
[8] The example of Meta’s LLaMA, mentioned earlier, provides both some support and rebuttal for this concern: Meta has insisted it plans to continue sharing access despite the leaks, but there are good reasons to think this event will discourage other companies from implementing similar access rules. Source: Vincent, “Meta’s Powerful AI Language Model Has Leaked Online.”
[9] By this, I am referring to systems such as highly autonomous cyber systems (which could conceivably cause unintended havoc on a scale far greater than Stuxnet), AI systems in nuclear forces or strategic operations (e.g., early warning systems, command and control, and tracking foreign nuclear assets such as missile submarines), or outright “human-level” artificial general intelligence (AGI).
[10] Surveys of AI experts provide a mixed range of forecasts, but in a 2022 survey a non-trivial portion of such experts forecasted a 50% chance that “human-level AI” (roughly defined as a system that is better than humans at practically all meaningful tasks) will exist by 2035. Additionally, half of the surveyed experts forecasted a 50% chance of this outcome by 2061. Notably however, some types of “very powerful systems” (e.g., highly autonomous cyber systems) may not even require “human-level AI.” For data and further discussion regarding these forecasts, see Roser, Max. 2023. “AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?” Our World in Data. February 7, 2023. https://ourworldindata.org/ai-timelines.
This shortform has been obsolesced by the following new version of my memo: https://forum.effectivealtruism.org/posts/jPxnAawQ9edXLRLRF/harrison-d-s-shortform?commentId=zPBhKQL2q3cafWdc5.
The following is a midterm assignment I submitted for my Cyber Operations class at Georgetown, regarding the risk of large AI model theft. I figured I would just publish this since it’s fairly relevant to recent discussions and events around AI model theft. (I also am posting this so I have a non-Google-Doc link to share with people)Note: In this assignment I had a 500-word limit and was only tasked to describe a problem’s relevance to my client/audience whilebriefly mentioningpolicy options. In an upcoming memo assignment I will need to actually go into more detail on the policy recommendations (and I’d be happy to receive suggestions for what CISA should do if you have any).(I also acknowledge that the recommendations I lay out here are a bit milquetoast, but I genuinely just didn’t know what else to say...)-------------Memorandum for the Cybersecurity and Infrastructure Security Agency (CISA)SUBJECT: Securing Large AI Models Against TheftLarge artificial intelligence (AI) models such as ChatGPT have increasingly demonstrated AI’s potential. However, as proprietary models become more powerful it is increasingly important to protect them against theft. CISA should work to facilitate information sharing that supports public policy and private responses. The following four sections will discuss some of the threat motivations/trends, potential consequences, market failures, and policy recommendations for CISA.Motivations and Relevant Trends Regarding AI Model TheftThere are many reasons to expect that hackers will attempt to exfiltrate proprietary AI models:China and other actors have repeatedly stolen sensitive data and intellectual property (IP).[1]Future models may prove to have such significant economic or military value that state actors are willing to expend substantial effort/assets to steal them.Current large models have high up-front development (“training”) costs/requirements but comparatively low operational costs/requirements after training.[2]This makes theft of models attractive even for non-state actors. Additionally, recent export controls on semiconductors to China could undermine China’s ability to train future large models,[3]which would further increase Beijing’s incentive to support model theft.Someone reportedly leaked Meta’s new large language model (LLaMA) within days of Meta providing model access to researchers.[4]Potential Consequences of AI Model TheftTheft of powerful AI models—or the threat thereof—could have significant negative consequences beyond straightforward economic losses:Many powerful AI models could be abused:Content generation models could enhance disinformation and spear phishing campaigns.[5]Image recognition models could empower semi-autonomous weapons or authoritarian surveillance.[6]Simulation models could facilitate the design of novel pathogens.[7]The merethreatof theft/leaks may discourage efforts to improve AI safety and interpretability that involve providing more access to powerful models.[8]Enhanced Chinese AI research could intensify AI racing dynamics that prove catastrophic if “very powerful systems”[9]are attainable over the next 15 years.[10]Why Traditional Market Incentives May Fail to Mitigate These RisksMany companies will havesomeincentives to protect their models, but there are some reasons to expect their efforts will be suboptimal relative to the risks:The risks described in the previous section are largely externalities and companies that do not appropriately guard against these risks may out-compete companies that do.Unauthorized use of models may be limited to foreign jurisdictions where the companies did not expect to make substantial profits (e.g., an off-limits Chinese economy).Market dynamics may disincentivize some prosocial actions such as cybersecurity incident disclosures.[11]Suggestions for CISACISA should explore some options to inform and facilitate public policy and private responses to these threats:Map relevant actors and stakeholders.Evaluate and/or propose platforms and frameworks for information sharing.Assess the presence and impact of market failures.Collect research relevant to actions that other actors could take (e.g., programs at DARPA/IARPA,[12]mandatory incident disclosure legislation).Begin drafting a report which incorporates the previous suggestions and elicits input from relevant actors.-------------ReferencesAllen, Gregory, Emily Benson, and William Reinsch. 2022. “Improved Export Controls Enforcement Technology Needed for U.S. National Security.” Center for Strategic and International Studies. November 30, 2022. https://www.csis.org/analysis/improved-export-controls-enforcement-technology-needed-us-national-security.Brooks, Chuck. 2023. “Cybersecurity Trends & Statistics for 2023: More Treachery and Risk Ahead as Attack Surface and Hacker Capabilities Grow.” Forbes. March 5, 2023. https://www.forbes.com/sites/chuckbrooks/2023/03/05/cybersecurity-trends—statistics-for-2023-more-treachery-and-risk-ahead-as-attack-surface-and-hacker-capabilities-grow/?sh=2c6fcebf19db.Calma, Justine. 2022. “AI Suggested 40,000 New Possible Chemical Weapons in Just Six Hours.” The Verge. March 17, 2022. https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx.Cottier, Ben. 2022. “The Replication and Emulation of GPT-3.” Rethink Priorities. December 21, 2022. https://rethinkpriorities.org/publications/the-replication-and-emulation-of-gpt-3.———. 2023. “Trends in the Dollar Training Cost of Machine Learning Systems.” Epoch. January 31, 2023. https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.Cox, Joseph. 2023. “How I Broke into a Bank Account with an AI-Generated Voice.” Vice. February 23, 2023. https://www.vice.com/en/article/dy7axa/how-i-broke-into-a-bank-account-with-an-ai-generated-voice.Dickson, Ben. 2020. “The GPT-3 Economy.” TechTalks. September 21, 2020. https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/.Feldstein, Steven. 2019. “The Global Expansion of AI Surveillance.” Carnegie Endowment for International Peace. September 17, 2019. https://carnegieendowment.org/2019/09/17/global-expansion-of-ai-surveillance-pub-79847.“Guaranteeing AI Robustness against Deception (GARD).” n.d. DARPA. Accessed March 11, 2023. https://www.darpa.mil/program/guaranteeing-ai-robustness-against-deception.Humphreys, Brian. 2021. “Critical Infrastructure Policy: Information Sharing and Disclosure Requirements after the Colonial Pipeline Attack.” Congressional Research Service. May 24, 2021. https://crsreports.congress.gov/product/pdf/IN/IN11683.Longpre, Shayne, Marcus Storm, and Rishi Shah. 2022. “Lethal Autonomous Weapons Systems & Artificial Intelligence: Trends, Challenges, and Policies.” Edited by Kevin McDermott.MIT Science Policy Review3 (August): 47–56. https://doi.org/10.38105/spr.360apm5typ.Nakashima, Ellen. 2015. “Chinese Breach Data of 4 Million Federal Workers.”The Washington Post, June 4, 2015. https://www.washingtonpost.com/world/national-security/chinese-hackers-breach-federal-governments-personnel-office/2015/06/04/889c0e52-0af7-11e5-95fd-d580f1c5d44e_story.html.“Not My Problem.” 2014. The Economist. July 10, 2014. https://www.economist.com/special-report/2014/07/10/not-my-problem.Rasser, Martijn, and Kevin Wolf. 2022. “The Right Time for Chip Export Controls.” Lawfare. December 13, 2022. https://www.lawfareblog.com/right-time-chip-export-controls.Roser, Max. 2023. “AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?” Our World in Data. February 7, 2023. https://ourworldindata.org/ai-timelines.Sganga, Nicole. 2022. “Chinese Hackers Took Trillions in Intellectual Property from about 30 Multinational Companies.” CBS News. May 4, 2022. https://www.cbsnews.com/news/chinese-hackers-took-trillions-in-intellectual-property-from-about-30-multinational-companies/.“TrojAI: Trojans in Artificial Intelligence.” n.d. IARPA. Accessed March 11, 2023. https://www.iarpa.gov/research-programs/trojai.Vincent, James. 2023. “Meta’s Powerful AI Language Model Has Leaked Online — What Happens Now?” The Verge. March 8, 2023. https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse.[1]Nakashima, Ellen. 2015. “Chinese Breach Data of 4 Million Federal Workers.” The Washington Post, June 4, 2015.https://www.washingtonpost.com/world/national-security/chinese-hackers-breach-federal-governments-personnel-office/2015/06/04/889c0e52-0af7-11e5-95fd-d580f1c5d44e_story.html; andSganga, Nicole. 2022. “Chinese Hackers Took Trillions in Intellectual Property from about 30 Multinational Companies.” CBS News. May 4, 2022.https://www.cbsnews.com/news/chinese-hackers-took-trillions-in-intellectual-property-from-about-30-multinational-companies/.[2]For example, a single successful training run of GPT-3 reportedly required dozens of terabytes of data and cost millions of dollars of GPU usage, but the trained model is a file smaller than a terabyte in size and actors can operate it on cloud services that cost under $40 per hour. Sources:Cottier, Ben. 2022. “The Replication and Emulation of GPT-3.” Rethink Priorities. December 21, 2022.https://rethinkpriorities.org/publications/the-replication-and-emulation-of-gpt-3; andDickson, Ben. 2020. “The GPT-3 Economy.” TechTalks. September 21, 2020.https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/.Additionally, one report suggested that by 2030, state-of-the-art models may cost hundreds of millions or even >$1B dollars to train (although the report highlights that these estimates could significantly change). Source: Cottier, Ben. 2023. “Trends in the Dollar Training Cost of Machine Learning Systems.” Epoch. January 31, 2023.https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.[3]For discussion regarding this claim, see: Allen, Gregory, Emily Benson, and William Reinsch. 2022. “Improved Export Controls Enforcement Technology Needed for U.S. National Security.” Center for Strategic and International Studies. November 30, 2022.https://www.csis.org/analysis/improved-export-controls-enforcement-technology-needed-us-national-security; andRasser, Martijn, and Kevin Wolf. 2022. “The Right Time for Chip Export Controls.” Lawfare. December 13, 2022.https://www.lawfareblog.com/right-time-chip-export-controls.[4]Vincent, James. 2023. “Meta’s Powerful AI Language Model Has Leaked Online — What Happens Now?” The Verge. March 8, 2023.https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-online-misuse.[5]Brooks, Chuck. 2023. “Cybersecurity Trends & Statistics for 2023: More Treachery and Risk Ahead as Attack Surface and Hacker Capabilities Grow.” Forbes. March 5, 2023.https://www.forbes.com/sites/chuckbrooks/2023/03/05/cybersecurity-trends—statistics-for-2023-more-treachery-and-risk-ahead-as-attack-surface-and-hacker-capabilities-grow/?sh=2c6fcebf19db;Cox, Joseph. 2023. “How I Broke into a Bank Account with an AI-Generated Voice.” Vice. February 23, 2023.https://www.vice.com/en/article/dy7axa/how-i-broke-into-a-bank-account-with-an-ai-generated-voice.[6]Feldstein, Steven. 2019. “The Global Expansion of AI Surveillance.” Carnegie Endowment for International Peace. September 17, 2019.https://carnegieendowment.org/2019/09/17/global-expansion-of-ai-surveillance-pub-79847;Longpre, Shayne, Marcus Storm, and Rishi Shah. 2022. “Lethal Autonomous Weapons Systems & Artificial Intelligence: Trends, Challenges, and Policies.” Edited by Kevin McDermott. MIT Science Policy Review 3 (August): 47–56.https://doi.org/10.38105/spr.360apm5typ(p. 49).[7]Calma, Justine. 2022. “AI Suggested 40,000 New Possible Chemical Weapons in Just Six Hours.” The Verge. March 17, 2022.https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx.[8]The example of Meta’s LLaMA, mentioned earlier, provides both some support and rebuttal for this concern: Meta has insisted it plans to continue sharing access despite the leaks, but there are good reasons to think this event will discourage other companies from implementing similar access rules. Source: Vincent, “Meta’s Powerful AI Language Model Has Leaked Online.”[9]By this, I am referring to systems such as highly autonomous cyber systems (which could conceivably cause unintended havoc on a scale far greater than Stuxnet), AI systems in nuclear forces or strategic operations (e.g., early warning systems, command and control, and tracking foreign nuclear assets such as missile submarines), or outright “human-level” artificial general intelligence (AGI).[10]Surveys of AI experts provide a mixed range of forecasts, but in a 2022 survey a non-trivial portion of such experts forecasted a 50% chance that “human-level AI” (roughly defined as a system that is better than humans at practically all meaningful tasks) will exist by 2035. Additionally, half of the surveyed experts forecasted a 50% chance of this outcome by 2061. Notably however, some types of “very powerful systems” (e.g., highly autonomous cyber systems) may not even require “human-level AI.” For data and further discussion regarding these forecasts, see Roser, Max. 2023. “AI Timelines: What Do Experts in Artificial Intelligence Expect for the Future?” Our World in Data. February 7, 2023.https://ourworldindata.org/ai-timelines.[11]For sources on this claim, see: “Not My Problem.” 2014. The Economist. July 10, 2014.https://www.economist.com/special-report/2014/07/10/not-my-problem; andHumphreys, Brian. 2021. “Critical Infrastructure Policy: Information Sharing and Disclosure Requirements after the Colonial Pipeline Attack.” Congressional Research Service. May 24, 2021.https://crsreports.congress.gov/product/pdf/IN/IN11683.[12]DARPA and IARPA are already working on some projects related to the security and reliability of AI models, including GARD at DARPA and TrojAI at IARPA. Sources: “Guaranteeing AI Robustness against Deception (GARD).” n.d. DARPA. Accessed March 11, 2023.https://www.darpa.mil/program/guaranteeing-ai-robustness-against-deception; and“TrojAI: Trojans in Artificial Intelligence.” n.d. IARPA. Accessed March 11, 2023.https://www.iarpa.gov/research-programs/trojai.