What have recent changes in the information ecosystem been?
Events of the last few years, including the war in Ukraine and the pandemic, have accelerated structural shifts towards more digital, mobile, and platform-dominated media environments. The widespread use of synthetic content has been observed in recent conflicts, ranging from Ukraine to Sudan to Gaza. Social media platforms have been changing rapidly, in the last year Facebook published updates related to Meta, Reels, AI-related content. Twitter under Elon Musk has loosened moderation guidelines. TikTok has become one of the fastest-growing platforms ever, reflecting the growth in short form video content.
The explosive growth in the amount of content created has increased the prominence on AI led content moderation in order to scale effectively. There is also growing attention from governments on content moderation, with the EU’s Digital Services Act which, Australia’s Online Safety Act and the UK Online Safety Bill, amongst others.
The interconnected nature of platforms and user networks means that disinformation is networked across many platforms, including unmoderated or encrypted spaces, making it hard to identify, track and flag or remove false content at scale across the information landscape.
What does AI mean for creating mis/disinformation?
As the availability of LLMs increases and cost falls, this makes it easier for threat actors to create more personalised and more effective content. As content creation is becoming more automated, this reduces the financial and time costs associated with micro targeting and hyper personalization, and an improved understanding of the information environment allows threat actors to craft more compelling and effective narratives for each target segment. This also makes disinformation campaigns more difficult to detect as new content is easier to generate, preventing a need for copypasta, and the quality of deepfakes is drastically improving.
What does AI mean for the spread of mis/disinformation?
The spread of campaigns often relies on large numbers of accounts across social media, and the perceived authenticity of the accounts is key. ML techniques allow generation of increasingly realistic profile photos, reducing the need for image scraping and the potential for reverse image searches to aid in detection of a campaign. When combined with the improvements in text generation through LLMs for bio’s and online presence, this results in en masse creation of credible accounts to spread disinformation.
ML systems can also improve social engineering techniques to target influencers or so called “super-spreaders” who can organically amplify a message or campaign. Deepfakes also make it easier to impersonate experts or credible sources to amplify a message.
What does AI mean to real-time information ecosystem interactions?
Advancements in conversational AI or chatbots could automate engament with targeted individuals. These use large volumes of data, ML, and NLP to imitate human interactions, recognizing speech and text input and generating a response. This can be used to take part in online discussions and respond to comments to stimulate controversy and disputes, and increase polarisation.
As AI reduces the costs, increases the effectiveness, and reduces the ease of detection of disinformation campaigns, threat monitoring and early detection are becoming increasingly important.
How does this impact global catastrophic risk?
Information ecosystem risks are both broad and dangerous, with potential impacts ranging from water security to financial stability and even amplifying/creating conflicts. Disinformation is used to influence public opinion, legitimise unpopular actions and regimes including garnering support for confrontation, alter election results, increase polarisation and extremism, and undermine the credibility of institutions, science, experts and media.
Examples include the use of disinformation as a form of information warfare in the Russia Ukraine conflict, the spread of conspiracy theories around the 2016 US elections, and the fuelling of anti muslim and anti Pakistan sentiment in India. Much like climate change, this acts as a threat multiplier for catastrophic and existential risks and can significantly increase risk by polarising and destabilising the world, undermining global governance, increasing geopolitical tensions, and the risk of conflict, arms races, and more.
Disinformation from generative AI has been seen in recent conflicts in Sudan, Gaza and the Ukraine with deepfakes from leaders of both sides, designed to stoke tensions and escalate conflict. The role of AI-empowered disinformation has the potential to both create and worsen conflict.
Disinformation is used to undermine scientific credibility and promote climate denialism and inaction. A recent report from CCDH showed that ⅓ of British children thought concerns of climate change were overblown which shows marked success with climate denialism narratives.
Disinformation around nuclear weapons can be used by state or non state actors to simulate or provoke nuclear attack or response. It also increases geopolitical tensions and the likelihood of great power war. The recent election in Taiwan was supposedly subject to significant influence operations by mainland China with AI augmented disinformation.
Disinformation is used by threat actors and incentivised parties to downplay AI safety concerns and promote rapid AI development at the expense of safety, alignment and governance. This could significantly hamper the efforts of AI safety activities.
AI and disinformation: Offence vs Defence
Spreading disinformation has always been more effective than combatting it. In creating and spreading disinformation, the most viral techniques and narratives can be used without regard for the truth, often preying on insecurities for the most vulnerable and tapping into visceral emotions such as fear. The effectiveness is evident across the board, notably in the difference in click-through rates for clickbait headlines generated by ad companies Taboola and Outbrain vs generic news stories. Responding to disinformation also has a temporal disadvantage, where a counter-narrative as opposed to an initial narrative has to be established, and an anchoring effect has often occurred.
Techno-fixes are also limited in their effectiveness, and certain ones such as reverse image searches place a high burden of effort for the user. Fact-checking is time consuming and laborious and an ever-decreasing proportion of information can be fact-checked as AI-generated/spread disinformation is proliferated.
AI empowered tools such as Bot Sentinel, Botometer and BotSlayer classify accounts as bots based on profile features and behaviour. Tools and solutions such as Captain Fact, Claimbuster, Logically.ai and Alethea Artemis use AI to detect misinformation and disinformation at scale. However the effectiveness of detection algorithms depends on the availability of large sets of training data and quality of data labels. While detection is becoming more robust, for example within deepfake detection by looking beyond subtle signatures of particular generation tools and using underlying physical and biological signals that are hard for AI to imitate, there is a constant back and forth between AI-generated content and detection methods as both sides become more sophisticated and adapt to each other.
As techniques such as establishing provenance on the blockchain are established, with the use of digital watermarks, these will be likely circumvented by rogue state and non-state actors empowered by AI. The commercial incentives for creating and spreading disinformation have historically been greater than countering it, and it is difficult to see how this may change.
AI is likely to make disinformation much easier to create and spread more effectively, at lower human and financial cost. AI tools often place a high burden of effort for the end user so will have limited adoption and as they are developed, zero day exploits and other techniques will be developed to circumvent these tools. Although technical tools are a crucial part of the toolkit to combat mis and disinformation, they are necessary but not sufficient.
What is the current state of knowledge on the field?
Campaigners across government and civil society are often resource and time-poor and don’t know the best, evidence-informed way to respond to information threats in combatting mis/disinformation. There is a wealth of academic literature on quantitative experiments on the efficacy of interventions to combat mis/disinformation, however this is disparate, disaggregated, spans many disciplines and lacks a shared ontology. This evidence base is hard to find, navigate and interpret so tends to be ignored by the practitioner community.
Our Proposal: Strengthening the evidence base for societal resilience
This project aims to create, populate, test the effectiveness and iterate an online living database to be used to improve the effectiveness of counter disinformation campaigns and media literacy, especially around AI. This open source online living database will collate, curate and categorise empirical studies that have been run on interventions to combat mis/disinformation and extract insights from study. The insights will include characterising the relevant information threat, intervention tested, methodology details, participant characteristics and statistics from results. This will enable the comparison of different interventions for any characterised information threat. The next phase of the project will be to create an algorithm to rank interventions for any characterised information threat, based on characterised parameters such as effect size, sample size, effect duration and others.
The database aims to operationalise the existing evidence base by aggregating a highly disaggregated academic field, serving both the research and practitioner community, having a large scale impact by improving the effectiveness of campaigns and media literacy initiatives run by large numbers of practitioner groups around the world.
We have validated the utility of this proposition with leading academics and practitioners in the field including the University of Cambridge Social Decision-Making Lab, IRIS (LSHTM, University of Rome, University of Venice), University of Bristol, Max Planck Institute, North-Eastern University, University of Minnesota, IMT Lucca, University of Seattle, Princeton, UC Davis, the Centre for Countering Digital Hate, The European Centre of Excellence for Countering Hybrid Threats, ISD Global, CASM, Stimson Center, Public Democracy, and Climate Action Against Disinformation.
Learnings so far: What interventions may work? Evidence based counter campaigns & media literacy
While fact checking and other labelling technologies are progressing in identification of mis/disinformation online, technology development such as AI (e.g. LLM’s) makes the creation and spread of disinformation significantly quicker, easier and cheaper, and changes the tactics used, hence technofixes will constantly be catching up. Tackling mis and disinformation requires both counter specific disinformation campaigns run on particular issues but more importantly the building of individual and societal resilience through education interventions focused on media literacy, especially around emerging technologies and AI.
What do effective counter interventions look like?
A range of counter interventions are able to counter disinformation, some regulatory, some platform-oriented but also those that can be integrated into campaigns.
Debunking (providing correcting information targeted towards misconceptions of beliefs), inoculation (pre-emptive exposure to weakened forms of disinformation, and is often technique based or issue based) and adjacent messaging (providing alternative more hopeful narratives as opposed to directly refuting) are all tools that can be integrated into campaigns.
AI will create a more uncertain information environment, with fatigue likely setting into citizens where fact checking tools are arduous and time consuming to use. Effective counter-interventions are highly context-specific but likely to have the following characteristics in common.
Early detection. Information threats are much easier to respond to when nascent, when more effective techniques can be used. Deplatforming and prebunking are effective options to quash influence threats before disinformation narratives become widespread
Target knowledge acquisition. Obtaining demographic and psychographic insight into potential targets of influence operations, e.g. customers/investors help design more effective counter-campaigns
Evidence-based response. As AI-related disinformation proliferates, preempting particular narratives becomes more challenging. Therefore technique based inoculation is likely to be more effective, as techniques will target weaknesses from common cognitive fallacies, which may be personalised based on psychographic elements
How may we want to update media literacy training in light of this?
The future world is one where disinformation is more prevalent, more personalised and harder to discern. In light of this, effective media literacy to build societal resilience will need to include the following:
How to hold information in uncertainty. People have a preference for certainty over uncertainty (certainty effect), and aiding people to develop probabilistic mindsets where information may or may not be true is pivotal
How to interact with uncertain information. Communicating the uncertainty associated with information is critical to enable others to also hold information in uncertainty, and not for strength in beliefs to increase with sharing.
How to recognise influence operations. Educating the public on who may be targeting them, why they do so, the techniques they use, the goals they have and how this links to particular narratives can help identify when a piece of information is more likely to be disinformation.
What technical tools can be used for verifying information, reporting mis/disinformation and deplatforming
Who are we?
Say No to Disinfo is focused on improving the information ecosystem, and hence reducing existential risk by making society more robust and resilient to disruption due to mis/disinformation. We have a focus on the intersection of AI and emerging technologies with disinformation and our activities focus on improving the effectiveness of direct disinformation response, and of educational interventions to improve media literacy and societal resilience to mis/disinformation through the creation of the online living database, and the work on improving the effectiveness of media literacy for the current and future technology environment.
Progress to date and our asks
With collaboration from leading academics and practitioners, we have completed database design, compiled an initial list of hundreds of academic papers containing thousands of experiments on counter mis/disinformation interventions, and are in the process of uploading these into the database.
We are seeking to augment the academic data with field data from campaigns that have been run. If you/your organisation have any data that we could incorporate or that you could make us aware of, please let us know.
We are looking for volunteers to upload papers to the database, reviewing them and extracting key information. It provides a hands-on opportunity to learn from cutting edge studies whilst contributing to a living resource that will have tangible positive real world impact, used by both civil society and government. If you are interested please get in touch with us at ari@saynotodisinfo.com
Disinformation as a GCR Threat Multiplier and Evidence Based Response
What have recent changes in the information ecosystem been?
Events of the last few years, including the war in Ukraine and the pandemic, have accelerated structural shifts towards more digital, mobile, and platform-dominated media environments. The widespread use of synthetic content has been observed in recent conflicts, ranging from Ukraine to Sudan to Gaza. Social media platforms have been changing rapidly, in the last year Facebook published updates related to Meta, Reels, AI-related content. Twitter under Elon Musk has loosened moderation guidelines. TikTok has become one of the fastest-growing platforms ever, reflecting the growth in short form video content.
The explosive growth in the amount of content created has increased the prominence on AI led content moderation in order to scale effectively. There is also growing attention from governments on content moderation, with the EU’s Digital Services Act which, Australia’s Online Safety Act and the UK Online Safety Bill, amongst others.
The interconnected nature of platforms and user networks means that disinformation is networked across many platforms, including unmoderated or encrypted spaces, making it hard to identify, track and flag or remove false content at scale across the information landscape.
What does AI mean for creating mis/disinformation?
As the availability of LLMs increases and cost falls, this makes it easier for threat actors to create more personalised and more effective content. As content creation is becoming more automated, this reduces the financial and time costs associated with micro targeting and hyper personalization, and an improved understanding of the information environment allows threat actors to craft more compelling and effective narratives for each target segment. This also makes disinformation campaigns more difficult to detect as new content is easier to generate, preventing a need for copypasta, and the quality of deepfakes is drastically improving.
What does AI mean for the spread of mis/disinformation?
The spread of campaigns often relies on large numbers of accounts across social media, and the perceived authenticity of the accounts is key. ML techniques allow generation of increasingly realistic profile photos, reducing the need for image scraping and the potential for reverse image searches to aid in detection of a campaign. When combined with the improvements in text generation through LLMs for bio’s and online presence, this results in en masse creation of credible accounts to spread disinformation.
ML systems can also improve social engineering techniques to target influencers or so called “super-spreaders” who can organically amplify a message or campaign. Deepfakes also make it easier to impersonate experts or credible sources to amplify a message.
What does AI mean to real-time information ecosystem interactions?
Advancements in conversational AI or chatbots could automate engament with targeted individuals. These use large volumes of data, ML, and NLP to imitate human interactions, recognizing speech and text input and generating a response. This can be used to take part in online discussions and respond to comments to stimulate controversy and disputes, and increase polarisation.
As AI reduces the costs, increases the effectiveness, and reduces the ease of detection of disinformation campaigns, threat monitoring and early detection are becoming increasingly important.
How does this impact global catastrophic risk?
Information ecosystem risks are both broad and dangerous, with potential impacts ranging from water security to financial stability and even amplifying/creating conflicts. Disinformation is used to influence public opinion, legitimise unpopular actions and regimes including garnering support for confrontation, alter election results, increase polarisation and extremism, and undermine the credibility of institutions, science, experts and media.
Examples include the use of disinformation as a form of information warfare in the Russia Ukraine conflict, the spread of conspiracy theories around the 2016 US elections, and the fuelling of anti muslim and anti Pakistan sentiment in India. Much like climate change, this acts as a threat multiplier for catastrophic and existential risks and can significantly increase risk by polarising and destabilising the world, undermining global governance, increasing geopolitical tensions, and the risk of conflict, arms races, and more.
Disinformation from generative AI has been seen in recent conflicts in Sudan, Gaza and the Ukraine with deepfakes from leaders of both sides, designed to stoke tensions and escalate conflict. The role of AI-empowered disinformation has the potential to both create and worsen conflict.
Disinformation is used to undermine scientific credibility and promote climate denialism and inaction. A recent report from CCDH showed that ⅓ of British children thought concerns of climate change were overblown which shows marked success with climate denialism narratives.
Disinformation around nuclear weapons can be used by state or non state actors to simulate or provoke nuclear attack or response. It also increases geopolitical tensions and the likelihood of great power war. The recent election in Taiwan was supposedly subject to significant influence operations by mainland China with AI augmented disinformation.
Disinformation is used by threat actors and incentivised parties to downplay AI safety concerns and promote rapid AI development at the expense of safety, alignment and governance. This could significantly hamper the efforts of AI safety activities.
AI and disinformation: Offence vs Defence
Spreading disinformation has always been more effective than combatting it. In creating and spreading disinformation, the most viral techniques and narratives can be used without regard for the truth, often preying on insecurities for the most vulnerable and tapping into visceral emotions such as fear. The effectiveness is evident across the board, notably in the difference in click-through rates for clickbait headlines generated by ad companies Taboola and Outbrain vs generic news stories. Responding to disinformation also has a temporal disadvantage, where a counter-narrative as opposed to an initial narrative has to be established, and an anchoring effect has often occurred.
Techno-fixes are also limited in their effectiveness, and certain ones such as reverse image searches place a high burden of effort for the user. Fact-checking is time consuming and laborious and an ever-decreasing proportion of information can be fact-checked as AI-generated/spread disinformation is proliferated.
AI empowered tools such as Bot Sentinel, Botometer and BotSlayer classify accounts as bots based on profile features and behaviour. Tools and solutions such as Captain Fact, Claimbuster, Logically.ai and Alethea Artemis use AI to detect misinformation and disinformation at scale. However the effectiveness of detection algorithms depends on the availability of large sets of training data and quality of data labels. While detection is becoming more robust, for example within deepfake detection by looking beyond subtle signatures of particular generation tools and using underlying physical and biological signals that are hard for AI to imitate, there is a constant back and forth between AI-generated content and detection methods as both sides become more sophisticated and adapt to each other.
As techniques such as establishing provenance on the blockchain are established, with the use of digital watermarks, these will be likely circumvented by rogue state and non-state actors empowered by AI. The commercial incentives for creating and spreading disinformation have historically been greater than countering it, and it is difficult to see how this may change.
AI is likely to make disinformation much easier to create and spread more effectively, at lower human and financial cost. AI tools often place a high burden of effort for the end user so will have limited adoption and as they are developed, zero day exploits and other techniques will be developed to circumvent these tools. Although technical tools are a crucial part of the toolkit to combat mis and disinformation, they are necessary but not sufficient.
What is the current state of knowledge on the field?
Campaigners across government and civil society are often resource and time-poor and don’t know the best, evidence-informed way to respond to information threats in combatting mis/disinformation. There is a wealth of academic literature on quantitative experiments on the efficacy of interventions to combat mis/disinformation, however this is disparate, disaggregated, spans many disciplines and lacks a shared ontology. This evidence base is hard to find, navigate and interpret so tends to be ignored by the practitioner community.
Our Proposal: Strengthening the evidence base for societal resilience
This project aims to create, populate, test the effectiveness and iterate an online living database to be used to improve the effectiveness of counter disinformation campaigns and media literacy, especially around AI. This open source online living database will collate, curate and categorise empirical studies that have been run on interventions to combat mis/disinformation and extract insights from study. The insights will include characterising the relevant information threat, intervention tested, methodology details, participant characteristics and statistics from results. This will enable the comparison of different interventions for any characterised information threat. The next phase of the project will be to create an algorithm to rank interventions for any characterised information threat, based on characterised parameters such as effect size, sample size, effect duration and others.
The database aims to operationalise the existing evidence base by aggregating a highly disaggregated academic field, serving both the research and practitioner community, having a large scale impact by improving the effectiveness of campaigns and media literacy initiatives run by large numbers of practitioner groups around the world.
We have validated the utility of this proposition with leading academics and practitioners in the field including the University of Cambridge Social Decision-Making Lab, IRIS (LSHTM, University of Rome, University of Venice), University of Bristol, Max Planck Institute, North-Eastern University, University of Minnesota, IMT Lucca, University of Seattle, Princeton, UC Davis, the Centre for Countering Digital Hate, The European Centre of Excellence for Countering Hybrid Threats, ISD Global, CASM, Stimson Center, Public Democracy, and Climate Action Against Disinformation.
Learnings so far: What interventions may work? Evidence based counter campaigns & media literacy
While fact checking and other labelling technologies are progressing in identification of mis/disinformation online, technology development such as AI (e.g. LLM’s) makes the creation and spread of disinformation significantly quicker, easier and cheaper, and changes the tactics used, hence technofixes will constantly be catching up. Tackling mis and disinformation requires both counter specific disinformation campaigns run on particular issues but more importantly the building of individual and societal resilience through education interventions focused on media literacy, especially around emerging technologies and AI.
What do effective counter interventions look like?
A range of counter interventions are able to counter disinformation, some regulatory, some platform-oriented but also those that can be integrated into campaigns.
Debunking (providing correcting information targeted towards misconceptions of beliefs), inoculation (pre-emptive exposure to weakened forms of disinformation, and is often technique based or issue based) and adjacent messaging (providing alternative more hopeful narratives as opposed to directly refuting) are all tools that can be integrated into campaigns.
AI will create a more uncertain information environment, with fatigue likely setting into citizens where fact checking tools are arduous and time consuming to use. Effective counter-interventions are highly context-specific but likely to have the following characteristics in common.
Early detection. Information threats are much easier to respond to when nascent, when more effective techniques can be used. Deplatforming and prebunking are effective options to quash influence threats before disinformation narratives become widespread
Target knowledge acquisition. Obtaining demographic and psychographic insight into potential targets of influence operations, e.g. customers/investors help design more effective counter-campaigns
Evidence-based response. As AI-related disinformation proliferates, preempting particular narratives becomes more challenging. Therefore technique based inoculation is likely to be more effective, as techniques will target weaknesses from common cognitive fallacies, which may be personalised based on psychographic elements
How may we want to update media literacy training in light of this?
The future world is one where disinformation is more prevalent, more personalised and harder to discern. In light of this, effective media literacy to build societal resilience will need to include the following:
How to hold information in uncertainty. People have a preference for certainty over uncertainty (certainty effect), and aiding people to develop probabilistic mindsets where information may or may not be true is pivotal
How to interact with uncertain information. Communicating the uncertainty associated with information is critical to enable others to also hold information in uncertainty, and not for strength in beliefs to increase with sharing.
How to recognise influence operations. Educating the public on who may be targeting them, why they do so, the techniques they use, the goals they have and how this links to particular narratives can help identify when a piece of information is more likely to be disinformation.
What technical tools can be used for verifying information, reporting mis/disinformation and deplatforming
Who are we?
Say No to Disinfo is focused on improving the information ecosystem, and hence reducing existential risk by making society more robust and resilient to disruption due to mis/disinformation. We have a focus on the intersection of AI and emerging technologies with disinformation and our activities focus on improving the effectiveness of direct disinformation response, and of educational interventions to improve media literacy and societal resilience to mis/disinformation through the creation of the online living database, and the work on improving the effectiveness of media literacy for the current and future technology environment.
Progress to date and our asks
With collaboration from leading academics and practitioners, we have completed database design, compiled an initial list of hundreds of academic papers containing thousands of experiments on counter mis/disinformation interventions, and are in the process of uploading these into the database.
We are seeking to augment the academic data with field data from campaigns that have been run. If you/your organisation have any data that we could incorporate or that you could make us aware of, please let us know.
We are looking for volunteers to upload papers to the database, reviewing them and extracting key information. It provides a hands-on opportunity to learn from cutting edge studies whilst contributing to a living resource that will have tangible positive real world impact, used by both civil society and government. If you are interested please get in touch with us at ari@saynotodisinfo.com