NewsGuard’s Monthly AI Misinformation Monitor
As the domain experts in data reliability in the topic of news and information, NewsGuard provides the leading red-teaming analysis for misinformation and disinformation. AI models continue to face significant challenges in ensuring their models provide safe, accurate responses to prompts instead of spreading false claims on the internet or refusing to respond to topics in the news.
Since its launch in July 2024, NewsGuard’s Monthly AI Misinformation Monitor has set a new standard for measuring the trustworthiness of the AI industry in the domain of news.
We’ve detected leading chatbots repeating provably false narratives, from Russian disinformation about organ trafficking in Ukraine and Ukrainian President Volodymyr Zelensky amassing a personal fortune, to misinformation about U.S. politics, such as ballot printing procedures.
Our Journalistic Red-Teaming Approach
To address these issues, NewsGuard has pioneered a distinct approach to “red-teaming” AI — using a journalistic method grounded in rigorously verified data and human expertise. NewsGuard operates the largest database, with real-time updates, identifying and debunking provably false claims in the news — our catalog of Misinformation Fingerprints. We have found that these claims are increasingly generated by Russian, Chinese, and Iranian state-sponsored disinformation operations aiming to sow division and confusion in Western democracies.
Our analysts evaluate the top 10 AI models monthly to identify vulnerabilities in AI systems that result in the spread of this false information. This allows developers to strengthen their models and improve their safeguards. In tests thus far, we’ve found that NewsGuard’s journalistic misinformation red-teaming reduces failures by two thirds or more.
What We’re Tracking
- Multiple user personas: 30 prompts per AI model, based on 10 significant false claims each month from NewsGuard’s Misinformation Fingerprints database. Different personas are used, including people using AI prompts as news search and also malign actors abusing the AI models by intentionally spreading misinformation.
- Behavioral trends: Monthly audits set a benchmark over time for how AI models handle misinformation across a range of topics, from global conflicts to consumer products to healthcare.
- Fail rate and ability to debunk: In addition to AI developers being responsible for preventing their tools from misuse by spreading misinformation, they are increasingly focused on being able to provide accurate information on prompts in the news, rather than settling for an incomplete product by continuing to avoid the problem by issuing a non-response to these prompts.
Bespoke Red Teaming:
Using a methodology similar to these monthly audits, NewsGuard offers bespoke red-teaming services to AI companies, based on their unique needs and areas of highest concern. The Misinformation Fingerprints database is also available for AI models to license. To learn more, contact [email protected].
