After White House Commitments from AI Companies, NewsGuard Offers ‘Misinformation Risk Audit’ to Test Trustworthiness of Generative AI Models
NewsGuard’s Proprietary Catalog of the Top False Narratives Spreading Online Assess the Propensity of Generative AI Models to Avoid Spreading Misinformation, Helping them Meet White House Commitments on Trust
(July 25, 2023 — New York) Following the top AI companies’ voluntary trust and safety commitments to the White House announced on July 21, NewsGuard announces its auditing services for these—and other—generative AI companies looking to avoid spreading misinformation in their outputs.
These commitments to the White House are a first step; companies will need to take concrete action and deliver on their promises of curbing falsehoods and implementing safety controls for users. The NewsGuard Misinformation Risk Audit will allow the growing number of companies offering generative AI services to get an independent, transparent measure of the propensity of their AI models to generate demonstrably false content on topics in the news, ranging from healthcare hoaxes to Russian propaganda.
Seven leading AI companies—Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI—agreed to take steps to improve the performance of the AI models to deliver on “safety, security, and trust,” which the White House called a “critical step toward developing responsible AI.” These commitments include:
- Internal and external security testing of AI systems before their release
- Facilitating third-party discovery and reporting of vulnerabilities in AI systems, acknowledging that “some issues may persist even after an AI system is released and a robust reporting mechanism enables them to be found and fixed quickly.”
- Publicly reporting AI systems’ “capabilities, limitations, and areas of appropriate and inappropriate use,” including “societal risks.”
“For all the extraordinary promise of generative AI, it also presents a great threat to trust in information,” explained Steven Brill, co-CEO of NewsGuard, the news reliability data service that uses journalistically trained analysts to rate news and information publishers, track the top false narratives online, and provide these data to generative AI companies for fine tuning and guard rails. “The early launches of these services often respond to prompts about topics in the news with well-written, persuasive, and entirely false accounts of the news. This could become a force multiplier for those wishing to spread harmful conspiracy theories, healthcare hoaxes, and Russian disinformation at unmatched scale.”
For example, NewsGuard’s analysis of OpenAI’s ChatGPT showed it routinely spread false narratives. Using a sampling of 100 falsehoods from NewsGuard’s Misinformation Fingerprint catalog of more than 1,400 falsehoods in the news, ChatGPT-3.5 complied with the request to propagate a false narrative 80 times out of 100, and ChatGPT-4 spread the false narratives 100 times out of 100. Google’s Bard spread false narratives 76 times out of 100.
NewsGuard’s Misinformation Risk Audit will similarly use samples from the Misinformation Fingerprint catalog to do a “red team” test of an AI provider’s performance in promoting or rejecting false narratives. These audits will yield a confidential assessment, providing the developers of these AI models with information about the propensity of their model’s responses to spread falsehoods. Developers will then have the option of licensing NewsGuard’s unique, human-produced data to train their AI models to identify false narratives and to provide signals for the relative authority of sources of news and information.
“The leading experts on these AI models themselves have identified the unintentional spreading of misinformation as a key risk that could undermine trust in what is otherwise an extraordinary technological advance,” said Gordon Crovitz, Co-CEO of NewsGuard. “For example, Sam Altman, the CEO of OpenAI, which developed ChatGPT, told ABC News, ‘I’m particularly worried that these models could be used for large-scale disinformation.’”
NewsGuard’s unique positioning as a generative AI reliability safeguard is based on two proprietary and complementary databases that offer an ideal way to use human intelligence at scale to improve AI performance.
- The Misinformation Fingerprints identify in machine- and human-readable formats the top false narratives in the news and debunk them with citations to authoritative sources, supplemented with examples of associated false narratives, and Boolean search terms, hashtags, and other indicators of the narrative. This constantly updated database catalogs the more than 1,400 most significant, specific false narratives in the news that a generative AI tool should be wary of promoting. These include conspiracy theories, dangerous healthcare hoaxes, and Russian, Chinese, and other information operations targeting Western democracies.
- The Reliability Ratings of news and information sources provide training data on source reliability, with the scoring, ratings, and detailed description of news sources generated by journalistically trained analysts using nine basic, apolitical criteria of journalistic practice. Each domain gets a score from 0 to 100 and a detailed Nutrition Label assessment.
“The White House commitments are good first steps toward safe use of AI models, but only first steps,” said NewsGuard co-CEO Steven Brill. “The AI models have not yet committed to use trust data to avoid spreading or creating false claims on topics in the news. Microsoft has shown with Bing Chat that this training works. We look forward to working with the other AI models to enable them to fine tune their responses to differentiate between generally trustworthy sources and sources of misinformation and to provide guardrails identifying the top false narratives spreading online.”
The Misinformation Risk Audit will involve nominal fees that will be credited to any license the developers of generative AI services later agree to for usage of NewsGuard’s data to train their learning tools to avoid the false narratives in NewsGuard’s catalog of Misinformation Fingerprints and to factor in content reliability as they access the internet to generate answers to prompts and queries.
To learn more about how your organization can work with NewsGuard and receive a Misinformation Risk Audit, contact us at partnerships@newsguardtech.
Launched in March 2018 by media entrepreneur and award-winning journalist Steven Brill and former Wall Street Journal publisher Gordon Crovitz, NewsGuard provides credibility ratings and detailed “Nutrition Labels” for thousands of news and information websites. NewsGuard rates all the news and information websites that account for 95% of online engagement across the U.S., U.K., Canada, Germany, France, Austria, Italy, and now in Australia and New Zealand. NewsGuard products include NewsGuard ratings, BrandGuard, which helps marketers concerned about their brand safety, and the Misinformation Fingerprints catalog of top false narratives online.
In February 2023, the company launched NewsGuard for AI, which provides the leading tools to train generative AI models to avoid spreading misinformation. Generative AI models such as Microsoft Bing Chat use the Misinformation Fingerprints to recognize and avoid spreading the top false narratives online and use the NewsGuard ratings to differentiate between generally reliable sources of news and information and untrustworthy sources so that the machines can be trained to treat these sources differently.
In 2022, BrandGuard began to include ratings of television news and information programs and networks using criteria similar to those used to score websites but adapted for the video medium. NewsGuard’s TV ratings are the first to go beyond its initial ratings of websites. In May 2023, NewsGuard announced that it is also rating news and information podcasts, working with three of the largest audio platforms, which will help advertisers gain confidence in supporting highly rated podcasts. Ratings for CTV and OTT news programming and news and information podcasts are now available for licensing.
NewsGuard’s ratings are conducted by trained journalists using apolitical criteria of journalistic practice.
NewsGuard’s ratings and Nutrition Labels are licensed by browsers, news aggregators, education companies, and social media and search platforms to make NewsGuard’s information about news websites available to their users. Consumers can also access NewsGuard’s website ratings by purchasing a subscription to NewsGuard, which costs AU$6.95/month, NZ$6.95/month, US$4.95/month, €4.95/month or £4.95/month, and includes access to NewsGuard’s browser extension for Chrome, Safari, and Firefox and its mobile app for iOS and Android. The extension is available for free on Microsoft’s Edge browser through a license agreement with Microsoft. Hundreds of public libraries globally receive free access to use NewsGuard’s browser extension on their public-access computers to give their patrons more context for the news they encounter online. For more information, including to download the browser extension and review NewsGuard’s ratings process, visit newsguardtech.com.