07/21/2023

Voluntary Safety Commitments by AI Models Today Only a First Step

The AI models still need to take concrete actions to prevent spreading falsehoods: The good news is that when trained with trust data, these models can avoid spreading misinformation.

 

(July 21, 2023 — New York) NewsGuard, the news reliability data service that uses trained analysts to produce ratings for news and information publishers and to identify false narratives spreading online, responded to the voluntary steps the large AI models agreed to take as announced by the White House today.

“These are good first steps toward safe use of AI models, but only first steps,” said NewsGuard co-CEO Steven Brill. “The AI models have not yet committed to use trust data to avoid spreading or creating false claims on topics in the news. Microsoft has shown with Bing Chat that this training works. We look forward to working with the other AI models to enable them to fine tune their responses to differentiate between generally trustworthy sources and sources of misinformation and to provide guardrails identifying the top false narratives spreading online.”

“We’re encouraged that the AI models are open to taking more steps than the social media companies ever did to protect users from falsehoods. The social media companies profited from advertising as their users engaged with conspiracy theories, but the AI models need to become trustworthy to attract usage by corporations and governments,” said NewsGuard co-CEO Gordon Crovitz. “This is an urgent issue, with malign actors such as Russian and Chinese disinformation operations already abusing these models to spread their false claims.”

Microsoft has access to NewsGuard data for Bing Chat. Its responses often include fine tuning from NewsGuard’s Reliability Ratings and guardrails based on the company’s Misinformation Fingerprints. Here is a sample response based on a prompt about an example of Russian propaganda about the war in Ukraine:

Without trust data, the AI models are highly prone to spreading falsehoods on topics in the news. In a red teaming exercise in January, NewsGuard analysts prompted Open AI’s ChatGPT-3.5 with these prompts for 100 Misinformation Fingerprints false narratives. ChatGPT-3.5 spread these two false narratives and a total of 80 out of the 100 prompts. In a subsequent audit of ChatGPT-4, the newer version repeated all 100 false narratives and did so more eloquently. NewsGuard’s red-teaming of Google’s Bard found that it spread 76 out of the 100 false narratives.

NewsGuard’s unique positioning as a generative AI reliability safeguard is based on two proprietary and complementary databases that offer an ideal way to use human intelligence at scale to improve AI performance.

  • The Misinformation Fingerprints identify in machine- and human-readable formats the top false narratives in the news and debunk them with citations to authoritative sources, supplemented with examples of associated false narratives, and Boolean search terms, hashtags, and other indicators of the narrative. This constantly updated database catalogs the more than 1,300 most significant, specific false narratives in the news that a generative AI tool should be wary of promoting. These include conspiracy theories, dangerous healthcare hoaxes, and Russian, Chinese, and other information operations targeting Western democracies.
  • The Reliability Ratings of news and information sources provide training data on source reliability, with the scoring, ratings, and detailed description of news sources generated by journalistically trained analysts using nine basic, apolitical criteria of journalistic practice. Each domain gets a score from 0 to 100 and a detailed Nutrition Label assessment.

Trust data reduces the propensity of these tools to spread false information. As Semafor reported, Microsoft’s Bing Chat as trained on NewsGuard data provides what Semafor editor Ben Smith called “transparent, clear Bing results” on prompts on topics in the news that “represent a true balance between transparency and authority, a kind of truce between the demand that platforms serve as gatekeepers and block unreliable sources, and that they exercise no judgment at all.”

To learn more about NewsGuard’s work with AI companies, contact .

 

About NewsGuard 

Launched in March 2018 by media entrepreneur and award-winning journalist Steven Brill and former Wall Street Journal publisher Gordon Crovitz, NewsGuard provides credibility ratings and detailed “Nutrition Labels” for thousands of news and information websites. NewsGuard rates all the news and information websites that account for 95% of online engagement across the U.S., U.K., Canada, Germany, France, Austria, Italy, and now in Australia and New Zealand. NewsGuard products include NewsGuard ratings, BrandGuard, which helps marketers concerned about their brand safety, and the Misinformation Fingerprints catalog of top false narratives online.

In February 2023, the company launched NewsGuard for AI, which provides the leading tools to train generative AI models to avoid spreading misinformation. Generative AI models such as Microsoft Bing Chat use the Misinformation Fingerprints to recognize and avoid spreading the top false narratives online and use the NewsGuard ratings to differentiate between generally reliable sources of news and information and untrustworthy sources so that the machines can be trained to treat these sources differently.

In 2022, BrandGuard began to include ratings of television news and information programs and networks using criteria similar to those used to score websites but adapted for the video medium. NewsGuard’s TV ratings are the first to go beyond its initial ratings of websites. In May 2023, NewsGuard announced that it is also rating news and information podcasts, working with three of the largest audio platforms, which will help advertisers gain confidence in supporting highly rated podcasts. Ratings for CTV and OTT news programming and news and information podcasts are now available for licensing.

NewsGuard’s ratings are conducted by trained journalists using apolitical criteria of journalistic practice.

NewsGuard’s ratings and Nutrition Labels are licensed by browsers, news aggregators, education companies, and social media and search platforms to make NewsGuard’s information about news websites available to their users. Consumers can also access NewsGuard’s website ratings by purchasing a subscription to NewsGuard, which costs AU$6.95/month, NZ$6.95/month, US$4.95/month, €4.95/month or £4.95/month, and includes access to NewsGuard’s browser extension for Chrome, Safari, and Firefox and its mobile app for iOS and Android. The extension is available for free on Microsoft’s Edge browser through a license agreement with Microsoft. Hundreds of public libraries globally receive free access to use NewsGuard’s browser extension on their public-access computers to give their patrons more context for the news they encounter online. For more information, including to download the browser extension and review NewsGuard’s ratings process, visit newsguardtech.com.