05/06/2022

New research from NYU shows NewsGuard helps those most exposed to misinformation

Study finds that NewsGuard helps improve the news diet quality of the biggest consumers of false news.

(New York, May 6th, 2022) Research published today by New York University demonstrates that individuals who frequently consume low-quality news improve their news diets after receiving access to NewsGuard’s Red and Green Reliability Ratings for news websites.

“This research shows that the intervention of NewsGuard’s ratings can help the folks who need them most,” said NewsGuard co-CEO Gordon Crovitz. “In other words, if social media platforms and other technology companies were to integrate NewsGuard’s ratings and Nutrition Labels into their products, it would have a meaningful impact on the ability of misinformation to spread and damage society.”

The abstract published in the peer-reviewed journal Science Advances summarized the problem that led to the research: “As the primary arena for viral misinformation shifts toward transnational threats, the search continues for scalable countermeasures compatible with principles of transparency and free expression.” The study found that access to NewsGuard’s transparent, apolitical ratings of sources of news and information online led to a “substantively meaningful increase in news diet quality among the heaviest consumers of misinformation.”

The authors, academics from New York University and Princeton through NYU’s Center for Social Media and Politics (CSMaP), set out the problem caused by the inability of news consumers to tell which sources are trustworthy and which are not, which they said has “contributed to pathologies in American political discourse including the spread of misinformation, disagreements about basic facts related to governance and policy, and lowered trust in established media. Of particular concern is the possibility that these problems are interlinked: As political divisions widen, partisan media alienate people from authoritative sources, which could make it more difficult to counteract potentially corrosive—and in the case of public health during a pandemic, life-threatening—misinformation.”

The researchers noted that their aim was to test whether labeling the reliability of news sources could succeed compared with approaches to counter misinformation that have been tried. “These range from relatively intrusive measures such as algorithmic downranking to subtle warnings and labels targeted at specific factual claims and to general efforts to boost digital media literacy skills,” they wrote. “In addition to being scalable relative to fact-checks, source labels are relevant for a broad array of publishers across the spectrum of reliability, rather than merely those designated as purveyors of misinformation.” They added, “NewsGuard provides a particularly comprehensive set of source ratings produced according to transparent criteria. Deploying these source ratings as visible labels across a user’s search and social feeds provides an opportunity to test the effectiveness of expert ratings as a general solution to online misinformation.”

The methodology was to track actual user behavior: “In an online field experiment, we randomly encouraged participants to install a prominent web browser extension, NewsGuard, which embeds straightforward source-level indicators of news reliability into users’ search engine results pages (SERPs), social feeds, and visited URLs.”

The study was conducted between May and June 2020, during which time researchers had a random sample of more than 3,000 participants install NewsGuard’s browser extension, which displays Red-Green ratings of news source reliability directly into users’ search engine results, social media feeds, and elsewhere on their browsers. The researchers then collected “anonymized digital trace data” for a subset of participants, enabling them to track the browsing habits of the users to observe which webpages the users visited after having access to NewsGuard. Using NewsGuard’s Reliability Rating dataset as a benchmark for source quality, the researchers then analyzed the quality of news sites visited by each user. 

Among the participants, the researchers analyzed the 10%-20% of participants who had news diets including a meaningful percentage of NewsGuard red-rated sources—one where the NewsGuard Nutrition Label urges users to “proceed with caution” and provides a detailed explanation of why the site fails key criteria of journalistic practice. These are the key news consumers whose reliance on unreliable sources can be reduced by access to warning labels. The researchers found “suggestive evidence of a substantively meaningful boost in news quality among the heaviest consumers of misinformation.” Researchers measured the first two weeks of access to NewsGuard ratings and the second two weeks: “We observed the quality of news diets among the lowest 10% of our sample increase by 5.4 and 8.6% from their pretreatment levels, respectively.” For the second two-week period, “we even observe a statistically significant increase in news reliability among those in the bottom 20% of pretreatment news consumption quality.”

This research did not find what the researchers called “statistically significant average treatment effects,” explaining that across all the participants the effect of labeling was on average small “potentially because of limited exposure to low-credibility sites to begin with.” The large majority of people surveyed only saw NewsGuard green-rated sites in their healthy news diets, so would not have seen red ratings, and thus could not have been affected by them. 

The research found that the majority of users did not rely on unreliable news on a regular basis. In fact, most participants (65 percent) did not visit a single unreliable news site before getting access to NewsGuard. The research thus found, unsurprisingly, that NewsGuard’s ratings did not cause a measurable shift in news habits for the average reader who does not encounter a meaningful number of misinformation sources while browsing. Less than 12% of the sample’s news diet consisted of at least 5% of visits to news sites deemed unreliable by NewsGuard. Only 1.5% of respondents’ news diets had an average NewsGuard reliability score below 60, which is the threshold for earning a red rating. 

In a press release describing the study’s findings, Andrew M. Guess, a faculty research affiliate at CSMaP and an assistant professor at Princeton University, who was involved with the research, said the following:

“In our partisan age, when attitudes about news sources are strongly correlated with partisanship, relatively subtle cues like source credibility labels may not be powerful enough to shift news habits and counteract misperceptions among the general public,” Guess said. “However, a key metric of success for this intervention is how it changes the behavior of those who consume the most low-quality news. The fact that it doesn’t work for the overall population doesn’t mean the tool is ineffective. It means it must be part of a far larger toolkit to combat the spread of online misinformation.”

These findings mirror those of 2019 research conducted by Gallup and the Knight Foundation, in which a representative sample of American adults installed the NewsGuard extension and reported their results. The study found the Nutrition Labels to be helpful to users and that seeing a red rating next to an unreliable source dissuaded users from sharing the content on social media. Specifically, the research found the following:

• 91% of users found the NewsGuard Nutrition Labels helpful. 
• 90% generally agreed with the ratings and respondents trusted the ratings more because NewsGuard ratings are done by “trained journalists with varied backgrounds.”
• 89% of users of social media sites and 83% overall wanted social media sites and search engines to integrate NewsGuard ratings and reviews into their news feeds and search results. 
• 78% said they would recommend NewsGuard toa friend or relative.
• 69% would trust social media and search companies more if they took the simple step of including NewsGuard in their products.
• 63% would be less likely to share news stories from red-rated websites, and 56% would be more likely to share news from green-rated websites.

Although, unlike the Gallup study, NYU’s research did not assess how NewsGuard data impacted the news sharing habits of users, future research may focus on that phenomenon. The authors suggested that future research could focus on the subset of news consumers whose news diet consists significantly of unreliable, NewsGuard red-rated sites. More than 40% of news and information sites accessed in the U.S. get red ratings from NewsGuard.

About NewsGuard

Launched in March 2018 by media entrepreneur and award-winning journalist Steven Brill and former Wall Street Journal publisher Gordon Crovitz, NewsGuard provides credibility ratings and detailed “Nutrition Labels” for thousands of news and information sources. NewsGuard rates all the news and information sources that account for 95% of online engagement across the US, UK, Canada, Germany, France, and Italy.

NewsGuard’s ratings are conducted by trained journalists using nine apolitical criteria of journalistic practice, including whether a news source repeatedly publishes false content, whether it regularly corrects or clarifies errors, and whether it avoids deceptive headlines. Based on the criteria, each source receives an overall trust rating, a trust score of 0-100, a score on each of the nine criteria, and a detailed “Nutrition Label” explaining the rating and providing examples of the site’s editorial practices. Advertisers, advertising agencies and advertising tech companies license NewsGuard’s ratings to direct their programmatic advertising toward legitimate journalism and avoid misinformation. For more information, including to download the browser extension and review the ratings process, visit newsguardtech.com.