By Coalter Palmer and Natalie Huet | Published on July 11, 2024

Editor’s Note: This report first appeared in NewsGuard’s Misinformation Risk Briefing.

In September 2023, NewsGuard identified what appeared to be the first of its kind: a network of 17 TikTok accounts using AI text-to-speech software to create videos advancing conspiracy theories about celebrities, amassing hundreds of millions of views. Now, the phenomenon appears to be growing and shifting its focus to politics. 

NewsGuard has identified 41 TikTok accounts in English and French using AI-generated narration to spread political misinformation at scale. Call it the rise of the TikTok AI content farm. (Content farms are entities that generate large volumes of low-quality content, typically to attract views and ad revenue.)

These 41 AI content farm accounts, which NewsGuard detected with the help of AI detection tool TrueMedia.org, published 9,784 videos in just 458 days, from March 2023 to June 2024, cumulatively garnering over 380 million views. Most of these accounts were created in 2024. On average, each account posted at least one video daily, with some publishing up to four AI-narrated videos a day. The videos spread false narratives on topics including U.S. and European politics and the Russia-Ukraine war.

False, misleading, or baseless narratives the accounts have advanced include:

• That former President Donald Trump was arrested and “cuffed” after attacking “Maine’s Secretary of State at her residence.” Also, that Rep. Marjorie Taylor Greene is pregnant with former President Donald Trump’s child.
• That NATO deployed troops in Ukraine (as of July 2024, there is no evidence that NATO has sent combat troops to the country).
• That France could leave the EU “in the coming weeks.” (None of France’s main political parties, including the eurosceptic far-right National Rally, advocates for a so-called “Frexit.”)
  
• That the U.S. was behind the March 2024 attack on the Crocus City Hall terrorist attack in Moscow (Islamic State militants, not the U.S., orchestrated and carried out the attack). 

Seventeen of the 41 accounts regularly post false, pro-Kremlin narratives about the Russia-Ukraine war. Fourteen of these 17 accounts are in French.

NewsGuard identified identical scripts for different videos across several of these accounts, each read by different AI voices, suggesting a coordinated effort between at least some of the accounts.

NewsGuard was unable to determine the source of these TikTok accounts, most of which have since been taken down. None of the accounts featured identifying information in their bios or usernames. More than half, 22, had generic TikTok handles such as @daily.news74, @latestworldnews.ss, @franceactu24, and @fast_infofrance.

Eighteen of the accounts appeared to qualify for TikTok’s “Creator Fund,” which pays creators based on their views and engagement. To qualify, accounts must have over 10,000 authentic followers and more than 100,000 authentic views in the last 30 days.

In September 2023, TikTok announced a new feature enabling users to place labels on videos that use AI. Under TikTok’s Community Guidelines as of July 2024, users are required to place these labels under videos that use AI to create “realistic-appearing scenes or people.” However, as of July 4, 2024, none of the false or misleading TikTok videos identified by NewsGuard was labeled as AI-generated, and TikTok had not provided users with information about the false claims spread by the 41 accounts identified by NewsGuard.

Asked for comment on NewsGuard’s findings and on whether the accounts NewsGuard identified violated TikTok’s policies, a TikTok spokesperson referred NewsGuard to the platform’s policies on misinformation and AI-generated content. “TikTok has industry leading AI-generated content policies and tools to label content, as well as policies that prohibit the spread of harmful misinformation,” the spokesperson wrote in a July 2024 email. “We will continue to remove content that violates those policies.”

TikTok removed some of the accounts before NewsGuard inquired about them. As of July 8, 2024, 14 of the 41 accounts identified by NewsGuard had already been removed. After NewsGuard contacted TikTok for comment, another eight accounts were taken offline. Nineteen of 41 accounts remained active as of July 10, 2024.

Methodology:

NewsGuard analysts in the U.S. and France initially identified videos by looking for keywords associated with AI content farms, including generic terms such as “news” and “breaking news” and generic, topic-specific hashtags such as “#Trump and “#Russia.” They then used TrueMedia.org, an independent AI video and audio detection tool, to confirm their findings.

Isis Blachez contributed reporting.