TikTok Targets AI-Generated Spam Accounts In High-Risk Topics

TikTok announced it will test improved detection systems for AI-generated spam accounts posting about politics, financial advice, and medical topics, aiming to curb misleading content that could affect public trust. The company also joined the steering committee of the Coalition for Content Provenance and Authenticity (C2PA) to help drive adoption of content provenance standards across the industry.

TikTok will test improvements to its detection systems for accounts posting AI-generated spam about politics and current events, financial advice, and medical topics, as announced in a newsroom post https://newsroom.tiktok.com/helping-people-spot-and-understand-aigc-on-tiktok?lang=en-150& sp=943ade1e-e60b-4b7a-b390-16ca078d72ed.1784053130787 . The testing is set to begin in the upcoming weeks and described those categories as areas where misleading content could affect public trust or well-being. Medical content is one of the three named targets, and it aligns with a Kapwing report I covered in June https://www.searchenginejournal.com/tiktok-shows-3x-more-ai-slop-than-youtube-report-finds/579521/ , which found health to be among the highest categories for AI-generated content on TikTok, well above categories where creators appear on camera. TikTok explained that testing is deployed at the account level, saying the company already removes spam on a large scale and has dismantled over 86 million fake accounts in the first three months of this year. How Other Platforms Have Responded This announcement is part of broader efforts to promote authentic content across social media platforms. Last July, YouTube updated its monetization guidelines to better address inauthentic https://www.searchenginejournal.com/youtube-targets-mass-produced-content-in-monetization-update/550337/ , repetitive videos, and Meta quickly followed https://www.searchenginejournal.com/meta-follows-youtube-in-crackdown-on-unoriginal-content/551096/ with their own measures against unoriginal content. Additionally, in June, Google published a paper https://www.searchenginejournal.com/google-generated-ai-detected/579987/ on a system designed to detect and remove coordinated groups of accounts posting AI-generated spam. The C2PA Committee Seat In addition to spam testing, TikTok announced its participation in the steering committee of the Coalition for Content Provenance and Authenticity, the organization responsible for developing the C2PA standard that tracks the origin and modifications of digital content. TikTok mentioned that it was the first video platform to implement C2PA Content Credentials two years ago. TikTok said the committee seat will let it help drive adoption of the technology across the industry. The company announced it has tagged over 3 billion videos as AI-generated content, utilizing Content Credentials, creator labels, and its proprietary invisible watermarking. In May, Google took a similar step by extending SynthID verification to Search https://www.searchenginejournal.com/google-brings-ai-content-verification-to-search/575320/ and promising that C2PA verification would be introduced soon. Why This Matters This falls within categories many marketers focus on, like health, which already has a lot of AI-generated content. Anyone creating finance, health, or politics-related videos on TikTok is working in topics that TikTok is especially watching for spam. What this means in practice will depend on how TikTok conducts the test, and the details are still unclear from the announcement. Looking Ahead The specifics to watch are how TikTok defines a qualifying account and what follows a detection, neither of which the announcement spells out. There’s also no firm start date beyond the coming weeks. On the C2PA side, TikTok didn’t announce any change to its existing labeling system. Featured Image: Mamun Sheikh/Shutterstock