Stability-Aware Feature Design for Robust Watermark Detection in Machine-Generated Text

Researchers introduced Pattern Stability Score (PSS), a watermark detection framework for machine-generated text that improves detection AUC by over 10-15 percentage points across different token lengths compared to prior baselines. The method, evaluated on PG-19, CNN/DailyMail, and WikiText using Llama-3-8B and Qwen2-7B, maintains above 87.8% AUC even when all components differ from training.

arXiv:2608.18102v1 Announce Type: new Abstract: The widespread adoption of large language models LLMs has intensified the demand for principled methods to distinguish human from machine-generated text. Watermarking provides a promising avenue, yet existing detectors exhibit sharp performance deterioration under multiple paraphrasing and when applied to shorter texts. We introduce Pattern Stability Score PSS , a novel detection framework that leverages local statistical features and stability dynamics across paraphrased variants. Specifically, the proposed method combines global and local z-score features with higher-order statistics of run-length patterns, enriched by autocorrelation signals and stability scores computed over paraphrase depth. Numerical evaluations are performed on three benchmark datasets PG-19, CNN/DailyMail, and WikiText using multiple LLMs Llama-3-8B, Qwen2-7B and paraphrasers Mistral-7B, Qwen2-7B, Gemma-7B , systematically stress-testing robustness under up to eight rounds of paraphrasing. Compared to prior z-score thresholding baselines and some state-of-the-art deep learning methods, our approach improves detection AUC area under the receiver operating characteristic curve by over 10-15 percentage points across different token lengths. Additionally, extensive cross-domain experiments demonstrate that a single universal classifier generalizes across different LLMs, paraphrasers, and text domains without retraining, maintaining above 87.8% AUC even when all components differ from training.