Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

A new arXiv study (2608.20349v1) presents the first large-scale, n-gram token-level mechanistic analysis of prompt stability, using 132,000 prompt variants, and finds a Scaling Law of Prompt Performance Stability: higher average task performance correlates with lower variance and greater robustness. The researchers identify Domain-Specific Terminology and Explicit Action Directives as key linguistic drivers, and their automated Prompt-Refining Agent reduces performance variance by 40.7% in code generation tasks while preserving or improving mean performance.

arXiv:2608.20349v1 Announce Type: new Abstract: Large Language Models LLMs exhibit extreme sensitivity to surface-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations. Moving beyond black-box optimization and coarse-grained templates, we present the first large-scale, n-gram token-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants. Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation. We identify two core linguistic drivers underlying this robustness: 1 Domain-Specific Terminology, which tightly anchors semantic boundaries, and 2 Explicit Action Directives, which formalize reasoning trajectories. Together, these elements constrain the model's interpretative space, effectively locking in'' more deterministic generation behavior. Building on these insights, we introduce an automated Prompt-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints. Empirical evaluation shows that our approach reduces performance variance by 40.7% in code generation task, while preserving or improving mean performance. These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering.