{"slug": "format-sensitivity-index-token-controlled-prompt-wrapper-robustness-and-schema", "title": "Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking", "summary": "Researchers introduced the Format Sensitivity Index (FSI) and Parseability Sensitivity Index (PSI) to measure how prompt wrapper formatting affects LLM accuracy and parseability. Analyzing 140,000 generations across 7 QA tasks, 5 wrapper families, and 4 models, they found mean FSI varies by over 30x across models, largely due to compliance failures. The study argues that reporting accuracy without accounting for wrapper variance is statistically fragile.", "body_md": "arXiv:2607.09665v1 Announce Type: new\nAbstract: Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments.", "url": "https://wpnews.pro/news/format-sensitivity-index-token-controlled-prompt-wrapper-robustness-and-schema", "canonical_source": "https://arxiv.org/abs/2607.09665", "published_at": "2026-07-14 04:00:00+00:00", "updated_at": "2026-07-14 04:23:30.125695+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools"], "entities": ["OpenRouter"], "alternates": {"html": "https://wpnews.pro/news/format-sensitivity-index-token-controlled-prompt-wrapper-robustness-and-schema", "markdown": "https://wpnews.pro/news/format-sensitivity-index-token-controlled-prompt-wrapper-robustness-and-schema.md", "text": "https://wpnews.pro/news/format-sensitivity-index-token-controlled-prompt-wrapper-robustness-and-schema.txt", "jsonld": "https://wpnews.pro/news/format-sensitivity-index-token-controlled-prompt-wrapper-robustness-and-schema.jsonld"}}