{"slug": "harness-delta-attribution-dissecting-gains-from-harness-evolution", "title": "Harness-Delta Attribution: Dissecting Gains From Harness Evolution", "summary": "A new arXiv paper (2610.22466v1) introduces Harness-Delta Attribution (HDA), an evaluation protocol that separates harness-evolution gains into artifact-driven overfitting, gains explained by test-time scaling, and a residual upper bound on generalizable improvement. Applied to 11 executor models and three harness-evolution frameworks across math, coding, creativity and agentic tasks, HDA found that in the 21 settings with statistically significant search-set gains, overfitting and test-time scaling accounted for 54% and 14% of the gain on average, with their combined share exceeding 50% in 14 settings. The authors report that search-set gains often fail to transfer to held-out tasks and that validation-based signals are insufficient to prevent overfitting, motivating attribution-aware evaluation in future harness-evolution research.", "body_md": "arXiv:2610.22466v1 Announce Type: new \nAbstract: Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.", "url": "https://wpnews.pro/news/harness-delta-attribution-dissecting-gains-from-harness-evolution", "canonical_source": "https://arxiv.org/abs/2610.22466", "published_at": "2026-10-19 04:00:00+00:00", "updated_at": "2026-10-19 04:18:28.712491+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "machine-learning", "large-language-models"], "entities": ["Harness-Delta Attribution", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/harness-delta-attribution-dissecting-gains-from-harness-evolution", "markdown": "https://wpnews.pro/news/harness-delta-attribution-dissecting-gains-from-harness-evolution.md", "text": "https://wpnews.pro/news/harness-delta-attribution-dissecting-gains-from-harness-evolution.txt", "jsonld": "https://wpnews.pro/news/harness-delta-attribution-dissecting-gains-from-harness-evolution.jsonld"}}