{"slug": "schrodinger-s-code-repository-have-llms-learned-swe-bench-or-memorized-it", "title": "Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?", "summary": "Repository-level coding benchmarks such as SWE-bench suffer from data leakage because they are built on popular open-source repositories repeatedly used for LLM training, meaning strong performance may reflect memorization of canonical solutions rather than learned software-engineering skill, according to an analysis titled \"Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?\" The finding matters because SWE-bench has become the standard for evaluating coding agents, so leaked training data can inflate reported capability.", "body_md": "Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical rep", "url": "https://wpnews.pro/news/schrodinger-s-code-repository-have-llms-learned-swe-bench-or-memorized-it", "canonical_source": "https://aiflash.com/news/125332/", "published_at": "2026-09-24 02:30:07+00:00", "updated_at": "2026-09-24 02:59:38.285424+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "machine-learning", "artificial-intelligence"], "entities": ["SWE-bench"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/schrodinger-s-code-repository-have-llms-learned-swe-bench-or-memorized-it", "markdown": "https://wpnews.pro/news/schrodinger-s-code-repository-have-llms-learned-swe-bench-or-memorized-it.md", "text": "https://wpnews.pro/news/schrodinger-s-code-repository-have-llms-learned-swe-bench-or-memorized-it.txt", "jsonld": "https://wpnews.pro/news/schrodinger-s-code-repository-have-llms-learned-swe-bench-or-memorized-it.jsonld"}}