AI Software Development – What Does the Data Say?

A compilation of recent studies and industry data indicates that truly autonomous and reliable long-horizon agentic software development using large language models (LLMs) is essentially science fiction, with maximum effective context limits orders of magnitude smaller than advertised and model-generated documentation often degrading performance. The analysis, which includes peer-reviewed and industry studies, shows that while LLM use increases output volume, it does not improve outcomes, and in some cases teams take longer to ship worse software, with LLM coding acting as an amplifier of existing development strengths and weaknesses.

I’m currently pulling together a bunch of sources – that are mostly recent – on the topic of LLMs and their use in software development. Some are peer-reviewed studies. Some are industry studies that haven’t been peer-reviewed. One is statistical physics. Expect more from that angle. Wanna’ know the limits of a technology? Ask a physicist. One is just a blog post, but very useful information about the effect of context size. Most are corroborated by personal experiments and also observations on teams. As time goes on and more data comes in, my picture comes more into focus. Before I cite the sources, a quick executive summary for all you busy executives out there: - Truly autonomous and reliable long-horizon agentic software development is so highly improbable using LLMs that it’s essentially science fiction. - The maximum effective context limits of LLMs – including hyperscale “frontier” LLMs – beyond which model outputs become unusably inaccurate is orders of magnitude smaller than advertised limits. The most common mechanism for extending inference over large contexts is what vendors call “compression”. This means that parts of the context are summarised by the model, which is a famously unreliable/lossy process. - LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training “dominant priors” , can often “outweigh” information we give it. To an LLM, it’s all just tokens, weights and probabilities. Right, wrong, new, old – the highest probability wins. Big contexts and “attention dilution” – where probabilities in the context become too small to compete with the ones in the model – are likely to make these effects worse. - Repo-level .md files tend to make model performance worse, probably because they add noise instead of signal in many specific tasks. Model-generated .md files are especially problematic in this respect, it seems. Upshot: including your team’s coding standards and an architecture summary for every task is probably counterproductive. - LLMs struggle with negation. Telling them not to do something can often have the same effect as telling them to do it. In case you were wondering why some of your guardrails are about as reliable as a coin-toss. - LLM inference is more accurate when we give them examples demonstrations rather than just describing what we want. They’re pattern-matchers. Show them the patterns – more “like this” and less “do this” and no “don’t do this” . - Large/long-scale industry studies show a clear trend – output is up more code, more commits, bigger diffs , but outcomes don’t reflect that trend. If anything, the average team is taking longer to ship worse software. If ever we needed proof that software development isn’t a production process… Some studies find a small % of teams getting modest gains in outcomes, and correlate that with their existing software development capability. AI coding is an amplifier of, not a fix for, development strengths and weaknesses. You’d think organisations would be lining up to do something about that… Sigh. - The psychological and cognitive factors in LLM use are a growing field of serious research. One study found a significant correlation between confidence in AI output and belief in the paranormal. Multiple studies found a negative impact on learning, cognition and critical thinking with greater LLM reliance. New research suggests that reports of developers feeling demotivated and burned-out with extensive use may have some real truth behind them. - Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model. They will always be “driving in fog”, with local, short-range probabilities crowding out long-range ones. In case you were wondering why they suck at the “big picture” – probabilistically, it’s a blur. - The energy and compute needed to train an LLM to be an order of magnitude more reliable – e.g., wrong 3% of the time instead of 30% – is 10^20 times what the current frontier models require. Don’t expect significantly more reliable models any time soon. Any future gains in reliability will have to made by better context engineering deciding what to include in the input and more effective quality gates deciding what to do with the output- and that’s exactly what we’re seeing AI companies focusing on these days. Models may get more powerful , but not significantly more reliable. This it folks – work with what you’ve got - Some AI champions will protest research that points to no significant improvements in model performance by pointing to the many published benchmarks that do indeed show LLMs getting better and better. But other research finds that we might wish to be more skeptical of benchmark performance, partly because many of the most popular ones measure what’s easy to measure algorithmically – and in that sense, they’re not really like real-world problems which are messy and unpredictable – and also because… well, the words “published benchmarks” are a bit of a clue. Perhaps inadvertently, but maybe even knowingly, increasingly models are being “trained to the test”. If it’s out there, then it’s probably in there. Code Evolution & Long-Horizon Agentic Workflows SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration https://arxiv.org/abs/2603.03823 https://arxiv.org/abs/2603.03823 SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks https://arxiv.org/html/2603.24755v1 https://arxiv.org/html/2603.24755v1 SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution https://arxiv.org/abs/2603.13428 https://arxiv.org/abs/2603.13428 Benchmark vs. Real-World Performance Measuring what Matters: Construct Validity in Large Language Model Benchmarks https://arxiv.org/abs/2511.04703 https://arxiv.org/abs/2511.04703 Research Update: Algorithmic vs. Holistic Evaluation METR – not peer-reviewed https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/ background https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/ background Evaluation data contamination in LLMs: how do we measure it and when does it matter? https://arxiv.org/abs/2411.03923 https://arxiv.org/abs/2411.03923 Context Engineering Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs https://arxiv.org/abs/2509.21361 https://arxiv.org/abs/2509.21361 Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding https://arxiv.org/abs/2509.21865 https://arxiv.org/abs/2509.21865 Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? https://arxiv.org/abs/2602.11988 https://arxiv.org/abs/2602.11988 blog post – informational, not peer-reviewed research The Hidden Science Behind LLM Token Limits And How Million-Token Models Actually Work https://www.ashisharora.ai/post/the-hidden-science-behind-llm-token-limits-and-how-million-token-models-actually-work https://www.ashisharora.ai/post/the-hidden-science-behind-llm-token-limits-and-how-million-token-models-actually-work Language models are not naysayers: An analysis of language models on negation benchmarks https://arxiv.org/abs/2306.08189 https://arxiv.org/abs/2602.11988 Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? https://aclanthology.org/2022.emnlp-main.759 https://aclanthology.org/2022.emnlp-main.759 STALE: Can LLM Agents Know When Their Memories Are No Longer Valid? https://arxiv.org/abs/2605.06527 https://arxiv.org/abs/2605.06527 Task Matters: Knowledge Requirements Shape LLM Responses to Context–Memory Conflict https://aclanthology.org/2026.findings-acl.202 https://aclanthology.org/2026.findings-acl.202 I call these “dominant priors”, when information in the model overrides that provided in the context Large-scale Industry Studies in Software Engineering What 28 million workflows reveal about AI coding’s biggest risk CircleCI https://www.linkedin.com/pulse/what-28-million-workflows-reveal-ai-codings-biggest-risk-circleci-j9syc/ https://www.linkedin.com/pulse/what-28-million-workflows-reveal-ai-codings-biggest-risk-circleci-j9syc/ The Acceleration Whiplash – AI Engineering Report 2026 Faros https://www.faros.ai/research/ai-acceleration-whiplash https://www.faros.ai/research/ai-acceleration-whiplash State of AI-assisted Software Development 2025 DORA https://dora.dev/research/2025/dora-report/ https://dora.dev/research/2025/dora-report/ Psychology, Cognition & Learning Super-intelligence or Superstition? Exploring Psychological Factors Influencing Belief in AI Predictions about Personal Behavior https://arxiv.org/html/2408.06602v3 https://arxiv.org/html/2408.06602v3 The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers https://www.researchgate.net/publication/391270185 The Impact of Generative AI on Critical Thinking Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers https://www.researchgate.net/publication/391270185 The Impact of Generative AI on Critical Thinking Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers Experimental evidence of the effects of large language models versus web search on depth of learning https://www.researchgate.net/publication/397000021 Experimental evidence of the effects of large language models versus web search on depth of learning https://www.researchgate.net/publication/397000021 Experimental evidence of the effects of large language models versus web search on depth of learning At What Cost? Software Developers’ Well-Being in the Age of GenAI https://ourarchive.otago.ac.nz/esploro/outputs/preprint/At-What-Cost-Software-Developers-Well-Being/9926870122801891 https://ourarchive.otago.ac.nz/esploro/outputs/preprint/At-What-Cost-Software-Developers-Well-Being/9926870122801891 Limits of LLMs & Deep Learning The wall confronting large language models – statistical mechanics study https://arxiv.org/abs/2507.19703 https://arxiv.org/abs/2507.19703 Learning long-term dependencies with gradient descent is difficult https://pubmed.ncbi.nlm.nih.gov/18267787/ https://pubmed.ncbi.nlm.nih.gov/18267787/ Why deep neural networks can’t see the bigger picture, at any scale of model What are the implications for teams using this technology? If you’re interested in a hype-free, evidence-based take about why the technical practices of Agile Software Development are so closely aligned with AI-assisted and agentic software engineering, join me on October 6th at 18:45 BST . Register: https://www.tickettailor.com/events/codemanship/2324138