I’m currently pulling together a bunch of sources – that are mostly recent – on the topic of LLMs and their use in software development.
Some are peer-reviewed studies. Some are industry studies that haven’t been peer-reviewed.
One is statistical physics. Expect more from that angle. Wanna’ know the limits of a technology? Ask a physicist.
One is just a blog post, but very useful information about the effect of context size.
Most are corroborated by personal experiments and also observations on teams. As time goes on and more data comes in, my picture comes more into focus.
Before I cite the sources, a quick executive summary for all you busy executives out there:
- Truly autonomous and reliablelong-horizon agentic software development is so highly improbable using LLMs that it’s essentially science fiction. - The maximum effective context limits of LLMs – including hyperscale “frontier” LLMs – beyond which model outputs become unusably inaccurate is orders of magnitude smaller than advertised limits. The most common mechanism for extending inference over large contexts is what vendors call “compression”. This means that parts of the context are summarised by the model, which is a famously unreliable/lossy process.
- LLMs cannot distinguish between recent and out-of-date information in the context, and information in the model itself, learned during training (“dominant priors”), can often “outweigh” information we give it. To an LLM, it’s all just tokens, weights and probabilities. Right, wrong, new, old – the highest probability wins. Big contexts and “attention dilution” – where probabilities in the context become too small to compete with the ones in the model – are likely to make these effects worse.
- Repo-level .md files tend to make model performance worse, probably because they add noise instead of signal in many specific tasks. Model-generated .md files are especially problematic in this respect, it seems. Upshot: including your team’s coding standards and an architecture summary for every task is probably counterproductive.
- LLMs struggle with negation. Telling them notto do something can often have the same effect as telling them to do it. In case you were wondering why some of your guardrails are about as reliable as a coin-toss. - LLM inference is more accurate when we give them examples (demonstrations) rather than just describing what we want. They’re pattern-matchers. Show them the patterns – more “like this” and less “do this” (and no “don’t do this”).
- Large/long-scale industry studies show a clear trend – output is up (more code, more commits, bigger diffs), but outcomesdon’t reflect that trend. If anything, the average team is takinglongerto shipworsesoftware. If ever we needed proof that software development isn’t a production process… Some studies find a small % of teams getting modest gains in outcomes, and correlate that with their existing software development capability. AI coding is an amplifier of, not a fix for, development strengths and weaknesses. (You’dthinkorganisations would be lining up to do something about that… Sigh.) - The psychological and cognitive factors in LLM use are a growing field of serious research. One study found a significant correlation between confidence in AI output and belief in the paranormal. Multiple studies found a negative impact on learning, cognition and critical thinking with greater LLM reliance. New research suggests that reports of developers feeling demotivated and burned-out with extensive use may have some real truth behind them.
- Deep neural networks, including LLMs, struggle to learn patterns with long-range dependencies, at any scale of model. They will always be “driving in fog”, with local, short-range probabilities crowding out long-range ones. In case you were wondering why they suck at the “big picture” – probabilistically, it’s a blur.
- The energy and compute needed to train an LLM to be an order of magnitude more reliable – e.g., wrong 3% of the time instead of 30% – is 10^20 times what the current frontier models require. Don’t expect significantly more reliable models any time soon. Any future gains in reliability will have to made by better context engineering (deciding what to include in the input) and more effective quality gates deciding what to do with the output- and that’s exactly what we’re seeing AI companies focusing on these days. Models may get more powerful, but not significantly more reliable. This it folks – work with what you’ve got! - Some AI champions will protest research that points to no significant improvements in model performance by pointing to the many published benchmarks that do indeed show LLMs getting better and better. But other research finds that we might wish to be more skeptical of benchmark performance, partly because many of the most popular ones measure what’s easy to measure algorithmically – and in that sense, they’re not really like real-world problems which are messy and unpredictable – and also because… well, the words “published benchmarks” are a bit of a clue. Perhaps inadvertently, but maybe even knowingly, increasingly models are being “trained to the test”. If it’s out there, then it’s probably in there.
Code Evolution & Long-Horizon Agentic Workflows
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integrationhttps://arxiv.org/abs/2603.03823 SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskshttps://arxiv.org/html/2603.24755v1
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolutionhttps://arxiv.org/abs/2603.13428 Benchmark vs. Real-World Performance
Measuring what Matters: Construct Validity in Large Language Model Benchmarkshttps://arxiv.org/abs/2511.04703
Research Update: Algorithmic vs. Holistic Evaluation (METR – not peer-reviewed)https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/#background
Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?https://arxiv.org/abs/2411.03923 Context Engineering
Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMshttps://arxiv.org/abs/2509.21361
Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding[https://arxiv.org/abs/2509.21865](https://arxiv.org/abs/2509.21865)
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?[https://arxiv.org/abs/2602.11988](https://arxiv.org/abs/2602.11988)
(blog post – informational, not peer-reviewed research)
The Hidden Science Behind LLM Token Limits (And How Million-Token Models Actually Work)https://www.ashisharora.ai/post/the-hidden-science-behind-llm-token-limits-and-how-million-token-models-actually-work
Language models are not naysayers: An analysis of language models on negation benchmarkshttps://arxiv.org/abs/2306.08189
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?[https://aclanthology.org/2022.emnlp-main.759](https://aclanthology.org/2022.emnlp-main.759)
STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?[https://arxiv.org/abs/2605.06527](https://arxiv.org/abs/2605.06527)
Task Matters: Knowledge Requirements Shape LLM Responses to
Context–Memory Conflicthttps://aclanthology.org/2026.findings-acl.202 (I call these “dominant priors”, when information in the model overrides that provided in the context)
Large-scale Industry Studies in Software Engineering
What 28 million workflows reveal about AI coding’s biggest risk (CircleCI)https://www.linkedin.com/pulse/what-28-million-workflows-reveal-ai-codings-biggest-risk-circleci-j9syc/
The Acceleration Whiplash – AI Engineering Report 2026 (Faros)[https://www.faros.ai/research/ai-acceleration-whiplash](https://www.faros.ai/research/ai-acceleration-whiplash)
State of AI-assisted Software Development 2025 (DORA)[https://dora.dev/research/2025/dora-report/](https://dora.dev/research/2025/dora-report/)
Psychology, Cognition & Learning
Super-intelligence or Superstition? Exploring Psychological Factors Influencing Belief in AI Predictions about Personal Behaviorhttps://arxiv.org/html/2408.06602v3
The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workershttps://www.researchgate.net/publication/391270185_The_Impact_of_Generative_AI_on_Critical_Thinking_Self-Reported_Reductions_in_Cognitive_Effort_and_Confidence_Effects_From_a_Survey_of_Knowledge_Workers
Experimental evidence of the effects of large language models versus web search on depth of learninghttps://www.researchgate.net/publication/397000021_Experimental_evidence_of_the_effects_of_large_language_models_versus_web_search_on_depth_of_learning
At What Cost? Software Developers’ Well-Being in the Age of GenAIhttps://ourarchive.otago.ac.nz/esploro/outputs/preprint/At-What-Cost-Software-Developers-Well-Being/9926870122801891
Limits of LLMs & Deep Learning
The wall confronting large language models – (statistical mechanics study)https://arxiv.org/abs/2507.19703 Learning long-term dependencies with gradient descent is difficulthttps://pubmed.ncbi.nlm.nih.gov/18267787/
(Why deep neural networks can’t see the bigger picture, at any scale of model)
What are the implications for teams using this technology? If you’re interested in a hype-free, evidence-based take about why the technical practices of Agile Software Development are so closely aligned with AI-assisted and agentic software engineering, join me on October 6th at 18:45 BST.
Register: https://www.tickettailor.com/events/codemanship/2324138