{"slug": "why-software-automation-is-hard", "title": "Why Software Automation Is Hard", "summary": "Software automation is proving more difficult than many expect, as larger organizations face growing challenges with dependencies, context, and technical debt that smaller teams do not. Industry leaders are betting heavily that coding agents will improve faster than these problems accumulate, but any significant delay in AI progress could leave many tech companies struggling with the consequences of hasty automation.", "body_md": "*Originally intended as a quick take, but got a bit longer, so why not turn it into a post. Just sharing my observations & assumptions here about the state of software automation. Happy to hear thoughts on where you think I'm off. I'm sure none of the thoughts in this post are totally original, many have been proposed in similar form elsewhere, and **I**'m*[[1]](https://www.lesswrong.com/feed.xml#fn8phj1qlod3x)* far from the first person to speak of the bottlenecks that AI progress and adoption are facing. It still seemed useful to compile my current views on the situation and summarize them to those with only an outside view on the impact of AI on the software industry.*\n\nThe software world is trying hard to automate itself. Undoubtedly, coding agents have made a step change since last November and now enable more and more use cases that were unthinkable a year ago. And yet it seems to me that there's still a big disconnect between how many people think coding agents *should* be affecting the software industry and what's really going on so far in most places.\n\nPlease feel encouraged share your views and disagreements about any of these in the comments.\n\nFirst, I do think the following things are all true:\n\nBut it's easy to extrapolate this too far, assuming that much of software engineering can now be automated and that the same number of engineers can now get done 10x as much as before. My impression is that this is partially true for very small teams, but gets less and less true the larger an organization is and the more dependencies and constraints they have. In particular, I think that it takes a lot of very deliberate effort to find the right ways to use current AI to actually become considerably faster at actually-useful work, and most naive / locally-optimizing approaches at doing so may not work out and even be detrimental.\n\nI don't know how AI will develop, if progress will continue at about its current pace, and how such progress will affect tech orgs. But I do think many of them are playing with fire and are betting *a lot* on the assumption that coding agents will become much more capable quickly, at a rate where they somehow manage to outpace the problems that are currently being caused. If progress gets delayed significantly - perhaps due to hardware bottlenecks, headwinds against data-center construction in the US, a Taiwan crisis, a cascade of investors losing trust in AI and pulling out their investments - then my current take is that many existing tech orgs will face considerable challenges caused by their current strategies of hasty automation.\n\nIt seems to me that what many people in software are betting on is that coding agents keep getting better at recent rates (or even accelerate), and this will allow them to eventually surpass pretty much all the problems mentioned above. If their capabilities grow faster than the problems accumulate, then it's a good thing to ride this wave as early as possible.\n\nThe problems I listed are not insurmountable, just difficult. For instance, the context issues could to some degree get solved. Coding agents may get much larger context windows, or continual learning gets solved or greatly improved. While the code itself does not contain all the relevant context, agents may eventually process not only entire code bases, but also the full history of company-internal communication tools, knowledge bases, and chat & thought process history of prior coding agents, and have all of these either in their context [10] or even their weights, allowing them to know pretty much all functional + non-functional requirements, reasons why certain decisions were made, and so on.\n\nSimilarly, while technical debt may accumulate, one can also argue that the viability of refactoring and rewriting code from scratch is increasing rapidly. At some point, technical debt may just be a non-issue because a fleet of coding agents can rewrite almost any piece of software overnight, if necessary.\n\nAnother argument I find compelling is that, perhaps, some startups will just figure out how to integrate coding agents properly *without* running into many of these issues, which for established larger orgs is much harder to do. I believe this could either happen by finding very suitable organizational structures and processes, or by finding particular use cases that are well-suited for AI automation. And these better-prepared and hence much more rapidly executing startups may, over the next years, just outcompete many of the established organizations that are failing to properly adapt. If, from the start, you establish rules and norms around standardized documentation, test coverage, centralized LLM-friendly communication channels, and focus your acceleration attempts to those areas where they have a good chance of working without causing too much trouble elsewhere, then maybe you really can achieve much higher velocity than other companies of similar size throughout the growth phase in a way that leads to a higher market share.\n\nEven then I'd think that in most domains, quality of strategic judgment is likely more important than speed. A company being 10x as fast at developing new stuff compared to another one may still lose if they just hastily follow the weak signals they pick up from the market.\n\nAll of the above is not to say that the world will not look very strange in five or ten years. I just see a lot of reasons why software automation in particular may not be as straightforward to accomplish as it may look on the surface. None of the challenges I mentioned are insurmountable, but they exist, and solving them will likely take some time.\n\nEven when we reach a point where some fully automated AI-only companies exist that do not involve any human employees as potential bottlenecks, I would expect these to not have *that* much of an immediate advantage. At least as long as they still cater to humans. Because, as humans are (initially) both the likely end users and those that hold most of the capital, the signals such companies get from the market will still mostly reach them at human time scales. Being able to develop software 1000x as quickly may not be all *that* useful when the market feedback still comes in at 1x speed.\n\nTo be clear, I'm not meaning to imply much about the alignment problem or existential risk here. Clearly, once we have fully AI-driven companies without human involvement, and they actually manage to be competitive, then we're deep in singularity territory, and I'd be very happy about internationally coordinated efforts to delay or prevent us from reaching that state of affairs anytime soon. For the most part, I'm arguing that AI automation really seems way trickier than I would have expected a few years ago. I was confused for some time why coding agents seem so incredible during personal use and yet don't seem to have that much of an impact on the productivity of most larger organizations yet [11]. This post is my attempt to make sense of this.\n\nFor context, I've been working as a programmer for close to 15 years and have been working a lot with Github Copilot agents and (since the Opus 4.5 release) with Claude Code, both privately and professionally.\n\nThis is another reason I found it worthwhile to write such a post - people who have no close ties to the tech world may primarily know coding agents from messy public debates as well as their own experiences, which on an individual level are often overwhelmingly positive. As individuals, we get empowered to solve all kinds of problems we couldn't solve before, and this makes coding agents seem almost magical. But I argue that these magical properties don't transfer that well to the software industry, at least currently.\n\nWhat I mean by this is that there are many very intangible things, like what kind of experience you strive for with your software, how \"dangerous\" certain modules are (maybe some particular part of the code requires adjustments, but the three times somebody tried that in the past, it always spectacularly failed, so you learned to not touch that part of the code and just live with its limitations), or knowing that a particular change that would be useful will lead to some conflict with another team that has strong views on doing things differently; things like that, which you don't necessarily think about deliberately, but they still steer your behavior in meaningful ways.\n\nWell, not all of it. There are certainly different types of context humans work with. Some context is in form of written documents (that live outside the code) - these could be processed equally well, or better even, by coding agents, given they have access to them and know they exist and when to query them. However, humans also have a constantly growing theory of the code, of the product, and of the organization as a whole, and know which things they need when. They know how many users their software has (if any), how consequential bugs of different types are and how much effort is warranted to prevent them, how severe an outage would be, broadly what future plans may exist, how much time pressure there is, and so on, and so on. All these things are like very particular glasses through which the human sees their work. The coding agent (of today) has almost none of that. Once coding agents get continual learning, they may be able to persist such things on their own, without having to rely on lossy text representations - but even then, they'll first have to *build* that context, which would require help from the humans who, until then, are the only ones who have all that context. So even once we have such capable coding agents, they could still take months to build a similar amount of context as a capable human software engineer.\n\nI often observe this when watching others prompt an LLM. In such situations, my impression is often \"Wow, this prompt is so vague and just uses terms that are never explained, no way the LLM will be able to work with that\", but in most cases, the LLM will just correctly infer what they are talking about and give a pretty solid answer. They just have learned to make *usually* correct assumptions when working with highly incomplete information. But this comes at the cost of sometimes making wrong assumptions without questioning them, and I don't think they have a way, even in principle, to distinguish right from wrong assumptions reliably.\n\nTo state this more clearly, I don't necessarily think that *the skill itself* degrades that quickly. But once you're high on the drug of \"just type the problem into a chat box and hope for it to magically get solved\", it's very hard to go back to the old world of expending serious cognitive effort for 8 hours a day.\n\nannual recurring revenue, one way to quantify the revenue of a company which is particularly popular among tech orgs.\n\nSome reasons why I think the link between development velocity and revenue is weaker than one might think:\n\nNaturally, when I make a claim like \"an acceleration may counterintuitively lead to a decrease in a company's performance\", I should ensure to check whether this would imply that *artificially slowing down a company* would be *good* for it. This would seem like a pretty wild claim. And I doubt it is typically true. But if slowing down is bad, then shouldn't speeding up be good, after all? Or why would pre-coding-agent tech orgs be at some optimum where neither a speed-up nor a slow-down would improve things? Well, firstly, they might be close to such an optimum, for the \"they evolved over decades into the shape they have today\" reason. Accelerating *parts* of the system without the rest of the system being able to catch up may indeed have an overall negative effect. Secondly, it could be that *some* (well-dosed) acceleration would be good, but \"everyone use coding agents and get 10x as fast\" + \"even non-tech people should start shipping things to prod\" does not seem, to me, like the kind of acceleration with such positive properties.\n\nSeems unlikely in the current paradigm, as this could easily reach tens of billions of tokens or so.\n\nSome exceptions exist, like Anthropic itself, which does seem to possibly have reached much higher development velocity in some areas, although I'd still say it's unclear a) how sustainable this practice will turn out to be or them, b) how big a part this plays in their extreme revenue growth, and c) to what degree their development model could be applied to other tech companies or is pretty specific to their use case (e.g. my understanding is that the rapid development cycles mostly apply to Claude Code, which is a very new piece of software which they likely already started as something to eventually be developed mostly by AI - an advantage that most established companies with their legacy code don't share).", "url": "https://wpnews.pro/news/why-software-automation-is-hard", "canonical_source": "https://www.lesswrong.com/posts/KzXptMbuDsYLrppss/why-software-automation-is-hard", "published_at": "2026-06-06 08:56:05+00:00", "updated_at": "2026-06-06 09:21:52.040953+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/why-software-automation-is-hard", "markdown": "https://wpnews.pro/news/why-software-automation-is-hard.md", "text": "https://wpnews.pro/news/why-software-automation-is-hard.txt", "jsonld": "https://wpnews.pro/news/why-software-automation-is-hard.jsonld"}}