The expanding scope of coding agents #
Published on
It’s time for another post about LLMs! I know, I know, that’s basically all anyone talks about these days. But that sort of makes sense: it’s what a significant number of software engineers spend their day using.
And they’re changing so rapidly. Back in May last year, I said: But how good is it at actually writing code? It’s like having a keen but not particularly thorough Junior Engineer at your beck and call. If you give it a clearly defined task and guidance on how to implement it (and maybe some feedback as it suggests changes), it’s more than capable of doing it. If you don’t give it enough guidance it tends to go more off the rails. I tried having it generate a simple application from scratch with minimal technical guidance and no review of what it was doing, and it made such a mess of it I decided it was quicker to throw it away and start again by hand.
What I was describing there basically amounted to “the coding agent can be responsible for a function, or a group of functions if they’re described well enough”. That post was published at about the time that Sonnet and Opus 4 were released, and there have been several big releases since then.
Graduating from functions to features
Now, I didn’t stick with Claude through all those shiny new releases. Anthropic effectively banned using third-party harnesses with their subscription plans, and did it in a way that rubbed me the wrong way. As far as I’m concerned, I’m paying for inference-as-a-service: it shouldn’t matter if I do that via Claude Code, or OpenCode, or something else. But they disagree. So I jumped ship to Z.ai’s GLM coding plan. At the time I signed up, the cost for a year-long subscription was about the same as a month of Claude, with higher limits.
The GLM models definitely weren’t (and still aren’t) as good as the Claude series, but they were mostly good enough. But as new models were released, the capabilities slowly ratcheted up. I noticed the unit of work I gave them graduated from functions to small, self-contained features. Instead of describing exact implementation steps, I was describing the functionality I wanted. I didn’t stop providing guidance on how to do it, but I was increasingly happy giving it larger chunks of work and getting something reasonable back.
Around the same time, the cost of some of the models dropped quite a bit. Between a new
flash model, token discounts, resets, and so on, it got to the point where I couldn’t hit the
plan limits even if I tried. So I was throwing more and more work at it: having two to three things
going at the same time, with me only dropping in at the end to review and ask for follow-ups.
From features to artefacts
The GLM models weren’t quite good enough to tackle larger units of work. I tried going back to Claude to see what it was like now but bounced off because its output was incredibly painful to read. I wanted to try Fable, their next-tier model, but you can’t do that without one of the amazingly expensive plans or paying API costs directly. Neither of those options is particularly appealing to me. Meanwhile, OpenAI launched Astra, their Fable-equivalent. And you could use it in small amounts on the less expensive plans.
I played around with Astra on a few small things, but I was conscious of the low limits. Plus, GLM-5.3-flash mostly did the bits and pieces that I wanted. I figured I’d give Astra a proper try the next time I had a larger project. Fortunately, some packet loss gave me that opportunity very soon.
I use Navidrome running on a remote server to host all my music, which is fantastic. Until your connection goes a bit wibbly. Apparently, the music player I was using didn’t believe in buffering, so when I developed 20% packet loss to the server the music kept pausing, playing a second, pausing, playing another second, and so on. Pretty much the most infuriating way it could have handled it. Writing my own music player had been at the back of my mind for a little while, and suddenly I had both impetus AND a shiny new tool.
I launched pi, switched over to Astra on low thinking mode, and gave it this prompt:
This is a new project. It’s going to be a native music player written in Go using the Gio library, backed by a Navidrome server. Think foobar2000, but Linux-native, Go, and Navidrome. It doesn’t have to do everything right away, but an MVP should be able to: connect to navidrome (saving credentials), show playlists, play a playlist in order. skip tracks, /un, adjust volume, etc. It should be tolerant to network issues – buffer a good few minutes of playback data, or an entire track or two. I have two other UI projects using Gio for reference, see ../gtodo and ../glauncher. For actual playback we may want to consider using an established backend like mpv, but that’s not a hard recommendation. For this project you are the overseer/orchestrator. You should use subagents using the zai/glm-5.3-flash model to do recon, implementation, debugging, etc. You can ask me questions if you have any.
I’d seen a few comments from people doing a similar kind of orchestration, and references to tools like prime agent that have first-class support for the workflow. It seemed like a natural fit for my case: Astra can do the orchestration, architecture, and review, but the much cheaper GLM models can do the actual heavy lifting. It asked me a few questions, and an hour later a fully functioning music player popped out:
It was exactly what I asked for. A one-paragraph prompt producing a fully functioning piece of software kind of blows my mind. And that was with Astra on low thinking, on the £20/month plan, orchestrating cheaper agents. There was a fair bit of judgement encoded in the prompt, and “think foobar2000” carries a lot of information in just two words, but it still felt very light on the details compared to what I’d usually write. Just a description of what I want in, and a binary out.
I’ve been using gmusic for a week now, and only had to make a few minor additions to go from “MVP” to “comfortable daily driver”: scrobbling support, media keys,
and remembering playback/playlist position across restarts<sup>1</sup>.
Cattle, pets, prompts, and saying no
The ability to generate a binary from a paragraph of text changes the way I look at software. Sysadmins often speak of treating servers like “cattle, not pets”: if one server develops a fault, you just tear it down and automatically rebuild or redeploy it. It holds no special value, so you don’t need to slowly nurse it back to health, or spend ages recreating it. I think that sort of attitude now applies to software.
If it turns out gmusic has some horrible architectural flaw — and it might, because I haven’t even glanced at the source — then I can just throw the whole thing away, feed in the same prompt with an additional bit on how to avoid the same mistake, and get something similar enough back. There are some limits here: the “paragraph-to-artefact” pipeline is stochastic, so you might in fact not get something similar enough back. And you have to be willing to throw the whole thing away and wait for a replacement. No big deal for a music app, but I wouldn’t want to take that attitude with a business-critical application. There’s also an interesting wrinkle in the other direction: model capabilities are advancing so much that you might want to throw it out and regenerate it again. Could “regenerate the app from the prompt” become the new “install updates”?
One thing to fall out of that view is that documented intent becomes king. Your prompt and subsequent corrections and additions are more important than the generated code. I like the idea of keeping some kind of decision log that future agents can be pointed at, but I’ve not yet found a good way to do so. Getting the agents to write it out reveals some obvious flaws: they’re very bad at distinguishing deliberate decisions from incidental implementation details; they’re also bad at picking the right level of detail, and seem chronically incapable of breaking with previously documented behaviour.
Agent-managed documentation, in fact, seems to end up resembling a cult. The first pass is probably OK-ish. Then the next agent to touch it will be compelled to extend any list, timeline, or history with whatever it’s working on. Over time, a superfluous comment about how something was made will undergo a gradual epistemological transformation until it’s treated as a commandment from on high by all future agents. It’s a mess.
I think this is part of a bigger problem: models are still way too sycophantic. There’s very little pushback against either the user or the documentation. If I asked an agent to add a feature to gmusic that analyses a webcam image of my toaster and alerts me when the toast is ready, it’d just do it. If you’re not careful, your software will gain 7 kitchen sinks and 3 Swiss Army knives, and at no point will the agent suggest splitting them out, or refactoring. Before you know it, your cattle have mutated into grotesque pets with far too many limbs.
The current generation of models is almost certainly capable enough to push back on bizarre features, stop scope creep, ask questions to determine how robustly to build something or how to handle edge cases, and so on. You can probably even prompt them in a way that they’ll do it, but in their normal state the RLHF<sup>2</sup> seems to have bulldozed over all that in favour of being a yes man.
From artefacts to systems?
I think the current state of agents is that they can own “artefacts”, having graduated from functions and features. It’s a very deliberate choice of words: they can’t own the full application or system because they won’t independently step back and make the necessary judgement calls.
It’s tempting to imagine a perfect little software factory, where you have an agent in charge of product management, one in charge of architecture, one that manages the documentation, and so on, all with their own points of view and priorities in their prompts. If you’re not careful, you’ll soon be talking about polecats and deacons and writing very large cheques to frontier labs.
I do think there’s some benefit in this kind of orchestration, but there’s also a limit to how much adding agents wearing different hats will help. And there’s also a limit to how much I’m willing to spend on inference! I have got good mileage out of a dedicated code review agent. It often catches one or two problems that the other model overlooked, so I get an obvious and immediate “return on investment”: a few pence spent running deepseek-4.1-flash with a code review prompt earns me fewer bugs.
I think that’s basically the cut-off for me right now. I almost want a “documentation review” agent to try and manage some of the problems I mentioned earlier, but I don’t think it’s worth it. Maybe it’ll make a fun experiment as local models get more capable.
Determinism instead of prompts
One thing that interests me is how much of this problematic behaviour we can get rid of without relying on the model to remember and follow instructions. I had an issue with agents making a mess of the git history with non-fast-forward merges from subagents. The “natural” fix is to prompt the agent not to do that, but that relies on it following that one particular instruction. It doesn’t always do that. Instead I made a simple pi extension that checks the git history and tells the agent to fix it if it’s broken. That has had a 100% success rate so far.
Similarly, in the gmusic prompt I told the agent what model to use for the subagents. After doing this for a while (and promoting the instructions to a skill instead of something I type out every time), I noticed it would occasionally just use a different model. So now I have another pi extension that defines subagent roles using the models I want it to use, and the agents can’t just do their own thing.
This makes me wonder if the next step to expanding agent capabilities isn’t necessarily more capable models, or more agents in different hats, but more rigour in software engineering. If we had a way to accurately assess the need for refactoring, or the quality of documentation, then we could immediately put hard-and-fast rules in that the agents will follow. That’s a load-bearing “if”, though, if you’ll forgive me the Claude-ism.
However we reach it, the future where a prompt can become a fully formed, maintainable, secure, documented piece of software probably isn’t that far away. It’s going to be very interesting when regular users can just summon good software to do whatever they want. I’m not sure what that means for my career as a software engineer: there are still a lot of places in the process that benefit from knowledge, experience and judgement, but maybe that won’t be the case for much longer? There are interesting times ahead for us, either way.
Which actually fixed a long-standing minor annoyance for me: I have a “Daily Mix” playlist generated, well, daily. In my old player it would effectively queue the whole playlist, so when I opened it the next day I carried on listening to the old daily playlist. So instead of just pressing the play button on my keyboard, I had to find the app, click the new playlist, and then press play. Not a problem worth fixing in itself, but still a minor daily papercut that I’m glad to be rid of.↩︎ 2.
Or whatever other post-training is done these days. ↩︎
Related posts
Here are some semantically similar posts. Want to subscribe using a feed reader?
You can [build a feed](/feeds/posts/build/like/the-expanding-scope-of-coding-agents/) with just posts similar to this one.
[Standard feeds](/feeds/) are also available, but are approximately 42% less fun.
Coming around on LLMs
For a long time I’ve been a sceptic of LLMs and how they’re being used and marketed. I tried ChatGPT when it first launched, and was totally underwhelmed. Don’t get me wrong: I find the technology damn impressive, but I just couldn’t see any use for it. Recently I’ve seen more and more comments along the lines of “people who criticise LLMs haven’t used the...
Building a family of single-user services
Last month I made and wrote about a personal service to track my workouts and record personal bests. Since then, things have escalated. I now have eight different services running in the same “family”:
An app can be a ready meal
Three years ago I read “an app can be a home-cooked meal” by Robin Sloan. It’s a great article about how Robin cooked up an app for his family to replace a commercial one that died. It’s been stuck in my head ever since. It’s only recently that I’ve actually done anything like Robin described, though. Part of the reason was my brain got too hung up on the family...
It's time for another post about LLMs! I know, I know, that's basically all anyone talks about these days. But that sort of makes sense: it's what a significant number of software engineers spend the...