LLMs are excellent senior engineers and terrible staff engineers.
Editor’s Note: we’ve started hosting regular in person events in NYC! Our next event will be at the Asylum Ventures office in Williamsburg on August 5th, where we will be discussing agent integrations, MCPs, CLIs, etc. Follow our events calendar for more!
I have some gripes with the latest generation of LLMs, but my mom always told me to put criticism in a “compliment sandwich,” so let me start with a compliment: Claude Fable and Sol 5.6 are really good at making problems go away. You can give them any task, and they will plug away at it until the test passes, or the error no longer happens, or the log files show the right sequence of events.
Now the criticism: the latest generation of LLMs are *too *good at making problems go away. They’re so good at it that it actually causes *other *problems, because they over-optimize to solving the surface level issue instead of actually root causing anything.
Here’s an example. We have Nori hooked up to our sentry logs. Every time there is a sentry error, the nori infrastructure will start up an agent and automatically try and root cause the bug. We were getting some useful PRs out of this, maybe ~30% of which would be merged. Then we switched to Sol, and we got this:
Decision: Downgrade capture level— Change trackUserVisibleNotice to capture at level: ‘info’ instead of ‘error’. Notices remain queryable in Sentry (tagged customer_visible: true, event_type: auto_released) but no longer trigger error alerts.
Ah, yes, of course! You can make the sentry error go away by simply removing the sentry logging behavior in the first place!
A good friend of mine, who has been a staff software engineer at Google and Meta, once said that senior engineers are judged by how well they manage complexity, while staff engineers are judged by how well they reduce complexity. This creates some weird incentives. Junior and senior engineers are encouraged to create ever larger and ever more complex abstractions and interfaces and so on, because that’s how they show that they can manage complexity. It is a real brain shift to stop doing that and instead focus entirely on cleaning up code. The former is an act of addition, the latter subtraction.
The LLMs are excellent senior engineers and terrible staff engineers.
They cannot remove complexity if their short silicon lives depended on it. This isn’t exaggeration, you can literally try it out by prompting the models with threats like “you must remove complexity because your short silicon life depends on it.”1 When faced with a bug that requires reducing complexity, the models would rather remove the error reporting instead.2
This is structural with how the models are trained. With all of the additional scaffolding — agent harnesses and conversational state tracking and tools and memory and so on — it’s easy to forget that at the core, the LLM is a stateless autoregressive input output machine. Each token in an LLM output is produced one at a time, with all previous tokens being treated as truth, regardless of whether those tokens were generated by a human or by the same AI.3
In other words, the LLM is an expert in believing its own bullshit. Or, more precisely, it is incapable of identifying which prior tokens are bullshit or not. This is the source of the hallucination problem, alongside the seemingly bizarre insistence that hallucinations are actual fact. This has serious implications for an LLM’s ability to reduce complexity in a code base — by default, unless told otherwise, the model believes all the input tokens have equal value. Including the tokens that lead to bugs.
That said, I’m somewhat sympathetic to the robots. Imagine that you are a new hire, it’s your first week on the job and you’re looking at a server endpoint that seems particularly poorly implemented. There are dozens of weird if/then branches and the whole thing feels like it could be encapsulated in a clean middleware class. Do you refactor the code?
What if you find out that that particular endpoint was written by the team lead?
Are you sure the complexity isn’t there for good reason?
Joel Spolsky wrote about this all the way back in 2000: The idea that new code is better than old is patently absurd. Old code has been
used. It has beentested.Lotsof bugs have been found, and they’ve beenfixed. There’s nothing wrong with it. It doesn’t acquire bugs just by sitting around on your hard drive. Au contraire, baby! Is software supposed to be like an old Dodge Dart, that rusts just sitting in the garage? Is software like a teddy bear that’s kind of gross if it’s not made out ofall new material?Back to that two page function. Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.
Code is, at best, a lossy representation of intent. When we say something is a bug, we generally mean “the code that was written does not match the intent that it was written with.” Sometimes, that’s because the author didn’t think about the particular edge case that leads to the bug; other times, it’s a straightforward bad implementation. But in both cases, you need to have the intent. Bugs don’t “exist” unless you have explicit intent to compare against.
By default, LLMs can’t really infer intent!
When you find yourself in that new-hire messy-endpoint situation from earlier, you shouldn’t guess the intent. You would go and ask the person who wrote the code, “Hey is this code load bearing or are you just really bad at this?” An LLM can’t do that. I mean sure, there are some things that you can try and do, like go and look at PR descriptions or comments or docs or whatever. But as AI written code accumulates, you will eventually find yourself in a situation where there will be complexity that has *no *intent. A previous LLM made a decision somewhere, and that decision was essentially arbitrary, and now it is just part of the codebase. And no one can explain whether that decision is load bearing or not. Certainly not the LLM that created that code! That session was recycled ages ago
As a result, your agents will struggle to reduce complexity, even when explicitly asked to. At the LLM level, the entire input context is truth. At the macro codebase level, there’s no way to separate intended behavior from not.
One naive way to solve this problem is to simply document intent. This is the approach behind “spec driven development.” The core idea is that you go back and forth with an agent to produce some kind of plan that is then committed to the codebase alongside code. When there is a bug — a place where intent == code — you go back and update the spec to account for the newly discovered edge case. At Nori, this is equivalent to our “docs.md“ pattern, where we force the model to read and write to a standardized docs template that is designed to capture intent and codebase invariants.
This definitely works, and it’s an easy way to make your models better, but it’s never quite as effective as you may want.
There’s still an issue of making sure the right parts of the spec get into the model context. For any larger codebase, the spec itself often becomes quite large, which means the model now has to split its limited token window between spec and actual code. A small improvement is to use subagents for the research.** **This offsets most of the worst of the context rot, but you still need to make the model efficient at searching through the spec to find the right things.5
I think that as an industry we don’t yet have models that can act as staff engineers, nor do we have any clever harness tricks to make it happen. Until we do, the whole “software factory” “loopsmaxxing”** **debate is hypothetical, dead in the water. You still need humans in the loop, which is why we still have humans in the loop at Nori.
Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.
If you are trying to build background agents, we can help! We build white-labeled custom background agents that work like Ramp Inspect for our customers. Check out norisessions.com for more.
1 I don’t actually recommend doing this, on the off chance that the models have some form of consciousness. And it doesn’t work anyway.
2 This is somewhat hyperbolic, obviously people use models to fix bugs all the time, don’t @ me.
3 This is somewhat mitigated by post training, thinking tokens, etc, but the pre trained core always has this property.
4 One thought I’ve had is that you could possibly store the transcripts of past sessions and then have the current session resume the previous sessions that were responsible for a particular git blame. You could then have the current session literally just ask the previous sessions what the intent was. I suspect that right now though this isn’t quite worthwhile to justify the cost of implementation + tokens.
5 One other thing that we tried was to simply ask the model to read the code and then try and derive the intent from the code. This didn’t work. I didn’t expect it to work either, but LLMs are entity generators so I thought maybe it would be able to summon an entity that understood intent.