If you've spent any time on LinkedIn in the last year or so, you'll likely be quite familiar with a certain genre of post generally along the lines of "this new model changes everything. Everything you thought you knew about AI before this was wrong!" These have, of course, been a thing since at least 2022, and they belie a lack of understanding about what it is that makes these models less useful than they promise to be. The posts largely ignore the fact that much of the behaviour of interest to us is dependent not on the model, but on how the model is constrained, and that furthermore technology for constraining LLMs is not very good. Specifically, we have precisely one method of doing it: the prompt. To explain why this is so wrong, we're going to have to do some maths.
System behaviour is defined on the boundaries #
The key principle that we tend to miss in a lot of our discussions about LLMs and what they can and can't do is that we tend to talk about LLMs in an unconstrained abstract "what can LLMs do?" situation, with no constraints or system boundaries. In that kind of situation, it's quite easy to conclude that the damned things can, in theory, produce anything. This is, however, absolutely the wrong mental model to use. I hope my less mathematical readers will forgive me for this, because at this point we have to talk a bit about mathematics, and specifically the kind of continuum mechanics problems that I studied at university and still love. I assure you, I'll make it as painless as possible.
One of the simplest models we work with in continuum mechanics is the Heat Equation, which I've shown here in its one-dimensional form:
This equation describes how heat diffuses through a material in one dimension over time: think the kind of situation where you dip a metal chopstick into some very hot ramen and you want to know how the chopstick heats up over time (for the sake of the presentation here, we'll assume that the sides of the chopstick are insulated so that heat doesn't leak out along the sides of the body: obviously this doesn't actually happen, but I am not writing a full boundary element solver for one blog post). It's a very famous equation (Joseph Fourier developed the Fourier transform to solve it, in fact), and on a one-dimensional domain, it's fairly simple to solve. I'll not bore you with the details (the technique used is Separation of Variables, if you're curious), but the general gist is that we split the function
Which is a standard eigenvalue problem in one dimension. The interesting thing here is that there's not simply one function all valid solutions to the problem. This, as you might imagine, is a bit useless for actually getting anything done.
To actually get some useful results out of our problem, we need to not only specify the law of physics in question (the Heat Equation, which is our generic "this is how heat behaves in materials" law of physics), but also the conditions on the boundary; that is, at the ends of our thin rod. These can look like quite a few different things: the end of the rod might be insulated, meaning that no heat can leak out the end of the rod. The end of the rod might be set to be a given temperature (we might have one end of the chopstick in the ramen and the other at ambient temperature), or, from information about how metal radiates heat, it might leak a given amount of heat. Or we might be putting more heat into it. These all represent very different systems with very different behaviours, which you can play around with in the animation below:
The point of all of this is that how a system actually behaves in practice is determined by the conditions on the boundaries: if you know everything about the boundaries of the system and the applicable law of physics, you can predict everything about how the system works in the domain. The heat equation is obviously a very simple system, and the real world will naturally be much more complex (you can't tell exactly what's going on inside from the behaviour at the boundaries most of the time because the applicable law of physics is too complex, as it were), but this principle, in a more general sense, applies to almost everything: the constraints placed on a system at the boundaries are a key constraint on how it behaves internally. Any discussion of a new kind of system and its performance, then, has to take system boundaries into account, and not discussing them and the conditions on them is intellectually irresponsible.
Of course, it's probably quite hard to directly make the analogy from the Heat Equation to ChatGPT, so it's worth unpicking how behaviour at the boundaries of a system constrains a system that's a bit more complicated.
Software engineering as a constrained optimisation task #
As it happens, the design of a software system is precisely such a situation (in a slightly confusing use of language, the design of a software system is itself a system). Software design falls into a class of problems that we can broadly call constrained-optimisation problems: we have an objective function, or some number that we want to make as small as possible (in software this might be cost, response time, click-through-rate, downtime... some parameter like that) and some set of constraints: technologies used, required performance, data models, how the interfaces with other systems need to behave. What's important to note here is that the constraints dictate the eventual shape of the system far, far more than the objective function does: the edges of the system are clamped, effectively, we know what we put in and what we want to get out and how the boundaries are structured, and optimisation is a matter of finding the most efficient solution for this goal given those boundary constraints.
PostgreSQL is an excellent example of the pattern here: the core constraint on the system is that it has to be able to execute queries written in SQL. This, in itself, constrains the shape of the system: SQL works on relations, or collections of tuples, which means that the system's underlying data structures also have to be relations. You have some flexibility as to how you model them (row-first or column-first), but the underlying concept has to exist. You also have to, by virtue of the fact that the system ingests SQL, implement a SQL parser. I could go on. Philosophically, the core of this is I/O: all software systems, without exception, are constrained by how you get information into them and how the software system gives useful results back to the user. Once that's fully described, you generally find that there are only comparatively few shapes that the internals of the system can take. The central question of software engineering is to find the version of the internals that best satisfies the interface constraints.
Constrained optimisation problems are fairly well-understood. For simpler mathematical formulations we can find exact solutions using something like the Revised Simplex Method, and for more complex problems there's a whole family of techniques, many with terrifying names such as Simulated Annealing, to find a solution close to optimality. While we obviously can't just apply those to the production of software systems (there's no one single objective function that you can use to optimise the design of software), the art of software engineering is in many ways a set of intuitive heuristics that we use to solve the constrained optimisation problem that software is.
What's the boundary of an LLM? #
And now we can loop back to LLMs. LLMs are clearly a system: they have inputs and outputs, you can prod them one way and they will react in some way (subject, of course, to statistical uncertainty). We could thus do quite well in understanding LLMs by understanding the boundary of the LLM system. But what is that boundary?
An LLM as a system has as its input interface natural language. And there's the issue: the boundary of an LLM is, to a first order of approximation, basically every possible utterance in every language the LLM has been trained on. This is, to put it lightly, absurdly large. A boundary or an interface that can take on any value in any language is one that it's more or less impossible to understand deterministically, or even statistically. These things are basically impossible to apply constraints to in any meaningful way.
This is because the only thing we can pass to an LLM is, in fact, a prompt. We can't actually set hard constraints in the way you might in physics modelling: you can't clamp the edge of a plate in an LLM, as it were. Constraints, initial conditions, any restrictions on the code: they all have to be packaged into the prompt. And the prompt is a very limited tool: natural language just isn't precise enough, when all's said and done. Our goal when trying to solve what are broadly some flavour of constrained optimisation problem ("produce something that meets the following criteria", basically) using LLMs then becomes a process of finding a prompt that appropriately encodes all the constraints and desiderata that we have for the output. But that, of course, is in general a hard task.
Coding harnesses like Opencode are probably the closest we've come to effectively constraining an LLM in a way that can make it do useful work, and it's for this reason that the major application of LLMs in a way that isn't totally unserious (just mostly unserious, you understand) is in code generation. This is because even if we can't really constrain what code the LLM generates, we can absolutely constrain the interfaces of it by way of unit tests, providing clear APIs and other such affordances (even in the case of natural language, we have access to precise forms of the language for developing functionality that are significantly more specific than what you can write without that language subset). Between all of that, it's possible to get LLM output that's, if not exactly what we want, at least good enough to play a role in a system and mostly behave as expected. Anywhere else, though, the constraint is considerably weaker: it's almost impossible to describe a painting in sufficient detail in natural language that a human could reproduce it without seeing the image itself, much less a generative model.
In general, getting something halfway acceptable out of an LLM means running what is essentially a Monte Carlo simulation: you prompt the model over and over again, changing the prompt slightly each time, until you hit something that meets your acceptance criteria. The fact that software engineering has thus far been the major industry where LLMs have had real impact falls out of this quite naturally: a coding harness, for all the talk about how sophisticated they are, is basically a way to automate acceptance testing for LLM output. Doing that work by hand is, quite simply, an extremely painful and frustrating process. This is exacerbated by the fact that even with automation in play, a lot of acceptance checking has to be manual: you have to manually test software that an LLM wrote quite extensively to confirm that it actually does whatever the fuck you want it to do. To do that, though, you need to have quite a clear picture in your head of what you want, and while doing a task yourself is often largely a process of iteratively figuring out what you want as you build it, this isn't an option here. Mental fatigue sets in, and if you don't care about the work too much in the first place, it becomes very tempting to pass off something that looks right without thinking too hard.
Importantly, it's unlikely that more powerful models are actually going to get around this. The LLM, in this analogy, is somewhat akin to the laws of physics for a given physical situation: it's some approximation of the world of code or the world of text that has some internal laws of motion for how languages behave. If you give it a snippet of language, it can predict some plausible way in which the trajectory of the language snippet might continue, as it were. Some laws of physics, obviously, are more accurate than others: quantum mechanics describes subatomic phenomena better than classical mechanics, and one LLM might describe a given programming language or certain kinds of complex codebase better than another. The way we set boundary constraints, however, hasn't changed at all: it's prompts all the way down, and that is a fundamental limit on what we can expect to get out of these machines. And it doesn't really matter, in the end, if you have a perfectly accurate model of radio wave propagation from an antenna (for example) if you have no way of accurately describing the antenna's geometry. Until we figure out a way to actually do that in an effective and cost-efficient manner, what generative models can effectively do is going to be sharply limited.
The methods being used to solve this by the major LLM labs has, thus far, been to throw more compute at the problem, combine LLMs together in something akin to an ensemble model, and hope that between that and better LLMs being built (the hope is broadly that if the model is accurate enough you can compensate for boundary difficulties to an extent) the problem will become less noticeable. This has kind of (I stress, only kind of) worked, but at an exorbitant cost in compute that makes pushing further in that direction unsustainable in every sense of the term. And, so long as the only way to interact with or constrain with an LLM is the prompt, finding different solutions to the problem is going to remain difficult.
I leave it to the reader to draw conclusions from this about the consequences of this for the LLM industry and its customers.
We've recently released a new module for Arca, discussing the fundamentals of git (how to set up local and remote repositories, commits, SSH authentication, that kind of thing). The value of the New Zealand dollar has also dropped sharply of late, so if you've been thinking about getting access to these resources while supporting Arca's continuing development during early access, this would be an excellent time to do it (two or three one-year subscriptions will cover this month's power bill and rent for the week, which I feel would be a good thing to be able to do).