Ask a language model how many times the letter "r" shows up in "strawberry" and there's a decent chance it says two. Ask it to compound $2,500 at 7% for eight years and it might quietly switch to simple interest and hand you $3,900 instead of $4,295.47. The reasoning sounds fine. The number is wrong. And the model is exactly as confident either way.
This isn't a bug you can prompt your way out of, because it's baked into what the model is. A language model predicts the next token. When it writes 1234 × 5678 =
, it isn't running a multiplication — it's guessing the most likely sequence of digit-tokens given everything it saw in training. There's no carry logic, no place value, no actual arithmetic underneath. You get a plausible-looking number: right length, believable leading digits, often just wrong. Anything that needs exact calculation, careful bookkeeping, or counting lands squarely in the model's weakest spot.
Program-Aided Language Models (PAL) fix this with one move: don't ask the model for the answer, ask it for a program.
🖩 Interactive demo (CoT slips the arithmetic; PAL's code is really executed in your browser): https://dev48v.infy.uk/prompt/day27-pal.html
Plain Chain-of-Thought asks one system — the model — to do two very different jobs: work out the method and crunch the numbers. The method part it's great at. The number-crunching is the part it's worst at. So a single slipped multiplication poisons an otherwise sound line of reasoning.
PAL splits those two jobs apart. The model still does the reasoning — it decides which quantities matter, names them as variables, and writes the operations that combine them. But instead of stating a final number, it emits a short program and stops. Then a real interpreter runs that program and produces the number.
Compound interest, the PAL way:
principal = 2500
rate = 0.07
for y in range(8):
principal *= (1 + rate) # interest ON interest
print(round(principal, 2)) # 4295.47
The model never computed 1.07**8
. It described how to compute it and let Python do the arithmetic. That's the whole trick: the part the model is bad at is delegated to a system that's exact by construction.
It's deterministic. An interpreter returns the same value every time — no sampling, no temperature, no drift. Run it once or a hundred times, you get 4295.47.
It's verifiable. A wrong Chain-of-Thought is a wall of prose you have to re-check by hand. A wrong PAL program is a bug on a specific line. You can read it, unit-test it, diff it against a spec, or run it on new inputs to confirm the method generalises.
The model can be a little sloppy and still be right. As long as the method in the code is sound, the exact arithmetic is handled downstream. That's a much lower bar than getting every multiplication right in your head.
Think of PAL as Chain-of-Thought where the chain is executable. Same reasoning skeleton — break the problem into steps — but the steps are code, and something actually runs them.
If you've seen the ReAct pattern or function calling, PAL will feel familiar. It's a specific, powerful case of tool use where the tool is a code interpreter and the model's "action" is emitting a program. This is exactly what a modern code-interpreter feature does: the model writes Python, a sandbox runs it, the output comes back. Framed that way, PAL drops straight into agents — for any sub-task that's cleanly computable (math, date arithmetic, parsing, tallying), the agent reaches for the code tool instead of doing it in its head. The generalisation is simple: don't reason about what you can compute; compute it.
PAL means executing text a model generated, so treat it as untrusted input. Never eval it in your own process with full privileges. A bad or manipulated program could read secrets, hit the network, delete files, or loop forever. Run it in an isolated sandbox — a locked-down subprocess, a container, a restricted VM — with no filesystem or network access, a memory cap, and a hard timeout. Capture only the output.
And PAL has limits. It only helps when the problem can be formalised as code: math, logic, conversions, dates, counting, data manipulation. It does nothing for taste, judgment, or genuinely open-ended questions. It also won't save you from a bad plan — if the model picks the wrong formula, the interpreter will compute the wrong answer precisely and confidently. PAL removes arithmetic slips, not reasoning slips.
The rule of thumb is easy to remember: if a calculator or a five-line script could solve it, let the model write that script — and let the machine that's good at math do the math.
Play with all three problems in the demo: https://dev48v.infy.uk/prompt/day27-pal.html