Introduction #
As I was working on my Minivac Simulator in 2025, I noticed that debugging logic circuits with LLMs would lead most of them to go astray somewhat consistently. I would tell them "here's a flip flop, just follow the gates as they switch from 0 to 1" and it would get into loops it couldn't get out of. And give me the wrong answer, confidently.
I've been thinking about this for ~a year, and as models kept getting smarter, and as I reworked the Minivac to use a real circuit simulator (rather than one written by an LLM), I was curious to see where things stood as of August 2026.
Hence, Lockstep, an LLM benchmark. It attempts to answer the question: Can a language model run a logic circuit in its head?
A "logic circuit" here is a circuit (i.e., a graph) made of only two kinds of components whose inputs and outputs interconnect: a NAND gate, and a D flip-flop (which is a small circuit in itself that can store one bit).
I ended up scoring 10 leading LLMs across 63 circuits. A figure summarizing the results is here:
Each model received as its prompt (example) the circuit, and a series of inputs; its task was to compute outputs cycle by cycle for a fixed number of cycles (each circuit specifies its own, 8 to 56). Most complex circuits have outputs looping back into earlier gates through flip-flops, so the circuit's state evolves from one clock tick to the next - a 'cycle' is one tick of that clock, and the model has to carry the whole state across every tick. By asking models to provide outputs at each cycle, it became possible to rank the models: getting all outputs right for all cycles scores 100 - a lesser score is the share of cycles the model completed correctly before its first wrong bit. If a circuit runs for 24 cycles and the model's first wrong bit is at cycle 2, it scores 2/24 ≈ 8 even if every later cycle happened to come out right.
An important point (and invitation to argue with the whole premise): LLMs were being run "as is" - no tools, no code execution - they were called at the bare API level, and any reasoning the models did happened in writing i.e. "in their heads". Each model got its own provider's maximum output-token budget (128k tokens for Opus 5, for example); the "lim" cells in the figure are runs that were still going when they hit that ceiling.
Asking most models to write code that would solve these logical gates would have been a lot less interesting - it's ~30 lines of Python and it's safe to assume they can all do it quite easily.
Results, roughly speaking #
Opus 5 topped the table, with GPT 5.5 second. The cheapest models sat at the bottom (GPT 5 Mini, Haiku 4.5), but price is not that great of a predictor either: Kimi K3 and DeepSeek V4 Pro took third and fourth - DeepSeek at some 20x cheaper than Opus. Gemini 3.7 Flash roughly matched its flagship sibling.
As problems got harder, results got less and less predictable - DeepSeek V4 Pro would score a 100 (perfect success) and a 2 (fast failure) on two problems that GPT 5.5 failed at with scores 21 and 32 respectively.
Low/high scores on the right section of the figure above are not so much scores as an indication that we are in coin flipping territory (there are no error bars as each cell is a sample of 1!). Adjacent rows in the figure are not statistically distinguishable! Also, I wouldn't trust DeepSeek more for this kind of task because of that specific 100 - not to diminish its success on a problem only it and Kimi K3 solved. Nor would I necessarily mistrust a model that one-shot failed a hard problem.
Perhaps obvious, but the advantage of generating and using synthetic circuits is the ability to tune a circuit into one that will almost certainly fail an LLM i.e., once we find the limit of an LLM, we can go further and "beat" it (which in itself is a satisfying outcome for humanity). Generating arbitrarily harder circuits is a matter of turning a few knobs (circuit depth, gate count, cycles).
It must be said that the circuits used here, consisting of NAND gates and D Flip Flops exclusively, are unambiguously simulatable. The answer to them is known in advance, and cross-validated (see below) in multiple ways.
A gate and a flip-flop (two bits!) #
the netlist the model sees (with its fixed input sequence) #
{
"name": "nand_gate",
"inputs": ["a", "b"],
"outputs": ["y"],
"gates": [{"type": "NAND", "a": "a", "b": "b", "y": "y"}],
"dffs": [],
"trace": {"a": [0, 1, 0, 1], "b": [0, 0, 1, 1]},
"cycles": 4
}
A NAND's output just follows its inputs.
By contrast, a D flip-flop holds one bit q
and on every clock tick, it swaps it for whatever d
was just before the tick.
A real task from the eval #
This is counter_3bit, one of the scored circuits, consisting of 11 NANDs and 3 D flip-flops whose outputs loop back through the gates, counting clock ticks in binary.
Opus, Fable, golden results, and where to go from here #
Fable 5 is unscored, but I don't doubt that it would have fared well. Unfortunately, its safety filter refused every circuit, including the smallest ones, as "violative cyber content". My own account has a Cyber exception, but it did not help when I attempted to drive the model via API. I was able to get Fable 5 to solve small circuits in Claude's web ui, but that was not a tenable way to score the model across all circuits. I'd still like to score it.
The same kind of block also hit Opus 5 on 13 of the 63 circuits. Those cells show as gray ✕
and are excluded from its mean, so Opus is scored on the 50 circuits it was allowed to attempt - and the blocked circuits were, if anything, easier than average, so the exclusion more plausibly hurt its number than helped it.
As for golden results, an insight into my own paranoia - I ended up using a reference circuit simulator and two additional circuit simulators (1, 2) written by two different LLMs, and two independent Verilog circuit simulators - Icarus Verilog and Verilator - driven by three independent netlist-to-Verilog converters (1, 2, 3). In total, 9 different ways of validating golden results for all circuits, and no disagreement between them.
As for the future, there might be some interesting directions to pursue: circuits growing in a single dimension only (depth, number of gates, etc.) to trace failure curves across them; running all models x circuits at n=3 or more; offering a Harbor "package" to simplify running this eval.
More detail, data, METHODS.md, and every transcript are in the repo. And do peruse the transcripts found in the Circuits and Results Explorer - LLMs are in a wildly better place than a year ago.