Can an LLM draw a high-level architecture diagram? I benchmarked six to find out

A developer built archdiagram-bench, a benchmark that tests whether large language models can produce usable high-level architecture diagrams from a real code repository, using draw.io XML output and a judge model validated against deliberately damaged reference diagrams. Across six models, scores clustered tightly between 0.38 and 0.44, with Claude Sonnet 5.5 and Claude Opus 5.5 tying at 0.44 despite Opus costing more than twice as much per attempt, while GPT-5.4 mini scored 0.38 at $0.04 per attempt. The author reports that the assumption a larger model produces a better diagram does not always hold.

This is a submission for the Kaggle Benchmarking Challenge https://dev.to/challenges/kaggle-2026-09-23 I've been writing design docs and decks at work, and I use LLMs to draw the architecture diagrams for them. The diagrams keep failing in one of two ways: So I built archdiagram-bench , a benchmark for one question: given a real code repository, can a model draw a high-level architecture diagram that a newcomer could actually use? The model gets a real repository BillTalk, a voice-operated billing app of about 80 files that I wrote , tools to list, search and read its files, and this request: Draw an architecture diagram that explains the high-level design of this code base to a developer who is new to it. - Show the end-to-end user flow from its starting point, for example an HTTP call or an interaction in an app. - If a database or other data store is involved, show the read and write operations. - Show every major part of the system and the interactions among them. - Group the parts that run or are deployed together, for example agents or services on the same host, or everything that runs on the user's device. It answers with draw.io XML, the format I'd actually paste into a design doc. I deliberately gave no box limit and no definition of "high-level". Judging the right level is the skill being measured, the same way I'd expect it from a colleague. Each model gets three attempts and the score is the average. A judge model is only useful if it notices what's wrong, so I tested the judge before scoring any model. I made copies of my reference diagram, each damaged in one known way: The judge has to mark each damaged copy below the clean reference, on the mark that damage should affect. The first version of the judge missed a defect. Giving it the same file tools the models have, plus a one-time study of the repository, fixed that: it then caught all five. The benchmark ships with this check as its own task, so anyone can rerun it when they swap the judge. | Model | Why | |---|---| | Claude Sonnet 5.5 | I usually use Opus, so I wanted to see whether Sonnet gives comparable results | | Claude Opus 5.5 | The one I use most. Does the most capable model actually draw the best diagram? | | GPT-5.5 | The same question for OpenAI's large model | | GPT-5.4 mini | A small, cheap model, to test the price trade-off | | Gemini 3.8 Flash | I rarely use Gemini and wanted to see how it compares | | Gemini 3.1 Pro | Google's larger model. It is also the judge, so read its score with that in mind | I normally reach for Claude and OpenAI models, so I wanted to see how Gemini compares. I also wanted to know whether the more expensive models are worth their tokens for an everyday task like this, so the lineup has a large and a small model from each provider where I could get one. | Model | Average score | Model cost per attempt | |---|---|---| | Claude Sonnet 5.5 | 0.44 | $0.19 | | Claude Opus 5.5 | 0.44 | $0.44 | | GPT-5.5 | 0.43 | $0.39 | | Gemini 3.1 Pro the judge | 0.43 | $0.25 | | Gemini 3.8 Flash | 0.43 | $0.22 | | GPT-5.4 mini | 0.38 | $0.04 | My reference scores 1.0. The best single attempt, from Claude Opus 5.5, scored 0.56. On average, Sonnet and Opus scored the same. These numbers are from my own runs of the task in the benchmark notebook. The Kaggle leaderboard runs the same task again for each model, so its scores will differ a little from run to run, and it may list fewer models while my quota catches up. This was the result I cared about most. Claude Opus 5.5 costs eleven times as much per attempt as GPT-5.4 mini, and the difference in score is 0.06, which is smaller than the variation between one model's own attempts. Sonnet matched Opus at less than half the price. For drawing an architecture diagram from a repository, the small models held their own. The assumption that a larger model will produce a better result doesn’t always hold, and that was something I learned along the way. Every model produced a valid, renderable draw.io diagram with a clear structure, which is a real achievement. But none came close to the reference. My sketch has about ten boxes; the models drew several times that, most with file names inside, and level of detail was marked at half or lower on every attempt. The clearest example: BillTalk's design is three voice agents, each with its own system prompt and logic, sharing one on-device model. In the diagrams I examined, the models didn't draw three agents. They drew three UI files, or one box called "system prompts". The benchmark has plenty of room for future models to improve. This is the insight I didn't expect, and it came from looking at where the diagrams and my sketch disagreed. "Component." My first prompt said "show every component". I meant an architectural building block: the database, an agent, the speech recogniser. But BillTalk is a React Native app with a folder called components/ , where the word means a piece of UI. The early diagrams had boxes for the files in that folder, down to the save button. "Belong together." My first prompt also asked to group components that "belong together". I grouped by where things run: everything on the user's device in one group. The early diagrams grouped by folder: UI layer, state layer, data layer. So I reworded both lines to what you see above: "every major part of the system", and groups defined by where things run. All the scores in this post are from that reworded prompt. Fixing the words wasn't enough. Every model still scored half or lower on level of detail, on every attempt. So the over-detail isn't a misunderstanding of a word. Given a repository, these models default to describing how the code is organised, and "high-level" on its own doesn't pull them away from that. A person reads "high-level design" and draws a dozen boxes. If you want that from an LLM today, be explicit: say roughly how many boxes you expect, give it a rough sketch or a list of the parts you want shown, and say that groups should reflect where things run, not which folder they're in. I looked at every picture next to its score, and that shaped the scoring more than anything else: The scoring I ended with agreed with my own judgement on the diagrams I compared by eye, including which one was best. This is version one. Here's what I'm building next: All the benchmark code is in the notebook, in plain cells you can read top to bottom. The dataset holds only the BillTalk repository and my reference diagram. Built with the kaggle-benchmarks https://github.com/Kaggle/kaggle-benchmarks SDK. Diagrams are rendered with the draw.io https://www.drawio.com/ viewer through Playwright.