Create a spec once from clear technical specs and then generate some high quality code. Sounds great, right. Reminds me of IaC, where you use a templating language to create infrastructure. Software as Code maybe. SaC anyone? Unfortunately, in practice it's not that straightforward. Thoughtworks have placed SDD into an "Assess" category and warned that it could be an anti-pattern for releasing software.
This article isn't about SDD. I'm more interested in discussing the output of SDD and how that is tested. Code can now be generated fast these days. So what better to test AI-written code than with AI itself. There are a lot of concepts and technical terms for the Quality Assurance part of AI generated code. One of these is the LLM-as-a-Judge idea. This idea is used to score the output of an LLM based on some explicit criteria. Traditionally, the way to evaluate an LLM was to judge its output on the helpfulness or faithfulness (using something called "exact-match" metrics). Sometimes it was usually down to a human to do this. It also changes the way that Quality is Assured when dealing with AI-written code.
Traditional QA is built on deterministic checks; either something does or does not fail. Something like expect(x).toContainText(y);
. A failing test means that something is wrong. Then the bug can be fixed in the code and the test will pass. However, the outputs of an LLM are probabilistic, so it breaks the traditional pass/fail model. This is where a judge comes in. Instead of pass/fail, it can assign a score based on how good the output is. This changes the focus of the team too, as now the focus becomes reliability and consistency and a continuous evaluation of the code.
Before I go into a few use cases and examples, I'd like to cover the term Golden Data Set and what it is. LLMs don't really have a good idea of what good looks like until you show it. There are different approaches to this which is where the Golden Data Set comes in. It is just a fancy way to say a versioned set of input/output test cases. The data set includes passes and fails, known issues, edge cases, and some of the tests have the expected outcomes and some do not. This data set can be taken from a working system whose outputs have already been tested and are known to be correct. These assertions can then be used in the first-pass testing of the AI-generated output. Then after each new version of the AI-generated code, we can compare the data sets and use those for testing the next version of the AI-generated code. Think of it like a growing versioned regression test suite. Importantly, the Golden Data Set needs to be curated by a human and it is best to use quality test cases rather than increase the quantity of test cases.
In this scenario there are 2 LLMs; one that generates the code and one that acts as The Judge. Every PR that touches a prompt, model or config is gated by an LLM Judge. The Judge can then be given rules to run on the output, as well as use the Golden Data Set as a reference to compare the results with. Instead of giving a score like 39/50 test cases passed, it will give a rating of confidence based on whatever was defined previously. Examples could be the quality, detail or even accuracy of the response. Usually, The Judge LLM would run at a temperature of 0 (0 meaning that there would be no creativity in the responses of the Judge). A threshold can be set that blocks the merge. The table below shows an example of what a CICD flow could look like, step-by-step.
| 1 | Trigger | A PR changes a prompt, model version or config | || | 2 | Testing | Tests run against a golden dataset | || | 3 | Output | Generator produces outputs | || | 4 | Scoring | The Judge does this | || | 5 | Checking | Scores are compared to a previous run, not evaluated in isolation. | || | 6 | Deploy | If a metric we care about regresses, the pipeline blocks merge/deploy | || | 7 | Logging | Store logs (timestamp, dataset version, judge model snapshot and judge prompt) |
There is a paper called "LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost" which as the title suggests was a study into LLM-judge framework scoring and a discussion of the idea of "execution-based vs. judgment-based testing". I'll leave a link at the bottom of the page.
100 Jira tickets were selected to simulate a realistic software development life cycle. Then using an LLM, Given-When-Then style BDD tests were generated for each of them. These tests were then reviewed by 3 senior QA Engineers experienced with the product that the test cases targeted.
The Judges, a selection of LLMs with different weightings and reasoning efforts (for example GPT-4, GPT-4o Mini and GPT-5), were tasked with "understanding" the tests. Each Judge received the text of the Jira ticket, the BDD test cases and a few guidelines on what different technical details meant (such as HTTP-methods). They could then assess whether the executed BDD tests address the specified requirements and output a coverage percentage and a written rationale about their scoring.
There is a team somewhere building an AI powered chatbot that offers a voice feature. The team are running single-turn eval tests, but customers are complaining the bot is going in circles and forgetting what the user has just said. The problem is that the team aren't using a good testing strategy. The single-turn eval tests are almost like individual unit tests. Just relying on unit tests to deliver quality software isn't a good approach. It's like grading a movie by looking at random frames instead of watching the film.
Below is a list of the important evaluation steps. As you can see, there's a lot of things to evaluate, so this is where the LLM-as-a-Judge can be used instead of manually checking.
Something that I haven't included in the list above is the testing of security in the response. This is because it should actually be a separate activity. There are 3 ways of testing evals:
The Judge is itself an LLM, so it inherits LLM vulnerabilities. Testing the adversarial-robustness of The Judge itself is also important through things like fake reasoning, prompt injection, adversarial phrasing and jailbreak-style manipulation. This is more down the path of Red Teaming activity which is getting off topic for this article.
At this point, you may be thinking "So you're saying that instead of minimising bugs, we're moving to the idea that we're probably going to find a few? That doesn't sound like a good way to do QA". And yeah, good point. With AI now in the loop of generating code it means that QA roles will change. Just as scripting made coming up with test scenarios and creating test data easier, and automated tests made running a suite of tests easier and deployments faster, LLMs become part of a test design approach.
Teams need to use different strategies to test AI-generated code. The fast, deterministic checks where "pass" is unambiguous are always going to be valuable, so they should be kept. These should be layered with a continuous cycle of evaluation. This means that teams should create Golden Data Sets and maintain them, as these collections will drift. The dataset evolves quickly as failures are discovered. Every time a new failure is discovered it should get added as a new case. This is the same philosophy as the pre-AI-coding way of managing suites of tests. The LLM-as-a-Judge model of testing introduces a different way of thinking about testing too. Every time The Judge prompt itself changes, old scores against the dataset aren't directly comparable anymore. In fact, whenever something changes, like the model's settings or preferences or even the model itself, then we need to set a new base-line for The Judge and the data sets.
Something I haven't covered explicitly is that this process has a sometimes mandatory requirement for a human to be in the loop, especially if the code is for an application (like in sensitive, highly regulated environments). Any failures in scores or drops in confidence need a human to investigate further. And that's a general theme in QA that always comes up; if software is being built for humans then a human needs to be in the loop at some point.
LLM-Judge Scoring: https://arxiv.org/abs/2512.01232