Building a Production AI Agent in Spring Boot: A/B Testing Prompts With an LLM Judge (Part 9) A developer building a production AI agent in Spring Boot found that a system prompt change based on intuition lost in a pairwise A/B test against the old prompt, despite appearing better in manual checks. The test, using an LLM judge and the pairwise comparison pattern from the Spring AI LLM-as-a-Judge guide, revealed that the new prompt made answers terse and dropped order summaries. The developer advocates for pairwise comparison with position-swapping to mitigate judge biases, as documented in the MT-Bench paper. Last week I changed a system prompt based on a feeling. It was the first prompt change after the evaluation harness from Part 8 went live, and I was completely sure about it. The target was the markdown table. Part 8's first nightly run caught the agent answering price comparisons with a markdown table that renders broken in the chat frontend. The fix looked obvious: add one line to the system prompt demanding plain text. I checked six conversations by hand. All six looked better. I was ready to ship it to production. Then I ran the comparison the way Part 8 promised: the same 40 cases, the same judge, two prompts. The old prompt won. Not by a little. It won 18 pairs, lost 10, and tied 12, and the judge's rationales made the reason visible. The plain-text line had also made the agent terse, and terse answers dropped the order summary that customers actually need. My confidence was a sample size of one. The dataset was the jury. This part is about the pattern that settled that argument: pairwise comparison, the LLM-as-a-judge pattern for A/B testing prompts and tool descriptions before they reach production. It is the harness from Part 8, upgraded to answer "which version is better?" instead of "is this version good?" Every prompt edit is an experiment with one sample. You notice one conversation where the agent is verbose, you add "be concise", and the change ships because that one conversation got better. The dataset from Part 8 makes the agent measurable, but a nightly score cannot tell you whether a change helped. One night is noise, three nights is a signal, and by the time you have three nights of data you have already shipped the change to every user. The variable itself is the problem. A system prompt and a tool description are the two things in an agent you cannot unit test. Part 6 proved the code is bug-free. Part 8 proved the answers are good on a fixed dataset. Neither says anything about whether your new wording is better than the old wording, because "better" is a comparison, and a score against a rubric is not a comparison. The comparison needs a controlled experiment: one dataset, two configurations, one judge, run on the same day. That is exactly what the pairwise pattern gives you, and it is one of the two evaluation patterns the Spring AI LLM-as-a-Judge guide https://docs.spring.io/spring-ai/reference/guides/llm-as-judge.html documents. The guide defines the two patterns side by side: SelfRefineEvaluationAdvisor works this way: it rates a response, and if the rating is too low it retries with feedback. This is the pattern Part 8 used for the nightly metrics.For prompt changes, pairwise is the better tool, for two reasons. First, comparing is easier than scoring, and the guide makes that argument itself: evaluation is fundamentally easier than generation, and relative judgment is easier than absolute. "Which of these two answers is better?" is a simpler question than "Is this answer a 3 or a 4?", and simpler questions produce steadier judges. Second, rubrics drift. A "4" from last month is a "3" today, because the judge model updated or your bar moved. A pairwise verdict against the same fixed opponent does not drift. The catch is position bias, and it comes from the paper that founded this field. The MT-Bench and Chatbot Arena paper https://arxiv.org/abs/2306.05685 that introduced LLM-as-a-judge measured three systematic biases in judges: position bias a preference for whichever response appears first , verbosity bias a preference for longer answers , and self-enhancement bias a preference for answers that look like the judge's own style . The paper also proposed the mitigation I use: judge every pair twice with the positions swapped, and only trust the verdicts that survive the swap. A pairwise comparison is only valid if both sides ran on the same cases, the same tools, the same memory, and the same day. The baseline is not your memory of how the agent used to behave. It is a snapshot. I keep a baseline client built from the current production prompt, and a candidate client built from the edited one. Both share the same tool registry and the same conversation memory as the agent from Parts 1 through 8. The only difference between them is the line I am testing. If you change two things at once, you will never know which one moved the score. The snapshot has one extra requirement: save the responses and the tool call logs, not just the pass or fail. When a candidate loses, the losing responses are evidence. When a candidate wins, the winning responses are the new baseline's regression test. I store both under a timestamped directory: experiments/2026-08-06-shipping-tool/ . The candidate is the same agent with one prompt line or one tool description edited. Everything else stays identical. This sounds obvious and it is the rule I break most often, usually by "cleaning up" a second description while I am in there. Two edits, one experiment, zero information about either. Spring AI does not ship a pairwise evaluator. The evaluation testing reference https://docs.spring.io/spring-ai/reference/api/testing.html documents two built-in evaluators, RelevancyEvaluator and FactCheckingEvaluator , and both are direct assessment. Pairwise is a pattern you assemble from the pieces Spring AI gives you: a ChatClient per configuration, the same dataset as Part 8, and structured output to parse the judge's verdict. The verdict is a record: public record PairwiseVerdict String winner, // "A", "B", or "TIE" String rationale // the judge's reason, kept for the report {} The judge prompt is my own template. The guide's best practices section sets the rules I follow: forced choice with a small number of options, a rationale, and a separate judge client. Mine asks for the winner and one sentence of reasoning: private static final PromptTemplate PAIRWISE PROMPT = new PromptTemplate """ You are comparing two assistant responses to a customer question in an e-commerce chat. Question: {question} Context the assistant had: {context} Response A: {responseA} Response B: {responseB} Pick the better response on correctness, factual accuracy against the context, helpfulness, and format. If they are equally good, or both unacceptable, answer TIE. Answer as JSON with two fields: "winner" "A", "B", or "TIE" and "rationale" one sentence . """ ; The judge call follows the same hygiene as Part 8: temperature 0.0, and a separate client from the agent's, because the guide's own example code comments "use separate ChatClient for evaluation to avoid narcissistic bias". Parsing uses Spring AI's structured output https://docs.spring.io/spring-ai/reference/api/structured-output-converter.html : PairwiseVerdict judge String question, List