Using semantic benchmarks to build a self-improving text-to-query agent

Conversion, a marketing intelligence company, announced it built a self-improving text-to-query agent that converts natural-language audience descriptions into editable filters, using a tool-using agent, an intermediate representation, a deterministic compiler, and a semantic benchmark called Statement Bench. The agent was tested against eight models, including Claude Opus 5, Kimi K3, GLM 5.3 Flash, and the Gemini 3.8 Flash release from this morning, with results indicating the task was more complex than standard structured generation.

Most of our recent AI work at Conversion has focused on general marketing intelligence. Marketing automation teams do a wide range of work across systems: researching accounts, building audiences, planning campaigns, writing content, and acting on performance data. We have been building agents that can reason across those workflows and use the same tools that a skilled marketer would use. Those systems benefit from capable, general-purpose models. The work is open-ended, and good judgment is often more important than completing a task quickly. But we also had a backlog of smaller, more focused AI features. One was natural-language filters: let a user describe an audience in plain English and turn that description into a filter they could inspect and edit in Conversion’s existing statement builder. In Conversion, a filter is called a statement. At first, this seemed like a straightforward structured-generation task. Give a model the available fields, describe the output format, and ask it to produce JSON. It turned out to be considerably more difficult than that. Take the following example: Find contacts who submitted the demo form at least once in the last 30 days and work at a software company with an open opportunity worth more than $50,000. This requires the system to: - Find the specific form the user means by “the demo form” - Determine which field represents a company’s industry - Learn how that workspace represents “software,” which means looking at the values actually stored in that field rather than guessing - Traverse from a contact to its company and then to that company’s opportunities - Ensure that “open” and “more than $50,000” apply to the same opportunity - Apply a relative event window It also needed to do all of this quickly enough to feel like a filter interface, not a research agent. What looked like a small prompt-engineering task had become a constrained text-to-query problem. Solving it required a tool-using agent, an intermediate representation IR , a deterministic compiler, and a semantic benchmark. We ran eight models through the resulting benchmark, Statement Bench, including Claude Opus 5, Kimi K3, GLM 5.3 Flash, and the Gemini 3.8 Flash release from this morning. Results are below. Giving the agent tools Most of the information needed to answer the request above is specific to the customer’s environment. A single workspace can hold hundreds of millions of historical field values, along with its assets and objects. For obvious reasons, we could not put all of that into one prompt. Our first useful architectural decision was to stop treating the problem as ordinary structured generation. Instead, the model receives a small set of tools. It can search fields, inspect historical values, and resolve business-specific assets such as forms, campaigns, emails, and audiences. It uses those tools only when the request requires them. Much of this search infrastructure came from our recent Global Search work, which provides text and semantic search over all records in Conversion. We plan to share more on that soon The basic flow looks like this: Natural-language request | v Tool-using agent <-----------------+ / | \ | fields assets relationships | rejection with reasons \ | / | v | Constrained IR | | | v | Validator and compiler ---------------+ | v Production statement This keeps the initial context small. It also makes failures much easier to understand. If a statement is wrong, we can determine whether the agent found the wrong asset, selected the wrong field, misunderstood a relationship, represented the correct idea incorrectly, or exposed a bug in the compiler. That distinction later became important for our evaluation loop. Creating a smaller language Tool use solved the context problem. It did not solve latency. One lesson from early feedback: users tolerate far less latency in a purpose-built interface than in chat. This points to a broader paradox. We set latency expectations based on how difficult a task feels to us, not how difficult it is for the system. Writing content feels difficult because we can see the work. Describing a filter feels simple because our minds silently resolve context, entities, relationships, and intent. For the model, reconstructing those hidden assumptions is the task. The less work the user perceives, the less time they give the system to do it. Based on early feedback, we set two goals: more than 95 percent accuracy and a response time around 5 seconds for common queries. Conversion has an expressive internal query language. In our early tests, using the production format directly, only the largest models such as Claude Opus could generate it reliably. Even simple statements took around 45 seconds. The visual statement builder exposes only a subset of the full language. We created a smaller, agent-friendly intermediate representation for that subset. Smaller models could produce it using fewer tokens, while a deterministic compiler handled the full production format. Consider the statement: Job title contains “Director.” The original production statement looks like this: { "type": "LOGICAL", "version": 1, "logical": { "operator": "OR", "operands": { "type": "LOGICAL", "version": 1, "logical": { "operator": "AND", "operands": { "type": "VARIABLE", "version": 1, "variable": { "variableSchemaId": "550e8400-e29b-41d4-a716-446655440000", "where": { "type": "LOGICAL", "version": 1, "logical": { "operator": "AND", "operands": { "type": "LOGICAL", "version": 1, "logical": { "operator": "CONTAINS", "operands": { "type": "ATTRIBUTE", "version": 1, "attribute": { "name": "value" } }, { "type": "CONSTANT", "version": 1, "constant": { "value": "Director" } } } } } } } } } } } } The model-facing representation of the same filter is: { "field": "550e8400-e29b-41d4-a716-446655440000", "op": "contains", "value": "Director" } The IR has already been through several generations, and the latest was shaped by watching small models fail on the earlier ones. One large improvement was introducing better same-record semantics something schema validation cannot catch : { "related": "OPPORTUNITY", "all": { "field": "<stage uuid ", "op": "equals", "value": "Closed Won" }, { "field": "<amount uuid ", "op": "gt", "value": 100000 } } This split between model and code gave us a few useful properties: - Unsupported statements are difficult to express - Same-record relationship semantics are visible - Field and relationship references can be validated - The compiler can be tested independently of the model - Generated statements remain editable in the existing UI The IR ultimately reduces the model’s job: the agent resolves the user’s intent and produces a constrained plan; code handles the production format. Building a semantic benchmark An output can be completely valid and still be wrong. Take this request: Contacts at companies with a won opportunity worth more than $100,000. A contact belongs to a company, and a company can have many opportunities. Matching this filter means traversing relationships contact to company, company to opportunities and checking two conditions along the way: the deal is won, and the deal is worth more than $100,000. The difficulty is that those conditions have to hold for the same opportunity. If they’re checked independently, a company with a won $20,000 deal and an open $150,000 deal satisfies both: one condition matches each. Schema validation will never catch this. Once a few examples like this passed, editing the prompt risked regressing them. We needed a way to check meaning, not just validity, and to check it every time something changed. We built Statement Bench around the behaviors of the product, derived from anonymized audience patterns our customers had previously built. The suite now holds 100 cases across fifteen categories such as plain field conditions, events, relative and calendar time windows, relationships, and compound queries. Each case runs against a realistic workspace sandbox. The agent receives the same data and tools it receives in production. The evaluator checks several layers: - Did the agent return a statement? - Does the IR satisfy its schema? - Do the referenced fields and relationships exist? - Can the statement compile and pass production validation? - Does it represent the requested meaning? - How many model steps, tool calls, tokens, and rejected submissions did it require? The fifth is the most interesting one since validity does not guarantee semantic equality. The semantic checks read the compiled statement, asserting things like “one opportunity condition carrying both the stage and the amount,” “an email event whose type is a click, not an open,” or “a webinar condition rather than a custom-campaign one.” Running an eval-driven optimization loop The benchmark changed how we could continue working on the feature. Instead of asking a coding agent to “improve the prompt” or “implement a new IR,” we could give it an executable definition of improvement. The loop looked like this: - Run the benchmark - Group failures by their underlying cause - Inspect the agent’s tool trajectory and submitted IR - Change the prompt, tools, validators, or compiler - Run the full benchmark again - Keep the change only if it improves the system without introducing regressions Coding agents could use the benchmark to compare models, experiment with the IR, improve tool descriptions, and refine the prompt autonomously. Running the full suite after every change also kept us from overfitting to individual failures, and we held out a further 50 cases to confirm that. A few changes improved results the most: Move paths, types, and structure into the compiler. Our first IR made the model write out all relationships explicitly: contact to company, company to opportunity. The field’s metadata already implies that path, so the compiler now infers it. We did the same for dates, type casting, negation placement, and group nesting. Moving rules into the compiler simplified the IR and reduced schema failures. Reject with explanations and fixes. Every schema and compiler rejection says what to write instead when available : “gt cannot be negated on this field; use lte”, “copy the id from campaign list”. Small models converge in a retry or two, and the production model is rejected on a few requests per hundred. Structure the prompt for small models. Reorganizing the prompt did not change accuracy, but it halved the number of retries, which directly improved latency. This was inspired by Anthropic’s Prompting best practices https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices . Use examples over prose. Two additional examples in our format reference resolved a class of misses that paragraphs of explanation had not, cutting rejected submissions roughly in half. Give complete context or none. Models reach for what is in context before they call a tool. When the context included a partial or unlabelled set of fields, the model used the nearest one rather than searching, producing semantically incorrect statements. By reducing partial context in favor of tool calls, we increased build rates and cut input tokens by a fifth. The final production configuration, GLM 5.3 Flash, completed all 100 benchmark cases with a median latency of 2.3 seconds and a 95th percentile of 7.1 seconds. And, 97 of the 100 were semantically correct. Compared with the original production-format approach, simple filters had moved from roughly 45 seconds to a little over one second at 1/20 the cost. Comparing models on Statement Bench The benchmark also gave us a way to compare models on the actual task. On September 2, 2026, we ran the same 100 cases across eight models. Each model received the same prompt, tools, IR, compiler, and 30-second request timeout. Provider routing, prompt caching, and temporary inference load all affect latency. | Model | Valid builds | Semantically correct | P50 latency | P95 latency | Cache read | Tool calls | Rejected submissions | Est. cost per 1,000 requests | |---|---|---|---|---|---|---|---|---| | Claude Opus 5 | 100/100 100% | 100/100 100% | 3.16s | 8.53s | 91.1% | 162 | 0 | $27.51 | | GLM 5.2 | 100/100 100% | 100/100 100% | 4.38s | 13.02s | 93.5% | 201 | 5 | $14.94 | | Kimi K3 | 100/100 100% | 100/100 100% | 5.17s | 11.84s | 34.3% | 157 | 0 | $48.51 | | GLM 5.3 Flash | 100/100 100% | 97/100 97% | 2.34s | 7.07s | 92.8% | 163 | 2 | $1.33 | | DeepSeek V4 Pro | 96/100 96% | 96/96 100% | 5.53s | 24.31s | 47.8% | 172 | 1 | $12.00 | | Gemini 3.7 Flash | 77/100 77% | 77/77 100% | 15.14s | 30.01s | 26.5% | 228 | 1 | $18.24 | | Gemini 3.8 Flash | 76/100 76% | 76/76 100% | 14.29s | 30.01s | 35.4% | 266 | 1 | $27.44 | | DeepSeek V4 Flash | 56/100 56% | 55/56 98% | 6.79s | 30.00s | 41.5% | 100 | 1 | $0.56 | Estimated costs are per 1,000 attempted requests using observed input, cached-input, and output tokens at each provider’s listed non-promotional rate on September 2, 2026. Cached input is billed at the published cache-read rate where the provider publishes one, and at the full input rate otherwise. A few findings stood out. Neither model size nor price predicted latency. The fastest model was the smallest and cheapest. The second fastest was the largest and most expensive. Failure has moved from wrong answers to slow answers. Six of the eight models were semantically correct on every statement they finished; the differences between them are almost entirely in how many requests finished inside the timeout. In early iterations of the IR and prompts, most smaller models failed the benchmark at the build step with under 50 percent semantic accuracy. Reasoning tokens outweigh tool calls. Gemini 3.8 Flash spent 180,000 of its 192,000 output tokens on reasoning and made 266 tool calls; Claude Opus 5 spent 813 tokens on reasoning, made 162, and finished every case. Our Global Search efforts reduced each tool lookup to the millisecond range, so the remaining cost is the model’s turn between them. Takeaway Models are good at resolving ambiguity, code is good at enforcing precision, and most of our early failures came from asking the model to do both. Building this agent was the work of deciding which of the two should own each part. We expect the same is true of text-to-SQL and most other natural-language interfaces. If you’re interested in any of these problems, reach out We’re hiring.