{"slug": "using-semantic-benchmarks-to-build-a-self-improving-text-to-query-agent", "title": "Using semantic benchmarks to build a self-improving text-to-query agent", "summary": "Conversion, a marketing intelligence company, announced it built a self-improving text-to-query agent that converts natural-language audience descriptions into editable filters, using a tool-using agent, an intermediate representation, a deterministic compiler, and a semantic benchmark called Statement Bench. The agent was tested against eight models, including Claude Opus 5, Kimi K3, GLM 5.3 Flash, and the Gemini 3.8 Flash release from this morning, with results indicating the task was more complex than standard structured generation.", "body_md": "Most of our recent AI work at Conversion has focused on general marketing intelligence.\n\nMarketing automation teams do a wide range of work across systems: researching accounts, building audiences, planning campaigns, writing content, and acting on performance data. We have been building agents that can reason across those workflows and use the same tools that a skilled marketer would use.\n\nThose systems benefit from capable, general-purpose models. The work is open-ended, and good judgment is often more important than completing a task quickly.\n\nBut we also had a backlog of smaller, more focused AI features. One was natural-language filters: let a user describe an audience in plain English and turn that description into a filter they could inspect and edit in Conversion’s existing statement builder. (In Conversion, a filter is called a statement.)\n\nAt first, this seemed like a straightforward structured-generation task. Give a model the available fields, describe the output format, and ask it to produce JSON. It turned out to be considerably more difficult than that.\n\nTake the following example:\n\nFind contacts who submitted the demo form at least once in the last 30 days and work at a software company with an open opportunity worth more than $50,000.\n\nThis requires the system to:\n\n- Find the specific form the user means by “the demo form”\n- Determine which field represents a company’s industry\n- Learn how that workspace represents “software,” which means looking at the values actually stored in that field rather than guessing\n- Traverse from a contact to its company and then to that company’s opportunities\n- Ensure that “open” and “more than $50,000” apply to the same opportunity\n- Apply a relative event window\n\nIt also needed to do all of this quickly enough to feel like a filter interface, not a research agent.\n\nWhat looked like a small prompt-engineering task had become a constrained text-to-query problem. Solving it required a tool-using agent, an intermediate representation (IR), a deterministic compiler, and a semantic benchmark.\n\nWe ran eight models through the resulting benchmark, Statement Bench, including Claude Opus 5, Kimi K3, GLM 5.3 Flash, and the Gemini 3.8 Flash release from this morning. Results are below.\n\n## Giving the agent tools\n\nMost of the information needed to answer the request above is specific to the customer’s environment. A single workspace can hold hundreds of millions of historical field values, along with its assets and objects. For obvious reasons, we could not put all of that into one prompt.\n\nOur first useful architectural decision was to stop treating the problem as ordinary structured generation. Instead, the model receives a small set of tools. It can search fields, inspect historical values, and resolve business-specific assets such as forms, campaigns, emails, and audiences. It uses those tools only when the request requires them.\n\nMuch of this search infrastructure came from our recent Global Search work, which provides text and semantic search over all records in Conversion. We plan to share more on that soon!\n\nThe basic flow looks like this:\n\n```\nNatural-language request\n          |\n          v\n   Tool-using agent  <-----------------+\n    /     |      \\                     |\nfields  assets  relationships          | rejection with reasons\n    \\     |      /                     |\n          v                            |\n    Constrained IR                     |\n          |                            |\n          v                            |\n Validator and compiler ---------------+\n          |\n          v\n Production statement\n```\n\nThis keeps the initial context small. It also makes failures much easier to understand. If a statement is wrong, we can determine whether the agent found the wrong asset, selected the wrong field, misunderstood a relationship, represented the correct idea incorrectly, or exposed a bug in the compiler. That distinction later became important for our evaluation loop.\n\n## Creating a smaller language\n\nTool use solved the context problem. It did not solve latency.\n\nOne lesson from early feedback: **users tolerate far less latency in a purpose-built interface than in chat.**\n\nThis points to a broader paradox. We set latency expectations based on how difficult a task feels to us, not how difficult it is for the system. Writing content feels difficult because we can see the work. Describing a filter feels simple because our minds silently resolve context, entities, relationships, and intent. For the model, reconstructing those hidden assumptions is the task. The less work the user perceives, the less time they give the system to do it.\n\nBased on early feedback, we set two goals: more than 95 percent accuracy and a response time around 5 seconds for common queries.\n\nConversion has an expressive internal query language. In our early tests, using the production format directly, only the largest models such as Claude Opus could generate it reliably. Even simple statements took around 45 seconds.\n\nThe visual statement builder exposes only a subset of the full language. We created a smaller, agent-friendly intermediate representation for that subset. Smaller models could produce it using fewer tokens, while a deterministic compiler handled the full production format.\n\nConsider the statement:\n\nJob title contains “Director.”\n\nThe original production statement looks like this:\n\n```\n{\n  \"type\": \"LOGICAL\",\n  \"version\": 1,\n  \"logical\": {\n    \"operator\": \"OR\",\n    \"operands\": [\n      {\n        \"type\": \"LOGICAL\",\n        \"version\": 1,\n        \"logical\": {\n          \"operator\": \"AND\",\n          \"operands\": [\n            {\n              \"type\": \"VARIABLE\",\n              \"version\": 1,\n              \"variable\": {\n                \"variableSchemaId\": \"550e8400-e29b-41d4-a716-446655440000\",\n                \"where\": {\n                  \"type\": \"LOGICAL\",\n                  \"version\": 1,\n                  \"logical\": {\n                    \"operator\": \"AND\",\n                    \"operands\": [\n                      {\n                        \"type\": \"LOGICAL\",\n                        \"version\": 1,\n                        \"logical\": {\n                          \"operator\": \"CONTAINS\",\n                          \"operands\": [\n                            {\n                              \"type\": \"ATTRIBUTE\",\n                              \"version\": 1,\n                              \"attribute\": { \"name\": \"value\" }\n                            },\n                            {\n                              \"type\": \"CONSTANT\",\n                              \"version\": 1,\n                              \"constant\": { \"value\": \"Director\" }\n                            }\n                          ]\n                        }\n                      }\n                    ]\n                  }\n                }\n              }\n            }\n          ]\n        }\n      }\n    ]\n  }\n}\n```\n\nThe model-facing representation of the same filter is:\n\n```\n{\n  \"field\": \"550e8400-e29b-41d4-a716-446655440000\",\n  \"op\": \"contains\",\n  \"value\": \"Director\"\n}\n```\n\nThe IR has already been through several generations, and the latest was shaped by watching small models fail on the earlier ones. One large improvement was introducing better same-record semantics (something schema validation cannot catch):\n\n```\n{\n  \"related\": \"OPPORTUNITY\",\n  \"all\": [\n    { \"field\": \"<stage uuid>\", \"op\": \"equals\", \"value\": \"Closed Won\" },\n    { \"field\": \"<amount uuid>\", \"op\": \"gt\", \"value\": 100000 }\n  ]\n}\n```\n\nThis split between model and code gave us a few useful properties:\n\n- Unsupported statements are difficult to express\n- Same-record relationship semantics are visible\n- Field and relationship references can be validated\n- The compiler can be tested independently of the model\n- Generated statements remain editable in the existing UI\n\nThe IR ultimately reduces the model’s job: the agent resolves the user’s intent and produces a constrained plan; code handles the production format.\n\n## Building a semantic benchmark\n\nAn output can be completely valid and still be wrong. Take this request:\n\nContacts at companies with a won opportunity worth more than $100,000.\n\nA contact belongs to a company, and a company can have many opportunities. Matching this filter means traversing relationships (contact to company, company to opportunities) and checking two conditions along the way: the deal is won, and the deal is worth more than $100,000.\n\nThe difficulty is that those conditions have to hold for the *same* opportunity. If they’re checked independently, a\ncompany with a won $20,000 deal and an open $150,000 deal satisfies both: one condition matches each. Schema validation\nwill never catch this.\n\nOnce a few examples like this passed, editing the prompt risked regressing them. We needed a way to check meaning, not just validity, and to check it every time something changed.\n\nWe built Statement Bench around the behaviors of the product, derived from anonymized audience patterns our customers had previously built. The suite now holds 100 cases across fifteen categories such as plain field conditions, events, relative and calendar time windows, relationships, and compound queries.\n\nEach case runs against a realistic workspace sandbox. The agent receives the same data and tools it receives in production.\n\nThe evaluator checks several layers:\n\n- Did the agent return a statement?\n- Does the IR satisfy its schema?\n- Do the referenced fields and relationships exist?\n- Can the statement compile and pass production validation?\n- Does it represent the requested meaning?\n- How many model steps, tool calls, tokens, and rejected submissions did it require?\n\nThe fifth is the most interesting one since validity does not guarantee semantic equality.\n\nThe semantic checks read the compiled statement, asserting things like “one opportunity condition carrying both the stage and the amount,” “an email event whose type is a click, not an open,” or “a webinar condition rather than a custom-campaign one.”\n\n## Running an eval-driven optimization loop\n\nThe benchmark changed how we could continue working on the feature. Instead of asking a coding agent to “improve the prompt” or “implement a new IR,” we could give it an executable definition of improvement.\n\nThe loop looked like this:\n\n- Run the benchmark\n- Group failures by their underlying cause\n- Inspect the agent’s tool trajectory and submitted IR\n- Change the prompt, tools, validators, or compiler\n- Run the full benchmark again\n- Keep the change only if it improves the system without introducing regressions\n\nCoding agents could use the benchmark to compare models, experiment with the IR, improve tool descriptions, and refine the prompt autonomously. Running the full suite after every change also kept us from overfitting to individual failures, and we held out a further 50 cases to confirm that.\n\nA few changes improved results the most:\n\n**Move paths, types, and structure into the compiler.** Our first IR made the model write out all relationships explicitly: contact to company, company to opportunity. The field’s metadata already implies that path, so the compiler now infers it. We did the same for dates, type casting, negation placement, and group nesting. Moving rules into the compiler simplified the IR and reduced schema failures.**Reject with explanations and fixes.** Every schema and compiler rejection says what to write instead (when available): “gt cannot be negated on this field; use lte”, “copy the id from campaign_list”. Small models converge in a retry or two, and the production model is rejected on a few requests per hundred.**Structure the prompt for small models.** Reorganizing the prompt did not change accuracy, but it halved the number of retries, which directly improved latency. This was inspired by Anthropic’s[Prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices).**Use examples over prose.** Two additional examples in our format reference resolved a class of misses that paragraphs of explanation had not, cutting rejected submissions roughly in half.**Give complete context or none.** Models reach for what is in context before they call a tool. When the context included a partial or unlabelled set of fields, the model used the nearest one rather than searching, producing semantically incorrect statements. By reducing partial context in favor of tool calls, we increased build rates and cut input tokens by a fifth.\n\nThe final production configuration, GLM 5.3 Flash, completed all 100 benchmark cases with a median latency of **2.3\nseconds** and a 95th percentile of 7.1 seconds. And, **97 of the 100** were semantically correct. Compared with the\noriginal production-format approach, simple filters had moved from roughly 45 seconds to a little over one second at\n1/20 the cost.\n\n## Comparing models on Statement Bench\n\nThe benchmark also gave us a way to compare models on the actual task.\n\nOn September 2, 2026, we ran the same 100 cases across eight models. Each model received the same prompt, tools, IR, compiler, and 30-second request timeout.\n\nProvider routing, prompt caching, and temporary inference load all affect latency.\n\n| Model | Valid builds | Semantically correct | P50 latency | P95 latency | Cache read | Tool calls | Rejected submissions | Est. cost per 1,000 requests |\n|---|---|---|---|---|---|---|---|---|\n| Claude Opus 5 | 100/100 (100%) | 100/100 (100%) | 3.16s | 8.53s | 91.1% | 162 | 0 | $27.51 |\n| GLM 5.2 | 100/100 (100%) | 100/100 (100%) | 4.38s | 13.02s | 93.5% | 201 | 5 | $14.94 |\n| Kimi K3 | 100/100 (100%) | 100/100 (100%) | 5.17s | 11.84s | 34.3% | 157 | 0 | $48.51 |\n| GLM 5.3 Flash | 100/100 (100%) | 97/100 (97%) | 2.34s | 7.07s | 92.8% | 163 | 2 | $1.33 |\n| DeepSeek V4 Pro | 96/100 (96%) | 96/96 (100%) | 5.53s | 24.31s | 47.8% | 172 | 1 | $12.00 |\n| Gemini 3.7 Flash | 77/100 (77%) | 77/77 (100%) | 15.14s | 30.01s | 26.5% | 228 | 1 | $18.24 |\n| Gemini 3.8 Flash | 76/100 (76%) | 76/76 (100%) | 14.29s | 30.01s | 35.4% | 266 | 1 | $27.44 |\n| DeepSeek V4 Flash | 56/100 (56%) | 55/56 (98%) | 6.79s | 30.00s | 41.5% | 100 | 1 | $0.56 |\n\n*Estimated costs are per 1,000 attempted requests using observed input, cached-input, and output tokens at each\nprovider’s listed non-promotional rate on September 2, 2026. Cached input is billed at the published cache-read rate\nwhere the provider publishes one, and at the full input rate otherwise.*\n\nA few findings stood out.\n\n**Neither model size nor price predicted latency.** The fastest model was the smallest and cheapest. The second fastest\nwas the largest and most expensive.\n\n**Failure has moved from wrong answers to slow answers.** Six of the eight models were semantically correct on every\nstatement they finished; the differences between them are almost entirely in how many requests finished inside the\ntimeout. In early iterations of the IR and prompts, most smaller models failed the benchmark at the build step with\nunder 50 percent semantic accuracy.\n\n**Reasoning tokens outweigh tool calls.** Gemini 3.8 Flash spent 180,000 of its 192,000 output tokens on reasoning and\nmade 266 tool calls; Claude Opus 5 spent 813 tokens on reasoning, made 162, and finished every case. Our Global Search\nefforts reduced each tool lookup to the millisecond range, so the remaining cost is the model’s turn between them.\n\n## Takeaway\n\nModels are good at resolving ambiguity, code is good at enforcing precision, and most of our early failures came from asking the model to do both. Building this agent was the work of deciding which of the two should own each part. We expect the same is true of text-to-SQL and most other natural-language interfaces.\n\nIf you’re interested in any of these problems, reach out! We’re hiring.", "url": "https://wpnews.pro/news/using-semantic-benchmarks-to-build-a-self-improving-text-to-query-agent", "canonical_source": "https://conversion.ai/blog/text-to-query-agent/", "published_at": "2026-09-03 18:43:24+00:00", "updated_at": "2026-09-03 18:53:40.196492+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "natural-language-processing", "ai-research"], "entities": ["Conversion", "Claude Opus 5", "Kimi K3", "GLM 5.3 Flash", "Gemini 3.8 Flash", "Statement Bench"], "alternates": {"html": "https://wpnews.pro/news/using-semantic-benchmarks-to-build-a-self-improving-text-to-query-agent", "markdown": "https://wpnews.pro/news/using-semantic-benchmarks-to-build-a-self-improving-text-to-query-agent.md", "text": "https://wpnews.pro/news/using-semantic-benchmarks-to-build-a-self-improving-text-to-query-agent.txt", "jsonld": "https://wpnews.pro/news/using-semantic-benchmarks-to-build-a-self-improving-text-to-query-agent.jsonld"}}