{"slug": "building-an-ultra-high-throughput-ai-sql-engine", "title": "Building an Ultra-High Throughput AI-SQL Engine", "summary": "Researchers are building AI-SQL query engines that push LLM inference into the database, but executing AI functions remains extremely expensive because a single query can trigger hundreds of thousands or millions of model calls, with a filter requiring one LLM call per row and a naive join requiring one call for every pair of rows. The work, spanning UC Berkeley's DocETL, Stanford's LOTUS, MIT's Palimpzest, and Cornell's ThalamusDB, primarily cuts cost by eliminating LLM calls and using cheaper models, and the team is building the QUAIL-B benchmark to evaluate these engines, using a BIO-4 query over 5,000 long medical reports and 4,144 reaction terms. The authors argue query plans themselves should control LLM inference rather than sending millions of related calls to a general-purpose engine such as vLLM as separate requests.", "body_md": "[Back to blog](https://fsdatalab.github.io/blog/)\n\n# Building an Ultra-High Throughput AI-SQL Engine\n\n# 1. AI-SQL makes unstructured data useful, but it is expensive.\n\nFast LLM classifiers have been taking over the internet lately.\n[Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) is\nthe clearest example: give a model a small, bounded decision and get an\nanswer almost immediately. What better place to run millions of those\ndecisions than… inside the database!\n\nIndeed, database vendors have recently begun to offer this kind of intelligence at scale through AI-SQL, also called AI functions. AI-SQL extends SQL with user-defined functions that invoke LLMs. Users specify each function with a natural-language prompt. A query can look like this:\n\n```\nSELECT *\nFROM reviews AS r\nWHERE AI.IF(PROMPT('Does this review discuss the ending?\\n\\n{0}', r.review));\n```\n\nMany database vendors support AI-SQL. For example, [Snowflake Cortex\nAISQL](https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql),\n[BigQuery AI\nfunctions](https://cloud.google.com/bigquery/docs/generative-ai-overview),\n[Databricks AI\nFunctions](https://docs.databricks.com/aws/en/large-language-models/ai-functions),\nand, recently, [MotherDuck](https://motherduck.com/blog/motherduck-supports-jev/)\nall support it.\n\nUnfortunately, executing AI-SQL is extremely expensive. An AI function evaluates its prompt row by row, so one SQL query can create hundreds of thousands or millions of model calls. A filter needs one LLM call per row. A naive join needs one LLM call for every pair of rows in its two input tables.\n\nThis line of work has become extremely popular in the database research\ncommunity. A number of open-source academic systems have emerged, including our work on\n[DocETL](https://docetl.org/) from UC Berkeley,\n[LOTUS](https://lotus-data.github.io/) from Stanford,\n[Palimpzest](https://palimpzest.org/) from MIT, and\n[ThalamusDB](https://github.com/itrummer/thalamusdb) from Cornell. These systems\n(and database vendors) primarily reduce cost by eliminating as many LLM calls as possible\n(e.g., [MOAR](https://arxiv.org/abs/2512.02289), [Task\nCascades](https://arxiv.org/abs/2601.05536), and\n[Abacus](https://arxiv.org/abs/2505.14661)) and by using cheaper models when\npossible (e.g., [BARGAIN](https://arxiv.org/abs/2509.02896)). Even after these\noptimizations, a query plan may still require hundreds of thousands or millions\nof LLM calls.\n\n# 2. Key Idea: Query plans should control LLM inference!\n\nA natural thought is to use a general-purpose inference engine such as vLLM to execute the query plan. However, sending millions of related model calls to vLLM as separate requests has a large cost! We’ll illustrate with the following query:\n\nGiven a dataset of medical reports and a dataset of\npossible adverse reactions, find serious adverse event reports that mention\nboth a cardiovascular reaction and a neurological reaction.[1](#note-1)\n\nWe call this query BIO-4 in\n[QUAIL-B](https://github.com/fsdatalab/quail-bench), a benchmark we are\nbuilding to evaluate AI-SQL query engines. Its inputs contain 5,000 long\nreports and 4,144 reaction terms (the latter is used twice, as there are\ntwo joins). The logical plan, shown in [Figure 1](#figure-1), works as follows:\n\n1. \nIt filters the reports for serious adverse events.\n2. \nIt filters the two reaction term dataset inputs (i.e., lists of possible adverse reaction terms) for cardiovascular and neurological reactions.\n3. \nIt joins the surviving reports with the cardiovascular terms, then with the neurological terms.\n\nHow might we execute BIO-4 with vLLM? Following what databases do, we’d\nrender one prompt for each filter input, and, for the join, one prompt\nfor candidate report and reaction *pair*. Each prompt would be a\nseparate inference request.<sup>[2](#note-2)</sup> For the filters, there’s just one document\nper prompt; for the join, we’d place the much longer medical report\nfirst as the anchor and the reaction term second as the partner to\nmaximize reuse of the prefix’s key and value state (KV) across join\nprompts. We’d execute one operator at a time and order its LLM requests\nso requests for the same document are evaluated together, maximizing\nKV reuse.\n\n**A cost estimate for the query plan.** Before running the vLLM\nbaseline, we first want to estimate the lowest possible runtime for the\nsame plan. We count the model’s arithmetic work and HBM traffic from the\ntoken lengths, then use a [roofline\nmodel](https://modal.com/gpu-glossary/perf/roofline-model) to estimate\nthe time. The estimate assumes peak GPU throughput, full overlap between\nCPU and GPU work, and unlimited space for retained KV. No implementation\ncan meet all of these assumptions, so this optimistic lower bound is our\n*speed of light estimate*, or SoL. For BIO-4, the SoL estimate is 894.37\nseconds, or 14.91 minutes.<sup>[3](#note-3)</sup> The [implementation in\nQuail](https://github.com/fsdatalab/quail-exploration/blob/0d24478a82100b518d6110f5c1c8cec0c26c6487/quail/planner/sol.py)\ncontains the full calculation (which we’ll discuss in a follow-up blog\npost).\n\n**How we hoped vLLM would perform.** Each request produces one token\nconstrained to TRUE or FALSE, so all model work is prefill. BIO-4\ncompiles to millions of requests, so there should always be a large\nbatch ready for the H100. With a large enough batch, vLLM should keep\nthe H100 busy, and get as close to the SoL estimate as possible.\n\n**How vLLM actually performs.** We run vLLM 0.26.0 with Qwen3 4B FP8 on\none H100. We give it enough batch capacity to use the GPU. At scale\nfactor 1.0, a vLLM baseline takes *6.84 hours*, or 27.55x the SoL\nestimate! There are two reasons for the inefficiency. [Figure 2](#figure-2) shows a\nrepresentative 3.5-second window from the first join.\n\n- \nThe primary reason is host overhead. The CPU spends long stretches scheduling and tracking requests while the H100 waits, as shown by the gaps in [Figure 2](#figure-2) .<sup>[4](#note-4)</sup>\n- \nThe second is *KV regret:* vLLM sometimes discards KV that it needs\nagain later. On BIO-4 at scale factor 1.0, it therefore processes 174.6\nmillion tokens instead of 124.3 million: 50.3 million extra tokens, or\n40% more work.\n\nWe can, and we should, reduce both sources of waste by optimizing inference for AI-SQL!\n\n# 3. We built Quail to run AI-SQL queries faster.\n\nWe will first show you how you can get started. To skip to read about\nhow Quail works, skip to [Section 3.2](#32-quail-jointly-plans-queries-and-inference).\n\n## 3.1 You can run your first Quail query in a few lines of code!This part is collapsible, for space reasons.\n\nWe can use Quail to run two AI filters over all 100,000 movie reviews in\nthe [Stanford IMDB\ndataset](https://huggingface.co/datasets/stanfordnlp/imdb). We first\ndownload the reviews from Hugging Face, and load them into an Arrow\ndataset.\n\n``` python\nimport pyarrow as pa\nimport pyarrow.dataset as ds\nfrom datasets import concatenate_datasets, load_dataset\nimport quail\n\nimdb = load_dataset(\n    \"stanfordnlp/imdb\",\n    revision=\"e6281661ce1c48d982bc483cf8a173c1bbeb5d31\",\n)\nall_reviews = concatenate_datasets([\n    imdb[\"train\"],\n    imdb[\"test\"],\n    imdb[\"unsupervised\"],\n])\nreviews = ds.dataset(pa.table({\n    \"review_id\": pa.array(f\"review-{i}\" for i in range(len(all_reviews))),\n    \"review\": all_reviews.data.table.column(\"text\"),\n}))\n```\n\nThe query keeps reviews that discuss the movie’s ending, and recommend watching the movie.\n\n```\n# This Python process has access to one H100.\nconfig = quail.EngineConfig(\n    gpus=1,\n    model=\"qwen3-4b-fp8\",\n    backend=\"quail\",\n    device=\"h100-sxm\",\n)\nwith quail.Session(config) as session:\n    session.register(\n        \"reviews\",\n        quail.DocumentProvider.from_dataset(reviews, id_col=\"review_id\"),\n    )\n\n    query = session.sql(\"\"\"\n        SELECT r.review_id\n        FROM reviews AS r\n        WHERE AI.IF(\n            PROMPT(\n                'Does this review discuss the ending of the movie?\\n\\n{0}',\n                r.review\n            ),\n            -- Optional, but helps Quail reorder filters.\n            {'selectivity': 0.25}\n        )\n        AND AI.IF(\n            PROMPT(\n                'Does the reviewer recommend watching the movie?\\n\\n{0}',\n                r.review\n            ),\n            {'selectivity': 0.5}\n        )\n    \"\"\", dialect=\"bq\")\n\n    print(query.explain())\n    result = query.run()\n    table = result.collect()\n```\n\nBefore running the query, query.explain() prints the logical and physical plans. The output below keeps only the parts that describe the two filters and their execution settings.\n\n```\nlogical:\nProject: r.review_id\nSemanticFilter\npredicate 1: discusses the ending (selectivity=25%)\npredicate 2: recommends the movie (selectivity=50%)\nScan reviews as r [review, review_id]\n\nphysical: backend=quail, model=qwen3-4b-fp8, workers=1\nKV=bf16\nchunk budget=110,376 tokens\nadmission budget=362,250 tokens\nProject: r.review_id (est. rows=12,500)\nAiFilter: r (est. rows=12,500; est. time=124 s)\nKV rewind=on\npredicate 1 (input rows=100,000; est. pass=25%)\npredicate 2 (input rows=25,000; est. pass=50%)\nScan reviews as r (rows=100,000)\ntokens=29,926,924, mean_doc_tokens=299.3\n```\n\nThe complete example in\n[demos/imdb_ending_filter.py](https://github.com/fsdatalab/quail/blob/d8d31f14f9d5c40d6cf683a74ba860788ce0d500/demos/imdb_ending_filter.py)\nprints the following results at the end of the run:\n\n```\nmatching reviews: 16057 of 100000\nstage evaluated 100000 reviews, 0.283 passed\nstage evaluated 28296 reviews, 0.568 passed\nboot_s: 57.65 (cold)\ntoken_wait_s: 0.0\nwall_s: 277.41\ntotal_s: 335.06 (boot + query)\nfresh_tokens: 32499738\ndocuments/second: 360.5\nGPU price: $3.9492/GPU-hour (Modal)\nGPU cost, including startup: $0.3675\n```\n\nThe full run costs $0.3675 at [Modal’s H100\nprice](https://modal.com/pricing), including model startup.<sup>[5](#note-5)</sup>\nAt current GPT-5 nano prices (including cached token prices), the same\ntwo-filter workload would cost about $1.75, or 4.8 times the measured\nQuail cost![6](#note-6)\n\n**Running on Modal**. If you don’t have a dedicated GPU, you can put the\nwhole query inside a Modal GPU function. The function creates a normal\nQuail session and runs it:\n\n``` python\nimport modal\n\napp = modal.App(\"quail-engine\")\nimage = (\n    modal.Image.from_registry(\n        \"nvidia/cuda:13.0.1-devel-ubuntu24.04\",\n        add_python=\"3.12\",\n    )\n    .entrypoint([])\n    .uv_pip_install(\"quail-engine==0.1.0\")\n)\n\n@app.function(image=image, gpu=\"H100!\", memory=32768, timeout=1200)\ndef run_query(sql, documents):\n    import quail\n\n    config = quail.EngineConfig(\n        gpus=1,\n        model=\"qwen3-4b-fp8\",\n        backend=\"quail\",\n        device=\"h100-sxm\",\n    )\n    with quail.Session(config) as session:\n        session.register(\n            \"docs\",\n            quail.DocumentProvider.from_table(documents, id_col=\"id\"),\n        )\n        result = session.sql(sql).run()\n        return result.collect(), result.report\n```\n\nModal allocates the H100, and Quail plans and runs the query inside the\nfunction. A complete example is in\n[demos/quickstart_modal.py](https://github.com/fsdatalab/quail/blob/d8d31f14f9d5c40d6cf683a74ba860788ce0d500/demos/quickstart_modal.py).\n\nCheck out the [Quail documentation](https://fsdatalab.github.io/quail)\nto learn more.\n\n## 3.2 Quail jointly plans queries and inference.\n\nThis section describes Quail’s main design ideas at a high level. We are still actively building Quail, and we will provide the full technical details in a future report.\n\nWe have three performance goals for Quail:\n\n1. \nMinimize KV regret.\n2. \nKeep the GPU busy by reducing CPU scheduling overhead.\n3. \nReach high model FLOP/s utilization (MFU) while the GPU is active.\n\nOur current evaluation focuses on the first two goals. We defer a full MFU study to future work.\n\nAs shown in [Figure 3](#figure-3), Quail consists of a query frontend, a query\nplanner, and an execution engine. Through the frontend, the user\nprovides Arrow tables or datasets, an AI-SQL or Python query, and the\nmodel and GPU or GPUs to use. The frontend creates a logical plan from\nthe query. The query planner orders the filters and joins, chooses the\nanchor for each join, and determines how many tokens each model forward\npass should process. The planner then lowers the logical plan into a\nphysical operator plan, which the execution engine runs.\n\nQuail is extensible, and its design is inspired by [Apache\nDataFusion](https://datafusion.apache.org/), an open source, extensible\nanalytical query engine. One can add new query\noperators, planning rules, execution backends, models, or support for\nother hardware.\n\n### 3.2.1 Quail turns AI-SQL into a logical query plan.\n\nUsers register data as an in-memory Arrow table or an Arrow dataset. Users can write queries in AI-SQL (we support both Snowflake’s and BigQuery’s spellings, AI_FILTER and AI.IF), or use a Python query builder similar to pandas. The current release of Quail supports AI filters and joins, along with relational projections and LIMIT.\n\nUsers define each [AI\noperator](https://fsdatalab.github.io/quail/docs/user-guide/sql#ai-operators)\nwith a prompt and can provide optional planning information. The\noptional selectivity gives the expected fraction of documents or\ndocument pairs that will pass; without it, Quail keeps predicates in\ntheir written order. For a join, the optional anchor chooses which input\ncomes first in the prompt for KV reuse; without it, the planner chooses\nthe anchor.\n\nUsers can specify the model and GPU count. Quail currently supports three models: Qwen3 4B FP8, Qwen3 32B FP8, and DiffusionGemma on H100 GPUs. We plan to add support for more models and hardware through the extension interface.\n\nIn Quail, each AI-SQL query is parsed with SQLGlot into a logical plan, which is then passed to the query planner.\n\n### 3.2.2 Quail plans operator order and KV reuse.\n\nDatabase query optimizers already use many rules, e.g., pushing down filters, reordering predicates, and choosing join order. AI-SQL adds several new decisions, e.g., which document should be the join anchor, which KV will be needed by a later operator, and how much model work should enter each forward pass. Quail plans both kinds of decisions together.\n\n**Overview.** Given the logical query plan, we do the following:\n\n1. \n*Compute dataset statistics.* We estimate the document lengths and\nbasic statistics for each input dataset.\n2. \n*Compute forward pass and KV limits.* From the selected model and\nGPU, we choose how many tokens to process in each model forward pass\nand calculate the fixed KV capacity.\n3. \n*Perform SQL query rewrites* . We push down projections and filters,\norder the filters, and choose the join order and anchor for each\njoin.\n4. \n*Perform Inference-specific query rewrites.* We lower AI operations\ninto physical operators and plan their execution order and KV use.\n\nWe describe these steps at a high level, in turn.\n\n**Dataset statistics.** We estimate the row count, average document\nlength, and maximum document length for each input dataset.\n\n**Forward pass and KV limits.** Given the user’s selected model size and\nGPU memory size, we calculate the maximum number of tokens to compute\nfor each model forward pass. We reserve HBM for the model weights and\ntwo forward passes, and leave the rest of the HBM for the KV cache.\n(This is more conservative than vLLM, which profiles one forward pass to\ndetermine how much activation memory to reserve, so we can probably\nimprove on this).\n\n**SQL query rewrites**. We push projections and filters down to the\nsource datasets. We order filters using their estimated cost and\nselectivity, following extremely well-known prior work ([Hellerstein and\nStonebraker](https://dsf.berkeley.edu/jmh/miscpapers/sigmod93.pdf) et\nal.). For joins, we use a\n[Selinger-style](https://doi.org/10.1145/582095.582099) search (i.e.,\nSystem R) to choose the join order and anchor for each join. The cost\nmodel uses the speed-of-light estimate from Section 2. We will explain\nthe calculation in a future post. For now, you can check out the [cost\nmodel code](https://github.com/fsdatalab/quail/tree/main/quail/cost).\n\n**Inference-specific query rewrites**. After the SQL rewrites, we\ntranslate the logical plan into a DAG of physical operators. You can\nfind the physical operators that Quail currently supports in our\n[physical plan\ndocumentation](https://fsdatalab.github.io/quail/docs/architecture/physical-plans).\nFor example, the AiFilter physical operator evaluates AI predicates over\ndocuments, while AiJoin evaluates AI predicates over document pairs that\nshare an anchor. Each AI physical operator also specifies its prompt,\nmodel, forward pass token budget, and KV settings (e.g., whether to\nwrite KV to HBM because there will be a subsequent operator in the\nquery).\n\nDrawing inspiration from vectorized query execution, Quail streams intermediate results directly between operators rather than materializing complete datasets on disk or in main memory. For example, in BIO-4, as soon as a batch of reports passes the initial filter operator, Quail immediately pipelines it to the first join operator. It maintains the report KV cache in HBM throughout both join operations, allowing direct comparisons against the filtered reaction terms without redundant KV recomputations.\n\n### 3.2.3 Quail runs the physical query plan.\n\nOverview. The execution engine has three main components:\n\n1. \n**Physical plan executor.** On the CPU, Quail pulls document batches\nthrough the physical operator DAG and prepares work for the GPU.\n2. \n**KV manager.** Quail allocates, pins, “rewinds” (i.e., only\npersists KV for the prefix we know will appear in a future operator,\nnot the entire LLM prompt which includes the document(s) and some\nnatural language instruction), and releases KV pages according to\nthe physical plan.\n3. \n**Inference program.** On the GPU, Quail runs a model forward pass\nfor each input batch.\n\n[Figure 4](#figure-4) shows how these components work together.\n\nWe’ll discuss the first two components; then we’ll describe the\ninference program in [Section 3.2.4](#324-quail-uses-specialized-inference-programs-for-ai-sql).\n\n**Physical plan executor.** Quail uses a pull-based executor, as in\n[Volcano](https://doi.org/10.1109/69.273032), but processes a batch at a\ntime, as in [MonetDB](https://www.cidrdb.org/cidr2005/papers/P19.pdf).\nBefore execution, Quail tokenizes every document column referenced by an\nAI filter or join with\n[Gigatoken](https://github.com/marcelroed/gigatoken)<sup>[7](#note-7)</sup>, then\nloads one model copy per GPU. During execution, the CPU prepares one\ninput batch while the GPU processes another.\n\n**KV manager.** Each GPU has a fixed pool of KV pages in HBM. After each\nmodel evaluation, Quail retains only the KV that a later evaluation can\nreuse. For a filter, Quail places the document before the\npredicate-specific question. After the predicate returns TRUE or FALSE,\nQuail discards the predicate KV and “rewinds” to the end of the document\nKV. Then, if the predicate returns TRUE and another AI operator uses the\ndocument, Quail will retain the “rewinded” KV in HBM; otherwise, Quail\nwill release it. Quail similarly retains “rewinded” KV for joins. If\neviction is necessary, Quail evicts the shortest documents, since longer\ndocuments take disproportionately longer to recompute, thanks to\nattention being a quadratic operation.\n\nNote that a general-purpose inference is different in that: (1) it\nretains *all* the KV associated with a request (no “rewinding”), even\nthough the suffix KV will never be used again in the query, (2) *all*\nrequests’ KV are wastefully saved in HBM, even if documents are filtered\nout in the query and never needed again, and (3) documents are evicted\nwith LRU.\n\n**Using multiple GPUs.** Our current multi-GPU support is quite basic.\nWe place one complete model copy and one KV pool on each GPU. We\npartition filter documents and join anchors randomly and uniformly\nacross the GPUs, run them independently, and combine the results on the\nCPU.\n\n### 3.2.4 Quail uses specialized inference programs for AI-SQL.\n\nDuring planning, Quail chooses which documents or document pairs require\nmodel evaluation. During execution, each evaluation follows an\n*inference program*: e.g., embedding lookup, transformer layers,\nattention, matrix multiplication, etc.\n\nHere, we first describe how physical operators are expressed as inference programs, then how vLLM represents an inference program (which we adopt), and finally, the changes we make to Quail’s inference program.\n\n**Physical operator interface.** Each AiFilter or AiJoin physical\noperator is expressed as an inference program. The program takes token\nIDs and positions, plus the locations of any reusable KV pages. It\nreturns TRUE and FALSE scores for each row or pair of rows. Quail runs\nthe program across all rows or pairs evaluated by the operator.\n\n**vLLM’s inference programs.** vLLM is a general-purpose engine designed\nto support any inference pattern, across various model architectures and\nhardware backends. How does vLLM *do it all*? As shown in [Figure 5](#figure-5),\ngiven the model choice and GPU, there are two primary paths through\nwhich vLLM creates an inference program (i.e., of GPU kernels): (1)\nPyTorch operations JIT-compiled with\n[torch.compile](https://docs.vllm.ai/en/stable/design/torch_compile/)\nand TorchInductor into generated Triton GPU kernels, and (2) custom\noperations (such as attention) expressed through highly specialized GPU\nkernels like, FlashAttention.\n\n**Quail’s inference program.** We did not reimplement every model and\nGPU operation from scratch. That would be silly. Instead, Quail uses\nvLLM’s model implementations to obtain the operations required for a\nforward pass, then runs them with its own scheduler and KV manager.\nHowever, Quail makes three small changes to the forward pass:\n\n**First, fuse small operations.** We write\n[Triton](https://triton-lang.org/) kernels that fuse normalization with\nFP8 quantization, Q/K normalization with RoPE, and activation with FP8\nquantization. This is extremely easy to do now with AI agents; it\nrequires no novel kernel design ideas. By fusing these operations, Quail\nreduces kernel launches and intermediate HBM traffic.[8](#note-8)\n\n**Second, specialize attention for joins.** An AI join compares one\nanchor with many partners. Standard vLLM treats each anchor and partner\nas a separate sequence. Attention therefore reads the same anchor KV\nagain for every partner, as the left side of [Figure 6](#figure-6) shows.\n\nQuail groups all partners that share an anchor and computes the anchor\nKV once (right side of [Figure 6](#figure-6)). [Figure 7](#figure-7) shows how Quail evaluates attention in two parts. One\n[FlashAttention 3](https://arxiv.org/abs/2407.08608) call\ncomputes causal attention within each partner. A second call applies all\npartner queries to the shared anchor KV, reducing repeated reads. Quail\ncombines the two results using their log-sum-exp values and the\n[online softmax formula](https://arxiv.org/abs/1805.02867),\nproducing the same output as attention over each full anchor and partner\nsequence. This is one level of “tree”-based attention.<sup>[9](#note-9)</sup> One\nTriton kernel combines the BF16 outputs and converts them to the FP8\nformat expected by the output projection.\n\n**Third, restrict the output head to** TRUE **and** FALSE. Normally, a\nmodel would use its final output head (“language modeling” head,\nlm_head) to compute a score for every token in its vocabulary. For AI\nfilters and joins, Quail needs only the scores for token IDs that\nrepresent TRUE or FALSE.<sup>[10](#note-10)</sup> Quail therefore multiplies the\nfinal hidden state by only the corresponding rows of the output/language\nmodeling head matrix. By using the smaller matrix, Quail reduces\ncomputation and GPU memory use by the output head.\n\n# 4. We evaluate Quail against vLLM.\n\nAt scale factor 0.1, Quail is faster than a “stock” vLLM baseline on 27\nof the 29 QUAIL-B queries. The **(geometric) mean speedup is 1.84x**, and\nthe **maximum speedup is 11.22x** on BIO-2. At scale factor 1.0, we find\na query for which Quail is 14.04x faster! The two queries where stock\nvLLM wins expose one missing feature clearly: Quail does not yet reuse\nmatching prefixes across different rows.\n\nIn this section, we first describe our [metrics and\nbaselines](#41-metrics-and-baselines-for-ai-sql-performance), then\npresent the [full QUAIL-B results](#42-overall-quail-is-184x-faster-across-quail-b),\nand finally examine [BIO-4](#43-quail-dominates-vllm-on-bio-4-1404x-faster)\nand [AGENT-1](#44-but-vllm-dominates-quail-on-agent-1-quail-takes-232x-as-long)\nin detail.\n\n## 4.1 Metrics and baselines for AI-SQL performance.\n\n**Metrics.** We report three metrics for each query: KV regret,\n$/query, and input tokens/second. KV regret is repeated model work:\nfresh input tokens beyond the minimum needed to compute each reusable\nprefix once. $/query is query runtime in hours multiplied by\n[$3.9492 per H100-hour](https://modal.com/pricing). Input\ntokens/second is the total requested input tokens divided by query\nruntime. Each evaluated prompt contributes its full input length,\nincluding tokens served from KV. Lower KV regret and cost are better;\nhigher throughput is better.\n\n**QUAIL-B.** We created\n[QUAIL-B](https://github.com/fsdatalab/quail-bench), a benchmark\nwith 29 AI-SQL queries. It covers IMDB reviews, medical reports,\nfact-checking claims, legal documents, and software-agent traces.\nQueries include filters, filter sequences, and one or more joins. Each\ndataset has scale factors 0.1, 0.5, and 1.0. We compare all 29 default\nqueries at scale factor 0.1. We examine BIO-4 at scale factor 1.0 and\nAGENT-1 in more detail.\n\n**Setup.** Every query uses Qwen3 4B FP8 with BF16 KV on one H100. Quail\nand each vLLM baseline run one after the other on the same physical GPU.\nThey use the same model, prompts, and logical query plan.\n\n**vLLM baselines.** Both baselines use the optimal operator ordering\nchosen by our query planner. For each operator, we prepare its requests\nand order them to improve KV reuse. We call the operator-at-a-time\nbaseline “stock vLLM.” For QUAIL-B queries with multiple filters or\njoins, we also report a “pipelined vLLM” baseline. It pipelines requests\nbetween consecutive filters and between consecutive joins. For fairness,\nboth baselines use Gigatoken for tokenization, as Quail does, instead of\nvLLM’s Hugging Face tokenizer.[11](#note-11)\n\n## 4.2 Overall, Quail is 1.84x faster across QUAIL-B.\n\n[Table 1](#table-1) averages tokens/second, KV regret, and cost per query within\neach dataset at scale factor 0.1. Cost multipliers are relative to Quail.\n\n| Dataset | Quail | Stock vLLM | \n|---|---|---|\n| BIO (4 queries) | 12,296,410 tokens/s 157,995 KV regret $0.0892/query (1.00x) | 1,420,421 tokens/s 1,441,816 KV regret $0.7771/query (8.72x) | \n| IMDB (10 queries) | 649,864 tokens/s 451,917 KV regret $0.0282/query (1.00x) | 382,818 tokens/s 1,129,174 KV regret $0.0467/query (1.66x) | \n| FEV (8 queries) | 1,692,276 tokens/s 408,592 KV regret $0.0383/query (1.00x) | 724,574 tokens/s 519,588 KV regret $0.0820/query (2.14x) | \n| LEP (5 queries) | 388,903 tokens/s 2,147 KV regret $0.0670/query (1.00x) | 316,617 tokens/s 70,982 KV regret $0.0853/query (1.27x) | \n| AGENT (2 queries) | 73,006 tokens/s 11,886,152 KV regret $0.2616/query (1.00x) | 169,201 tokens/s 23,928 KV regret $0.1129/query (0.43x) | \n\n[Figure 8](#figure-8) summarizes throughput by dataset, and [Figure 9](#figure-9) reports latency\nfor all 29 queries. The geometric mean of Quail’s per-query speedups\nover stock vLLM is 1.84x. In total, Quail completes the benchmark in\n1,643.74 seconds, compared with 4,451.95 seconds for stock vLLM. Quail\ntakes 3.35x longer than the combined SoL estimate of 491.17 seconds, so\nthere is substantial room to improve.\n\n[Figure 10](#figure-10) focuses on the eight queries where pipelining changes how vLLM\nsubmits requests. Pipelined vLLM is faster than stock vLLM on seven of\nthem, by 1.12x on average and up to 1.27x on IMDB-6.\n\nStock vLLM is faster than Quail only on AGENT-1 and AGENT-2. Quail does not yet reuse matching prefixes across rows, so it recomputes far more KV tokens on each query. Section 4.4 examines AGENT-1.\n\n## 4.3 Quail dominates vLLM on BIO-4: 14.04x faster!\n\nBIO-4 contains the kind of reuse Quail currently handles well: long shared documents, two joins, and millions of related model calls whose order is known before execution. At scale factor 1.0, BIO-4 filters 5,000 medical reports and two uses of the same 4,144 reaction terms, then runs two joins over the surviving inputs.\n\n[Table 2](#table-2) reports throughput, cost, and KV regret.\n\n| Metric | Quail | Stock vLLM | SoL estimate | \n|---|---|---|---|\n| Requested input tokens/s | 19.03 million | 1.36 million | 37.37 million | \n| GPU cost per query | $1.93 | $27.03 | $0.98 | \n| KV regret | 18.0 million | 50.3 million | 0 (assumed) | \n\nQuail takes 29.26 minutes, compared with 6.84 hours for stock vLLM. Quail is 14.04x faster. It is 1.96x the SoL estimate, while stock vLLM is 27.55x the estimate. Even with pipelining, vLLM still takes 4.00 hours.\n\nQuail costs $1.93 per query, compared with $27.03 for stock vLLM. The SoL cost estimate is $0.98 per query. Quail processes 19.03 million requested input tokens per second, compared with 1.36 million for stock vLLM.\n\nQuail also recomputes less KV. It recomputes 18.0 million tokens, compared with 50.3 million for stock vLLM.\n\n## 4.4 But, vLLM dominates Quail on AGENT-1: Quail takes 2.32x as long.\n\nAGENT-1 contains a different kind of reuse. It filters 1,772 cumulative\nsnapshots from software agent runs. Separate rows contain overlapping\nprefixes from the same agent trace, and stock vLLM’s automatic prefix\ncaching recognizes them. Quail does not yet recognize that relationship,\nso stock vLLM wins. [Table 3](#table-3) shows two example rows.\n\n| id | trajectory_id | turn_index | trace | \n|---|---|---|---|\n| trace_42_turn_5 | trace_42 | 5 | [USER] Fix the failing parser. [ASSISTANT] Tries approach A. [TOOL] The test fails. | \n| trace_42_turn_10 | trace_42 | 10 | <complete trace from turn 5> [ASSISTANT] Finds the mistake, and tries approach B. [TOOL] The tests pass. | \n\nHere is the AGENT-1 query, simplified for this post:\n\n```\nSELECT t.id\nFROM agent_traces AS t\nWHERE AI.IF(PROMPT(\n    'Did the agent recover after trying an approach that did not work?\\n\\n{0}',\n    t.trace\n));\n```\n\n[Table 4](#table-4) reports throughput, cost, and KV regret.\n\n| Metric | Quail | Stock vLLM | SoL estimate | \n|---|---|---|---|\n| Requested input tokens/s | 73,006 | 169,201 | 367,400 | \n| GPU cost per query | $0.2623 | $0.1131 | $0.0521 | \n| KV regret | 11,886,152 | 23,928 | 0 (assumed) | \n\nStock vLLM finishes AGENT-1 in 103.07 seconds, compared with Quail’s 239.12 seconds. Quail takes 2.32x as long. The SoL estimate is 47.47 seconds, so stock vLLM still takes 2.17x longer than the estimate.\n\nStock vLLM wins because its automatic prefix caching can reuse KV across rows with matching token prefixes. Quail currently reuses KV only when the same document appears again in the query, not across different documents. As a result, Quail incurs 11.89 million KV regret tokens, while stock vLLM incurs only 23,928.\n\nWe plan to add automatic prefix caching to Quail, but the lookup must remain cheap at the request volumes that AI-SQL queries can produce.\n\n# 5. Put another way: Quail brings Jev-like speeds and intelligence to database-scale workloads.\n\nQuail also supports [DiffusionGemma 26B-A4B\nFP8](https://huggingface.co/RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic),\na larger mixture-of-experts model with 4B active parameters per token.\nThis gives Quail a higher-intelligence option that is still extremely\nfast. On IMDB-2 at scale factor 0.1, DiffusionGemma matched 88.89% of\nQwen3 32B’s answers, compared with 76.41% for Qwen3 4B. It ran the query\nin 32.41 seconds, or 1.53x as long as Qwen3 4B’s 21.20 seconds, on one H100.\n\nThis fits a broader class of workloads that need fast, bounded model\ndecisions instead of long generated responses.\n[Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev)\nhas highlighted the demand for this pattern in application backends.\nQuail targets its batch, online analytical processing (OLAP) version:\none query creates thousands or millions of related decisions over a\ndataset, and Quail plans and runs them together. This makes Quail a good\nfit for LLM judge workflows, trace compaction, labeling, and other\nlarge-scale data transformations.\n\n# 6. We are just getting started with Quail!\n\nSome next steps are obvious. E.g., we want to support more AI-SQL operators, more models, and more hardware. We especially want to support tiny hybrid models, so you can run Quail on a MacBook.\n\nWe are also interested many research ideas fusing the database and inference worlds; here are just a few:\n\n**Use the full memory hierarchy for KV.** Quail currently keeps reusable\nKV in GPU HBM or recomputes it. We want to move KV to host memory or\nlocal SSD when it does not fit on the GPU, then bring it back before\nreuse. We also want automatic prefix caching across rows. Perhaps we\nwill also want to do KV compression (we know it is good to make indexes\nsmaller).\n\n**Improve model FLOP/S utilization.** Quail currently relies on DeepGEMM\nand FlashAttention for its main GPU kernels. We have not optimized the\nkernels themselves, and we are stoked to be working with Modal and\nDoubleword, inference experts, on kernel optimization.\n\n**Train models for execution and planning.** [Google’s work on\nlightweight proxy models for\nAI-SQL](https://arxiv.org/abs/2603.15970) suggests small models can\nevaluate filters cheaply. The same models could predict selectivity and\nlikely survivors for the query planner, helping Quail choose operator\norder and decide which KV to keep. The big systems question is how to\nrun and train many specialized models alongside a larger model on the\nsame GPU.\n\n**Can an AI join work like a hash join?** Today, Quail reuses an anchor’s KV within\none join loop, but it recomputes every partner for each new anchor. The\nsame document is therefore encoded once per pair. Could we instead\nencode every document once and use its KV as a position-independent\nindex entry? A document from the other relation could then search those\nentries for matches, like probing a hash table, without recomputing the\nindexed documents. This may require removing or separating the position\ninformation that RoPE adds to KV.\n\n**New methods for using AI agents to build systems.** We used AI coding\nagents heavily to build the current version of Quail. We expect to keep using agents to build many of\nthe features above. How do we do this correctly? We want better ways to specify what the system must do, and to\ncheck that every agent-written change keeps answers correct and runtime\nclose to SoL. It feels inevitable that agents will do the bulk of the\ncoding, and we are excited to build Quail in public and share the\nmeta-learnings from building it with agents.\n\nMore blog posts, and eventually a technical report, are coming soon. For\nnow, please try [Quail](https://github.com/fsdatalab/quail)! If these ideas sound interesting, reach out to\nget involved! And if you want to build an application on top of Quail,\nsuch as an LLM judge workflow in AI-SQL, Quail has an MIT license. It is\nnow orders of magnitude cheaper to add intelligence to your data\nprocessing workflows, and we would love to see what you build :-)\n\n# Acknowledgements\n\nWe thank [Modal](https://modal.com/) for sponsoring the compute used in\nthis research.\n\n# Notes\n\n**1.** The query is based on the [BioDEX\ndataset](https://aclanthology.org/2023.findings-emnlp.896/). The SQL form\nof BIO-4 is shown below. In each prompt, `{0}` and `{1}` refer to the first\nand second arguments.\n\n```\nSELECT r.id,\n    n.id AS neurological_reaction_id,\n    c.id AS cardiovascular_reaction_id\nFROM reports AS r\nJOIN reaction_terms AS n\n    ON AI.IF(PROMPT(\n        'Does the medical report in {0} describe the reaction in {1} as '\n        'something the patient experienced?',\n        r.report,\n        n.term\n    ))\nJOIN reaction_terms AS c\n    ON AI.IF(PROMPT(\n        'Does the medical report in {0} describe the reaction in {1} as '\n        'something the patient experienced?',\n        r.report,\n        c.term\n    ))\nWHERE AI.IF(PROMPT(\n    'Does {0} describe a serious or life-threatening adverse event?',\n    r.report\n))\nAND AI.IF(PROMPT(\n    'Is this reaction neurological, affecting the nervous system? {0}',\n    n.term\n))\nAND AI.IF(PROMPT(\n    'Is this reaction cardiovascular, affecting the heart or blood vessels? {0}',\n    c.term\n));\n```\n\n**2.** Prompts use numbered placeholders, such as `{0}` and `{1}`, to refer to\nthe arguments after the prompt string in the `PROMPT` call. The SQL call\nand the model input it produces are shown below.\n\n```\nAI.IF(PROMPT(\n    'Does {0} mention {1}?',\n    r.report,\n    n.term\n))\n```\n\nThe documents do not have to appear exactly where their placeholders occur in the question. Quail can place the report first so its KV can be reused when the same report is compared with another reaction term.\n\nThe model receives the following input:\n\n```\nDOCUMENT:\n[contents of r.report]\n\n(The document above is DOCUMENT {0}.)\n\nEvaluate TRUE or FALSE for the following question:\nDoes {0} mention {1}?\n\nDOCUMENT {1}:\n[contents of n.term]\nANSWER:\n```\n\nOf course, whether other prompt layouts affect accuracy remains an open question, though we expect this to matter less as models improve.\n\n**3.** The speed of light estimate assumes 100 percent model FLOP/s utilization\n(MFU), meaning every forward pass sustains the GPU’s peak arithmetic\nthroughput. Real systems cannot reach that rate, so the estimate is an\noptimistic lower bound.\n\n**4.** Modal provides useful background on [GPU\nutilization](https://modal.com/blog/gpu-utilization-guide) and\n[host\noverhead](https://modal.com/blog/host-overhead-inference-efficiency)\nin inference engines.\n\n**5.** The IMDB dataset was already on disk, so the measurement excludes\nthe time and cost of downloading it.\n\n**6.** We use OpenAI’s [cached-token\nprice](https://developers.openai.com/api/docs/models/gpt-5-nano) in this\nestimate and assume an “infinite” cache, so every reusable document token\nreceives that rate.\n\n**7.** Marcel Rød built the fast\n[Gigatoken](https://github.com/marcelroed/gigatoken) tokenizer;\nthank you!\n\n**8.** Kernel fusion can substantially improve prefill MFU. In [“Chasing\nSpeed of Light on TPU\nv6e,”](https://www.sailresearch.com/blog/tpu-v6e-gemma) Sail\nResearch reports increasing Gemma 4 31B prefill MFU from about 32\npercent to 63 percent through several optimizations, including folding\nactivation, normalization, and RoPE work into surrounding kernels.\n\n**9.** We follow a long line of “Tree”-based attention approaches, which\nevaluate several branches that share a prefix without allowing one\nbranch to attend to another. E.g.,\n[SpecInfer](https://arxiv.org/abs/2305.09781) uses a tree mask\nduring speculative decoding to verify several possible continuations at\nonce. Also, [Hydragen](https://arxiv.org/abs/2402.05099) uses\nshared-prefix attention during decoding to generate several outputs from\none input. Quail applies the same structure, but during prefill.\n\n**10.** One might expect two token IDs, one for each answer. For Qwen,\nQuail scores four spellings of each answer. The TRUE tokens are “TRUE”\n(20611), “␠TRUE” (8214), “True” (2514), and “␠True” (3007). The FALSE\ntokens are “FALSE” (30351), “␠FALSE” (7833), “False” (4049), and\n“␠False” (3557). Here, ␠ marks a leading space.\n\n**11.** Both baselines use vLLM 0.26.0 with automatic prefix caching. We use\nthe largest stable settings: 25,305 maximum batched tokens, 4,096\nsequences, GPU memory utilization of 0.91, and one CUDA graph for 8,192\ntokens. Larger settings ran out of GPU memory.\n\n# Cite this post\n\n```\n@misc{shankar2026quail,\n  title = {Building an Ultra-High Throughput AI-SQL Engine},\n  author = {Shankar, Shreya and Frye, Charles and Finn, Fergus and\n            Dhariya, Arnav and Barrow, Joseph and Arik, Meryem},\n  year = {2026},\n  month = sep,\n  url = {https://fsdatalab.github.io/blog/introducing-quail/}\n}\n```\n\n", "url": "https://wpnews.pro/news/building-an-ultra-high-throughput-ai-sql-engine", "canonical_source": "https://fsdatalab.github.io/blog/introducing-quail/", "published_at": "2026-09-24 22:56:37+00:00", "updated_at": "2026-09-24 23:31:19.399586+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-research", "mlops", "ai-tools"], "entities": ["DocETL", "UC Berkeley", "LOTUS", "Stanford", "Palimpzest", "MIT", "ThalamusDB", "Cornell"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-an-ultra-high-throughput-ai-sql-engine", "markdown": "https://wpnews.pro/news/building-an-ultra-high-throughput-ai-sql-engine.md", "text": "https://wpnews.pro/news/building-an-ultra-high-throughput-ai-sql-engine.txt", "jsonld": "https://wpnews.pro/news/building-an-ultra-high-throughput-ai-sql-engine.jsonld"}}