I Tried to Make Natural Language a SQL Predicate. The Numbers Said No.

A few weeks ago I got stuck on a question: what if part of a WHERE clause could be written in plain English?

SELECT *
FROM support_tickets
WHERE SEM_PREDICT(
    ticket_text,
    'customer is asking for a refund'
);

I didn't want an LLM writing SQL for me, and I didn't want a chatbot. I also didn't want to send every row to an API. I wanted the natural-language condition to behave like any other predicate the engine can evaluate.

I built a prototype called SemPred, benchmarked it, and stopped when it missed the bar I'd set beforehand. The idea holds up. My first model doesn't. This post covers both.

Code: github.com/Yudeeswaran/SemPred

Databases are great at structured predicates. amount > 1000 AND currency = 'USD' is deterministic, cheap, indexable, and easy to reason about.

Plenty of real data doesn't fit that shape. Take a ticket table:

"Find the tickets where the customer wants a refund" has no SQL operator. You can write keyword rules, pre-classify everything, or call an LLM per row. I wanted to see if there was a fourth option, where the semantic condition is just another function in the query.

Two functions:

SEM_SCORE(text, predicate) returns a score.SEM_PREDICT(text, predicate) returns TRUE, FALSE, or UNKNOWN.

SELECT
    ticket_text,
    SEM_SCORE(ticket_text, 'customer is asking for a refund')   AS score,
    SEM_PREDICT(ticket_text, 'customer is asking for a refund') AS decision
FROM tickets;

The third state matters most. In DuckDB, UNKNOWN maps to SQL NULL, so the engine can tell "the model thinks this is false" apart from "the model has no idea". I'd rather get a NULL than a confident wrong answer, and three-valued logic already gives SQL a place to put it.

The SQL stays SQL. The semantic part is one more predicate next to your joins and filters.

The constraints I set up front:

The model. A frozen all-MiniLM-L6-v2 encoder turns text into an embedding. Each predicate gets its own small logistic-regression head on top of it. The encoder isn't trained at all.

Embed once, score many. If one ticket needs checking against 50 predicates, the encoder runs once and the 50 heads run on the same vector. Encoding is the expensive part, and the heads are nearly free by comparison.

A bounded embedding cache. Repeated text skips the encoder. This is a small detail in a notebook, but over millions of rows it's an architecture decision, and it was the point where the project started to feel like a data-systems problem.

DuckDB integration. Where PyArrow is available, the functions use DuckDB's Arrow UDF path, so values arrive in batches. The naive alternative (DuckDB → Python → model → DuckDB, once per row) looks fine at 100 rows and falls over at scale.

Abstention. The decision isn't p >= 0.5. There's a margin around the threshold. With a threshold of 0.5 and a margin of 0.1:

Both numbers are configurable.

I used Banking77: 10,003 training examples, 3,080 test examples, 77 intents, official split. I framed it as a few-shot predicate problem. Given a handful of labeled examples per predicate, can the system decide whether unseen text satisfies it?

Before any neural model, I ran TF-IDF. If a lexical baseline gets most of the way there, embeddings aren't earning their cost.

At 16 examples, MiniLM beats the matched TF-IDF setup by about 11.9 points. That looked like a win until I trained TF-IDF on all the training data:

"Improves accuracy by 12 points" was true and also misleading. The useful question is always "compared to what?"

I swapped in MPNet with the same frozen-encoder setup. Its best 16-shot result was 74.93% ± 1.77 pp. That's better than MiniLM, but below the 80% continuation threshold I'd defined before running it.

It was also much slower. Encoding 13,072 unique texts:

That's about 7× slower. A slower model can be worth it if quality jumps, but here it didn't.

Accuracy alone wasn't the real problem. The real problem was how accuracy trades against coverage.

At an abstention margin of 0.10, the 16-shot system committed to a decision on about 44.1% of ticket-predicate pairs. Of those committed decisions, about 96.3% matched the binary labels. That looks strong until you notice it's answering less than half the questions.

Widening the margin pushes accuracy up and coverage down. At one operating point I got roughly 90% ticket accuracy, but coverage fell to about 21%.

Before running anything, I'd written a gate:

≥ 90% committed-ticket accuracy AND ≥ 50% ticket coverage

No margin I evaluated met both. You can make almost any model look accurate by letting it answer fewer questions, so every abstention number needs its coverage figure next to it.

This is where I could have kept trying encoders and margins until something looked good. I didn't, because the gate existed to prevent exactly that. The gate wasn't met, so the conclusion is that this architecture isn't production-ready.

Having a written stopping condition changed how I ran everything. "Is this model better?" became "Did it meet the requirement?", and the second question is much harder to fudge.

Averages hide a lot. The card swallowed predicate did far better than the overall numbers, with the 16-shot model at the selected abstention setting:

Matched low-shot TF-IDF scored around 35.7% F1 on the same predicate. Some predicates have distinctive language, while others overlap heavily with neighboring intents. A real system probably shouldn't treat every predicate as equally hard.

In the current design, the predicate is the classifier head. The text goes through the encoder, and a per-predicate model decides. But the task is really pairwise: does this text satisfy this predicate? The predicate's wording carries information the head never sees.

Text:      "My account was charged twice."
Predicate: "customer is reporting a duplicate charge"

The relationship between those two strings is the signal. So the next hypothesis is to make the predicate an input to the model, not the name of a head.

This is a hypothesis to test, not a result. I haven't claimed it works.

An LLM given (ticket, predicate) would likely reason better. It also brings cost, latency, throughput, hosting, batching, privacy, and determinism concerns. At a million rows, a million LLM calls is hard to justify.

I see it more as a teacher or an escalation path. A compact model handles the bulk, and only the UNKNOWN cases go to a bigger model.

That's future work as well.

The systems side held up better than the quality side. In the benchmark environment, the DuckDB path scored 237,160 predicate pairs in 5.57 seconds, about 42,582 pairs/sec.

That's possible because unique texts are embedded once and the embeddings are reused across predicates. With N unique texts and P predicates, the cost is one encoder pass over N texts plus N × P cheap head evaluations, instead of N × P encoder runs.

I started out thinking the hard part was classifying text. After the prototype I'm less sure. The harder part may be running uncertain predicates safely over large tables:

NOT? None of those are NLP questions.

I'm not going to try five more embedding models, because that's benchmark roulette. The next experiment changes the architecture (pairwise scoring with the predicate as input), judged against the same gate and the same evaluation setup. After that, in rough order:

Nothing on that list counts as progress until it beats a benchmark defined in advance.

As a product, no. The model I evaluated misses its own quality and coverage bar.

As an experiment, yes. It turned "can SQL take a natural-language condition?" into a more specific question: what would it take for semantic reasoning to be a measurable, cacheable, safely executable primitive inside a data system? I don't have the full answer, but I now have a prototype, a benchmark, performance numbers, and a clear picture of where the design breaks.

The repo is open if you want to reproduce the benchmark or tell me where the architecture is wrong: github.com/Yudeeswaran/SemPred

If you work on semantic query execution, database-native ML, DuckDB extensions, or cheap inference at scale, I'd like to hear from you.

SemPred is a research prototype. It hasn't been validated as a general-purpose classifier or a production decision system, and the current benchmark results don't support using it for high-impact decisions.