TypeSafe Jev vs. Mistral Small and Gemini Flash-Lite for Local Event Validation

Blog

A practical comparison of accuracy, response time and cost, using our best individually tuned prompts for a narrow event-validation task.

By Jon ReedPublished A toddler playtime session belongs on an events website. A room-hire price list does not. Somewhere between those two are the listings that make automated event discovery interesting: a club supper, an online Bible study, a school notice, or a perfectly good activity with no description.

At Near Here, we check millions of venues each month across our wider pipeline. Once a listing has passed our deterministic checks, we sometimes need a model to answer one narrow question: do these words describe something we should exclude? Our goal is the smallest, cheapest model that performs well enough for that job.

TypeSafe's Jev looked particularly relevant. It returns typed decisions and probabilities rather than writing a chat response. We compared it with Mistral Small 4 and Gemini 3.5 Flash-Lite using individually tuned prompts and retained event listings. These are our best complete prompt configurations selected in this round, tested on 16 September 2026.

The job we gave each model #

Challenge: reject explicitly unsuitable local-event listings from a title and optional description, after deterministic checks; approve everything else without browsing or verifying field accuracy.

That includes excluding generic products and services, venue hire, empty navigation, private or internal activities, and cancelled or online-only occurrences. A short title, missing description, booking requirement or admission price is not, by itself, a reason to reject an event. The model is judging meaning, not checking dates, addresses or the quality of extraction.

We wanted practical testing and figures out quickly. We tuned each model's prompt separately, then chose the strongest complete configuration across our development cases. Different prompts or a different event sample could produce different results. This is a use-case study, not a general model ranking.

The selected results #

|---|---|---|---|
| Accuracy on 50-case test set | 84% (42/50) | 86% (43/50) | 96% (48/50) | 

| Expected-valid events rejected | 5 / 13 | 1 / 13 | 0 / 13 | | Expected exclusions approved | 3 / 37 | 6 / 37 | 2 / 37 | | Purpose-written test cases | 31 / 32 | 32 / 32 | 32 / 32 | | Average response time | 2.90s | 3.40s | 0.59s | | Cost per 1,000 decisions | $0.370 | $2.496 | $0.043 | | USD per 1M input / output tokens | $0.15 / $0.60 | $0.30 / $2.50 | $0.042 / free | | Model settings / response | High reasoning; decision + explanation | High thinking; decision + explanation | Native Choice + probabilities |

Accuracy here means matching the expected decisions fixed before testing. The same 50 cases were used for all three models, and those cases informed prompt selection. Speed and cost come from the separate 21-record test described below. Neither sample estimates production accuracy.

On the main set, Jev matched 48 of 50 expected decisions and rejected none of the 13 expected-valid events. Gemini matched 43; Mistral matched 42. Looking at the errors matters as much as the headline: accepting an unsuitable listing and hiding a valid event are different product mistakes.

Why the output format matters #

Mistral and Gemini returned a structured decision with a short explanation. Both used high reasoning settings. Jev selected a category and returned probabilities; it did not generate an explanation. One category meant no explicit exclusion, and the other categories represented reasons to reject.

Those differences are part of the configurations we measured. The cost and timing comparison is not a claim about the fastest or cheapest possible way to run every model. We did not test lower-effort chat settings in this round. Typed output also does not guarantee that the semantic decision is correct.

How we tested #

1. Fixed the task and expected decisions. The main set had 50 real records: 13 expected approvals and 37 rejections. We also checked 50 earlier cases, comprising 42 captured examples and eight synthetic cases, plus 32 purpose-written cases covering policy boundaries.
2. Tuned each provider separately. We kept the same decision policy and selected the strongest complete configuration by total matching decisions across all 132 cases. We retained failed and interrupted trials and checked false rejections separately. We did not add website-specific prompt exceptions.
3. Tested the selected prompts on additional records. We fixed the prompts before inspecting their model results on 21 unused real listings. The records had been inspected to write expected decisions, so this was not an independently administered blind test.
4. Measured complete responses and token usage. Development used three concurrent calls per provider. The additional-record test used one call at a time per provider, with the providers running concurrently. Timings include client and network overhead. There was one observation per record and no retries.

The real examples came from retained rejections, not a random sample of everything we find. Expected decisions were written by the assistant and fixed before calls; they include debatable product judgements and have not been independently human-adjudicated. We have kept the inputs, responses, prompts and hashes so the measurements can be traced back to their evidence.

The detailed measurements #

The 21 additional listings contained only two expected approvals and 19 rejections, mainly car listings and dental services. They provide a narrow additional check, not evidence of performance across the full variety of local events.

Estimated cost is measured tokens multiplied by standard uncached USD prices, including reasoning output for the chat models. Jev output is free at the published rate. We exclude credits, tax and batch or cache discounts. Cost per 1,000 decisions is the total recorded cost divided by 21, multiplied by 1,000. These are list-price estimates, not invoice totals.

What this means for Near Here #

Jev is a promising fit for this narrow decision. In the measured configuration it combined low cost and short response times with a strong result on our main test set. The additional sample was too small and too narrow to establish a general accuracy advantage. Mistral's valid-event rejections also show why we need to look beyond a single percentage.

These are the prompt configurations we selected for prospective production use; this report does not claim they have already been deployed. The useful question for us is whether a model can make this particular decision well, at the volume and cost our pipeline needs.

About Near Here #

Near Here helps people find local events across the UK, from village fêtes and pub quizzes to craft workshops and community gatherings. Built by Jon and Ellie, it brings together events scattered across venue websites, including the smaller occasions that never reach the big ticketing platforms.

This work helps us keep those listings useful: finding real things to do while filtering out pages that describe a service, a product or a room for hire. Explore www.nearhere.events or read the story behind Near Here.

Sources, report and contact #

• Download the complete comparison PDF — three pages, dated 16 September 2026.
• Original Jev video announcement on X andTypeSafe's announcement and pricing .

- [Mistral pricing](https://docs.mistral.ai/inference/pricing) and[Small 4 model details](https://docs.mistral.ai/models/mistral-small-4-0-26-03) . The API returned the`mistral-small-latest` alias.
- [Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing) . We requested and received`gemini-3.5-flash-lite` .
- [TypeSafe Choice documentation](https://docs.typesafe.ai/primitives/choice) . We requested`jev-latest` ; responses identified`jev-1.13.0` .

Prices were checked on 16 September 2026. Questions about the evaluation? Email hello@nearhere.events.