We benchmarked a fine-tuned E2B model against frontier APIs on a real production classification task. It matched their quality in under 100ms, at about a hundredth of the cost. #
At Airbnb, we use AI models for classification tasks, such as detecting risky user-generated content, identifying why customers contact support, and recognizing topics in reviews. These tasks require subtle language understanding and the common sense to interpret what people leave unsaid.
Frontier models, with hundreds of billions of parameters and internet-scale pretraining, are often good at these tasks. But for interactive products and high-volume workloads, accuracy is only part of the requirement. We also need low latency, manageable serving costs, and control over the tradeoff between false positives and false negatives.
Smaller open-weight models are faster and cheaper to run at scale. By extracting logprobs, we’re able to produce a score for thresholding. But these small models are trained to handle a vast range of conversations and our tasks occupy just a tiny corner of that training distribution. Out of the box, they often miss subtle distinctions that matter immensely for our product.
Better prompting has limits. (I’m often reminded of an assignment from elementary school: write instructions for how to make a PB&J. “Put the knife in the peanut butter,” you’d write, only to have the teacher stab through the jar or insert the handle instead of the blade. There seem to be no instructions detailed enough to avoid every possible lapse in common sense.)
GLOWC, an internal framework used by engineers at Airbnb, makes it easy to fine-tune open-weight language (and vision-language) models into fast, accurate classifiers. In this blog post, we put one of these classifiers head-to-head with open-weight and proprietary models and find that it matches frontier-level quality in under 100ms at roughly 1% of the cost.
Case study: benchmarking a production classifier #
Public benchmarks rarely capture a real product’s constraints. Instead, we test a production classifier that screens content for violations of Airbnb’s policy on pricing and payment transparency. The task requires recognizing subtle policy distinctions with high precision, since false positives prevent legitimate content from being published. The model is called thousands of times per minute, and because it gates publication in real time, a score must return within 200ms. We compare a fine-tuned classifier with 2 billion effective parameters (E2B) against open-weight and proprietary models on a held-out evaluation dataset of 10,000 weighted examples. The fine-tuned model is trained on de-identified examples from users who have not opted out of AI training. Open-weight models are self-hosted with vLLM 0.28.0 on a single NVIDIA RTX Pro 6000 GPU. Models up to 12B parameters run in bfloat16. Larger models use publicly available FP8 or NVFP4 checkpoints. Proprietary models are called through their vendor APIs. Unless noted, every model receives the same concise ~400-word prompt.
Quality vs. latency #
Any model we deploy must be accurate enough to trust with a policy decision and fast enough to avoid disrupting the user experience. These two criteria often trade off, forming a quality-latency frontier that we plot below.
We report max F1 as the measure of classification quality. While we would normally prefer PR-AUC for imbalanced classification, most proprietary models return only a True or False verdict, so there is no curve to integrate. Max F1 puts both kinds of models on one axis: for open-weight models and decision models, we sweep over thresholds and report the best F1. For other proprietary models, we report the F1 of their verdict. The conclusions do not depend on the choice: PR-AUC, where available, and accuracy rank the models nearly identically to max F1. Latency is server-side handling time as recorded by our serving gateway.
Open-weight models are fast but sacrifice quality. Many open-weight models have latencies below our budget. Classification quality and latency tend to climb with model size. Off the shelf, their performance plateaus meaningfully lower than the best proprietary models or our fine-tuned classifier.
Proprietary generative models are accurate but slow and inflexible. Their classification quality does not meaningfully improve with more thinking, so we report the low-thinking results. Even in low-thinking mode, GPT-6.1 Sol and Claude Sonnet 5.5 have a median latency of about 900ms and a p99 of multiple seconds, many times our latency budget. GPT-6 Luna is the fastest proprietary generative model we tested, yet is both slower and less accurate than Gemma4-31B running on a single GPU. And because each model returns a verdict rather than a score, it gives a single, fixed precision-recall point. With the concise prompt, GPT-6.1 Sol lands at 0.82 precision and 0.96 recall, favoring recall in a task where false positives are costly. Moving that point means rewriting the prompt.
Decision models have strong performance for their speed. TypeSafe’s Jev returns a probability instead of generating a verdict, so it isn’t held back by token generation and can be thresholded like our own classifier. It is the most accurate off-the-shelf model we tested under 100ms.
Fine-tuning moves the quality-latency frontier instead of trading along it. Fine-tuning Gemma4-E2B on hundreds of thousands of domain-specific examples lifts its F1 from 0.563 to 0.902, on par with the best proprietary models, while median latency stays at 35ms. No other model we tested is close on both axes.
A detailed prompt improves classification quality but doesn’t close the gap. We also test every model with a detailed 2,700-word prompt that spells out each behavior type with worked examples and common false positives. Proprietary generative models gain one to three points and roughly match the fine-tuned classifier, albeit at much higher latency. For models under our latency budget, the gains are modest and a large gap remains. The detailed prompt also increases latency and cost. Together, these indicate that for models fast enough to deploy, prompt tuning does not necessarily close the quality gap.
Cost at scale #
A GPU costs the same whether it’s busy or idle, therefore self-hosting is only cheap once you have sufficient traffic. The fine-tuned classifier can support thousands of RPM on a single GPU with low latency, therefore its serving cost is constant until another worker is needed. Proprietary models, called via vendor APIs, charge based on token usage. This forms a linear relationship with volume: the more traffic you send, the more expensive it becomes.
Costs for proprietary models are calculated from the input and output tokens observed on our evaluation runs, priced at each vendor’s publicly available list rates as of October 2026. They do not reflect any negotiated pricing, discounts, or other non-public commercial terms. For models that offer discounted pricing on cached input tokens, we assume a 71% cache hit rate based on the average shared prompt prefix. This is higher than the hit rate we observed in practice, so it favors the hosted models. GPU costs are based on publicly available cloud rental rates for an NVIDIA RTX Pro 6000, and all costs are shown as multiples of that single-GPU cost.
Self-hosting is cheaper at high volume. At thousands of RPM, the frontier models using the concise prompt cost two orders of magnitude more than a single self-hosted GPU, and the detailed prompt raises their cost by a further 20 to 40%. Lower-cost proprietary options still run nearly five times the cost of the GPU, with lower accuracy than the fine-tuned model. GPT-6 Luna and TypeSafe’s Jev cost about the same on this task, so we report Jev, the better-performing of the two. With the detailed prompt, Jev’s cost quadruples, as its API offers no discount for cached input.
Proprietary models only win at low traffic: below roughly 15 to 30 RPM, Claude Sonnet 5.5 and GPT-6.1 Sol are cheaper than a dedicated GPU.
When should you fine-tune? #
Fine-tuning on in-domain data delivers quality, latency, and cost at the same time. But it isn’t always the right tool. For low-volume tasks, or tasks that change week-to-week, a frontier model behind a good prompt is often the right call. In our benchmark, fine-tuning paid off when:
-
Volume is high. Above roughly 30 RPM on our task, one self-hosted GPU costs less than frontier APIs. At our volume, it costs orders of magnitude less.
-
Latency is tight. Our fine-tuned E2B classifier returned a score in about 85ms p99, while frontier APIs took around 900ms at the median and seconds at p99 to achieve similar quality. Faster tiers exist, but vendors charge a premium for them.
-
You need a threshold, not a verdict . Scores let you choose your tradeoff between recall and precision without rewriting the prompt.
-
You have labeled data. Examples teach distinctions and edge cases that instructions struggle to spell out. Hundreds of thousands of in-domain examples did what a 2,700-word prompt couldn’t.
This classifier is just one example, but the same tradeoffs have held across other models we’ve fine-tuned. At Airbnb, GLOWC provides shared infrastructure for fine-tuning, batch inference, and serving, so teams can focus on their task while improvements carry over to every model. If you’re interested in pushing the tradeoff between model quality, latency, and cost, check out our open positions!
Appendix
-
Example weights correct for how the evaluation set was sampled from production traffic. We validated a subset of models on a larger set of 184k examples; their max F1 was within 0.01 of the results reported here.
-
Latency doesn’t track parameter count alone. Attention implementation matters too. vLLM’s current kernels are less optimized for hybrid linear-attention layers than for standard or sliding-window attention, so models with similar parameter counts can have quite different latencies.
-
See OpenAI’s orAnthropic’s fast mode for more details.
Acknowledgments #
We would like to thank John Romano and Chuiwen Ma for their help fine-tuning the classifier and curating evaluation datasets, Anirudh Agarwal and Ben Mendler for the serving infrastructure and latency measurement behind the models in this benchmark, and Mukund Narasimhan, Nathan Brixius, and Ahmad Al-Dahle for their thoughtful reviews.
All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.