How we use decision models

PostHog has moved decision models into production as cheap gates and routers ahead of expensive LLM calls, reporting that decision models are up to 444.6x cheaper and 193.6x faster than frontier models. In PostHog's Replay Vision PII-scrubbing pipeline, two yes/no decision-model questions with a 0.4 threshold let 98.7% of session summaries skip a Gemini call entirely, and the PostHog in Slack bot uses a decision model to route messages to the cheapest suitable model. PostHog cautions that a 0.85 probability score still means roughly 150 wrong answers per 1,000 checks, so thresholds should be set according to which error costs more.

LLMs love to yap. Ask a simple question and you get a few hundred words of "it depends." It's annoying as hell. This is why we love decision models like Jev https://typesafe.ai/ for simpler tasks. You ask a structured question and get a typed answer with a probability, which looks something like this: - Noul. Are hedgehogs mammals? → Yes, 0.99 - Choice. Which animal family do hedgehogs belong to? → Mammal, 0.99 - Score. What is the current conservation status of hedgehogs? → Least Concern, 0.77 Whereas LLMs return free-form and overly confident prose, decision models make model uncertainty explicit. In a calibrated model, answers that score 0.8 are right about 80% of the time. That means a score of 0.5 is a coin flip. We've been experimenting with them at PostHog and several use cases have made it to production. This is everything we've learned about using decision models so far: when to use them, how to use them, and mistakes to avoid. 1. Use them to gate expensive LLM calls Decision models are up to 444.6x cheaper and 193.6x faster than frontier models, 1 footnote-1 so you can use them to check whether you even need to make an expensive LLM call at all. We do this in a pipeline that scrubs personal info in Replay Vision https://posthog.com/replay-vision?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models , our product that watches and summarizes user session recordings for you. Here's how it works: - Each session summary goes through 2 yes/no questions from a decision model. Using narrow questions like this cuts false flags to about a tenth of what a single question produced. - 1. “Does this name a real person?” - 2. “Does this show contact details or an ID?” - If either scores 0.4 or above, Gemini rewrites the summary without the personal info and it gets checked again. - Otherwise, it's clean and passes through the gate. In production https://github.com/PostHog/posthog/pull/120387 , 98.7% skip the LLM call entirely. Without the decision model, we'd have to keep fussing with prompts and regexes that miss things, or sending every summary to Gemini to catch the 1% that need it. But remember they make mistakes A probability score of 0.85 still means ~150 wrong answers per 1,000 checks, so plan ahead for false positives and false negatives. In the PII example, a false positive only costs us an extra LLM call, whereas a false negative stores personal data as clean. That's why we set the threshold low at 0.4. When false positives are the costlier mistake, use higher thresholds. For example, we recently updated the PostHog in Slack bot, so it can answer questions without being tagged https://github.com/PostHog/posthog/blob/master/products/slack app/backend/temporal/activities/untagged question.py . To avoid being annoying, it checks every message in a channel to see 1 whether it's asking for information, and 2 whether PostHog could answer it. Only messages that pass both at 0.85+ go through. Takeaway: Use decision models to filter jobs before they hit an expensive LLM call, and set the threshold based on which mistake costs you more. 2. Use them as model routers Our PostHog in Slack https://posthog.com/slack?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models bot also uses a decision model as a model router https://github.com/PostHog/posthog/blob/master/products/slack app/backend/services/model router.py . When the bot gets tagged, it first checks whether the message asked for a specific model. If not, one choice question picks from the list: “Pick the cheapest option that will most likely finish the task correctly on the first try. A failed run costs more than a smarter option.” This works because decision models are calibrated to work out of the box. Before, you'd have to use a cheap LLM, or go through all the effort of training your own classifier. But don't assume they're interchangeable Different tasks, models, and providers all have different thresholds. We were experimenting with using them to auto-merge similar logs in Error Tracking https://posthog.com/error-tracking?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models and found that thresholds changed between v1 and v2 of the same model. We had to make its thresholds configurable per PostHog project and keep the judge in shadow mode while recalibrating based on hand-labeled data https://github.com/PostHog/posthog/pull/117014 . This is one of the reasons why Fernando https://posthog.com/community/profiles/42799?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models on our growth team pushed for a model-agnostic interface early. We now have an internal AI gateway for swapping and testing models, including self-hosted ones. Takeaway: Use a choice question to route free-form LLM requests to the cheapest model that can handle it. Keep in mind that thresholds don't carry over, so set them per task and per model. 3. Use them for face-value checks If a question requires more context or chains of reasoning, it's better to use an LLM. However, you can still use a decision model if you have a way to flatten, or pre-process the input, so it's checkable at face-value. In Replay Vision, when a user searches for a theme like "onboarding friction," there's a chance for the query to return a few unexpected results because of the way embedding distance works. A decision model can’t "Rank the top 10 sessions that contain summaries that match the user's search query." It’s too complicated. Instead, we use the embeddings first to derive an initial list of session recordings to pass to the decision model. Then, the only task left for the decision model is to look at how closely each result in the list matches the user's original search term: "Does this summary describe a session that matches what the search looks for?" Results are then reranked https://github.com/PostHog/posthog/pull/107372 by that probability. On 20 test queries, this improved ranking quality by about 25%. But avoid multi-step reasoning We've tried using a decision model https://github.com/PostHog/posthog/pull/114045 for multi-step reasoning before. Before embeddings were ready in Replay Vision, we tried ranking a "Highlights" feed of sessions worth a human's time. The questions were: 1. Is this session worth watching? 2. If so, why is it worth watching? With choices like “error on screen,” “slow or stuck,” “user confused”, “user gave up", etc. Processing these two questions in order required an hourly job that sent large requests – up to 32 sessions and 12-38k tokens each – which flooded the model server and blocked other products' calls for hours. So yes, technically, you can chain decision models into multi-step judgments, but it's not operationally worth it. Leave that to LLMs, or do the heavy lifting before the decision model sees the data like in the embedding search example above. Takeaway: Decision models judge what's in front of them in a single pass. Leave multi-step reasoning to LLMs. 4. Use them in place of overqualified LLMs Decision models can take any job where an LLM is overkill, either because it's too slow or too expensive. One example is our smarter chart picker https://github.com/PostHog/posthog/pull/108065 for SQL queries. PostHog used to pick a visualization using deterministic rules like: - A date column, a number column and multiple rows → Line chart - Two or more text columns plus a number → Heatmap - Any number column → Bar chart Now, a decision model looks at both the SQL itself if your org opts in, and the first three rows of the results to suggest a better chart. The example below is based on a simple query about pageviews by browser. The original rules picked a basic aggregate line chart; with Jev, it chose a bar chart split by browser: But keep LLMs for open-ended jobs Anything more open-ended should still go to an LLM. Still, out of curiosity, we tried Jev on ranking our internal PR review agent's findings with five noul questions that roughly asked if the change is worth a developer's attention. The results were inconclusive with an AUC of 0.52. If Jev were an LLM, we’d solve this by sending more context, but Jev can’t handle large requests. Its long-context mode is also slower and less accurate. 2 footnote-2 Typical decision models just aren’t built for this kind of reasoning. Hybrid decision models like Jeeves https://github.com/PostHog/jeeves can help bridge this gap, though. Developed by our in-house AI research team https://posthog.com/teams/ai-research?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models , Jeeves is a decision model that reasons before deciding when it’s unsure. It's now our default for any decision call at PostHog that doesn't name a model, and there are similar variations by other developers, like Kev https://github.com/jaredpalmer/kev . Takeaway: Use decision models as evaluators for inputs that are too simple for a language model, yet too fuzzy for a regex. Anything more complex is still too much to hand over to a decision model. For now, at least. Words by Jina Yoon, who expects this article to be obsolete in two months. 📚 Good reads at face-value - What people are prompting in PostHog AI https://posthog.com/blog/what-people-are-prompting?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models – Cleo Lant - Someone behind the software https://karrisaarinen.com/posts/someone-behind-the-software/ – Karri Saarinen - Why we’re moving our warehouse product off DuckDB https://posthog.com/blog/moving-off-duckdb?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models – Benjamin Knofe-Vider - The sleep framework: A new way for LLMs to retain knowledge in continual learning https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/ – Ali Behrouz, Vahab Mirrokni - We made experiment queries 4x faster using precomputation https://posthog.com/blog/how-we-made-experiment-queries-fast?utm source=posthog-newsletter&utm medium=post&utm campaign=decision-models – Juraj Majerik 💼 You’d love it here, 0.99 1 footnote-anchor-1 According to TypeSafe AI https://typesafe.ai/blog/introducing-system-one-models-and-jev . 2 footnote-anchor-2 Jev takes 64k tokens per request and 32k per question by default. And even though TypeSafe added an opt-in header that raises both to 262k, they still warn that the long-context mode is slower and less accurate. Source: TypeSafe Docs https://docs.typesafe.ai/models