{"slug": "i-built-an-oracle-that-refuses-to-answer", "title": "I built an oracle that refuses to answer", "summary": "A developer built a public \"oracle\" application around Perplexity's pplx-decider-flash-v1.2, a 4B-parameter decision model with its language head removed that returns calibrated probabilities over 255 answer codes in a single prefill pass with zero output tokens. Running on an M4 Mac Mini with 64 GB of unified memory, the developer patched the vendor loader to keep BF16 weights at 8.5 GiB instead of doubling to float32, achieving 5.2-second warm loads and 2.7–3.7-second reads, and designed the interface to ask four separate questions per request after the model card noted single-question accuracy beats grouped queries by up to 20 points on some categories.", "body_md": "**A 4B decision model on a Mac Mini, and the 30 lines of UI that make it honest.**\n\nAsk a chatbot \"should I take this job?\" and you get three paragraphs. They will be well written, they will mention trade-offs, and they will quietly commit to an answer that the model did not actually compute. The prose manufactures confidence. That is the failure mode nobody puts in the eval table.\n\nSo when Perplexity released `pplx-decider-flash-v1.2`, a decision model with the language head removed, the interesting thing was not the benchmark score. It was the shape. This model cannot write a paragraph. It takes a state, answers typed questions about it, and returns calibrated probabilities. Zero output tokens. One prefill pass per read.\n\nThat is a machine for honest answers. I built a public oracle around it and spent most of the time on the opposite of a normal ML project: not making the number better, but making the interface refuse to oversell it.\n\nThe checkpoint is 4B parameters, distilled from Perplexity's 27B decision models. The backbone is Qwen3.5-4B. The language head is gone, replaced by a linear readout over 255 answer codes, each code mapping to a token id.\n\nYou do not send a prompt. You send a request:\n\n```\n{\n  \"state\": \"My integration keeps failing with a 401 since we rotated our API key.\",\n  \"questions\": {\n    \"urgent\": { \"type\": \"noul\", \"instructions\": \"Is the customer blocked right now?\" },\n    \"team\": {\n      \"type\": \"choice\",\n      \"criteria\": { \"billing\": \"Charges and refunds\", \"support\": \"Integration errors\" }\n    },\n    \"frustration\": { \"type\": \"score\", \"criteria\": [\"calm\", \"mildly annoyed\", \"frustrated\", \"angry\"] }\n  }\n}\n```\n\nand you get probabilities back for every question at once, in a single forward pass, at the cost of reading the prompt.\n\nThe card also notes something that shaped the whole build: for this smaller checkpoint, **one question per request is more accurate than grouping** on most multi-question tasks, by up to 20 points on some categories. The 27B model behaves the other way. So the app asks four separate questions and batches them into one call.\n\nThe card says CUDA. I have an M4 Mac Mini with 64 GB of unified memory and no NVIDIA silicon anywhere in the house. Two things had to be true: the hybrid linear-attention backbone had to run on MPS, and the 8.5 GiB of BF16 weights had to fit without becoming 18 GB.\n\nThe first part was free. `torch 2.14` with MPS loaded the Qwen3.5 class and produced logits on the first try. The second part needed a two line seam, because the vendor loader picks the dtype by device:\n\n```\ndtype = torch.bfloat16 if self.device.type == \"cuda\" else torch.float32\n```\n\nOn a Mac that means float32, which doubles the footprint for no accuracy gain, since the weights ship in BF16. I did not patch their source. I wrapped the classmethod around the load and put it back afterwards:\n\n``` python\ndef patch_qwen3_5_dtype(torch, target):\n    from transformers.models.qwen3_5 import modeling_qwen3_5 as m\n    original = m.Qwen3_5Model.from_pretrained.__func__\n\n    def wrapper(cls, *args, **kwargs):\n        kwargs[\"dtype\"] = target\n        return original(cls, *args, **kwargs)\n\n    m.Qwen3_5Model.from_pretrained = classmethod(wrapper)\n    return lambda: setattr(m.Qwen3_5Model, \"from_pretrained\", classmethod(original))\n```\n\nLoad to GPU takes 5.2 seconds warm. A full read, four passes over roughly 750 input tokens, takes 2.7 to 3.7 seconds. The backbone falls back to reference kernels because `flash-linear-attention` and `causal_conv1d` are not available on macOS, and that is the honest reason a read is three seconds instead of three hundred milliseconds.\n\nThe plan was simple: question in, probability out, bar underneath. I wrote the wrapper, asked it a few things, and immediately hit the thing that made the project worth building.\n\nWith no context, the model is confidently wrong in a very specific way.\n\n| Question (no context) | p(yes) | evidence | \n|---|---|---|\n| Is it worth learning Rust in 2026? | 0.93 | 0.02 | \n| Will this interview go well? | 0.83 | 0.02 | \n| Should I get a dog? | 0.89 | 0.01 | \n| Will my startup be profitable in two years? | 0.76 | 0.02 | \n| Will Bitcoin be higher in a year? | 0.44 | 0.01 | \n| Should I text my ex? | 0.47 | 0.02 | \n\nNothing in the text decides any of these. The model is returning a prior, and the prior leans yes, because the proposition being tested is phrased as \"the answer is yes\". A naive UI would render 93% in 130px type and call it a forecast. That is the exact failure the model's design was supposed to prevent, reintroduced by the front end.\n\nThe same model with real evidence behaves completely differently:\n\n| Question with the situation pasted | p(yes) | evidence | \n|---|---|---|\n| Should I quit my job to build this full time? | 0.07 | 0.92 | \n| Should I take the offer? (bad commute, 20% raise) | 0.44 | 0.72 | \n| Will this launch land on time? (vendor blocked) | 0.12 | 0.71 | \n| Should I raise the price from 12 to 29 dollars? | 0.70 | 0.90 | \n\nSame weights, same temperature. The number only means something when the text carries something to read.\n\nThe fix was to stop treating the probability as the product and make the *state* the product. Four states, checked in this order:\n\n``` js\nfunction verdict(p, spread, evidence) {\n  const d = Math.abs(p - 0.5);\n  if (evidence < 0.15) return { label: 'guessing', kind: 'guess' };\n  if (spread >= 0.18) return { label: 'unstable', kind: 'unstable' };\n  if (d <= 0.06) return { label: 'coin flip', kind: 'flip' };\n  const side = p > 0.5 ? 'yes' : 'no';\n  if (d <= 0.12) return { label: `barely ${side}`, kind: 'lean' };\n  if (d <= 0.25) return { label: `leans ${side}`, kind: 'lean' };\n  return { label: side, kind: 'clear' };\n}\n```\n\n`guessing` comes first on purpose. If the evidence score is under 0.15, nothing downstream matters: the bar turns grey, the accent colour is removed, and the card says the number is a prior, not a prediction. The most useful screen in the entire app is the one where it refuses.\n\nCalibration is a property of the checkpoint that I cannot verify in an afternoon. What I *can* measure, on every single read, is whether the answer survives being asked differently. So each question is asked three ways:\n\n```\nPHRASINGS = (\n    'The answer to the question \"{q}\" is yes.',\n    'If someone asks \"{q}\", the correct answer is yes.',\n    'Taking the situation into account, \"{q}\" should be answered yes.',\n)\n```\n\n`p` is the mean of the three, `spread` is max minus min. On evidence-backed questions the spread sits around 0.03 to 0.06. On evidence-free ones it regularly hits 0.16 to 0.24, and the card says so in plain language: *\"the three readings landed 24 points apart, so treat this as unstable.\"*\n\nThis costs two extra passes and about two seconds. It is the cheapest credibility the app has.\n\nThe fourth pass is the evidence score, an ordinal question with four levels:\n\n```\nEVIDENCE_LEVELS = (\n    \"no evidence, only the question itself\",\n    \"a vague hint, nothing checkable\",\n    \"some relevant specifics\",\n    \"concrete numbers, constraints and details\",\n)\n```\n\nNormalised to 0..1 and shown as `none / a hint / some / strong`. It is not a measure of truth. It is a measure of how much the model had to work with, which is the thing users actually need to know before they act on a number.\n\nAn oracle that argues with you is only interesting if other people can see the argument. The share card is generated client-side on a 1080x1350 canvas: the question, the verdict word, the number, the bar with a notch at 50, one sentence of explanation, and the byline. No server round trip, no image library, no dependencies.\n\nThat choice matters more than it looks. The card carries `guessing` and `unstable` as loudly as it carries `yes`, so the thing people post when they are annoyed is also the thing that explains the model's limits.\n\n| Model load | 5.2 s warm | \n| One read (4 passes) | 2.7 to 3.7 s | \n| Output tokens | 0 | \n| Disk | 8.5 GiB | \n| Memory | ~10 GB unified | \n| Hardware | one M4 Mac Mini | \n\nNo API key, no per-token bill, no rate limit to design around. One process-wide lock serialises reads, which is the right amount of concurrency for a demo and an obvious thing to replace with a queue if this ever met traffic.\n\nThe missing piece is a calibration harness. Two hundred questions with known outcomes, predicted p on one axis and observed frequency on the other, published as a reliability curve. Until that exists, the calibration claim belongs to Perplexity's development set, not to my question distribution, and the README says exactly that.\n\nAfter that: vision reads, since the checkpoint already accepts images up to 1.6 megapixels and \"should I buy this listing?\" with a screenshot is a two hour change. Then a decision journal, so the app can show you the reads where the model beat you and the ones where you beat it.\n\nThe whole project is five files, one checkpoint, and a SQLite table. It runs on a machine that sits on my desk and pulls about 40 watts.\n\nIf you build on it, keep the rule: no number without its evidence, and never let a state pretend to be a prediction.\n\nCode & more: [https://www.dailybuild.xyz/project/295-coin](https://www.dailybuild.xyz/project/295-coin)", "url": "https://wpnews.pro/news/i-built-an-oracle-that-refuses-to-answer", "canonical_source": "https://dev.to/harishkotra/i-built-an-oracle-that-refuses-to-answer-2ka4", "published_at": "2026-10-26 13:32:57+00:00", "updated_at": "2026-10-26 13:44:51.526583+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools"], "entities": ["Perplexity", "pplx-decider-flash-v1.2", "Qwen3.5-4B", "Apple M4 Mac Mini", "PyTorch"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-an-oracle-that-refuses-to-answer", "markdown": "https://wpnews.pro/news/i-built-an-oracle-that-refuses-to-answer.md", "text": "https://wpnews.pro/news/i-built-an-oracle-that-refuses-to-answer.txt", "jsonld": "https://wpnews.pro/news/i-built-an-oracle-that-refuses-to-answer.jsonld"}}