{"slug": "ask-ten-questions-read-the-text-once-making-a-decision-model-6-7x-faster", "title": "Ask ten questions, read the text once: making a decision model 6.7x faster", "summary": "A developer built cbjev, an open decision model that packs multiple typed questions about a document into a single encoder pass, making inference 6.7x faster than the Laya model it is fine-tuned from. On one RTX 4090, cbjev answers 10 questions over a 500-token document in 11.4 ms versus Laya's 75.8 ms, while also scoring higher mean accuracy (0.741 vs 0.710) across 15 English suites and cutting option-order sensitivity from 7.8% to 0.2%. The project reuses Laya's Apache-2.0 weights and ships a server compatible with TypeSafe Jev's /v1/systemone wire format.", "body_md": "A lot of \"AI\" in production isn't generation at all. It's a pile of small decisions about a piece of text: *which team should handle this ticket? how urgent is it? is the customer threatening to leave? is this e-mail phishing?* Sending each of those to an LLM and parsing the answer is slow, expensive and occasionally creative in ways you didn't ask for.\n\n[Laya](https://github.com/NandhaKishorM/laya) is an open model for exactly this job: you send a state (a ticket, an e-mail, a JSON document) and typed questions — `choice`, `score`, or a yes/no `noul` — and a ModernBERT encoder returns calibrated probabilities in one forward pass. I built **[cbjev](https://github.com/tomek7667/cbjev)** on top of it. This post is about the one idea that made it fast, how it could reuse Laya's weights, and what didn't work.\n\nLaya builds one sequence per question:\n\n```\n[CLS] question 1 [SEP] options [SEP] document [SEP]\n[CLS] question 2 [SEP] options [SEP] document [SEP]\n...\n```\n\nAsk ten questions about a 500-token document and the encoder processes the document ten times — about 5,500 tokens for a call whose unique content is about 1,000.\n\ncbjev packs a whole call into one row:\n\n```\n[CLS] q1 | q2 | ... | q10 | document [SEP]\n```\n\nand shapes the attention mask so that:\n\n`[CLS]`\nThat last detail is the important one. With a single question, the row is token for token and position for position exactly what Laya sees. So a Laya checkpoint dropped into this layout already works (I measured 0.749 on the 5-question typed-decisions benchmark before any training, against 0.768 in its own layout), and fine-tuning starts from Laya's full ability instead of relearning the task.\n\nMy first attempt didn't have this: the document came first and could not read the questions. It was just as fast, but fine-tuning had to rebuild skills Laya already had, and it kept losing on half the benchmarks. Switching to the \"shared\" layout above is what made the accuracy numbers work.\n\nOnce the token count is down, a small call is dominated by overhead, not math:\n\n`transformers` model at runtime) with bf16 matmuls over an fp32 residual stream,`torch.compile`\nMeasured side by side with Laya on one RTX 4090, same cases, through both libraries' public `predict` APIs:\n\n|  | cbjev | Laya (better checkpoint) | \n|---|---|---|\n| 10 questions, 500-token document | **11.4 ms** | 75.8 ms | \n| 1 question, short ticket | **3.0 ms** | 5.4 ms (TileLang fast path) | \n| mean accuracy, 15 English suites | **0.741** | 0.710 | \n| typed-decisions, 2,000 decisions | **0.783** | 0.768 | \n| answers that change when options are reordered | **0.2 %** | 7.8 % | \n| MASSIVE intent, 51 languages | **0.436** | 0.401 | \n\nThe option-order number comes from a cheap trick the packed layout makes almost free: every choice and score question is also asked with its options reversed, and the two answers are averaged. That's one extra short segment, not another pass.\n\n```\npip install \"cbjev[serve] @ git+https://github.com/tomek7667/cbjev\"\npython\nimport cbjev\n\nagent = cbjev.load()   # weights download from Hugging Face\nres = agent.predict(\n    {\"body\": \"Billed twice for March. Refund it today or we cancel.\"},\n    {\"team\": {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n              \"criteria\": {\"billing\": \"invoices, refunds\", \"technical\": \"bugs\", \"other\": \"anything else\"}},\n     \"churn\": {\"type\": \"noul\", \"instructions\": \"Does the customer threaten to cancel their subscription?\"}},\n)\n```\n\nIt also ships a server that speaks TypeSafe Jev's `/v1/systemone` wire format, so an existing Jev client can point at it.\n\nThanks to Convai Innovations for releasing Laya openly; cbjev is fine-tuned from their Apache-2.0 checkpoints. I'd love to hear where it breaks on your data.", "url": "https://wpnews.pro/news/ask-ten-questions-read-the-text-once-making-a-decision-model-6-7x-faster", "canonical_source": "https://dev.to/_tomek7667/ask-ten-questions-read-the-text-once-making-a-decision-model-67x-faster-k0e", "published_at": "2026-09-25 15:34:47+00:00", "updated_at": "2026-09-25 16:01:17.881849+00:00", "lang": "en", "topics": ["natural-language-processing", "ai-tools", "machine-learning", "ai-infrastructure"], "entities": ["cbjev", "Laya", "ModernBERT", "Convai Innovations", "TypeSafe Jev", "Hugging Face", "RTX 4090", "tomek7667"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ask-ten-questions-read-the-text-once-making-a-decision-model-6-7x-faster", "markdown": "https://wpnews.pro/news/ask-ten-questions-read-the-text-once-making-a-decision-model-6-7x-faster.md", "text": "https://wpnews.pro/news/ask-ten-questions-read-the-text-once-making-a-decision-model-6-7x-faster.txt", "jsonld": "https://wpnews.pro/news/ask-ten-questions-read-the-text-once-making-a-decision-model-6-7x-faster.jsonld"}}