{"slug": "show-hn-matching-jev-on-banking77-at-a-thousandth-of-the-cost", "title": "Show HN: Matching Jev on BANKING77 at a thousandth of the cost", "summary": "Cymetica's Tuatara Vector Model scored 91.79% on the 3,080-message BANKING77 test set, a statistical tie with the pinned jev-1.13.0 model's 92.40% at roughly a thousandth of the cost, according to the company's blog post. The run used all 77 intents as options in a single question and up to 24 retrieved training examples per message; jev-1.13.0 answered 2,846 of 3,080 messages correctly. BANKING77 is PolyAI's public dataset of 13,083 bank messages split into 10,003 training and 3,080 test examples.", "body_md": "We developed an early vector embedding model at Lawrence Berkeley National Lab and extended it called the Tuatara Vector Model. It's a blend and scored against Jev's 3,080 BANKING77 test messages resulting in 91.79% versus 92.40%, a statistical tie with some good cost savings.\n\nAs most may know, BANKING77 is a public dataset from PolyAI with 13,083 messages to a bank, each labelled with intents. The split was 10,003 messages for training and 3,080 for testing.\n\nThe run used the pinned model jev-1.13.0, all 77 intents as options in a single question, and up to 24 training examples retrieved for each message. Jev got 2,846 of 3,080 right.\n\nRecomputing Jev’s accuracy from the file gives 92.40%, the same figure the experiment reports.\n\nMore stats: [https://cymetica.com/blog/matching-jev-on-banking77-at-a-tho...](https://cymetica.com/blog/matching-jev-on-banking77-at-a-thousandth-of-the-cost)\n\nComments URL: [https://news.ycombinator.com/item?id=49867086](https://news.ycombinator.com/item?id=49867086)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-matching-jev-on-banking77-at-a-thousandth-of-the-cost", "canonical_source": "https://news.ycombinator.com/item?id=49867086", "published_at": "2026-09-27 14:47:05+00:00", "updated_at": "2026-09-27 15:01:33.327683+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-research"], "entities": ["Cymetica", "Tuatara Vector Model", "jev-1.13.0", "BANKING77", "PolyAI", "Lawrence Berkeley National Lab"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-matching-jev-on-banking77-at-a-thousandth-of-the-cost", "markdown": "https://wpnews.pro/news/show-hn-matching-jev-on-banking77-at-a-thousandth-of-the-cost.md", "text": "https://wpnews.pro/news/show-hn-matching-jev-on-banking77-at-a-thousandth-of-the-cost.txt", "jsonld": "https://wpnews.pro/news/show-hn-matching-jev-on-banking77-at-a-thousandth-of-the-cost.jsonld"}}