Show HN: Matching Jev on BANKING77 at a thousandth of the cost

Cymetica's Tuatara Vector Model scored 91.79% on the 3,080-message BANKING77 test set, a statistical tie with the pinned jev-1.13.0 model's 92.40% at roughly a thousandth of the cost, according to the company's blog post. The run used all 77 intents as options in a single question and up to 24 retrieved training examples per message; jev-1.13.0 answered 2,846 of 3,080 messages correctly. BANKING77 is PolyAI's public dataset of 13,083 bank messages split into 10,003 training and 3,080 test examples.

We developed an early vector embedding model at Lawrence Berkeley National Lab and extended it called the Tuatara Vector Model. It's a blend and scored against Jev's 3,080 BANKING77 test messages resulting in 91.79% versus 92.40%, a statistical tie with some good cost savings. As most may know, BANKING77 is a public dataset from PolyAI with 13,083 messages to a bank, each labelled with intents. The split was 10,003 messages for training and 3,080 for testing. The run used the pinned model jev-1.13.0, all 77 intents as options in a single question, and up to 24 training examples retrieved for each message. Jev got 2,846 of 3,080 right. Recomputing Jev’s accuracy from the file gives 92.40%, the same figure the experiment reports. More stats: https://cymetica.com/blog/matching-jev-on-banking77-at-a-tho... https://cymetica.com/blog/matching-jev-on-banking77-at-a-thousandth-of-the-cost Comments URL: https://news.ycombinator.com/item?id=49867086 https://news.ycombinator.com/item?id=49867086 Points: 1 Comments: 0