{"slug": "typesafe-jev-vs-rapidfuzz-vs-sentence-transformers-for-duplicate-records", "title": "TypeSafe Jev vs RapidFuzz vs Sentence Transformers for Duplicate Records", "summary": "A CodeCut benchmark test found that TypeSafe Jev, a fixed-answer classification model, can catch duplicate product records that RapidFuzz and Sentence Transformers miss, such as \"Haier 13' Silver Tube TV - HTR13\" and \"13in Crt Tv W/atsc Tuner - HTR13,\" which share model number HTR13 but no wording. The test used the Abt-Buy entity-resolution dataset of 1,081 Abt products, 1,092 Buy.com products and 1,097 known matches, with rapidfuzz v3.14.6, sentence-transformers v5.6.0, typesafe-sdk v0.7.5 and pandas v2.3.3. The author positions Jev as cheaper than running an LLM on every product because it picks from fixed answers rather than generating text, while still allowing custom matching rules.", "body_md": "Table of Contents\n\nIntroduction\n\nIn a [previous article](https://codecut.ai/text-similarity-fuzzy-matching-guide/), I compared four text similarity tools, from regex to Sentence Transformers. Fuzzy matching handled typos and word order, and embeddings handled different words with the same meaning:\n\nJon Smith    John Smith       <- typo, fuzzy matching catches it\n\nNYC          New York City    <- abbreviation, embeddings catch it\n\nOne case none of them handled well is two records that describe the same product in different words:\n\nHaier 13' Silver Tube TV - HTR13\n\n13in Crt Tv W/atsc Tuner - HTR13     <- same model (HTR13), different words\n\nAn LLM could decide what makes two products the same, for example with [PydanticAI forcing a yes or no answer](https://codecut.ai/enforce-structured-outputs-from-llms-with-pydanticai/), but running one on every product would be slow and expensive. [TypeSafe Jev](https://docs.typesafe.ai/models) is a cheaper option: it picks from fixed answers instead of writing a reply, and you can still give it your own rules.\n\nThis article tests whether Jev can catch the mistakes RapidFuzz and Sentence Transformers make, and what it costs to run.\n\n💻 Get the Code: The full script and setup are in the [GitHub companion folder](https://github.com/khuyentran1401/codecut-blog/tree/main/match_duplicate_records).\n\n  Stay Current with CodeCut\n\n  Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.\n\n    .codecut-subscribe-wrap {\n\n        display: flex;\n\n        justify-content: center;\n\n    }\n\n    .codecut-subscribe-btn {\n\n        background: #72BEFA !important;\n\n        color: #2F2D2E !important;\n\n        border: none;\n\n        border-radius: 8px;\n\n        padding: 12px 28px;\n\n        font-family: inherit;\n\n        font-size: 16px;\n\n        font-weight: 700;\n\n        cursor: pointer;\n\n        text-decoration: none !important;\n\n        display: inline-flex;\n\n        align-items: center;\n\n        justify-content: center;\n\n        transition: background 0.3s ease;\n\n    }\n\n    .codecut-subscribe-btn:hover,\n\n    .codecut-subscribe-btn:focus {\n\n        background: #5aa8e8 !important;\n\n        color: #2F2D2E !important;\n\n        text-decoration: none !important;\n\n    }\n\n    /* Mobile responsive */\n\n    @media (max-width: 480px) {\n\n        .codecut-subscribe-btn {\n\n            width: 100%;\n\n            text-align: center;\n\n        }\n\n    }\n\nSetup\n\nInstall the libraries:\n\npip install rapidfuzz sentence-transformers typesafe-sdk pandas python-dotenv\n\nThis article uses rapidfuzz v3.14.6, sentence-transformers v5.6.0, typesafe-sdk v0.7.5, and pandas v2.3.3.\n\nThe [Abt-Buy benchmark](https://dbs.uni-leipzig.de/research/projects/benchmark-datasets-for-entity-resolution) has two product lists, 1,081 products from Abt (the first list) and 1,092 from Buy.com (the second list), plus a file of the 1,097 known matches. I use it to count each tool’s correct and wrong merges.\n\nfirst list (1,081)          second list (1,092)\n\nSamsung ... SBH500  ──────  Samsung SBH500 ...\n\nSony ... CDPCX455   ──────  Sony CDP-CX455 ...\n\n...                         Sony CDP-CX355 ...\n\n                            ...\n\nThe code below shows only the call that matters for each tool. The data download and the full code are in the [companion folder](https://github.com/khuyentran1401/codecut-blog/tree/main/match_duplicate_records).\n\nMatch Names That Look Alike with RapidFuzz\n\n[RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) is a fast fuzzy string matching library. It scores how similar two strings look from 0 to 100.\n\nJohn Smith   vs   Smith, John      100   same words\n\nJohn Smith   vs   Jon Smith         95   one letter off\n\nJohn Smith   vs   Mary Jones        50   different names\n\nLet’s start with an easy case for RapidFuzz: the same product, with the same words in a different order.\n\nFirst list:    Samsung Stereo Bluetooth Headset In Black - SBH500\n\nRight match:   Samsung SBH500 Stereo Bluetooth Headset\n\nAsk RapidFuzz for the closest match in the second list:\n\nfrom rapidfuzz import fuzz, process, utils\n\ndef closest_match(product_name: str) -> tuple:\n\n    best_name, score, _ = process.extractOne(\n\n        product_name,\n\n        second_list.name,\n\n        scorer=fuzz.token_set_ratio,\n\n        processor=utils.default_process,  # lowercase and strip punctuation\n\n    )\n\n    return best_name, score\n\nbest_name, score = closest_match(\"Samsung Stereo Bluetooth Headset In Black - SBH500\")\n\nprint(f\"Closest match: {best_name}  (score {score:.1f})\")\n\nOutputClosest match: Samsung SBH500 Stereo Bluetooth Headset (score 100.0)\n\nEvery word in the second list’s name also appears in the first list’s name, so RapidFuzz gives it a perfect score and finds the right match.\n\nNow try a harder product, where the right match adds many extra words:\n\nFirst list:    GE GSD4000NWW White Built-In Dishwasher - GSD4000WH\n\nRight match:   GE GSD4000NWW Dishwasher with 5 Wash Cycles STD TUB ESTAR ( White) - gsd4000nww\n\nThey are the same product because both carry the model number GSD4000NWW.\n\nproduct_name = \"GE GSD4000NWW White Built-In Dishwasher - GSD4000WH\"\n\nright_match = \"GE GSD4000NWW Dishwasher with 5 Wash Cycles STD TUB ESTAR ( White) - gsd4000nww\"\n\nbest_name, best_score = closest_match(product_name)\n\nright_score = fuzz.token_set_ratio(product_name, right_match, processor=utils.default_process)\n\nprint(f\"Closest match: {best_name}  (score {best_score:.1f})\")\n\nprint(f\"Right match:   {right_match}  (score {right_score:.1f})\")\n\nOutputClosest match: LG Dishwasher  (score 87.0)\n\nRight match:   GE GSD4000NWW Dishwasher with 5 Wash Cycles STD TUB ESTAR ( White) - gsd4000nww  (score 75.9)\n\nThe right match shares more words, but it also adds more extra words, so it scores lower.\n\nMerging every product in the first list with its closest match gives these results:\n\nMerged\n\nCorrect\n\nWrong\n\n1,081\n\n740\n\n341\n\nNearly 1 in 3 merges is wrong.\n\nMatch Meaning with Sentence Transformers\n\n[Sentence Transformers](https://github.com/huggingface/sentence-transformers) turns text into a list of numbers, called an embedding, so texts with similar meaning end up with similar numbers. The similarity between two embeddings runs from 0 to 1.\n\nnumbers = similarity to \"cell phone\"\n\n        ★ cell phone\n\n          ● mobile phone (0.87)\n\n● laptop (0.44)\n\n● banana (0.31)\n\nBecause Sentence Transformers compares meaning instead of letters, it might do better on the dishwasher example that RapidFuzz got wrong:\n\nFirst list:    GE GSD4000NWW White Built-In Dishwasher - GSD4000WH\n\nRight match:   GE GSD4000NWW Dishwasher with 5 Wash Cycles STD TUB ESTAR ( White) - gsd4000nww\n\nAsk Sentence Transformers for the closest match:\n\nfrom sentence_transformers import SentenceTransformer, util\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\nsecond_list_vectors = model.encode([utils.default_process(name) for name in second_list.name])\n\nquery = model.encode(utils.default_process(\"GE GSD4000NWW White Built-In Dishwasher - GSD4000WH\"))\n\nbest = util.semantic_search(query, second_list_vectors, top_k=1)[0][0]\n\nprint(f\"Closest match: {second_list.name[best['corpus_id']]}  (similarity {best['score']:.2f})\")\n\nOutputClosest match: GE GSD4000NWW Dishwasher with 5 Wash Cycles STD TUB ESTAR ( White) - gsd4000nww (similarity 0.83)\n\nAs expected, Sentence Transformers finds the right match. This is because it rewards shared details such as the brand, model number, and color. Here is how much each detail adds:\n\nsimilarity to: GE GSD4000NWW White Built-In Dishwasher\n\nDishwasher                        0.65\n\n+ GE                              0.75   +0.10   brand matches\n\n+ GSD4000NWW                      0.91   +0.16   model number matches\n\n+ White                           0.95   +0.04   color matches\n\nNow let’s try a product where a wrong match shares more words than the right one. I expect Sentence Transformers to pick the wrong one:\n\nFirst list:    Sony 400 Disc MegaStorage CD Changer - CDPCX455\n\nRight match:   Sony CDP-CX455 CD Player/Changer - CDPCX455        <- same model number\n\nWrong match:   Sony CDP-CX355 300 Disc MegaStorage CD Changer     <- more words in common\n\nRun Sentence Transformers and show its top 2 matches:\n\nquery = model.encode(utils.default_process(\"Sony 400 Disc MegaStorage CD Changer - CDPCX455\"))\n\nfor hit in util.semantic_search(query, second_list_vectors, top_k=2)[0]:\n\n    print(f\"{hit['score']:.2f}  {second_list.name[hit['corpus_id']]}\")\n\nOutput0.93  Sony CDP-CX355 300 Disc MegaStorage CD Changer\n\n0.86  Sony CDP-CX455 CD Player/Changer - CDPCX455\n\nAs I expected, Sentence Transformers picks the wrong match.\n\nMerging every product in the first list with its closest match gives these results:\n\n1,081\n\n763\n\n318\n\nStill nearly 1 in 3 merges is wrong, only slightly better than RapidFuzz.\n\nAsk Jev Which Product Is the Same\n\n[TypeSafe Jev](https://docs.typesafe.ai/models) is a decision model. Instead of generating text, it answers a question with fixed answers and returns the probability of each one.\n\nFor example, here is the same question asked to a chat model and to Jev:\n\nreview:     \"Cheaper than the store, though the left earbud crackles.\"\n\nquestion:   What is this review about? (product quality, price, shipping)\n\nChat model (Claude Haiku 5.5):\n\n  **Product quality.** The review's complaint is that the left earbud\n\n  crackles. ...\n\nJev:\n\n  product quality   0.79   <- picked\n\n  price             0.21\n\n  shipping          0.00\n\nUnlike RapidFuzz and Sentence Transformers, Jev:\n\nFollows a rule you give it, so you can specify what “the same product” means in your data\n\nAnswers with a probability, so you can match the pairs with high confidence and review the rest\n\nRuns fast and cheap, so checking thousands of pairs is practical\n\nStart with the CD changer that Sentence Transformers got wrong:\n\nFirst list:    Sony 400 Disc MegaStorage CD Changer - CDPCX455\n\nRight match:   Sony CDP-CX455 CD Player/Changer - CDPCX455\n\nLet’s give Jev the product and 3 similar names from the second list, plus “none of these”. We will also specify what makes two products the same: the same model number, capacity, and color.\n\nfrom typesafe_sdk import Choice, TypeSafeClient\n\nclient = TypeSafeClient(model=\"jev-latest\")\n\ndef ask_jev(product_name: str, shortlist: list) -> dict:\n\n    response = client.system_one(\n\n        state=product_name,\n\n        questions={\"match\": Choice(\n\n            instructions=(\n\n                \"Which candidate is the same product? Same model, same capacity or size, \"\n\n                \"and same color or variant. Different model numbers mean different products.\"\n\n            ),\n\n            criteria={name: None for name in shortlist + [\"none of these\"]},\n\n        )},\n\n    )\n\n    return response.choices[\"match\"].probabilities\n\ncd_changer = ask_jev(\n\n    \"Sony 400 Disc MegaStorage CD Changer - CDPCX455\",\n\n    [\n\n        \"Sony CDP-CX355 300 Disc MegaStorage CD Changer\",\n\n        \"Sony CDP-CX455 CD Player/Changer - CDPCX455\",\n\n        \"Sony CDP-CE375 CD Player/Changer - CDPCE375\",\n\n    ],\n\n)\n\nfor name, probability in cd_changer.items():\n\n    print(f\"{probability:.2f}  {name}\")\n\nOutput0.00  Sony CDP-CX355 300 Disc MegaStorage CD Changer\n\n0.93  Sony CDP-CX455 CD Player/Changer - CDPCX455\n\n0.00  Sony CDP-CE375 CD Player/Changer - CDPCE375\n\n0.07  none of these\n\nJev gives the right match the highest probability, 0.93. This means it follows the rule we gave it: different model numbers mean different products.\n\nLet’s compare this result with the scores from the other two tools:\n\nCandidate\n\nRapidFuzz\n\nSentence Transformers\n\nJev\n\nSony CDP-CX455 CD Player/Changer (right match)\n\n73.8\n\n0.86\n\n0.93\n\nSony CDP-CX355 300 Disc MegaStorage CD Changer\n\n94.5\n\n0.93\n\n0.00\n\nSony CDP-CE375 CD Player/Changer\n\n62.8\n\n0.83\n\n0.00\n\nOnly Jev ranks the right match first. It is also confident the other two are wrong.\n\nWhat does Jev do when none of the choices is the right match? Let’s find out by giving it three wrong candidates:\n\nFirst list:    Sony Stereo Audio Receiver - STRDE197\n\nCandidates:    Sony STR-DG720 A/V Receiver - STRDG720    <- not a match\n\n               Sony STR-DG920 A/V Receiver - STRDG920    <- not a match\n\n               Sony STR-DG820 A/V Receiver - STRDG820    <- not a match\n\nreceiver = ask_jev(\n\n    \"Sony Stereo Audio Receiver - STRDE197\",\n\n    [\n\n        \"Sony STR-DG720 A/V Receiver - STRDG720\",\n\n        \"Sony STR-DG920 A/V Receiver - STRDG920\",\n\n        \"Sony STR-DG820 A/V Receiver - STRDG820\",\n\n    ],\n\n)\n\nfor name, probability in receiver.items():\n\n    print(f\"{probability:.2f}  {name}\")\n\nOutput0.00  Sony STR-DG720 A/V Receiver - STRDG720\n\n0.00  Sony STR-DG920 A/V Receiver - STRDG920\n\n0.00  Sony STR-DG820 A/V Receiver - STRDG820\n\n1.00  none of these\n\nJev picks “none of these”, which is the right answer. RapidFuzz and Sentence Transformers always return a closest name, so they would merge a wrong one.\n\nCombine Sentence Transformers and Jev\n\nWe can’t give Jev all 1,092 names from the second list as answers, because a Choice question takes at most 255. So let’s use Sentence Transformers to keep the 3 closest names for each product, and Jev chooses among them.\n\n1,092 names in the second list\n\n              │  Sentence Transformers\n\n              ▼\n\n3 closest names\n\n              │  Jev\n\n              ▼\n\n1 match, or none\n\nBecause Jev returns a probability, each product can be merged, left unmatched, or sent to a person to review.\n\nTop answer \"none of these\"?  ── yes ──▶  leave unmatched\n\n          │ no\n\n          ▼\n\nProbability above 0.7?       ── yes ──▶  merge\n\n          │ no\n\n          ▼\n\nProbability below 0.3?       ── yes ──▶  leave unmatched\n\n          │ no\n\n          ▼\n\nsend to a person\n\nRunning this on every product in the first list gives these results:\n\nRoute\n\nPairs\n\nCorrect\n\nWrong\n\nMerge (p > 0.7)\n\n808\n\n807\n\n1\n\nSend to a person\n\n62\n\nLeave unmatched\n\n211\n\nJev merges 808 pairs with only one wrong. The run took about 35 seconds and cost about $0.03 for 1,081 questions. With one wrong merge out of 808, the merges are reliable enough to apply automatically.\n\nWhich Approach Should You Use?\n\nHere are the three approaches side by side:\n\nApproach\n\nMerged\n\nCorrect\n\nWrong merges\n\nRapidFuzz\n\n1,081\n\n740\n\n341\n\nSentence Transformers\n\n1,081\n\n763\n\n318\n\nTop 3 + Jev (p > 0.7)\n\n808\n\n807\n\n1\n\nRapidFuzz and Sentence Transformers are free, but about 1 in 3 of their merges is wrong, so you still have to check every one of them. Jev costs about $0.03 for the whole list and makes only one wrong merge, so you only need to look at the few products it is unsure about. This means you end up saving a lot by spending a little.\n\nSo should you always use Jev? No. I recommend starting with the free tools to see how well they work for your data, and moving to more complicated tools only if the current ones are not good enough.\n\nAs a rule of thumb:\n\nUse RapidFuzz when names look almost the same but differ in typos, case, punctuation, or word order.\n\nUse Sentence Transformers when names use different words for the same product, and no near-identical products are in the list.\n\nUse Jev when you need rules to decide if two products are the same, such as matching the model number, capacity, or color.\n\nRelated Tutorials\n\n[Structured Output Tools for LLMs Compared](https://codecut.ai/structured-llm-outputs-tools-comparison/): Get fixed, typed answers from a chat model\n\n[Auto-Optimize an LLM Classifier with DSPy](https://codecut.ai/dspy-auto-optimize-llm-classifier/): Build a classifier from prompts and let DSPy tune them\n\n[langextract vs spaCy](https://codecut.ai/langextract-vs-spacy-entity-extraction-comparison/): AI-powered against rule-based extraction of entities from text\n\nThe post [TypeSafe Jev vs RapidFuzz vs Sentence Transformers for Duplicate Records](https://codecut.ai/typesafe-jev-vs-rapidfuzz-sentence-transformers/) appeared first on [CodeCut](https://codecut.ai).", "url": "https://wpnews.pro/news/typesafe-jev-vs-rapidfuzz-vs-sentence-transformers-for-duplicate-records", "canonical_source": "https://codecut.ai/typesafe-jev-vs-rapidfuzz-sentence-transformers/", "published_at": "2026-11-11 02:40:44+00:00", "updated_at": "2026-11-11 02:51:54.164070+00:00", "lang": "en", "topics": ["ai-tools", "machine-learning", "natural-language-processing", "ai-products"], "entities": ["TypeSafe Jev", "RapidFuzz", "Sentence Transformers", "CodeCut", "Abt-Buy", "typesafe-sdk", "PydanticAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/typesafe-jev-vs-rapidfuzz-vs-sentence-transformers-for-duplicate-records", "markdown": "https://wpnews.pro/news/typesafe-jev-vs-rapidfuzz-vs-sentence-transformers-for-duplicate-records.md", "text": "https://wpnews.pro/news/typesafe-jev-vs-rapidfuzz-vs-sentence-transformers-for-duplicate-records.txt", "jsonld": "https://wpnews.pro/news/typesafe-jev-vs-rapidfuzz-vs-sentence-transformers-for-duplicate-records.jsonld"}}