{"slug": "dksplit-fast-word-segmentation-for-python", "title": "DKSplit – Fast Word Segmentation for Python", "summary": "DKSplit v1.0.2, a fast character-level word segmentation library for Python, achieves 86.5% strict exact match accuracy on a 1,000-sample benchmark of domain prefixes, outperforming WordSegment (65.2%) and WordNinja (51.0%). The 9 MB ONNX model runs CPU-only and is designed for web-style concatenated strings such as domain names, hashtags, and usernames. Top-3 candidate coverage reaches 98.5% on the 1,000-sample set and 97.8% on a 5,000-sample set.", "body_md": "Fast character-level segmentation for web-style concatenated strings — domain names, hashtags, usernames, slugs. 9 MB ONNX model, CPU-only.\n\n```\npip install dksplit\n```\n\nRequires Python >= 3.8. Dependencies: numpy, onnxruntime.\n\n``` python\nimport dksplit\n\n# Single best segmentation\ndksplit.split(\"kubernetescluster\")\n# ['kubernetes', 'cluster']\n\n# Batch (faster for large volumes; results identical to split())\ndksplit.split_batch([\"openaikey\", \"microsoftoffice\", \"bitcoinprice\"])\n# [['openai', 'key'], ['microsoft', 'office'], ['bitcoin', 'price']]\n\n# Ranked candidates for ambiguous inputs\ndksplit.split3(\"noranite\")        # top-3, best first\n# [['nora', 'nite'], ['noranite'], ['nor', 'anite']]\n\ndksplit.split5(\"pikahug\")         # top-5\n# [['pikahug'], ['pika', 'hug'], ['pik', 'ahug'], ['pikah', 'ug'], ['pi', 'kahug']]\n\ndksplit.split_topk(\"chatgptlogin\", k=3)   # any k\n# [['chatgpt', 'login'], ['chatgptlogin'], ['chatgpt', 'log', 'in']]\n```\n\nTypical uses: spotting brands and lookalikes in newly registered domains\n(`yourbrandlogin`\n\n, `getyourbrand`\n\n), extracting keywords from domains, hashtags,\nand URLs, normalizing concatenated identifiers before matching and dedup,\nunderstanding spaceless search queries.\n\n— one answer per input; pipelines, aggregation, statistics.`split()`\n\n— ranked candidates for recall-sensitive matching or for reranking with your own signals (brand lists, frequency data); an acceptable segmentation is in the top-3 candidates 98.5% of the time (top-5: 99.3%).`split_topk()`\n\nBugfix: `split_batch()`\n\ncould differ from `split()`\n\non rare inputs; results\nare now guaranteed identical. Pass `exact=False`\n\nto keep the old ~2x faster\nbehavior.\n\n1,000 hand-audited domain prefixes drawn from the\n[Newly Registered Domains Database (NRDS)](https://domainkits.com/download/nrds)\n(.com feed). No filtering or cherry-picking on segmentation difficulty. Ground\ntruth was established through multi-model cross-validation (BiLSTM, Qwen 9B\nLoRA, Gemma 31B) and human audit. Each row provides a primary `truth`\n\nand an\noptional `might_right`\n\nfield for genuinely ambiguous cases (e.g.\nbrand-versus-compound).\n\nBoth benchmark sets ship in this repo's\n[ /benchmark](https://github.com/ABTdomain/dksplit/tree/main/benchmark)\ndirectory:\n\n`sample_1000.csv`\n\nand `benchmark_5000.csv`\n\n, a larger set built the\nsame way (also on Hugging Face as\n[ABTdomain/dksplit-benchmark](https://huggingface.co/datasets/ABTdomain/dksplit-benchmark)). To explore domain data yourself, register at\n\n[domainkits.com](https://domainkits.com)— fresh .com NRD downloads are free.\n\n| Model | Strict EM | Lenient EM |\n|---|---|---|\nDKSplit v1.0.2 |\n86.5% |\n91.5% |\n| WordSegment | 65.2% | 69.5% |\n| WordNinja | 51.0% | 54.0% |\n\nStrict EM counts only exact matches against `truth`\n\n. Lenient EM also accepts\nthe `might_right`\n\nalternative when present.\n\nTop-k coverage (an acceptable segmentation is present within the candidates):\n\n| Benchmark | top-1 | top-3 | top-5 |\n|---|---|---|---|\n| 1,000 samples | 91.5% | 98.5% | 99.3% |\n| 5,000 samples | 90.4% | 97.8% | 99.0% |\n\n```\ngit clone https://github.com/ABTdomain/dksplit.git\ncd dksplit/benchmark\npip install dksplit wordsegment wordninja\npython run_benchmark.py                     # 1,000-sample set\npython run_benchmark.py benchmark_5000.csv  # 5,000-sample set\n```\n\nAdding your own segmenter to the comparison is a one-line change in\n`run_benchmark.py`\n\n. Pull requests for ambiguous samples are welcome.\n\n| Input | DKSplit v1.0.2 | WordSegment | WordNinja |\n|---|---|---|---|\n`chatgptprompts` |\nchatgpt prompts |\nchat gpt prompts | chat gp t prompts |\n`spotifywrapped` |\nspotify wrapped |\nspot if y wrapped | spot if y wrapped |\n`ethereumwallet` |\nethereum wallet |\ne there um wallet | e there um wallet |\n`kubernetescluster` |\nkubernetes cluster |\nku bernet es cluster | ku berne tes cluster |\n`whatsappstatus` |\nwhatsapp status |\nwhat sapp status | what s app status |\n`drwatsonai` |\ndr watson ai |\ndr watson a i | dr watson a i |\n`escribirenvozalta` |\nescribir en voz alta |\nescribir env oz alta | es crib ire nv oz alta |\n`tuvasou` |\ntu vas ou |\ntuva sou | tuva so u |\n`candidiasenuncamais` |\ncandidiase nunca mais |\ncandid iase nunca mais | can didi as e nun cama is |\n\nDKSplit treats segmentation as a character-level sequence labeling task. The training data includes LLM-labeled domain segmentations, brand names, personal name combinations, multilingual phrases (English, French, German, Spanish, and more), and tech product names. At inference, the BiLSTM runs as an INT8-quantized ONNX model and CRF decoding is performed in NumPy. No GPU required; around 800 samples per second on a single CPU thread.\n\n**Why BiLSTM-CRF:** character precision, CPU-only inference, a 9 MB\nartifact — built for millions of strings per day. Design rationale and\nfailure-mode comparisons (dictionary segmenters, DeBERTa-V3, LLMs):\n[blog post](https://abtdomain.com/blog/2026/04/dksplit-update-cleaner-benchmark-first-deberta-run-different-failure-modes/).\n\n**Brand-aware:** recognizes thousands of brands, tech products, and proper nouns**Multilingual:** English, French, German, Spanish, and romanized text**Lightweight:** 9 MB model, minimal dependencies (numpy + onnxruntime)**Offline:** no API keys, no internet required**Top-k candidates:**`split3`\n\n/`split5`\n\n/`split_topk`\n\nreturn ranked alternative segmentations\n\n**Characters:**`a-z`\n\nand`0-9`\n\n, auto-lowercased. For best results pass letter-only runs: split off digits and separators (`-`\n\n,`.`\n\n,`_`\n\n) with simple rules first — those boundaries are a job for rules, not the model.**Max length:** 64 characters.**Script:** Latin script only. Non-Latin scripts (汉字, かな, 한글, العربية) are not supported.**Ambiguity:** some inputs are genuinely ambiguous.`split()`\n\noptimizes for the most common interpretation; use`split_topk()`\n\nwhen you need the alternatives.**Rare languages:** accuracy is highest on English and major European languages.\n\n- Read more about DKSplit:\n[DKSplit on EuroHPC](https://abtdomain.com/blog/tag/eurohpc) - Website:\n[domainkits.com](https://domainkits.com),[abtdomain.com](https://abtdomain.com) - PyPI:\n[pypi.org/project/dksplit](https://pypi.org/project/dksplit) - Hugging Face (LLM variant):\n[ABTdomain/dksplit-qwen-lora](https://huggingface.co/ABTdomain/dksplit-qwen-lora) - Issues:\n[GitHub Issues](https://github.com/ABTdomain/dksplit/issues)\n\n[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\nAttribution required: credit \"DKSplit by [ABTdomain](https://abtdomain.com)\"\nin your README, documentation, about page, or API response metadata.\n\nThe model was trained on the\n[Leonardo Booster](https://www.hpc.cineca.it/systems/hardware/leonardo/)\nsupercomputer at CINECA, Italy, with computing resources provided by the\n[EuroHPC Joint Undertaking](https://eurohpc-ju.europa.eu/) through the\nPlayground Access program (EHPC-AIF-2026PG01-281). We thank EuroHPC JU for\nenabling SMEs to explore new possibilities with world-class HPC infrastructure.", "url": "https://wpnews.pro/news/dksplit-fast-word-segmentation-for-python", "canonical_source": "https://github.com/ABTdomain/dksplit", "published_at": "2026-07-23 22:08:18+00:00", "updated_at": "2026-07-23 22:22:33.769420+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "developer-tools"], "entities": ["DKSplit", "WordSegment", "WordNinja", "ONNX", "ABTdomain", "Newly Registered Domains Database", "domainkits.com", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/dksplit-fast-word-segmentation-for-python", "markdown": "https://wpnews.pro/news/dksplit-fast-word-segmentation-for-python.md", "text": "https://wpnews.pro/news/dksplit-fast-word-segmentation-for-python.txt", "jsonld": "https://wpnews.pro/news/dksplit-fast-word-segmentation-for-python.jsonld"}}