Fast character-level segmentation for web-style concatenated strings — domain names, hashtags, usernames, slugs. 9 MB ONNX model, CPU-only.
pip install dksplit
Requires Python >= 3.8. Dependencies: numpy, onnxruntime.
import dksplit
dksplit.split("kubernetescluster")
dksplit.split_batch(["openaikey", "microsoftoffice", "bitcoinprice"])
dksplit.split3("noranite") # top-3, best first
dksplit.split5("pikahug") # top-5
dksplit.split_topk("chatgptlogin", k=3) # any k
Typical uses: spotting brands and lookalikes in newly registered domains
(yourbrandlogin
, getyourbrand
), extracting keywords from domains, hashtags, and URLs, normalizing concatenated identifiers before matching and dedup, understanding spaceless search queries.
— one answer per input; pipelines, aggregation, statistics.split()
— ranked candidates for recall-sensitive matching or for reranking with your own signals (brand lists, frequency data); an acceptable segmentation is in the top-3 candidates 98.5% of the time (top-5: 99.3%).split_topk()
Bugfix: split_batch()
could differ from split()
on rare inputs; results
are now guaranteed identical. Pass exact=False
to keep the old ~2x faster behavior.
1,000 hand-audited domain prefixes drawn from the
Newly Registered Domains Database (NRDS)
(.com feed). No filtering or cherry-picking on segmentation difficulty. Ground
truth was established through multi-model cross-validation (BiLSTM, Qwen 9B
LoRA, Gemma 31B) and human audit. Each row provides a primary truth
and an
optional might_right
field for genuinely ambiguous cases (e.g. brand-versus-compound).
Both benchmark sets ship in this repo's /benchmark directory:
sample_1000.csv
and benchmark_5000.csv
, a larger set built the same way (also on Hugging Face as ABTdomain/dksplit-benchmark). To explore domain data yourself, register at
domainkits.com— fresh .com NRD downloads are free.
| Model | Strict EM | Lenient EM |
|---|---|---|
| DKSplit v1.0.2 | ||
| 86.5% | ||
| 91.5% | ||
| WordSegment | 65.2% | 69.5% |
| WordNinja | 51.0% | 54.0% |
Strict EM counts only exact matches against truth
. Lenient EM also accepts
the might_right
alternative when present.
Top-k coverage (an acceptable segmentation is present within the candidates):
| Benchmark | top-1 | top-3 | top-5 |
|---|---|---|---|
| 1,000 samples | 91.5% | 98.5% | 99.3% |
| 5,000 samples | 90.4% | 97.8% | 99.0% |
git clone https://github.com/ABTdomain/dksplit.git
cd dksplit/benchmark
pip install dksplit wordsegment wordninja
python run_benchmark.py # 1,000-sample set
python run_benchmark.py benchmark_5000.csv # 5,000-sample set
Adding your own segmenter to the comparison is a one-line change in
run_benchmark.py
. Pull requests for ambiguous samples are welcome.
| Input | DKSplit v1.0.2 | WordSegment | WordNinja |
|---|---|---|---|
chatgptprompts |
|||
| chatgpt prompts | |||
| chat gpt prompts | chat gp t prompts | ||
spotifywrapped |
|||
| spotify wrapped | |||
| spot if y wrapped | spot if y wrapped | ||
ethereumwallet |
|||
| ethereum wallet | |||
| e there um wallet | e there um wallet | ||
kubernetescluster |
|||
| kubernetes cluster | |||
| ku bernet es cluster | ku berne tes cluster | ||
whatsappstatus |
|||
| whatsapp status | |||
| what sapp status | what s app status | ||
drwatsonai |
|||
| dr watson ai | |||
| dr watson a i | dr watson a i | ||
escribirenvozalta |
|||
| escribir en voz alta | |||
| escribir env oz alta | es crib ire nv oz alta | ||
tuvasou |
|||
| tu vas ou | |||
| tuva sou | tuva so u | ||
candidiasenuncamais |
|||
| candidiase nunca mais | |||
| candid iase nunca mais | can didi as e nun cama is |
DKSplit treats segmentation as a character-level sequence labeling task. The training data includes LLM-labeled domain segmentations, brand names, personal name combinations, multilingual phrases (English, French, German, Spanish, and more), and tech product names. At inference, the BiLSTM runs as an INT8-quantized ONNX model and CRF decoding is performed in NumPy. No GPU required; around 800 samples per second on a single CPU thread.
Why BiLSTM-CRF: character precision, CPU-only inference, a 9 MB artifact — built for millions of strings per day. Design rationale and failure-mode comparisons (dictionary segmenters, DeBERTa-V3, LLMs): blog post.
Brand-aware: recognizes thousands of brands, tech products, and proper nounsMultilingual: English, French, German, Spanish, and romanized textLightweight: 9 MB model, minimal dependencies (numpy + onnxruntime)Offline: no API keys, no internet requiredTop-k candidates:split3
/split5
/split_topk
return ranked alternative segmentations
Characters:a-z
and0-9
, auto-lowercased. For best results pass letter-only runs: split off digits and separators (-
,.
,_
) with simple rules first — those boundaries are a job for rules, not the model.Max length: 64 characters.Script: Latin script only. Non-Latin scripts (汉字, かな, 한글, العربية) are not supported.Ambiguity: some inputs are genuinely ambiguous.split()
optimizes for the most common interpretation; usesplit_topk()
when you need the alternatives.Rare languages: accuracy is highest on English and major European languages.
- Read more about DKSplit: DKSplit on EuroHPC - Website: domainkits.com,abtdomain.com - PyPI: pypi.org/project/dksplit - Hugging Face (LLM variant): ABTdomain/dksplit-qwen-lora - Issues: GitHub Issues
CC BY 4.0. Attribution required: credit "DKSplit by ABTdomain" in your README, documentation, about page, or API response metadata.
The model was trained on the Leonardo Booster supercomputer at CINECA, Italy, with computing resources provided by the EuroHPC Joint Undertaking through the Playground Access program (EHPC-AIF-2026PG01-281). We thank EuroHPC JU for enabling SMEs to explore new possibilities with world-class HPC infrastructure.