Deterministic, tenant-scoped resolution of company jargon to canonical entities

Lexiqr, a new pip-installable Python library, resolves tenant-specific jargon to canonical entities deterministically, with typo tolerance and multilingual support. The library, requiring Python 3.10+ and using RapidFuzz, maps terms like 'flooff' to entities with character spans and score tiers, and includes CLI tools for validation and testing. It aims to replace hardcoded synonym tables and LLM guessing with an explainable, tenant-scoped resolution layer.

Deterministic, tenant-scoped resolution of company jargon to canonical entities. Quickstart quickstart • Usage usage • Docs /bmeunier1974/lexiqr/blob/main/docs • Contributing /bmeunier1974/lexiqr/blob/main/CONTRIBUTING.md Every tenant calls the same thing something different. A tenant writes a lexicon mapping their private word — flooff — to one of your canonical entities: product . lexiqr loads that lexicon and turns free-form prompts into identified matches, each carrying the character span it covers, its score tier, and any typo it corrected. Teams usually solve this by hardcoding synonym tables, retraining embeddings, or letting an LLM guess. lexiqr is a pip-installable resolution layer instead: deterministic, explainable, and scoped to one tenant. Requires Python 3.10+ . The wheel is pure Python with a single runtime dependency RapidFuzz https://github.com/rapidfuzz/RapidFuzz : pip install lexiqr or, with uv https://docs.astral.sh/uv/ : uv add lexiqr A lexicon maps one tenant's private jargon to canonical entities. Here a German tenant maps flooff to the product entity — this is the file the examples below run against: { "schemaVersion": "1", "defaultLocale": "de-DE", "entities": { "product": { "locales": { "de-DE": { "preferred": { "singular": "flooff" } } } } } } Now resolve a prompt. Typo tolerance is on by default, so floof still resolves and the match names what was typed: python from lexiqr import EntityResolver resolver = EntityResolver.from file "lexicon.json" "flooff" resolves to the product entity, with its character span and tier. match = resolver.transform "wo ist flooff", locale="de-DE" .matches 0 print f"{match.canonical id} <- {match.surface form r} at {match.span}, tier {match.score tier.value}" The typo "floof" still resolves; the match names what was typed. typo = resolver.transform "wo ist floof", locale="de-DE" .matches 0 print f"corrected {typo.correction r} - {typo.surface form r}" php product <- 'flooff' at 7, 13 , tier preferred corrected 'floof' - 'flooff' Lexicon authors don't need Python. The same lexicon checks and runs from the command line, so lexiqr validate confirms the file is well-formed — lexiqr validate lexicon.json lexicon.json: valid lexicon. — and lexiqr try resolves a prompt against it, showing the same match the developer sees: lexiqr try lexicon.json --locale de-DE "wo ist flooff" prompt: "wo ist flooff " resolved via: de-DE 1 match: 1 product ← "flooff" tier: preferred locale: de-DE text: "flooff" Note Those blocks are the test suite. CI extracts them from this file, runs them, and compares the output to what you just read, so the quickstart cannot drift from the shipped API. Deterministic by contract. The same lexicon, prompt, and configuration produce an identical result across runs, platforms, and Pythons — a tested guarantee, not an aspiration. Typo-tolerant, and explainable about it. Edit budgets scale with word length, and every fuzzy match carries the correction it applied. Turn the pass off with fuzzy=False . Multilingual , with per-locale surface forms and fallback chains. Latin scripts match accent-insensitively; Arabic matches script-preserving. Spans you can trust — character offsets index the text the user typed, never a normalized copy.- Several terms can resolve to one entity, each carrying a tenant-defined filter verbatim. A match report replaces the per-tenant lookup table in your service. Typed, tested, bounded : py.typed and strict-mypy clean, a round-tripping report serialization, documented input limits, and a CI-enforced performance envelope. lexiqr validate and lexiqr try work without writing Python , with exit codes a script can branch on. The quickstart above is the five-minute story. This is the long one: a single command that resolves a realistic tenant lexicon and prints twelve narrated sections, each stating a claim, showing what lexiqr produced, and asserting it. uv run python examples/demo.py It exits non-zero , naming the section that failed, so it is a verification and not a brochure. It reads examples/medien.lexicon.json /bmeunier1974/lexiqr/blob/main/examples/medien.lexicon.json , a German media tenant. The run itself — — is one flat file you can read top to bottom and copy a section out of, importing nothing but lexiqr and the standard library. /bmeunier1974/lexiqr/blob/main/examples/demo.py examples/demo.py What the twelve sections claim - a tenant lexicon loads from a file — validation is construction C2 - a rejected lexicon names the entry, locale and field at fault C3 - an exact match reports its entity, surface form, span into the original prompt, and score tier C4 - preferred, alternate and canonical tiers each resolve and each name their tier C4 - two entries resolve to one product , each reporting its own entry ID and filter C19 - a typo resolves and carries its correction; the same prompt with fuzzy=False does not C5 - a prompt in an undeclared locale variant resolves through the fallback chain, and the report names the locale that answered C6 - accented and unaccented spellings both match with spans still on the typed text; Arabic matches script-preserving C7 - two entities in one prompt come back ordered by position, and an overlap resolves to the longest span C4 - a prompt over the documented maximum length is refused before any matching; whitespace-only is an empty report, not an error C8 - a report round-trips through the canonical serialization and serializes byte-identically twice C9 - the same lexicon through lexiqr validate and lexiqr try , with the exit codes a script reads C14, C15 A transcript this long should not be read as the whole guarantee. Three claims it deliberately does not make, each owned by a gate of its own: the performance envelope — owned by the CI perf gate uv run pytest -m perf . See Performance envelope performance-envelope below. cross-platform report equality — owned by the report-equality matrix job, which compares every OS and Python against scripts/report equality.golden.json . installability from PyPI — owned by the release workflow's clean-virtualenv leg, which installs the published wheel and runs against it. Below is an abridged excerpt of the real output — sections 5 and 6, with the other ten elided. The full transcript is committed as examples/demo.golden.txt /bmeunier1974/lexiqr/blob/main/examples/demo.golden.txt , and the test suite compares the command's output to it: lexiqr — a sample run: every claim printed, every claim asserted. lexicon: examples/medien.lexicon.json --- 5. Two entries resolve to one entity, each with its own filter C19 --- match "zeig mir die filme" → product ← "filme" span= 13, 18 tier=preferred locale=de-DE entry=movie filter={genre=drama|thriller, productType=Movie} match "zeig mir die serien" → product ← "serien" span= 13, 19 tier=preferred locale=de-DE entry=series filter={episodic=true, productType=Series} --- 6. A typo resolves and carries its correction; with fuzzy off it does not C5 --- prompt "zeig mir die flme" tolerant product ← "filme" span= 13, 17 tier=preferred locale=de-DE entry=movie filter={genre=drama|thriller, productType=Movie} correction="flme" exact EntityResolver.from file ..., fuzzy=False → 0 matches, resolved via de-DE OK: every section held. The quickstart resolves one word. These are the pieces you reach for next, in the order they usually come up. Validation is construction. Lexicon.from file and from dict either returns a lexicon lexiqr can trust or raises ValidationError naming the entity, locale, and field at fault. So you can check a tenant's file on the way in — at deploy time, in an upload handler, in your own tests — without building a throwaway resolver to find out: python from lexiqr import Lexicon, ValidationError try: lexicon = Lexicon.from file "lexicon.json" except ValidationError as invalid: print f"rejected: {invalid}" else: print f"valid: {sorted lexicon.entries } in {lexicon.default locale}" valid: 'product' in de-DE A Lexicon you already hold goes straight into a resolver — EntityResolver lexicon — so nothing is parsed or validated twice. A file that is not JSON at all raises MalformedDocumentError . It is a ValidationError , so the except above already covers it. Catch it by name only to tell "that file is not a lexicon document" apart from "that lexicon says the wrong thing" — the distinction the CLI turns into its two exit codes. EntityResolver ... , from file and from dict all accept a fuzzy keyword, defaulting to True . Pass fuzzy=False for exact-only behaviour. The keyword is public, semver-governed API. transform accepts a prompt of at most 10,000 characters Unicode code points , exported as MAX PROMPT LENGTH . A longer prompt raises ValidationError before any matching work happens, so a pasted document is rejected cheaply instead of taking a request thread with it. Reject or truncate upstream if your callers can paste arbitrary text. A single surface form is bounded too: at most 128 characters , exported as MAX SURFACE FORM LENGTH . That one is enforced when the lexicon loads rather than when a prompt is matched — see docs/lexicon-semantic-checks.md /bmeunier1974/lexiqr/blob/main/docs/lexicon-semantic-checks.md . Code that generates labels should size them against the constant, not against a copy of the number. Both limits are fixed parts of the contract, not per-call arguments or configuration knobs. Changing either is a semver-visible change. serialize report report turns a MatchReport into a canonical string: sorted keys, no insignificant whitespace, pure ASCII, and the match list in the report's own order. Two byte-equal serializations mean two equal reports and nothing else. So you can snapshot a result in your test suite, diff two snapshots to see real behaviour change, or store one and compare it months later. deserialize report text is its inverse — the form round-trips. python from lexiqr import serialize report, deserialize report snapshot = serialize report resolver.transform "wo ist flooff", locale="de-DE" ... store snapshot , compare it later, or check it into your tests Both functions are public, semver-governed API. The serialized shape can only change on a major release, so a patch or minor upgrade never silently invalidates a stored snapshot. lexiqr resolves one tenant's lexicon per resolver and deliberately ships no tenant registry. Mapping tenants to resolvers is your composition, not lexiqr's, which keeps it a thin layer you control. The recipe is a cache of resolvers keyed by tenant, each built once: Illustrative recipe — not run in CI. Adapt the loader and cache to your stack. from functools import lru cache from pathlib import Path from lexiqr import EntityResolver, MatchReport @lru cache maxsize=None def resolver for tenant id: str - EntityResolver: """One resolver per tenant, built once and reused across requests.""" lexicon = Path "lexicons" / f"{tenant id}.lexicon.json" return EntityResolver.from file lexicon def resolve tenant id: str, prompt: str, locale: str - MatchReport: return resolver for tenant id .transform prompt, locale A resolver is built once and then only read, so one instance per tenant is safe to share across requests. Size the cache to your tenant count, or swap lru cache for whatever eviction your deployment already uses. lexiqr is built to sit in a request path, so its performance is a stated, CI-enforced contract. Both numbers are measured against the seeded 1,000-surface-form benchmark lexicon: transform p95 < 10 ms initialization < 1 second cold How it is measured , so you can reproduce it: initialization is timed cold — one resolver built once, nothing warmed. For transform , a fixed set of warm-up calls is discarded, then p95 is taken over a fixed number of timed iterations. A long-but-under-limit prompt is measured too, so the 10,000-character limit is the only performance cliff rather than a hidden one before it. The gate is not the guarantee. The numbers above are the guarantee. The CI perf gate asserts that envelope times a 3× headroom factor p95 < 30 ms, init < 3 s on a single fixed runner. Shared CI runners are noisy, and the headroom turns that noise into a re-run rather than a false failure. Matching has to get roughly an order of magnitude slower to trip the gate, so catching subtle drift is not its job. That is why the raw timings are also recorded, un-gated, on every run. lexiqr runs on Python 3.10, 3.11, 3.12, and 3.13 and follows Semantic Versioning https://semver.org/spec/v2.0.0.html . A version constraint is only as trustworthy as the surface the promise covers, so that surface is named explicitly. Semver governs: The public API — everything exported from the lexiqr package: EntityResolver and its from file / from dict / transform methods, including the fuzzy keyword. The lexicon model — Lexicon , the type EntityResolver takes, with its validating from file / from dict constructors. Under it sits Entry , the named set of surface forms an entity is keyed by, carrying the entity it resolves to and the filter it holds. Then SurfaceForms , the shape an entry holds per locale, and Metadata / MetadataValue , that filter and the values it may hold. The structured error types — ValidationError and its coordinates canonical id , locale , field , which the CLI renders verbatim, plus MalformedDocumentError , the subclass raised when a file is not JSON at all. The match report types — MatchReport , EntityMatch , and ScoreTier , and the fields a caller reads off them: span, tier, correction, the entry that answered, and its metadata. The canonical report serialization — the byte-level shape produced by serialize report and consumed by deserialize report . The two documented limits — MAX PROMPT LENGTH and MAX SURFACE FORM LENGTH , whose values are part of the contract. A breaking change to any of these is a major-version change. Everything else — internal modules, private helpers, log wording — can change in a patch. Read the CHANGELOG /bmeunier1974/lexiqr/blob/main/CHANGELOG.md before upgrading; every release documents what changed. This is a single-repo project: both the meta-repo vision and blueprint and the product repo, with all four C4 containers shipping from here as one wheel. | Path | Container | What it is | |---|---|---| src/lexiqr/ excl. cli/ | core | The deterministic resolution engine and public typed API | src/lexiqr/cli/ | cli | lexiqr validate / lexiqr try for lexicon authors | schema/ | schema | The versioned JSON Schema for lexicon files, plus the shared fixture corpus | .github/workflows/ , pyproject.toml | delivery | CI gates and tag→PyPI trusted publishing | Guides for using lexiqr: docs/lexicon-authoring.md /bmeunier1974/lexiqr/blob/main/docs/lexicon-authoring.md — writing and validating a lexicon file, the lexiqr validate / lexiqr try CLI, and its scriptable exit-code contract docs/matching-rules.md /bmeunier1974/lexiqr/blob/main/docs/matching-rules.md — normalization, spans, score tiers, overlap resolution and ordering: the behavior determinism makes public docs/lexicon-semantic-checks.md /bmeunier1974/lexiqr/blob/main/docs/lexicon-semantic-checks.md — the complete list of checks core enforces beyond the published schema, and why CHANGELOG.md /bmeunier1974/lexiqr/blob/main/CHANGELOG.md — every release, recorded by hand The project's cross-container truth: VISION.md /bmeunier1974/lexiqr/blob/main/VISION.md — problem, actors, capabilities, non-goals, constraints CONTEXT.md /bmeunier1974/lexiqr/blob/main/CONTEXT.md — the project glossary docs/adr/ /bmeunier1974/lexiqr/blob/main/docs/adr — architecture decision records repo shape, contracts Two steps, no setup document to drift out of date — uv https://docs.astral.sh/uv/ does the rest: git clone https://github.com/bmeunier1974/lexiqr.git && cd lexiqr uv sync creates the venv and installs lexiqr plus its dev tools uv run pytest the same suite CI runs on every push and pull request CONTRIBUTING.md /bmeunier1974/lexiqr/blob/main/CONTRIBUTING.md describes the pull-request gate: lint, strict type-check, and tests on every supported Python. The release process, including the one-time PyPI trusted-publisher registration, lives in RELEASING.md /bmeunier1974/lexiqr/blob/main/RELEASING.md . To report a security issue, see SECURITY.md /bmeunier1974/lexiqr/blob/main/SECURITY.md . MIT — see LICENSE /bmeunier1974/lexiqr/blob/main/LICENSE .