Deterministic, tenant-scoped resolution of company jargon to canonical entities.
Quickstart • Usage • Docs • Contributing
Every tenant calls the same thing something different. A tenant writes a lexicon
mapping their private word — flooff
— to one of your canonical entities:
product
. lexiqr loads that lexicon and turns free-form prompts into identified matches, each carrying the character span it covers, its score tier, and any typo it corrected.
Teams usually solve this by hardcoding synonym tables, retraining embeddings, or letting an LLM guess. lexiqr is a pip-installable resolution layer instead: deterministic, explainable, and scoped to one tenant.
Requires Python 3.10+. The wheel is pure Python with a single runtime dependency (RapidFuzz):
pip install lexiqr
or, with uv:
uv add lexiqr
A lexicon maps one tenant's private jargon to canonical entities. Here a German
tenant maps flooff to the product
entity — this is the file the examples below run against:
{
"schemaVersion": "1",
"defaultLocale": "de-DE",
"entities": {
"product": {
"locales": {
"de-DE": { "preferred": { "singular": "flooff" } }
}
}
}
}
Now resolve a prompt. Typo tolerance is on by default, so floof
still resolves and the match names what was typed:
from lexiqr import EntityResolver
resolver = EntityResolver.from_file("lexicon.json")
match = resolver.transform("wo ist flooff", locale="de-DE").matches[0]
print(
f"{match.canonical_id} <- {match.surface_form!r} at {match.span}, tier {match.score_tier.value}"
)
typo = resolver.transform("wo ist floof", locale="de-DE").matches[0]
print(f"corrected {typo.correction!r} -> {typo.surface_form!r}")
php
product <- 'flooff' at (7, 13), tier preferred
corrected 'floof' -> 'flooff'
Lexicon authors don't need Python. The same lexicon checks and runs from the
command line, so lexiqr validate
confirms the file is well-formed —
lexiqr validate lexicon.json
lexicon.json: valid lexicon.
— and lexiqr try
resolves a prompt against it, showing the same match the developer sees:
lexiqr try lexicon.json --locale de-DE "wo ist flooff"
prompt: "wo ist [flooff]"
resolved via: de-DE
1 match:
[1] product ← "flooff"
tier: preferred locale: de-DE text: "flooff"
Note
Those blocks are the test suite. CI extracts them from this file, runs them, and compares the output to what you just read, so the quickstart cannot drift from the shipped API.
Deterministic by contract. The same lexicon, prompt, and configuration produce an identical result across runs, platforms, and Pythons — a tested guarantee, not an aspiration.Typo-tolerant, and explainable about it. Edit budgets scale with word length, and every fuzzy match carries the correction it applied. Turn the pass off withfuzzy=False
.Multilingual, with per-locale surface forms and fallback chains. Latin scripts match accent-insensitively; Arabic matches script-preserving.Spans you can trust— character offsets index the text the user typed, never a normalized copy.- Several terms can resolve to one entity, each carrying a
tenant-defined filter verbatim. A match report replaces the per-tenant lookup table in your service. Typed, tested, bounded:py.typed
and strict-mypy clean, a round-tripping report serialization, documented input limits, and a CI-enforced performance envelope.lexiqr validate
andlexiqr try
workwithout writing Python, with exit codes a script can branch on.
The quickstart above is the five-minute story. This is the long one: a single command that resolves a realistic tenant lexicon and prints twelve narrated sections, each stating a claim, showing what lexiqr produced, and asserting it.
uv run python examples/demo.py
It exits non-zero, naming the section that failed, so it is a verification and not a brochure. It reads examples/medien.lexicon.json, a German media tenant. The run itself —
examples/demo.py
What the twelve sections claim
- a tenant lexicon loads from a file — validation isconstruction (C2) - a rejected lexicon names the entry, locale and field at fault (C3)
- an exact match reports its entity, surface form, span into the original prompt, and score tier (C4)
- preferred, alternate and canonical tiers each resolve and each name their tier (C4)
- two entries resolve to one
product
, each reporting its own entry ID and filter (C19) - a typo resolves and carries its correction; the same prompt with
fuzzy=False
does not (C5) - a prompt in an undeclared locale variant resolves through the fallback chain, and the report names the locale that answered (C6)
- accented and unaccented spellings both match with spans still on the typed text; Arabic matches script-preserving (C7)
- two entities in one prompt come back ordered by position, and an overlap resolves to the longest span (C4)
- a prompt over the documented maximum length is refused before any matching; whitespace-only is an empty report, not an error (C8)
- a report round-trips through the canonical serialization and serializes byte-identically twice (C9)
- the same lexicon through
lexiqr validate
andlexiqr try
, with the exit codes a script reads (C14, C15)
A transcript this long should not be read as the whole guarantee. Three claims it deliberately does not make, each owned by a gate of its own:
the performance envelope— owned by the CI perf gate (uv run pytest -m perf
). SeePerformance envelopebelow.cross-platform report equality— owned by the report-equality matrix job, which compares every OS and Python againstscripts/report_equality.golden.json
.installability from PyPI— owned by the release workflow's clean-virtualenv leg, which installs the published wheel and runs against it.
Below is an abridged excerpt of the real output — sections 5 and 6, with the other ten elided. The full transcript is committed as examples/demo.golden.txt, and the test suite compares the command's output to it:
lexiqr — a sample run: every claim printed, every claim asserted.
lexicon: examples/medien.lexicon.json
--- 5. Two entries resolve to one entity, each with its own filter [C19] ---
match "zeig mir die filme" → product ← "filme" span=(13, 18) tier=preferred
locale=de-DE entry=movie filter={genre=drama|thriller, productType=Movie}
match "zeig mir die serien" → product ← "serien" span=(13, 19) tier=preferred
locale=de-DE entry=series filter={episodic=true, productType=Series}
--- 6. A typo resolves and carries its correction; with fuzzy off it does not [C5] ---
prompt "zeig mir die flme"
tolerant product ← "filme" span=(13, 17) tier=preferred locale=de-DE entry=movie
filter={genre=drama|thriller, productType=Movie} correction="flme"
exact EntityResolver.from_file(..., fuzzy=False) → 0 matches, resolved via de-DE
OK: every section held.
The quickstart resolves one word. These are the pieces you reach for next, in the order they usually come up.
Validation is construction. Lexicon.from_file
(and from_dict
) either
returns a lexicon lexiqr can trust or raises ValidationError
naming the entity, locale, and field at fault. So you can check a tenant's file on the way in — at deploy time, in an upload handler, in your own tests — without building a throwaway resolver to find out:
from lexiqr import Lexicon, ValidationError
try:
lexicon = Lexicon.from_file("lexicon.json")
except ValidationError as invalid:
print(f"rejected: {invalid}")
else:
print(f"valid: {sorted(lexicon.entries)} in {lexicon.default_locale}")
valid: ['product'] in de-DE
A Lexicon
you already hold goes straight into a resolver —
EntityResolver(lexicon)
— so nothing is parsed or validated twice.
A file that is not JSON at all raises MalformedDocumentError
. It is a
ValidationError
, so the except
above already covers it. Catch it by name only to tell "that file is not a lexicon document" apart from "that lexicon says the wrong thing" — the distinction the CLI turns into its two exit codes.
EntityResolver(...)
, from_file
and from_dict
all accept a fuzzy
keyword,
defaulting to True
. Pass fuzzy=False
for exact-only behaviour. The keyword is public, semver-governed API.
transform()
accepts a prompt of at most 10,000 characters (Unicode code
points), exported as MAX_PROMPT_LENGTH
. A longer prompt raises
ValidationError
before any matching work happens, so a pasted document is rejected cheaply instead of taking a request thread with it. Reject or truncate upstream if your callers can paste arbitrary text.
A single surface form is bounded too: at most 128 characters, exported as
MAX_SURFACE_FORM_LENGTH
. That one is enforced when the lexicon loads rather than when a prompt is matched — see docs/lexicon-semantic-checks.md. Code that generates labels should size them against the constant, not against a copy of the number.
Both limits are fixed parts of the contract, not per-call arguments or configuration knobs. Changing either is a semver-visible change.
serialize_report(report)
turns a MatchReport
into a canonical string:
sorted keys, no insignificant whitespace, pure ASCII, and the match list in the
report's own order. Two byte-equal serializations mean two equal reports and
nothing else. So you can snapshot a result in your test suite, diff two snapshots
to see real behaviour change, or store one and compare it months later.
deserialize_report(text)
is its inverse — the form round-trips.
from lexiqr import serialize_report, deserialize_report
snapshot = serialize_report(resolver.transform("wo ist flooff", locale="de-DE"))
Both functions are public, semver-governed API. The serialized shape can only change on a major release, so a patch or minor upgrade never silently invalidates a stored snapshot.
lexiqr resolves one tenant's lexicon per resolver and deliberately ships no tenant registry. Mapping tenants to resolvers is your composition, not lexiqr's, which keeps it a thin layer you control. The recipe is a cache of resolvers keyed by tenant, each built once:
from functools import lru_cache
from pathlib import Path
from lexiqr import EntityResolver, MatchReport
@lru_cache(maxsize=None)
def resolver_for(tenant_id: str) -> EntityResolver:
"""One resolver per tenant, built once and reused across requests."""
lexicon = Path("lexicons") / f"{tenant_id}.lexicon.json"
return EntityResolver.from_file(lexicon)
def resolve(tenant_id: str, prompt: str, locale: str) -> MatchReport:
return resolver_for(tenant_id).transform(prompt, locale)
A resolver is built once and then only read, so one instance per tenant is safe
to share across requests. Size the cache to your tenant count, or swap
lru_cache
for whatever eviction your deployment already uses.
lexiqr is built to sit in a request path, so its performance is a stated, CI-enforced contract. Both numbers are measured against the seeded 1,000-surface-form benchmark lexicon:
transform()
p95 < 10 msinitialization < 1 second(cold)
How it is measured, so you can reproduce it: initialization is timed cold —
one resolver built once, nothing warmed. For transform()
, a fixed set of warm-up calls is discarded, then p95 is taken over a fixed number of timed iterations. A long-but-under-limit prompt is measured too, so the 10,000-character limit is the only performance cliff rather than a hidden one before it.
The gate is not the guarantee. The numbers above are the guarantee. The CI perf gate asserts that envelope times a 3× headroom factor (p95 < 30 ms, init < 3 s) on a single fixed runner. Shared CI runners are noisy, and the headroom turns that noise into a re-run rather than a false failure. Matching has to get roughly an order of magnitude slower to trip the gate, so catching subtle drift is not its job. That is why the raw timings are also recorded, un-gated, on every run.
lexiqr runs on Python 3.10, 3.11, 3.12, and 3.13 and follows Semantic Versioning. A version constraint is only as trustworthy as the surface the promise covers, so that surface is named explicitly. Semver governs:
The public API— everything exported from thelexiqr
package:EntityResolver
and itsfrom_file
/from_dict
/transform
methods, including thefuzzy
keyword.The lexicon model—Lexicon
, the typeEntityResolver
takes, with its validatingfrom_file
/from_dict
constructors. Under it sitsEntry
, the named set of surface forms an entity is keyed by, carrying the entity it resolves to and the filter it holds. ThenSurfaceForms
, the shape an entry holds per locale, andMetadata
/MetadataValue
, that filter and the values it may hold.The structured error types—ValidationError
and its coordinates (canonical_id
,locale
,field
), which the CLI renders verbatim, plusMalformedDocumentError
, the subclass raised when a file is not JSON at all.The match report types—MatchReport
,EntityMatch
, andScoreTier
, and the fields a caller reads off them: span, tier, correction, the entry that answered, and its metadata.The canonical report serialization— the byte-level shape produced byserialize_report
and consumed bydeserialize_report
.The two documented limits—MAX_PROMPT_LENGTH
andMAX_SURFACE_FORM_LENGTH
, whose values are part of the contract.
A breaking change to any of these is a major-version change. Everything else — internal modules, private helpers, log wording — can change in a patch. Read the CHANGELOG before upgrading; every release documents what changed.
This is a single-repo project: both the meta-repo (vision and blueprint) and the product repo, with all four C4 containers shipping from here as one wheel.
| Path | Container | What it is |
|---|---|---|
src/lexiqr/ (excl. cli/ ) |
||
| core | ||
| The deterministic resolution engine and public typed API | ||
src/lexiqr/cli/ |
||
| cli | ||
lexiqr validate / lexiqr try for lexicon authors |
||
schema/ |
||
| schema | ||
| The versioned JSON Schema for lexicon files, plus the shared fixture corpus | ||
.github/workflows/ , pyproject.toml |
||
| delivery | ||
| CI gates and tag→PyPI trusted publishing |
Guides for using lexiqr:
docs/lexicon-authoring.md— writing and validating a lexicon file, thelexiqr validate
/lexiqr try
CLI, and its scriptable exit-code contractdocs/matching-rules.md— normalization, spans, score tiers, overlap resolution and ordering: the behavior determinism makes publicdocs/lexicon-semantic-checks.md— the complete list of checks core enforces beyond the published schema, and whyCHANGELOG.md— every release, recorded by hand
The project's cross-container truth:
VISION.md— problem, actors, capabilities, non-goals, constraintsCONTEXT.md— the project glossarydocs/adr/— architecture decision records (repo shape, contracts)
Two steps, no setup document to drift out of date — uv does the rest:
git clone https://github.com/bmeunier1974/lexiqr.git && cd lexiqr
uv sync # creates the venv and installs lexiqr plus its dev tools
uv run pytest # the same suite CI runs on every push and pull request
CONTRIBUTING.md describes the pull-request gate: lint, strict type-check, and tests on every supported Python. The release process, including the one-time PyPI trusted-publisher registration, lives in RELEASING.md. To report a security issue, see SECURITY.md.
MIT — see LICENSE.