When evaluating third-party API gateways, proxy routers, or anonymous arena models, prompt-based identification is essentially useless. A basic system prompt or lightweight fine-tune can make Claude claim it is GPT-4, make Llama sound like Mistral, or alter refusal behaviors entirely.
Persona and tone are malleable. But the underlying serving stack (tokenizers, merge tables, serving harnesses, and error validation boundaries) is rigid. To fake those, a proxy provider would need to intercept streams, translate tokens bidirectionally in real time, and rewrite metadata on the fly, which introduces severe latency and stream desyncs.
Here is how our client-side fingerprinting engine identifies models deterministically using infrastructure artifacts instead of conversational cues.
1. Tokenizer Vocabularies and Byte Fallbacks #
Every lab trains a distinct tokenizer with custom vocabulary sizes, merge tables, and fallback rules. Because the tokenizer determines the exact input matrix shape of the model weights, a model cannot alter how it segments text without a full retrain.
We send small, fixed text inputs to the endpoint and inspect the returned usage.prompt_tokens
:
Latin Pangrams:A fixed 212-character English string locks in baseline BPE merge behavior.
CJK Segmentation:Chinese text shows high divergence. Chinese frontier models (Qwen, GLM, DeepSeek) encode CJK with high vocabulary density (low token count), while Western models fragment the same string into 2 to 4 times more tokens.
Indentation and Code:Tests how 2-space vs 4-space whitespace blocks and code syntax brackets are compressed.
Emoji and Unicode:Tests whether multi-byte unicode sequences resolve to dedicated tokens, grapheme clusters, or raw byte fallbacks.
2. Template Offsets and Hidden Harnesses #
Inference engines (vLLM, SGLang, TGI, TensorRT-LLM) wrap user prompts in chat templates (like ChatML or Jinja templates) and often inject default system prompts.
By sending an empty prompt or a single 1-token payload and comparing the returned usage.prompt_tokens
against the raw payload token count, we calculate the static overhead:
offset = usage.prompt_tokens - payload_tokens
This delta exposes hidden guardrail templates, platform wrappers, and chat formatting artifacts applied upstream before the prompt hits the model.
3. Error Taxonomy and Validation Ceilings #
Validating normal responses only tells part of the story. Feeding intentionally invalid parameters triggers backend validation written by specific engineering teams:
Temperature ceilings: Sending temperature: 2.5
causes some runtimes to fail at >1.0, others at >2.0, with distinct error strings.Context refusals: Requesting max_tokens: 1000000000
forces the backend to reject the request and echo its true physical generation limit (e.g. "max_tokens must be ≤ 131072").Vendor error codes: Safety filter triggers often return proprietary numeric codes (e.g. Zhipu AI internal safety code 1301
).
4. Response Serialization Dialects #
Minor implementation details in how the server formats JSON reveal the runtime:
- The exact
choices[0].finish_reason
string (e.g. stop
vs end_turn
vs content_filter
). - How null or optional fields are serialized in the response body.
- Server-Sent Events (SSE) framing quirks during streaming.
Summary of Probe Vectors #
| Vector | Input | Observed Artifact | |---| | Tokenizer | Fixed Pangram / CJK block | usage.prompt_tokens | | Harness | 1-token minimal payload | Token delta (Δ offset) | | Boundaries | Extreme max_tokens / temp | Status and error string | | Error Code | Triggered filter payload | Vendor code (e.g. 1301) | | Dialect | Short completion | choices[0].finish_reason |
Client-Side Execution #
Because probes are purely deterministic, we run them directly in the browser. Requests are dispatched straight from the client tab to the target endpoint without passing through any intermediate proxy or logging server.
Each probe exports a simple runner contract that returns a deterministic value:
Why This Matters #
As API wrappers and proxy routers become more prevalent, verification cannot rely on trust or conversational vibes. By testing deterministic infrastructure artifacts like tokenizers, template deltas, and boundary conditions, we can accurately verify what model is actually serving your traffic.