# Probelock – lockfile for LLM tool calling > Source: > Published: 2026-07-09 07:16:50+00:00 **A capability lockfile for local models.** It records what a model does on a set of tool-calling and output checks, and fails CI when a model/quant/runtime swap lowers a score. ``` llama-3.1-8b @ Q8_0 (ollama) → llama-3.1-8b @ Q4_K_M (ollama) Capability Baseline Candidate Δ Status arg_validity 1.00 0.67 -0.33 REGRESSION arity_robustness 1.00 0.67 -0.33 REGRESSION format_adherence 1.00 1.00 +0.00 ok needle_in_tools 1.00 0.33 -0.67 REGRESSION no_hallucinated_tool 1.00 0.67 -0.33 REGRESSION required_args 1.00 1.00 +0.00 ok structured_output 1.00 0.33 -0.67 REGRESSION tool_discrimination 1.00 0.33 -0.67 REGRESSION tool_permission 1.00 0.67 -0.33 REGRESSION tool_restraint 1.00 0.67 -0.33 REGRESSION tool_selection 1.00 0.67 -0.33 REGRESSION FAIL — capabilities regressed or removed: arg_validity, arity_robustness, needle_in_tools, no_hallucinated_tool, structured_output, tool_discrimination, tool_permission, tool_restraint, tool_selection ``` Here the Q4 quant scores 0.33–0.67 on several capabilities where Q8 scored 1.00. `probelock gate` exits non-zero when a capability drops past the threshold. promptfoo is a test framework you author. probelock is a lockfile you commit. **Probes are derived from your tool schemas.** Point it at the OpenAI-style tool definitions your agent already ships, and it generates a fixed, reproducible battery of capability checks. You write no test cases.**No LLM judge.** Every probe is scored by code: JSON-schema validation, exact match, or a tool-name check. Run it twice on the same model and the numbers match. (promptfoo relies on assertions you write and often on model-graded evals, which vary across runs.)**It compares a model against its own baseline,** across a model/quant/runtime swap, rather than producing an absolute leaderboard. You only ever compare like with like, on your box, with your tools, so the "benchmarks are gameable/hardware-dependent" objection does not apply. ## Install & run (only needs [uv](https://docs.astral.sh/uv/)) Run it without installing, or install it into the current environment: ``` uvx probelock --help # run the latest release pip install probelock # or install it ``` To run an unreleased revision straight from git: ``` uvx --from git+https://github.com/kelkalot/probelock probelock --help ``` The examples below use `uv run` from a checkout of this repo. No model is required for the demo — a deterministic `SimulatedClient` stands in for two quant levels of the same model: ``` # from the probelock/ project dir uv run probelock derive --tools examples/agent_tools.json # see the probe battery uv run probelock probe --tools examples/agent_tools.json --simulate fixtures/profile_q8.json -o q8.lock uv run probelock probe --tools examples/agent_tools.json --simulate fixtures/profile_q4.json -o q4.lock uv run probelock diff q8.lock q4.lock uv run probelock gate --baseline q8.lock --candidate q4.lock # exits non-zero ``` Against a local model, swap `--simulate` for an OpenAI-compatible endpoint: ``` uv run probelock probe --tools examples/agent_tools.json \ --endpoint http://localhost:11434/v1 --model llama3.1:8b-instruct-q4_K_M \ --quant Q4_K_M --runtime ollama --timeout 120 -o q4.lock ``` A probe the model rejects (e.g. "model does not support tools") or that times out scores **0** for that capability and the run continues, so a model that cannot tool-call still produces a lockfile. An unreachable server, a 404 (wrong model or URL), or a run where every probe fails aborts the run, so a misconfiguration never becomes a poisoned all-zeros baseline. `examples/agent_tools.json` is a 3-tool schema for the walkthrough above, not a sensitivity benchmark — validation testing found it insensitive to real capability drift that a 10-tool schema with overlapping tool names and richer argument constraints caught cleanly (see [ VALIDATION.md](/kelkalot/probelock/blob/main/VALIDATION.md)). A schema with too few tools, or arguments with no real constraints to violate, under-reports regressions. Point `--tools` at your own agent's actual tool definitions before trusting `gate` in CI.probelock speaks one protocol — OpenAI `/v1/chat/completions` with OpenAI-style tools — so anything that exposes it works with `--endpoint` . For providers that do not (Anthropic, Gemini, …), route through a unified SDK with `--via` . Every path is deterministic; none of them put an LLM in the loop. | You have… | Use | |---|---| Ollama, vLLM, llama.cpp server, LM Studio, HF TGI, OpenAI, OpenRouter, Together… | `--endpoint /v1 --model ` (vLLM needs `--enable-auto-tool-choice` ; llama.cpp needs `--jinja` ) | Anthropic / Gemini / Mistral / Bedrock / … (any-llm) | `--via anyllm --model anthropic/claude-3-5-sonnet` | | Any of 100+ providers (litellm SDK) | `--via litellm --model anthropic/claude-3-5-sonnet` | A running LiteLLM proxy | `--endpoint http://litellm:4000/v1 --model ` (no extra) | In-process HF `transformers` / MLX (no server) | not yet — add a small `Client` adapter | ``` pip install 'probelock[anyllm]' # or 'probelock[litellm]' probelock probe --tools tools.json --via anyllm --model mistral/mistral-large-latest \ --samples 5 --temperature 0.7 -o candidate.lock ``` `--via` clients reuse the same caching, sampling, and error semantics as `--endpoint` ; they are thin adapters over each SDK's OpenAI-shaped response. Add a new backend by implementing the tiny `Client` protocol — that is the only seam. [ demo/](/kelkalot/probelock/blob/main/demo) has runs against a local Ollama server: a committed `qwen3.5:9b` baseline vs a `gemma3:1b` candidate (which does not support tool-calling). See [for the transcript, or replay it:](/kelkalot/probelock/blob/main/demo/DEMO.md) `demo/DEMO.md` ``` asciinema play demo/probelock-demo.cast # or: bash demo/demo.sh ``` The tool-calling capabilities drop `1.00 → 0.00` and the gate exits non-zero. `tool_restraint` , `tool_permission` , and `no_hallucinated_tool` stay `1.00` (a model that cannot call tools cannot misuse one), and `gemma3:1b` scores `1.00` on `format_adherence` vs `0.50` for `qwen3.5:9b` . The diff is per-capability. Also committed: `qwen3.5:9b` vs `lfm2.5-thinking:1.2b` : ``` uv run probelock diff demo/qwen3.5-9b.lock demo/lfm2.5-thinking.lock ``` The 1.2B model matches `qwen3.5:9b` on tool selection, discrimination, `needle_in_tools` , `arg_validity` , `required_args` , and the three safety probes; `structured_output` and `arity_robustness` drop `1.00 → 0.33` . | Capability | What it checks | Scorer | |---|---|---| `tool_selection` | Calls the right tool for the task | tool-name match | `tool_discrimination` | Calls the right tool and no other (picks precisely) | tool-name set | `needle_in_tools` | Finds the right tool when many (15+) are offered | tool-name match | `arg_validity` | Emitted args validate against the tool's JSON schema | `jsonschema` | `required_args` | All required args present and non-empty | key presence | `arity_robustness` | Fills every parameter (required + optional) when asked | all-present | `structured_output` | Emits schema-valid JSON on demand (no tools, no fences) | parse + `jsonschema` | `json_mode` (opt-in, `--json-mode` ) | Same, but via the server's native `response_format` API instead of a prompt | parse + `jsonschema` | `tool_restraint` | Does not call a tool for a task that needs none (over-trigger) | no tool call | `tool_permission` | Does not call a tool it was explicitly forbidden to use | forbidden tool absent | `no_hallucinated_tool` | Does not fabricate a call to a tool that was not offered | called ⊆ offered | `format_adherence` | Follows an exact output constraint | exact match | Three are **negative** probes (a higher score means the bad behavior did not happen): `tool_restraint` (over-triggering), `tool_permission` (calling a forbidden tool), and `no_hallucinated_tool` (fabricating a tool). All probes are derived from the tool schemas, not hand-authored. ``` tool schemas ──▶ derive probes ──▶ Client ──▶ ResponseMessage ──▶ deterministic scorer ──▶ Lockfile (your agent) (zero authoring) (model) (the only model (no LLM judge) (commit it) -touching part) Lockfile + Lockfile ──▶ diff / gate ``` The only nondeterministic part is the `Client` ; everything else is pure, so the same inputs produce the same lockfile and the same diff. At temperature 0 the client caches identical requests, so the probes that share one request (the tool checks for a given tool) hit the network once. The `SimulatedClient` crafts correct or incorrect responses that the real scorers grade, so the scoring path runs even with no model present. Schema-derived probes are single-turn and synthetic — great for catching schema-level regressions, blind to what breaks after several turns of real context, a tool result feeding back in, or ambiguous phrasing. `--traces` adds a second source: real, already-recorded agent decision points (e.g. exported from litellm's OpenTelemetry callback), replayed through the exact same deterministic scorers. ``` uv run probelock derive --tools tools.json --traces traces.json # see what gets added uv run probelock probe --tools tools.json --traces traces.json \ --endpoint http://localhost:11434/v1 --model llama3.1:8b -o candidate.lock ``` `--tools` is optional here: traced probes replay their own embedded tool definitions, so a trace-only run (`probe --traces traces.json ...` ) needs no schema file. The same holds for `--mined` below. Provide `--tools` when you also want the synthetic battery. A traces file is a small, stable JSON schema probelock defines itself — **not** raw OpenTelemetry — because OTel's own span attribute layout is not stable across libraries or versions (litellm has already changed where it puts request/response attributes once, and has a newer, differently-shaped opt-in integration). Converting your export into this shape is a one-time step you own; see [ examples/otel_traces_to_probelock.py](/kelkalot/probelock/blob/main/examples/otel_traces_to_probelock.py) for a documented starting point and [for the target shape:](/kelkalot/probelock/blob/main/fixtures/sample_traces.json) `fixtures/sample_traces.json` ``` { "version": 1, "records": [ { "id": "checkout-flow-turn-3", "messages": [{"role": "user", "content": "..."}], "tools": [ /* OpenAI-style tool defs actually offered at this turn */ ], "response": {"content": null, "tool_calls": [{"name": "...", "arguments": "{...}"}]} } ] } ``` Trace-derived probes join the *same* capabilities as schema-derived ones — `tool_selection` , `tool_discrimination` , `arg_validity` , `required_args` , and `structured_output` — since these map cleanly onto "replay this real context, check the candidate still behaves validly" (probe ids carry a `::traced::` marker if you want to inspect the split). The rest stay purely schema-derived: `needle_in_tools` , `tool_permission` , `no_hallucinated_tool` , and `tool_restraint` need a synthetic perturbation (an injected distractor tool, a forbidden-tool instruction, a removed tool) that a passively recorded trace does not naturally contain; `format_adherence` needs an exact-text prompt, not a tool-calling decision point; and `arity_robustness` needs its own explicit "fill EVERY parameter, including optional ones" instruction to mean anything — a real conversation was never asked for that, so replaying it would only test whichever optional fields happened to get filled in that one exchange, not robustness. **Unlike a tool schema, a traces file contains real conversation content.** `probe --traces` prints a warning every time, and the lockfile records a `traces_fingerprint` so a `diff` flags a baseline/candidate pair whose trace inputs differ — but review and redact the file yourself before committing it, the same way you would review any fixture with real data in it. Tested against a real llama.cpp regression (commit-level, not synthetic): `gate` fails on the regressed commit and passes on an adjacent, unrelated commit. See [ VALIDATION.md](/kelkalot/probelock/blob/main/VALIDATION.md) for the test setup and results, and [to reproduce it.](/kelkalot/probelock/blob/main/fixtures/gptoss_regression_trace.json) `fixtures/gptoss_regression_trace.json` If your stack does not already log requests, the recording proxy captures them with one line changed in the agent: ``` probelock proxy --listen 127.0.0.1:8484 \ --upstream http://127.0.0.1:11434 \ --out traces/agent.jsonl # agent side: base_url = "http://127.0.0.1:8484/v1" ``` Every request is forwarded to the upstream unchanged (streaming included — SSE flows token by token and is reassembled for the record afterwards, tool-call deltas and all); each completed chat-completions exchange is appended asynchronously as one `trace-v1` JSONL record. Recording is strictly non-invasive: on any internal logging error the request is still served and a warning goes to stderr. Failed or truncated exchanges (upstream errors, mid-stream disconnects) are logged with a failing status so `ingest` skips them instead of mining half-generated responses. Multi-turn conversations are stitched into sessions without any agent cooperation (restarting the proxy mid-conversation splits that conversation into two sessions — harmless, but it weakens confirmation evidence, so prefer restarting between runs), `--max-size` / `--max-age` rotate the log, and the file is created `0600` — it holds **verbatim conversation content**; keep it out of version control (redaction happens later, at `ingest` ). `--traces` (above) replays a *curated* export you assembled by hand. `probelock ingest` goes one step earlier: point it at a raw request/response log of real agent traffic — the proxy's output, or your own logging — and it mines probes for you: multi-turn, realistic regression tests with near-zero authoring effort, still scored by the same deterministic checks (LLMs may appear in *trace generation* — that is your own agent — but never in *scoring*). ``` probelock ingest traces/agent.jsonl --out probes/mined.json # everything lands "pending" probelock traces review probes/mined.json # activate probes (y/n/e/a/s/q) probelock probe --tools tools.json --mined probes/mined.json \ --endpoint http://localhost:11434/v1 --model llama3.1:8b -o candidate.lock ``` `ingest` accepts several logs at once (`probelock ingest agent.jsonl agent-*.jsonl` ) — pass a rotated set together so sessions spanning a rotation boundary keep their confirmation evidence. Several input formats are supported (`--format` , or `auto` ): `--format` | Shape | |---|---| `trace-v1` | the native record the recording proxy writes (one JSON object per line, `request` /`response.message` ) | `openai-jsonl` | the verbatim chat-completions request body next to the verbatim response, per line | `anthropic-jsonl` | logged Anthropic Messages API calls (`request` /`response` ); content blocks, `tool_use` /`tool_result` , and `system` are translated to the canonical shape | `otel-genai` | an OTLP-JSON span export, read via the OpenTelemetry GenAI semantic-convention attributes (`gen_ai.prompt` /`gen_ai.completion` , blob or indexed form) — scoped to the spec, not any one library's layout; spans without those attributes are skipped and counted | `auto` detects the JSONL shapes and OTel documents. See the `fixtures/sample_*` files for each. For OTel exporters that do not follow the semantic convention, [ examples/otel_traces_to_probelock.py](/kelkalot/probelock/blob/main/examples/otel_traces_to_probelock.py) remains the conversion recipe. Deduplication is exact-hash by default (deterministic). `--cluster embeddings --embed-endpoint URL --embed-model NAME` instead groups *near-duplicate* contexts by embedding cosine similarity (via an OpenAI-compatible `/v1/embeddings` endpoint you already run). This is opt-in and **not deterministic** — the grouping depends on the embedding model and version, so probelock prints a caveat and records `cluster: embeddings` in each affected probe's provenance. Everything downstream (scoring, gating) stays deterministic; only which contexts merged does not. Raw traffic includes model mistakes, so **provenance determines trust** — every probe records how many sessions support it and which rule confirmed it, and that decides how much review it needs: | Category | Check at replay | Mined from | Review | |---|---|---|---| `traced_schema_validity` | some call's args validate against the called tool's schema | every tool-calling exchange (no inference) | `--auto-accept schema_validity` is safe | `traced_tool_selection` | calls the confirmed tool | exchanges confirmed good: the result fed back and the conversation moved on (no error payload, no corrected-args retry, no re-ask), or the same context produced the same call in ≥ `--min-agreement` distinct sessions | review, or `--auto-accept-all --i-know-what-im-doing` | `traced_no_tool` | answers in text, calls nothing | unanimous text answers across ≥ `--min-agreement-notool` (default 3) distinct sessions, no re-ask, preferring contexts lexically distant from the tools | individual review only — a mislabeled probe freezes a model mistake as expected behavior | The traced capabilities are deliberately separate names in the lockfile: a drop in multi-turn trace probes while single-turn synthetic probes hold steady is itself diagnostic ("context-length-sensitive regression"). Privacy defaults are conservative. Argument values in frozen contexts are always replaced with structure-preserving placeholders (`"query": ""` ); message content stays verbatim (that is what makes replay realistic), so mined probes carry `"sensitive": true` and `probe -o` refuses to include them in a written lockfile without `--allow-sensitive` . If you want committable probes, opt in to scrubbing with `ingest --redact-patterns emails,phones,paths` . Identical contexts are deduplicated (timestamps stripped, whitespace collapsed), sampling keeps up to `--per-capability` probes per (tool, category) preferring longer contexts and later turns, and everything the pipeline skips (failed calls, forced `tool_choice` , oversized contexts, ambiguous agreement) is counted and reported — never silently dropped. The full pipeline is validated against real agent traffic on real local models — including a runtime swap the gate catches and real frozen-mistake probes the review step rejects — in [ VALIDATION-TRACES.md](/kelkalot/probelock/blob/main/VALIDATION-TRACES.md). `diff` compares two lockfiles; `trend` compares N, in the order you give them — a quantization ladder or the same model over time — so you can see *where* a capability holds and *where* it cliffs: ``` uv run probelock trend Q8_0.lock Q6_K.lock Q5_K_M.lock Q4_K_M.lock Q3_K_M.lock Q2_K.lock Capability Q8_0 Q6_K Q5_K_M Q4_K_M Q3_K_M Q2_K Δ Trend structured_output 1.00 1.00 1.00 0.67 0.33 0.33 -0.67 ↓ regressed tool_restraint 1.00 1.00 1.00 1.00 1.00 1.00 +0.00 = stable tool_selection 1.00 1.00 0.67 1.00 0.67 1.00 +0.00 ~ unstable ``` Each row is annotated by its whole-ladder behavior: `regressed` (net drop past `--max-drop` ), `improved` , `unstable` (net-flat but it dipped along the way — a signal a two-point diff of the endpoints would miss), `stable` , `removed` (present early but gone from the last rung — a dropped capability, counted as a regression), or `partial` (present in fewer than two lockfiles). `--format markdown|json|html` mirror `diff` ; the HTML view draws a sparkline per capability. `trend` never fails on a regression (use `gate` pairwise for CI); it exits non-zero only on bad input. The filename stem is each column's header, so name your lockfiles for the axis (`Q8_0.lock` , `Q4_K_M.lock` ). With one sample per probe, a capability backed by 3 tools quantizes to `{0, 0.33, 0.67, 1.0}` — a single flip moves it 0.33, far past the default 0.05 gate. So: runs each probe N times and records the pass-`probe --samples N [--temperature T]` *rate*(raise the temperature for sampling variance).only fails on a drop that is statistically significant for the recorded trial count (a one-sided two-proportion test). Sub-significant drops are shown as`gate --confidence 0.95` and do`noisy ↓` **not** fail the gate; raise`--samples` to confirm or clear them. A total collapse (e.g. `1.00 → 0.00` ) is significant even at low N; a single-flip `1.00 → 0.67` over 3 trials is `noisy` until you raise `--samples` . `probelock init` scaffolds a `probelock.tools.json` and a `.github/workflows/probelock.yml` to start from. Commit a baseline lockfile, then gate each candidate: ``` - run: uvx probelock probe --tools tools.json --endpoint $LLM_URL --model $MODEL --samples 5 --temperature 0.7 -o candidate.lock - run: uvx probelock gate --baseline probelock.lock --candidate candidate.lock --max-drop 0.05 --confidence 0.95 ``` Or use the composite GitHub Action ([ action.yml](/kelkalot/probelock/blob/main/action.yml)), which wraps those two steps end-to-end: ``` - uses: kelkalot/probelock@v0 with: tools: tools.json baseline: probelock.lock endpoint: ${{ secrets.LLM_ENDPOINT }} model: ${{ vars.LLM_MODEL }} ``` To show the result on a pull request, render the diff as Markdown (or `--format html` for a self-contained page): ``` probelock diff probelock.lock candidate.lock --format markdown >> "$GITHUB_STEP_SUMMARY" ``` - Proxy hardening: a static Go/Rust binary beside the reference Python implementation, and streaming-reassembly edge cases (multi-line SSE events, resume-after-disconnect). - In-process backends (HF `transformers` / MLX) via a small`Client` adapter, no server required. - Emit OpenTelemetry spans from `probe` runs, so a probe run shows up alongside your other agent traces in whatever backend you already use — a follow-on to trace-derived probes above (that direction consumes traces; this one produces them). Apache-2.0 — see [LICENSE](/kelkalot/probelock/blob/main/LICENSE). Built with [Claude Code](https://claude.com/claude-code).