Probelock – lockfile for LLM tool calling

A capability lockfile for local models. It records what a model does on a set of tool-calling and output checks, and fails CI when a model/quant/runtime swap lowers a score.

llama-3.1-8b @ Q8_0 (ollama)  →  llama-3.1-8b @ Q4_K_M (ollama)
Capability            Baseline   Candidate     Δ   Status
arg_validity              1.00        0.67  -0.33  REGRESSION
arity_robustness          1.00        0.67  -0.33  REGRESSION
format_adherence          1.00        1.00  +0.00  ok
needle_in_tools           1.00        0.33  -0.67  REGRESSION
no_hallucinated_tool      1.00        0.67  -0.33  REGRESSION
required_args             1.00        1.00  +0.00  ok
structured_output         1.00        0.33  -0.67  REGRESSION
tool_discrimination       1.00        0.33  -0.67  REGRESSION
tool_permission           1.00        0.67  -0.33  REGRESSION
tool_restraint            1.00        0.67  -0.33  REGRESSION
tool_selection            1.00        0.67  -0.33  REGRESSION

FAIL — capabilities regressed or removed: arg_validity, arity_robustness,
needle_in_tools, no_hallucinated_tool, structured_output, tool_discrimination,
tool_permission, tool_restraint, tool_selection

Here the Q4 quant scores 0.33–0.67 on several capabilities where Q8 scored 1.00. probelock gate

exits non-zero when a capability drops past the threshold.

promptfoo is a test framework you author. probelock is a lockfile you commit.

Probes are derived from your tool schemas. Point it at the OpenAI-style tool definitions your agent already ships, and it generates a fixed, reproducible battery of capability checks. You write no test cases.No LLM judge. Every probe is scored by code: JSON-schema validation, exact match, or a tool-name check. Run it twice on the same model and the numbers match. (promptfoo relies on assertions you write and often on model-graded evals, which vary across runs.)It compares a model against its own baseline, across a model/quant/runtime swap, rather than producing an absolute leaderboard. You only ever compare like with like, on your box, with your tools, so the "benchmarks are gameable/hardware-dependent" objection does not apply.

Install & run (only needs uv) #

Run it without installing, or install it into the current environment:

uvx probelock --help          # run the latest release
pip install probelock         # or install it

To run an unreleased revision straight from git:

uvx --from git+https://github.com/kelkalot/probelock probelock --help

The examples below use uv run

from a checkout of this repo. No model is required for the demo — a deterministic SimulatedClient

stands in for two quant levels of the same model:

uv run probelock derive --tools examples/agent_tools.json          # see the probe battery
uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q8.json -o q8.lock
uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q4.json -o q4.lock
uv run probelock diff   q8.lock q4.lock
uv run probelock gate   --baseline q8.lock --candidate q4.lock     # exits non-zero

Against a local model, swap --simulate

for an OpenAI-compatible endpoint:

uv run probelock probe --tools examples/agent_tools.json \
    --endpoint http://localhost:11434/v1 --model llama3.1:8b-instruct-q4_K_M \
    --quant Q4_K_M --runtime ollama --timeout 120 -o q4.lock

A probe the model rejects (e.g. "model does not support tools") or that times out scores 0 for that capability and the run continues, so a model that cannot tool-call still produces a lockfile. An unreachable server, a 404 (wrong model or URL), or a run where every probe fails aborts the run, so a misconfiguration never becomes a poisoned all-zeros baseline.

examples/agent_tools.json

is a 3-tool schema for the walkthrough above, not a sensitivity benchmark — validation testing found it insensitive to real capability drift that a 10-tool schema with overlapping tool names and richer argument constraints caught cleanly (see VALIDATION.md). A schema with too few tools, or arguments with no real constraints to violate, under-reports regressions. Point

--tools

at your own agent's actual tool definitions before trusting gate

in CI.probelock speaks one protocol — OpenAI /v1/chat/completions

with OpenAI-style tools — so anything that exposes it works with --endpoint

. For providers that do not (Anthropic, Gemini, …), route through a unified SDK with --via

. Every path is deterministic; none of them put an LLM in the loop.

You have…	Use
Ollama, vLLM, llama.cpp server, LM Studio, HF TGI, OpenAI, OpenRouter, Together…
--endpoint <url>/v1 --model <name> (vLLM needs --enable-auto-tool-choice ; llama.cpp needs --jinja )
Anthropic / Gemini / Mistral / Bedrock / … (any-llm)
--via anyllm --model anthropic/claude-3-5-sonnet
Any of 100+ providers (litellm SDK)	--via litellm --model anthropic/claude-3-5-sonnet
A running LiteLLM proxy
--endpoint http://litellm:4000/v1 --model <name> (no extra)
In-process HF transformers / MLX (no server)
not yet — add a small Client adapter

pip install 'probelock[anyllm]'   # or 'probelock[litellm]'
probelock probe --tools tools.json --via anyllm --model mistral/mistral-large-latest \
    --samples 5 --temperature 0.7 -o candidate.lock

--via

clients reuse the same caching, sampling, and error semantics as --endpoint

; they are thin adapters over each SDK's OpenAI-shaped response. Add a new backend by implementing the tiny Client

protocol — that is the only seam.

demo/ has runs against a local Ollama server: a committed

qwen3.5:9b

baseline vs a gemma3:1b

candidate (which does not support tool-calling). See for the transcript, or replay it:

demo/DEMO.md

asciinema play demo/probelock-demo.cast   # or: bash demo/demo.sh

The tool-calling capabilities drop 1.00 → 0.00

and the gate exits non-zero. tool_restraint

, tool_permission

, and no_hallucinated_tool

stay 1.00

(a model that cannot call tools cannot misuse one), and gemma3:1b

scores 1.00

on format_adherence

vs 0.50

for qwen3.5:9b

. The diff is per-capability.

Also committed: qwen3.5:9b

vs lfm2.5-thinking:1.2b

:

uv run probelock diff demo/qwen3.5-9b.lock demo/lfm2.5-thinking.lock

The 1.2B model matches qwen3.5:9b

on tool selection, discrimination, needle_in_tools

, arg_validity

, required_args

, and the three safety probes; structured_output

and arity_robustness

drop 1.00 → 0.33

.

Capability	What it checks	Scorer
tool_selection
Calls the right tool for the task	tool-name match
tool_discrimination
Calls the right tool and no other (picks precisely)
tool-name set
needle_in_tools
Finds the right tool when many (15+) are offered	tool-name match
arg_validity
Emitted args validate against the tool's JSON schema	jsonschema
required_args
All required args present and non-empty	key presence
arity_robustness
Fills every parameter (required + optional) when asked
all-present
structured_output
Emits schema-valid JSON on demand (no tools, no fences)	parse + jsonschema
json_mode (opt-in, --json-mode )
Same, but via the server's native response_format API instead of a prompt
parse + jsonschema
tool_restraint
Does not call a tool for a task that needs none (over-trigger)
no tool call
tool_permission
Does not call a tool it was explicitly forbidden to use
forbidden tool absent
no_hallucinated_tool
Does not fabricate a call to a tool that was not offered
called ⊆ offered
format_adherence
Follows an exact output constraint	exact match

Three are negative probes (a higher score means the bad behavior did not happen): tool_restraint

(over-triggering), tool_permission

(calling a forbidden tool), and no_hallucinated_tool

(fabricating a tool). All probes are derived from the tool schemas, not hand-authored.

tool schemas ──▶ derive probes ──▶ Client ──▶ ResponseMessage ──▶ deterministic scorer ──▶ Lockfile
 (your agent)    (zero authoring)  (model)    (the only model      (no LLM judge)          (commit it)
                                              -touching part)
                                                                    Lockfile + Lockfile ──▶ diff / gate

The only nondeterministic part is the Client

; everything else is pure, so the same inputs produce the same lockfile and the same diff. At temperature 0 the client caches identical requests, so the probes that share one request (the tool checks for a given tool) hit the network once. The SimulatedClient

crafts correct or incorrect responses that the real scorers grade, so the scoring path runs even with no model present.

Schema-derived probes are single-turn and synthetic — great for catching schema-level regressions, blind to what breaks after several turns of real context, a tool result feeding back in, or ambiguous phrasing. --traces

adds a second source: real, already-recorded agent decision points (e.g. exported from litellm's OpenTelemetry callback), replayed through the exact same deterministic scorers.

uv run probelock derive --tools tools.json --traces traces.json      # see what gets added
uv run probelock probe  --tools tools.json --traces traces.json \
    --endpoint http://localhost:11434/v1 --model llama3.1:8b -o candidate.lock

--tools

is optional here: traced probes replay their own embedded tool definitions, so a trace-only run (probe --traces traces.json ...

) needs no schema file. The same holds for --mined

below. Provide --tools

when you also want the synthetic battery.

A traces file is a small, stable JSON schema probelock defines itself — not raw OpenTelemetry — because OTel's own span attribute layout is not stable across libraries or versions (litellm has already changed where it puts request/response attributes once, and has a newer, differently-shaped opt-in integration). Converting your export into this shape is a one-time step you own; see examples/otel_traces_to_probelock.py for a documented starting point and

for the target shape:

fixtures/sample_traces.json

{
  "version": 1,
  "records": [
    {
      "id": "checkout-flow-turn-3",
      "messages": [{"role": "user", "content": "..."}],
      "tools": [ /* OpenAI-style tool defs actually offered at this turn */ ],
      "response": {"content": null, "tool_calls": [{"name": "...", "arguments": "{...}"}]}
    }
  ]
}

Trace-derived probes join the same capabilities as schema-derived ones — tool_selection

, tool_discrimination

, arg_validity

, required_args

, and structured_output

— since these map cleanly onto "replay this real context, check the candidate still behaves validly" (probe ids carry a ::traced::

marker if you want to inspect the split). The rest stay purely schema-derived: needle_in_tools

, tool_permission

, no_hallucinated_tool

, and tool_restraint

need a synthetic perturbation (an injected distractor tool, a forbidden-tool instruction, a removed tool) that a passively recorded trace does not naturally contain; format_adherence

needs an exact-text prompt, not a tool-calling decision point; and arity_robustness

needs its own explicit "fill EVERY parameter, including optional ones" instruction to mean anything — a real conversation was never asked for that, so replaying it would only test whichever optional fields happened to get filled in that one exchange, not robustness.

Unlike a tool schema, a traces file contains real conversation content. probe --traces

prints a warning every time, and the lockfile records a traces_fingerprint

so a diff

flags a baseline/candidate pair whose trace inputs differ — but review and redact the file yourself before committing it, the same way you would review any fixture with real data in it.

Tested against a real llama.cpp regression (commit-level, not synthetic): gate

fails on the regressed commit and passes on an adjacent, unrelated commit. See VALIDATION.md for the test setup and results, and

to reproduce it.

fixtures/gptoss_regression_trace.json

If your stack does not already log requests, the recording proxy captures them with one line changed in the agent:

probelock proxy --listen 127.0.0.1:8484 \
                --upstream http://127.0.0.1:11434 \
                --out traces/agent.jsonl

Every request is forwarded to the upstream unchanged (streaming included — SSE flows token by token and is reassembled for the record afterwards, tool-call deltas and all); each completed chat-completions exchange is appended asynchronously as one trace-v1

JSONL record. Recording is strictly non-invasive: on any internal logging error the request is still served and a warning goes to stderr. Failed or truncated exchanges (upstream errors, mid-stream disconnects) are logged with a failing status so ingest

skips them instead of mining half-generated responses. Multi-turn conversations are stitched into sessions without any agent cooperation (restarting the proxy mid-conversation splits that conversation into two sessions — harmless, but it weakens confirmation evidence, so prefer restarting between runs), --max-size

/ --max-age

rotate the log, and the file is created 0600

— it holds verbatim conversation content; keep it out of version control (redaction happens later, at ingest

).

--traces

(above) replays a curated export you assembled by hand. probelock ingest

goes one step earlier: point it at a raw request/response log of real agent traffic — the proxy's output, or your own logging — and it mines probes for you: multi-turn, realistic regression tests with near-zero authoring effort, still scored by the same deterministic checks (LLMs may appear in trace generation — that is your own agent — but never in scoring).

probelock ingest traces/agent.jsonl --out probes/mined.json   # everything lands "pending"
probelock traces review probes/mined.json                     # activate probes (y/n/e/a/s/q)
probelock probe --tools tools.json --mined probes/mined.json \
    --endpoint http://localhost:11434/v1 --model llama3.1:8b -o candidate.lock

ingest

accepts several logs at once (probelock ingest agent.jsonl agent-*.jsonl

) — pass a rotated set together so sessions spanning a rotation boundary keep their confirmation evidence.

Several input formats are supported (--format

, or auto

):

--format | Shape | |---|---| trace-v1 | the native record the recording proxy writes (one JSON object per line, request /response.message ) | openai-jsonl | the verbatim chat-completions request body next to the verbatim response, per line | anthropic-jsonl | logged Anthropic Messages API calls (request /response ); content blocks, tool_use /tool_result , and system are translated to the canonical shape | otel-genai | an OTLP-JSON span export, read via the OpenTelemetry GenAI semantic-convention attributes (gen_ai.prompt /gen_ai.completion , blob or indexed form) — scoped to the spec, not any one library's layout; spans without those attributes are skipped and counted |

auto

detects the JSONL shapes and OTel documents. See the fixtures/sample_*

files for each. For OTel exporters that do not follow the semantic convention, examples/otel_traces_to_probelock.py remains the conversion recipe.

Deduplication is exact-hash by default (deterministic). --cluster embeddings --embed-endpoint URL --embed-model NAME

instead groups near-duplicate contexts by embedding cosine similarity (via an OpenAI-compatible /v1/embeddings

endpoint you already run). This is opt-in and not deterministic — the grouping depends on the embedding model and version, so probelock prints a caveat and records cluster: embeddings

in each affected probe's provenance. Everything downstream (scoring, gating) stays deterministic; only which contexts merged does not.

Raw traffic includes model mistakes, so provenance determines trust — every probe records how many sessions support it and which rule confirmed it, and that decides how much review it needs:

Category	Check at replay	Mined from	Review
traced_schema_validity
some call's args validate against the called tool's schema	every tool-calling exchange (no inference)	--auto-accept schema_validity is safe
traced_tool_selection
calls the confirmed tool	exchanges confirmed good: the result fed back and the conversation moved on (no error payload, no corrected-args retry, no re-ask), or the same context produced the same call in ≥ --min-agreement distinct sessions
review, or --auto-accept-all --i-know-what-im-doing
traced_no_tool
answers in text, calls nothing	unanimous text answers across ≥ --min-agreement-notool (default 3) distinct sessions, no re-ask, preferring contexts lexically distant from the tools
individual review only — a mislabeled probe freezes a model mistake as expected behavior

The traced capabilities are deliberately separate names in the lockfile: a drop in multi-turn trace probes while single-turn synthetic probes hold steady is itself diagnostic ("context-length-sensitive regression").

Privacy defaults are conservative. Argument values in frozen contexts are always replaced with structure-preserving placeholders ("query": "<str:47ch>"

); message content stays verbatim (that is what makes replay realistic), so mined probes carry "sensitive": true

and probe -o

refuses to include them in a written lockfile without --allow-sensitive

. If you want committable probes, opt in to scrubbing with ingest --redact-patterns emails,phones,paths

. Identical contexts are deduplicated (timestamps stripped, whitespace collapsed), sampling keeps up to --per-capability

probes per (tool, category) preferring longer contexts and later turns, and everything the pipeline skips (failed calls, forced tool_choice

, oversized contexts, ambiguous agreement) is counted and reported — never silently dropped.

The full pipeline is validated against real agent traffic on real local models — including a runtime swap the gate catches and real frozen-mistake probes the review step rejects — in VALIDATION-TRACES.md.

diff

compares two lockfiles; trend

compares N, in the order you give them — a quantization ladder or the same model over time — so you can see where a capability holds and where it cliffs:

uv run probelock trend Q8_0.lock Q6_K.lock Q5_K_M.lock Q4_K_M.lock Q3_K_M.lock Q2_K.lock
Capability          Q8_0   Q6_K   Q5_K_M   Q4_K_M   Q3_K_M   Q2_K       Δ   Trend
structured_output   1.00   1.00     1.00     0.67     0.33   0.33   -0.67   ↓ regressed
tool_restraint      1.00   1.00     1.00     1.00     1.00   1.00   +0.00   = stable
tool_selection      1.00   1.00     0.67     1.00     0.67   1.00   +0.00   ~ unstable

Each row is annotated by its whole-ladder behavior: regressed

(net drop past --max-drop

), improved

, unstable

(net-flat but it dipped along the way — a signal a two-point diff of the endpoints would miss), stable

, removed

(present early but gone from the last rung — a dropped capability, counted as a regression), or partial

(present in fewer than two lockfiles). --format markdown|json|html

mirror diff

; the HTML view draws a sparkline per capability. trend

never fails on a regression (use gate

pairwise for CI); it exits non-zero only on bad input.

The filename stem is each column's header, so name your lockfiles for the axis (Q8_0.lock

, Q4_K_M.lock

).

With one sample per probe, a capability backed by 3 tools quantizes to {0, 0.33, 0.67, 1.0}

— a single flip moves it 0.33, far past the default 0.05 gate. So:

runs each probe N times and records the pass-probe --samples N [--temperature T]

rate(raise the temperature for sampling variance).only fails on a drop that is statistically significant for the recorded trial count (a one-sided two-proportion test). Sub-significant drops are shown asgate --confidence 0.95

and donoisy ↓

not fail the gate; raise--samples

to confirm or clear them.

A total collapse (e.g. 1.00 → 0.00

) is significant even at low N; a single-flip 1.00 → 0.67

over 3 trials is noisy

until you raise --samples

.

probelock init

scaffolds a probelock.tools.json

and a .github/workflows/probelock.yml

to start from. Commit a baseline lockfile, then gate each candidate:

- run: uvx probelock probe
       --tools tools.json --endpoint $LLM_URL --model $MODEL --samples 5 --temperature 0.7 -o candidate.lock
- run: uvx probelock gate
       --baseline probelock.lock --candidate candidate.lock --max-drop 0.05 --confidence 0.95

Or use the composite GitHub Action ( action.yml), which wraps those two steps end-to-end:

- uses: kelkalot/probelock@v0
  with:
    tools: tools.json
    baseline: probelock.lock
    endpoint: ${{ secrets.LLM_ENDPOINT }}
    model: ${{ vars.LLM_MODEL }}

To show the result on a pull request, render the diff as Markdown (or --format html

for a self-contained page):

probelock diff probelock.lock candidate.lock --format markdown >> "$GITHUB_STEP_SUMMARY"

• Proxy hardening: a static Go/Rust binary beside the reference Python implementation, and streaming-reassembly edge cases (multi-line SSE events, resume-after-disconnect).
• In-process backends (HF transformers

/ MLX) via a smallClient

adapter, no server required. - Emit OpenTelemetry spans from probe

runs, so a probe run shows up alongside your other agent traces in whatever backend you already use — a follow-on to trace-derived probes above (that direction consumes traces; this one produces them).

Apache-2.0 — see LICENSE.

Built with Claude Code.

source & further reading

github.com — original article