pytest plugin to mock OpenAI and Anthropic API calls — record real responses once, replay them in tests forever. No API key needed in CI, no cost per run, no flaky non-determinism.
with llm_mock(mode="record", fixture="tests/fixtures/summarize"):
result = my_pipeline("Summarize this document...")
@pytest.mark.llm_replay(fixture="summarize")
def test_summarize():
result = my_pipeline("Summarize this document...")
assert "key points" in result
Works with the Anthropic SDK (claude-*
models) and the OpenAI SDK (gpt-*
models) out of the box — no changes to your application code required.
Cost. A CI pipeline hitting real LLM APIs can cost dollars per run at scale.Flakiness. LLM outputs are non-deterministic — eventemperature=0
varies across model versions.Speed. Replayed fixtures return instantly; no network round-trip.Offline. Tests run without credentials in CI, on a plane, in a container.
llm-mock intercepts at the HTTP transport layer (via httpx
/respx
) — your production code is never touched. Fixtures are plain JSON files you commit to git, diff in PRs, and refresh on demand.
pip install llm-mock
Or install from source:
git clone https://github.com/autopost/llm-mock.git
cd llm-mock
pip install -e .
Runtime dependencies: httpx
, respx
, pydantic
import anthropic
client = anthropic.Anthropic()
def summarize(text: str) -> str:
message = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=100,
messages=[{"role": "user", "content": f"Summarize: {text}"}],
)
return message.content[0].text
pipeline.py
has zero knowledge of llm-mock. No imports, no changes needed.
Create a small script or a dedicated test that runs with mode="record"
. You need a real API key for this step.
from llm_mock import llm_mock
from my_app.pipeline import summarize
with llm_mock(mode="record", fixture="tests/fixtures/summarize"):
result = summarize("Long article about climate change...")
print(result) # real response from the API
ANTHROPIC_API_KEY=sk-... python record_fixtures.py
This creates tests/fixtures/summarize.json
. Commit this file to git.
Use the pytest decorator — no with
block needed inside the test:
import pytest
from my_app.pipeline import summarize
@pytest.mark.llm_replay(fixture="summarize")
def test_summarize():
result = summarize("Long article about climate change...")
assert "climate" in result
pytest # no API key needed, runs offline, instant
The decorator auto-discovers the fixture path relative to the test file — fixture="summarize"
looks for tests/fixtures/summarize.json
when the test lives in tests/
.
llm-mock intercepts the httpx call the Anthropic SDK makes internally and returns the saved response — your test code calls summarize()
exactly as it would in production.
Alternative: use the context manager directly if you need more control:
from llm_mock import llm_mock
def test_summarize():
with llm_mock(mode="replay", fixture="tests/fixtures/summarize"):
result = summarize("Long article about climate change...")
assert "climate" in result
If you change the prompt, update the model, or want to refresh fixtures:
ANTHROPIC_API_KEY=sk-... python record_fixtures.py # overwrites old fixture
git add tests/fixtures/summarize.json
git commit -m "refresh summarize fixture"
A complete working example from scratch.
pip install llm-mock
echo 'export ANTHROPIC_API_KEY=sk-ant-api03-...' > .env
echo '.env' >> .gitignore
Create try_record.py
:
import anthropic
from llm_mock import llm_mock
client = anthropic.Anthropic()
with llm_mock(mode="record", fixture="fixtures/hello"):
message = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print("Response:", message.content[0].text)
print("Fixture saved to fixtures/hello.json")
source .env && .venv/bin/python try_record.py
You should see the real response printed and fixtures/hello.json
created.
llm-mock list tests/fixtures/hello
Create try_replay.py
:
import anthropic
from llm_mock import llm_mock
client = anthropic.Anthropic(api_key="fake-key") # key is irrelevant in replay
with llm_mock(mode="replay", fixture="fixtures/hello"):
message = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print("Replayed:", message.content[0].text)
.venv/bin/python try_replay.py
The exact same response is returned instantly — no network call made.
import anthropic
import pytest
client = anthropic.Anthropic(api_key="fake-key")
@pytest.mark.llm_replay(fixture="hello")
def test_hello():
message = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
assert message.content[0].text # replayed from fixtures/hello.json
.venv/bin/pytest tests/test_hello.py -v
Inspect and manage fixture files from the terminal.
Note:activate your virtual environment first sollm-mock
is on your PATH:
source .venv/bin/activate
Or run it directly with
.venv/bin/llm-mock <command>
.
Set up llm-mock in your project. Run once in the project root:
llm-mock init
Creates:
tests/fixtures/
— directory for fixture filesrecord_fixtures.py
— ready-to-run record scripttests/test_example.py
— example test using@pytest.mark.llm_replay
Then follow the printed next steps:
Next steps:
1. Record fixtures (needs API key, run once):
ANTHROPIC_API_KEY=sk-ant-... python record_fixtures.py
2. Commit the fixtures:
git add tests/fixtures/ && git commit -m 'add llm-mock fixtures'
3. Run tests (no API key needed):
pytest
Show all recorded interactions in a fixture file:
$ llm-mock list tests/fixtures/summarize
Fixture : tests/fixtures/summarize.json
Provider: anthropic
Interactions: 2
1. a3f2c1d4e5b6… claude-sonnet-4-6 2026-04-23T10:00:00
"Summarize this document about climate change..."
2. b4g3d2e5f6c7… claude-haiku-4-5-20251001 2026-04-24T11:00:00
"What is the capital of France?"
Delete an entire fixture file:
llm-mock clear tests/fixtures/summarize
Delete a single interaction by hash:
llm-mock clear tests/fixtures/summarize --hash a3f2c1d4e5b6
Record mode:
Your code → Anthropic/OpenAI SDK → httpx
→ llm-mock intercepts → forwards to real API
→ saves response to fixture JSON
→ returns response to your code
Replay mode:
Your code → Anthropic/OpenAI SDK → httpx
→ llm-mock intercepts → looks up fixture by SHA256(model + messages + temperature)
→ returns saved response — no network call made
Request matching uses SHA256 of (model, messages, temperature)
. Same request always hits the same fixture entry. Different temperature or different message content → different fixture entry.
Streaming (stream=True
) is fully supported. In record mode the full SSE event stream is captured and saved to the fixture. In replay mode the saved events are reconstructed and returned as a real SSE response — the SDK receives and processes them exactly as if they came from the live API.
with llm_mock(mode="record", fixture="tests/fixtures/stream_summary"):
with client.messages.stream(...) as stream:
text = stream.get_final_text()
@pytest.mark.llm_replay(fixture="stream_summary")
def test_streaming():
with client.messages.stream(...) as stream:
text = stream.get_final_text()
assert "climate" in text
Context manager that activates record, replay, or auto mode.
| Parameter | Type | Description |
|---|---|---|
mode |
||
"record" |
"replay" |
"auto" |
record hits the real API and saves; replay returns from fixture; auto replays if fixture exists, records if not |
||
fixture |
||
str |
||
Path to the fixture file. .json extension added automatically if omitted |
||
provider |
||
"anthropic" |
"openai" |
"all" |
Which provider(s) to intercept. Default: "all" |
||
match_on |
||
list[str] |
||
Fields used to match requests to fixtures. Default: ["model", "messages", "temperature"] |
** auto mode** is the recommended default for most projects — it self-heals when new requests appear without manual mode switches:
@pytest.mark.llm_replay(fixture="summarize", mode="auto")
def test_summarize():
...
python
from llm_mock import llm_mock
with llm_mock(mode="replay", fixture="tests/fixtures/my_test", provider="anthropic"):
...
By default requests are matched by model + messages + temperature
. You can customise this with match_on
:
with llm_mock(mode="replay", fixture="tests/fixtures/summary",
match_on=["model", "messages"]):
...
with llm_mock(mode="replay", fixture="tests/fixtures/summary",
match_on=["model", "messages", "system"]):
...
Supported fields:
| Field | Default | Description |
|---|---|---|
"model" |
||
| included | The model name, e.g. "claude-sonnet-4-6" , "gpt-4o" |
|
"messages" |
||
| included | The full messages array — role + content | |
"temperature" |
||
| included | Sampling temperature. Remove from match_on to make tests temperature-agnostic |
|
"system" |
||
| excluded | Top-level system prompt. Add to match_on when different system prompts should produce separate fixture entries |
When to change the defaults:
Exclude— your app varies temperature between environments (dev vs prod) but you want a single fixture to cover bothtemperature
Include— your app uses system prompts and you need separate fixtures per system prompt (e.g. different personas or instructions)system
| Method | Effect |
|---|---|
LLM_MOCK_DISABLED=1 |
|
| Disables all interception — LLM calls go to the real API as normal | |
pytest --llm-mock-disabled |
|
| Same as above, but as a pytest flag — no env var needed |
Useful for refreshing all fixtures in one shot without touching test code:
LLM_MOCK_DISABLED=1 ANTHROPIC_API_KEY=sk-... pytest
pytest --llm-mock-disabled
Or in a weekly CI job that validates against the live model.
| Exception | When raised |
|---|---|
FixtureNotFoundError |
|
| Replay mode: fixture file missing, or no matching hash in file | |
FixtureParseError |
|
| Fixture file exists but contains invalid JSON |
from llm_mock import llm_mock, FixtureNotFoundError
try:
with llm_mock(mode="replay", fixture="tests/fixtures/missing"):
client.messages.create(...)
except FixtureNotFoundError as e:
print(e) # includes hint to run in record mode first
Fixture files are plain JSON — readable, diffable, committable.
{
"version": "2.0",
"provider": "anthropic",
"interactions": [
{
"hash": "a3f2c1...",
"request": {
"model": "claude-sonnet-4-6",
"messages": [{"role": "user", "content": "Say hello."}],
"max_tokens": 64
},
"response": {
"id": "msg_01XYZ",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "Hello! How can I help you today?"}],
"model": "claude-sonnet-4-6",
"stop_reason": "end_turn",
"usage": {"input_tokens": 10, "output_tokens": 9}
},
"recorded_at": "2026-04-23T10:00:00+00:00"
}
]
}
Multiple interactions (from different requests) are stored in the same file. Re-recording an existing hash overwrites only that entry.
| Provider | Intercepted endpoint | Status |
|---|---|---|
| Anthropic | api.anthropic.com/v1/messages |
|
| Supported | ||
| OpenAI | api.openai.com/v1/chat/completions |
|
| Supported | ||
Streaming (stream=True ) |
||
| Anthropic + OpenAI | Supported |
| Tool | Record mode | Native SDK support | In-process |
|---|---|---|---|
| llm-mock | |||
| yes | yes (Anthropic + OpenAI) | yes | |
git clone https://github.com/autopost/llm-mock
cd llm-mock
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
v0.2—auto
mode,LLM_MOCK_DISABLED
env var ✓v0.3—match_on
configurable match keys,--llm-mock-disabled
pytest flag,llm-mock init
command ✓v1.0— streaming support, fixture schema v2.0,llm-mock list
streaming flag ✓v2— shared fixtures for teams, semantic matching, web dashboard
pytest mock openai
· pytest mock anthropic
· mock LLM calls python
· record replay LLM
· vcr cassette openai
· fake openai response pytest
· test without API key
· offline LLM testing
· deterministic LLM tests