{"slug": "show-hn-llm-mock-record-and-replay-openai-anthropic-calls-in-pytest-v1-0", "title": "Show HN: LLM-mock – Record and replay OpenAI/Anthropic calls in pytest (v1.0)", "summary": "LLM-mock, a pytest plugin that records and replays OpenAI and Anthropic API calls, has been released in version 1.0. The tool intercepts HTTP calls to LLM APIs, allowing developers to record real responses once and replay them in tests without API keys, costs, or non-deterministic flakiness. It works with the Anthropic and OpenAI SDKs without requiring changes to application code.", "body_md": "**pytest plugin to mock OpenAI and Anthropic API calls** — record real responses once, replay them in tests forever. No API key needed in CI, no cost per run, no flaky non-determinism.\n\n```\n# Record once against the real API (run locally with your API key)\nwith llm_mock(mode=\"record\", fixture=\"tests/fixtures/summarize\"):\n    result = my_pipeline(\"Summarize this document...\")\n\n# Replay in tests — no API key, no cost, deterministic\n@pytest.mark.llm_replay(fixture=\"summarize\")\ndef test_summarize():\n    result = my_pipeline(\"Summarize this document...\")\n    assert \"key points\" in result\n```\n\nWorks with the **Anthropic SDK** (`claude-*`\n\nmodels) and the **OpenAI SDK** (`gpt-*`\n\nmodels) out of the box — no changes to your application code required.\n\n**Cost.** A CI pipeline hitting real LLM APIs can cost dollars per run at scale.**Flakiness.** LLM outputs are non-deterministic — even`temperature=0`\n\nvaries across model versions.**Speed.** Replayed fixtures return instantly; no network round-trip.**Offline.** Tests run without credentials in CI, on a plane, in a container.\n\nllm-mock intercepts at the **HTTP transport layer** (via `httpx`\n\n/`respx`\n\n) — your production code is never touched. Fixtures are plain JSON files you commit to git, diff in PRs, and refresh on demand.\n\n```\npip install llm-mock\n```\n\nOr install from source:\n\n```\ngit clone https://github.com/autopost/llm-mock.git\ncd llm-mock\npip install -e .\n```\n\n**Runtime dependencies:** `httpx`\n\n, `respx`\n\n, `pydantic`\n\n``` python\n# my_app/pipeline.py\nimport anthropic\n\nclient = anthropic.Anthropic()\n\ndef summarize(text: str) -> str:\n    message = client.messages.create(\n        model=\"claude-sonnet-4-6\",\n        max_tokens=100,\n        messages=[{\"role\": \"user\", \"content\": f\"Summarize: {text}\"}],\n    )\n    return message.content[0].text\n```\n\n`pipeline.py`\n\nhas zero knowledge of llm-mock. No imports, no changes needed.\n\nCreate a small script or a dedicated test that runs with `mode=\"record\"`\n\n. You need a real API key for this step.\n\n``` python\n# record_fixtures.py\nfrom llm_mock import llm_mock\nfrom my_app.pipeline import summarize\n\nwith llm_mock(mode=\"record\", fixture=\"tests/fixtures/summarize\"):\n    result = summarize(\"Long article about climate change...\")\n    print(result)  # real response from the API\nANTHROPIC_API_KEY=sk-... python record_fixtures.py\n```\n\nThis creates `tests/fixtures/summarize.json`\n\n. **Commit this file to git.**\n\nUse the pytest decorator — no `with`\n\nblock needed inside the test:\n\n``` python\n# tests/test_pipeline.py\nimport pytest\nfrom my_app.pipeline import summarize\n\n@pytest.mark.llm_replay(fixture=\"summarize\")\ndef test_summarize():\n    result = summarize(\"Long article about climate change...\")\n    assert \"climate\" in result\npytest  # no API key needed, runs offline, instant\n```\n\nThe decorator auto-discovers the fixture path relative to the test file — `fixture=\"summarize\"`\n\nlooks for `tests/fixtures/summarize.json`\n\nwhen the test lives in `tests/`\n\n.\n\nllm-mock intercepts the httpx call the Anthropic SDK makes internally and returns the saved response — your test code calls `summarize()`\n\nexactly as it would in production.\n\n**Alternative:** use the context manager directly if you need more control:\n\n``` python\nfrom llm_mock import llm_mock\n\ndef test_summarize():\n    with llm_mock(mode=\"replay\", fixture=\"tests/fixtures/summarize\"):\n        result = summarize(\"Long article about climate change...\")\n        assert \"climate\" in result\n```\n\nIf you change the prompt, update the model, or want to refresh fixtures:\n\n```\nANTHROPIC_API_KEY=sk-... python record_fixtures.py  # overwrites old fixture\ngit add tests/fixtures/summarize.json\ngit commit -m \"refresh summarize fixture\"\n```\n\nA complete working example from scratch.\n\n```\npip install llm-mock\necho 'export ANTHROPIC_API_KEY=sk-ant-api03-...' > .env\necho '.env' >> .gitignore\n```\n\nCreate `try_record.py`\n\n:\n\n``` python\nimport anthropic\nfrom llm_mock import llm_mock\n\nclient = anthropic.Anthropic()\n\nwith llm_mock(mode=\"record\", fixture=\"fixtures/hello\"):\n    message = client.messages.create(\n        model=\"claude-haiku-4-5-20251001\",\n        max_tokens=64,\n        messages=[{\"role\": \"user\", \"content\": \"Say hello in one sentence.\"}],\n    )\n    print(\"Response:\", message.content[0].text)\n    print(\"Fixture saved to fixtures/hello.json\")\nsource .env && .venv/bin/python try_record.py\n```\n\nYou should see the real response printed and `fixtures/hello.json`\n\ncreated.\n\n```\nllm-mock list tests/fixtures/hello\n```\n\nCreate `try_replay.py`\n\n:\n\n``` python\nimport anthropic\nfrom llm_mock import llm_mock\n\nclient = anthropic.Anthropic(api_key=\"fake-key\")  # key is irrelevant in replay\n\nwith llm_mock(mode=\"replay\", fixture=\"fixtures/hello\"):\n    message = client.messages.create(\n        model=\"claude-haiku-4-5-20251001\",\n        max_tokens=64,\n        messages=[{\"role\": \"user\", \"content\": \"Say hello in one sentence.\"}],\n    )\n    print(\"Replayed:\", message.content[0].text)\n.venv/bin/python try_replay.py\n```\n\nThe exact same response is returned instantly — no network call made.\n\n``` python\n# tests/test_hello.py\nimport anthropic\nimport pytest\n\nclient = anthropic.Anthropic(api_key=\"fake-key\")\n\n@pytest.mark.llm_replay(fixture=\"hello\")\ndef test_hello():\n    message = client.messages.create(\n        model=\"claude-haiku-4-5-20251001\",\n        max_tokens=64,\n        messages=[{\"role\": \"user\", \"content\": \"Say hello in one sentence.\"}],\n    )\n    assert message.content[0].text  # replayed from fixtures/hello.json\n.venv/bin/pytest tests/test_hello.py -v\n```\n\nInspect and manage fixture files from the terminal.\n\nNote:activate your virtual environment first so`llm-mock`\n\nis on your PATH:\n\n```\nsource .venv/bin/activate\n```\n\nOr run it directly with\n\n`.venv/bin/llm-mock <command>`\n\n.\n\nSet up llm-mock in your project. Run once in the project root:\n\n```\nllm-mock init\n```\n\nCreates:\n\n`tests/fixtures/`\n\n— directory for fixture files`record_fixtures.py`\n\n— ready-to-run record script`tests/test_example.py`\n\n— example test using`@pytest.mark.llm_replay`\n\nThen follow the printed next steps:\n\n```\nNext steps:\n  1. Record fixtures (needs API key, run once):\n       ANTHROPIC_API_KEY=sk-ant-... python record_fixtures.py\n  2. Commit the fixtures:\n       git add tests/fixtures/ && git commit -m 'add llm-mock fixtures'\n  3. Run tests (no API key needed):\n       pytest\n```\n\nShow all recorded interactions in a fixture file:\n\n``` bash\n$ llm-mock list tests/fixtures/summarize\n\nFixture : tests/fixtures/summarize.json\nProvider: anthropic\nInteractions: 2\n\n  1. a3f2c1d4e5b6…  claude-sonnet-4-6        2026-04-23T10:00:00\n       \"Summarize this document about climate change...\"\n  2. b4g3d2e5f6c7…  claude-haiku-4-5-20251001  2026-04-24T11:00:00\n       \"What is the capital of France?\"\n```\n\nDelete an entire fixture file:\n\n```\nllm-mock clear tests/fixtures/summarize\n```\n\nDelete a single interaction by hash:\n\n```\nllm-mock clear tests/fixtures/summarize --hash a3f2c1d4e5b6\nRecord mode:\n  Your code → Anthropic/OpenAI SDK → httpx\n    → llm-mock intercepts → forwards to real API\n    → saves response to fixture JSON\n    → returns response to your code\n\nReplay mode:\n  Your code → Anthropic/OpenAI SDK → httpx\n    → llm-mock intercepts → looks up fixture by SHA256(model + messages + temperature)\n    → returns saved response — no network call made\n```\n\n**Request matching** uses SHA256 of `(model, messages, temperature)`\n\n. Same request always hits the same fixture entry. Different temperature or different message content → different fixture entry.\n\n**Streaming** (`stream=True`\n\n) is fully supported. In record mode the full SSE event stream is captured and saved to the fixture. In replay mode the saved events are reconstructed and returned as a real SSE response — the SDK receives and processes them exactly as if they came from the live API.\n\n```\n# Record streaming response\nwith llm_mock(mode=\"record\", fixture=\"tests/fixtures/stream_summary\"):\n    with client.messages.stream(...) as stream:\n        text = stream.get_final_text()\n\n# Replay — no API key, instant, deterministic\n@pytest.mark.llm_replay(fixture=\"stream_summary\")\ndef test_streaming():\n    with client.messages.stream(...) as stream:\n        text = stream.get_final_text()\n    assert \"climate\" in text\n```\n\nContext manager that activates record, replay, or auto mode.\n\n| Parameter | Type | Description |\n|---|---|---|\n`mode` |\n`\"record\"` | `\"replay\"` | `\"auto\"` |\n`record` hits the real API and saves; `replay` returns from fixture; `auto` replays if fixture exists, records if not |\n`fixture` |\n`str` |\nPath to the fixture file. `.json` extension added automatically if omitted |\n`provider` |\n`\"anthropic\"` | `\"openai\"` | `\"all\"` |\nWhich provider(s) to intercept. Default: `\"all\"` |\n`match_on` |\n`list[str]` |\nFields used to match requests to fixtures. Default: `[\"model\", \"messages\", \"temperature\"]` |\n\n** auto mode** is the recommended default for most projects — it self-heals when new requests appear without manual mode switches:\n\n``` python\n@pytest.mark.llm_replay(fixture=\"summarize\", mode=\"auto\")\ndef test_summarize():\n    ...\npython\nfrom llm_mock import llm_mock\n\nwith llm_mock(mode=\"replay\", fixture=\"tests/fixtures/my_test\", provider=\"anthropic\"):\n    ...\n```\n\nBy default requests are matched by `model + messages + temperature`\n\n. You can customise this with `match_on`\n\n:\n\n```\n# Ignore temperature — different temperature values hit the same fixture\nwith llm_mock(mode=\"replay\", fixture=\"tests/fixtures/summary\",\n              match_on=[\"model\", \"messages\"]):\n    ...\n\n# Include system prompt in matching — different system prompts get separate fixture entries\nwith llm_mock(mode=\"replay\", fixture=\"tests/fixtures/summary\",\n              match_on=[\"model\", \"messages\", \"system\"]):\n    ...\n```\n\n**Supported fields:**\n\n| Field | Default | Description |\n|---|---|---|\n`\"model\"` |\nincluded | The model name, e.g. `\"claude-sonnet-4-6\"` , `\"gpt-4o\"` |\n`\"messages\"` |\nincluded | The full messages array — role + content |\n`\"temperature\"` |\nincluded | Sampling temperature. Remove from `match_on` to make tests temperature-agnostic |\n`\"system\"` |\nexcluded | Top-level system prompt. Add to `match_on` when different system prompts should produce separate fixture entries |\n\n**When to change the defaults:**\n\n**Exclude**— your app varies temperature between environments (dev vs prod) but you want a single fixture to cover both`temperature`\n\n**Include**— your app uses system prompts and you need separate fixtures per system prompt (e.g. different personas or instructions)`system`\n\n| Method | Effect |\n|---|---|\n`LLM_MOCK_DISABLED=1` |\nDisables all interception — LLM calls go to the real API as normal |\n`pytest --llm-mock-disabled` |\nSame as above, but as a pytest flag — no env var needed |\n\nUseful for refreshing all fixtures in one shot without touching test code:\n\n``` js\n# via env var\nLLM_MOCK_DISABLED=1 ANTHROPIC_API_KEY=sk-... pytest\n\n# via pytest flag\npytest --llm-mock-disabled\n```\n\nOr in a weekly CI job that validates against the live model.\n\n| Exception | When raised |\n|---|---|\n`FixtureNotFoundError` |\nReplay mode: fixture file missing, or no matching hash in file |\n`FixtureParseError` |\nFixture file exists but contains invalid JSON |\n\n``` python\nfrom llm_mock import llm_mock, FixtureNotFoundError\n\ntry:\n    with llm_mock(mode=\"replay\", fixture=\"tests/fixtures/missing\"):\n        client.messages.create(...)\nexcept FixtureNotFoundError as e:\n    print(e)  # includes hint to run in record mode first\n```\n\nFixture files are plain JSON — readable, diffable, committable.\n\n```\n{\n  \"version\": \"2.0\",\n  \"provider\": \"anthropic\",\n  \"interactions\": [\n    {\n      \"hash\": \"a3f2c1...\",\n      \"request\": {\n        \"model\": \"claude-sonnet-4-6\",\n        \"messages\": [{\"role\": \"user\", \"content\": \"Say hello.\"}],\n        \"max_tokens\": 64\n      },\n      \"response\": {\n        \"id\": \"msg_01XYZ\",\n        \"type\": \"message\",\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"Hello! How can I help you today?\"}],\n        \"model\": \"claude-sonnet-4-6\",\n        \"stop_reason\": \"end_turn\",\n        \"usage\": {\"input_tokens\": 10, \"output_tokens\": 9}\n      },\n      \"recorded_at\": \"2026-04-23T10:00:00+00:00\"\n    }\n  ]\n}\n```\n\nMultiple interactions (from different requests) are stored in the same file. Re-recording an existing hash overwrites only that entry.\n\n| Provider | Intercepted endpoint | Status |\n|---|---|---|\n| Anthropic | `api.anthropic.com/v1/messages` |\nSupported |\n| OpenAI | `api.openai.com/v1/chat/completions` |\nSupported |\nStreaming (`stream=True` ) |\nAnthropic + OpenAI | Supported |\n\n| Tool | Record mode | Native SDK support | In-process |\n|---|---|---|---|\nllm-mock |\nyes | yes (Anthropic + OpenAI) | yes |\n|\n\n[AIMock](https://github.com/CopilotKit/aimock)[vcr-langchain](https://github.com/amosjyng/vcr-langchain)\n\n```\ngit clone https://github.com/autopost/llm-mock\ncd llm-mock\npython -m venv .venv && source .venv/bin/activate\npip install -e \".[dev]\"\npytest\n```\n\n**v0.2**—`auto`\n\nmode,`LLM_MOCK_DISABLED`\n\nenv var ✓**v0.3**—`match_on`\n\nconfigurable match keys,`--llm-mock-disabled`\n\npytest flag,`llm-mock init`\n\ncommand ✓**v1.0**— streaming support, fixture schema v2.0,`llm-mock list`\n\nstreaming flag ✓**v2**— shared fixtures for teams, semantic matching, web dashboard\n\n`pytest mock openai`\n\n· `pytest mock anthropic`\n\n· `mock LLM calls python`\n\n· `record replay LLM`\n\n· `vcr cassette openai`\n\n· `fake openai response pytest`\n\n· `test without API key`\n\n· `offline LLM testing`\n\n· `deterministic LLM tests`", "url": "https://wpnews.pro/news/show-hn-llm-mock-record-and-replay-openai-anthropic-calls-in-pytest-v1-0", "canonical_source": "https://github.com/autopost/llm-mock", "published_at": "2026-07-13 10:21:38+00:00", "updated_at": "2026-07-13 10:35:35.913409+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["OpenAI", "Anthropic", "llm-mock", "pytest", "httpx", "respx", "pydantic"], "alternates": {"html": "https://wpnews.pro/news/show-hn-llm-mock-record-and-replay-openai-anthropic-calls-in-pytest-v1-0", "markdown": "https://wpnews.pro/news/show-hn-llm-mock-record-and-replay-openai-anthropic-calls-in-pytest-v1-0.md", "text": "https://wpnews.pro/news/show-hn-llm-mock-record-and-replay-openai-anthropic-calls-in-pytest-v1-0.txt", "jsonld": "https://wpnews.pro/news/show-hn-llm-mock-record-and-replay-openai-anthropic-calls-in-pytest-v1-0.jsonld"}}