{"slug": "testing-ai-pipelines-without-calling-the-model-every-time", "title": "Testing AI Pipelines Without Calling the Model Every Time", "summary": "A developer outlined a pattern for testing LLM pipelines without hitting the model on every run: wrap the model behind a small Protocol interface, inject a FakeLLM in tests that returns canned replies and records prompts, and reserve real-model calls for a separately marked live suite. The approach lets prompt building, parsing, retrieval and routing reach high coverage in CI via `pytest -m \"not live\"`, while parametrized fakes cover failure modes like empty strings, non-JSON text and truncated JSON.", "body_md": "Tests that call a real LLM are slow, cost money on every run, and fail randomly because the output changes. So many teams quietly skip testing their AI features altogether.\n\nThe fix is to notice that most of an \"AI pipeline\" isn't the model at all. Building the prompt, retrieving context, parsing the reply and deciding what to do next are ordinary, deterministic code, and ordinary code is easy to test.\n\nMake your pipeline depend on a small interface instead of a specific SDK:\n\n``` python\nfrom typing import Protocol\n\nclass LLM(Protocol):\n    async def complete(self, prompt: str) -> str: ...\n\nclass FakeLLM:\n    def __init__(self, reply: str) -> None:\n        self.reply = reply\n        self.prompts: list[str] = []\n\n    async def complete(self, prompt: str) -> str:\n        self.prompts.append(prompt)\n        return self.reply\n```\n\nProduction passes in the real client. Tests pass in `FakeLLM`, which returns whatever you tell it to and records what it was asked.\n\nNow you can check that the right information reaches the prompt and that replies are handled correctly, with no network calls:\n\n``` python\nimport pytest\n\n@pytest.mark.asyncio\nasync def test_extracts_order_id_from_reply() -> None:\n    llm = FakeLLM('{\"order_id\": \"A-1042\", \"intent\": \"refund\"}')\n\n    result = await classify_message(\"Where is my refund for A-1042?\", llm=llm)\n\n    assert result.order_id == \"A-1042\"\n    assert \"A-1042\" in llm.prompts[0]\n```\n\n(These examples use the `pytest-asyncio` plugin for async tests.)\n\nReal models sometimes return empty strings, plain text instead of JSON, or JSON that's cut off halfway. Those are exactly the cases that crash production, and they're trivial to test with a fake:\n\n```\n@pytest.mark.asyncio\n@pytest.mark.parametrize(\"reply\", [\"\", \"not json\", '{\"intent\": \"refund\"'])\nasync def test_bad_model_output_falls_back(reply: str) -> None:\n    result = await classify_message(\"hello\", llm=FakeLLM(reply))\n    assert result.intent == \"unknown\"\n```\n\nYou still want some tests against the real model, to catch prompt regressions and provider changes. Mark them and keep them out of the normal run:\n\n```\n@pytest.mark.live\n@pytest.mark.asyncio\nasync def test_live_refund_classification(real_llm) -> None:\n    result = await classify_message(\"I want my money back\", llm=real_llm)\n    assert result.intent == \"refund\"\n```\n\nCI runs `pytest -m \"not live\"` on every commit. The live suite runs on a schedule or before a release.\n\nWith the model behind an interface, prompt building, parsing, retrieval and routing can all reach high coverage, because none of them needs a network call. That's where most bugs live anyway.\n\nFast tests on every commit, a small live suite for real-model behaviour, and confidence when you change a prompt or a parser. AI features don't have to be the untested corner of your codebase.\n\nHow do you test your LLM features? I'm curious what's working for other teams.", "url": "https://wpnews.pro/news/testing-ai-pipelines-without-calling-the-model-every-time", "canonical_source": "https://dev.to/deependra_rajput_360d4024/testing-ai-pipelines-without-calling-the-model-every-time-4h6o", "published_at": "2026-11-20 12:09:45+00:00", "updated_at": "2026-11-20 12:12:39.641213+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "mlops"], "entities": ["pytest", "pytest-asyncio"], "also_reported_by": [], "takeaway": "Put the LLM behind a Protocol interface and inject a FakeLLM that returns canned replies and records prompts, so prompt building, parsing and routing are tested deterministically in CI while a small `live`-marked suite covers real-model behavior.", "recommendation": {"advice": "Wrap your LLM client behind a Protocol and inject a fake in unit tests, then run real-model tests only under a `live` marker excluded from CI.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/testing-ai-pipelines-without-calling-the-model-every-time", "markdown": "https://wpnews.pro/news/testing-ai-pipelines-without-calling-the-model-every-time.md", "text": "https://wpnews.pro/news/testing-ai-pipelines-without-calling-the-model-every-time.txt", "jsonld": "https://wpnews.pro/news/testing-ai-pipelines-without-calling-the-model-every-time.jsonld"}}