Testing AI Pipelines Without Calling the Model Every Time A developer outlined a pattern for testing LLM pipelines without hitting the model on every run: wrap the model behind a small Protocol interface, inject a FakeLLM in tests that returns canned replies and records prompts, and reserve real-model calls for a separately marked live suite. The approach lets prompt building, parsing, retrieval and routing reach high coverage in CI via `pytest -m "not live"`, while parametrized fakes cover failure modes like empty strings, non-JSON text and truncated JSON. Tests that call a real LLM are slow, cost money on every run, and fail randomly because the output changes. So many teams quietly skip testing their AI features altogether. The fix is to notice that most of an "AI pipeline" isn't the model at all. Building the prompt, retrieving context, parsing the reply and deciding what to do next are ordinary, deterministic code, and ordinary code is easy to test. Make your pipeline depend on a small interface instead of a specific SDK: python from typing import Protocol class LLM Protocol : async def complete self, prompt: str - str: ... class FakeLLM: def init self, reply: str - None: self.reply = reply self.prompts: list str = async def complete self, prompt: str - str: self.prompts.append prompt return self.reply Production passes in the real client. Tests pass in FakeLLM , which returns whatever you tell it to and records what it was asked. Now you can check that the right information reaches the prompt and that replies are handled correctly, with no network calls: python import pytest @pytest.mark.asyncio async def test extracts order id from reply - None: llm = FakeLLM '{"order id": "A-1042", "intent": "refund"}' result = await classify message "Where is my refund for A-1042?", llm=llm assert result.order id == "A-1042" assert "A-1042" in llm.prompts 0 These examples use the pytest-asyncio plugin for async tests. Real models sometimes return empty strings, plain text instead of JSON, or JSON that's cut off halfway. Those are exactly the cases that crash production, and they're trivial to test with a fake: @pytest.mark.asyncio @pytest.mark.parametrize "reply", "", "not json", '{"intent": "refund"' async def test bad model output falls back reply: str - None: result = await classify message "hello", llm=FakeLLM reply assert result.intent == "unknown" You still want some tests against the real model, to catch prompt regressions and provider changes. Mark them and keep them out of the normal run: @pytest.mark.live @pytest.mark.asyncio async def test live refund classification real llm - None: result = await classify message "I want my money back", llm=real llm assert result.intent == "refund" CI runs pytest -m "not live" on every commit. The live suite runs on a schedule or before a release. With the model behind an interface, prompt building, parsing, retrieval and routing can all reach high coverage, because none of them needs a network call. That's where most bugs live anyway. Fast tests on every commit, a small live suite for real-model behaviour, and confidence when you change a prompt or a parser. AI features don't have to be the untested corner of your codebase. How do you test your LLM features? I'm curious what's working for other teams.