cd /news/ai-tools/testing-ai-pipelines-without-calling… · home › topics › ai-tools › article
[ARTICLE · art-173210] src=dev.to ↗ pub= topic=ai-tools verified=true sentiment=↑ positive

Testing AI Pipelines Without Calling the Model Every Time

A developer outlined a pattern for testing LLM pipelines without hitting the model on every run: wrap the model behind a small Protocol interface, inject a FakeLLM in tests that returns canned replies and records prompts, and reserve real-model calls for a separately marked live suite. The approach lets prompt building, parsing, retrieval and routing reach high coverage in CI via `pytest -m "not live"`, while parametrized fakes cover failure modes like empty strings, non-JSON text and truncated JSON.

recommendation

Wrap your LLM client behind a Protocol and inject a fake in unit tests, then run real-model tests only under a `live` marker excluded from CI.

more in playbook: Evals and testing for LLM apps
by read2 min views7 publishedNov 20, 2026

Tests that call a real LLM are slow, cost money on every run, and fail randomly because the output changes. So many teams quietly skip testing their AI features altogether.

The fix is to notice that most of an "AI pipeline" isn't the model at all. Building the prompt, retrieving context, parsing the reply and deciding what to do next are ordinary, deterministic code, and ordinary code is easy to test.

Make your pipeline depend on a small interface instead of a specific SDK:

from typing import Protocol

class LLM(Protocol):
    async def complete(self, prompt: str) -> str: ...

class FakeLLM:
    def __init__(self, reply: str) -> None:
        self.reply = reply
        self.prompts: list[str] = []

    async def complete(self, prompt: str) -> str:
        self.prompts.append(prompt)
        return self.reply

Production passes in the real client. Tests pass in FakeLLM, which returns whatever you tell it to and records what it was asked.

Now you can check that the right information reaches the prompt and that replies are handled correctly, with no network calls:

import pytest

@pytest.mark.asyncio
async def test_extracts_order_id_from_reply() -> None:
    llm = FakeLLM('{"order_id": "A-1042", "intent": "refund"}')

    result = await classify_message("Where is my refund for A-1042?", llm=llm)

    assert result.order_id == "A-1042"
    assert "A-1042" in llm.prompts[0]

(These examples use the pytest-asyncio plugin for async tests.)

Real models sometimes return empty strings, plain text instead of JSON, or JSON that's cut off halfway. Those are exactly the cases that crash production, and they're trivial to test with a fake:

@pytest.mark.asyncio
@pytest.mark.parametrize("reply", ["", "not json", '{"intent": "refund"'])
async def test_bad_model_output_falls_back(reply: str) -> None:
    result = await classify_message("hello", llm=FakeLLM(reply))
    assert result.intent == "unknown"

You still want some tests against the real model, to catch prompt regressions and provider changes. Mark them and keep them out of the normal run:

@pytest.mark.live
@pytest.mark.asyncio
async def test_live_refund_classification(real_llm) -> None:
    result = await classify_message("I want my money back", llm=real_llm)
    assert result.intent == "refund"

CI runs pytest -m "not live" on every commit. The live suite runs on a schedule or before a release.

With the model behind an interface, prompt building, parsing, retrieval and routing can all reach high coverage, because none of them needs a network call. That's where most bugs live anyway.

Fast tests on every commit, a small live suite for real-model behaviour, and confidence when you change a prompt or a parser. AI features don't have to be the untested corner of your codebase.

How do you test your LLM features? I'm curious what's working for other teams.

── more in #ai-tools 4 stories · sorted by recency
── more on @pytest 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/testing-ai-pipelines…] indexed:0 read:2min 2026-11-20 · —