How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)

A developer built CreativeAudit, a production-oriented pipeline that evaluates ad creatives against a brief using a local LLM and strict Pydantic contracts. The system returns a PASS, NEEDS_REVISION, or FAIL verdict with weighted scores, reducing first-pass review time from tens of minutes to seconds. The developer emphasizes that the hardest part was designing contracts and failure modes to ensure reliability when the model behaves badly.

Art directors routinely spend 20–40 minutes per creative just checking brand guidelines, mandatory elements, and forbidden techniques. I wanted to automate the first-pass review — without turning it into another unreliable ChatGPT wrapper. So I built CreativeAudit : a production-oriented pipeline that evaluates ad creatives against a brief and returns a clear PASS / NEEDS REVISION / FAIL verdict. Automated ad creative evaluation against a brief using a local LLM. CreativeAudit takes a brief + creatives, builds a precise prompt, gets a structured evaluation from a language model, strictly validates the response, and returns a clear verdict PASS / NEEDS REVISION / FAIL , weighted score, and explanation. This is not "just another AI chat". It is a production-oriented first-pass review pipeline: the machine catches routine and critical violations so humans only need to look at the borderline cases. Campaign summary metrics, creative cards with verdict, detailed scores and model explanation. Smart-input mode converts free-form text into structured JSON with human review before evaluation starts. Manual first-pass creative review is slow, expensive and inconsistent: A plain chat with an LLM has several fatal flaws for this use case: For any real workflow this is unacceptable. I set a few hard requirements: Brief + Creatives → Jinja2 Prompt → Local LLM Ollama ↓ Pydantic Validation ↓ Score + Verdict + Feedback Architecture diagram showing the flow from Brief and Creatives through Jinja2 Prompt and Ollama, followed by Pydantic Validation, resulting in Score, Verdict, and Feedback Key components: app/schemas.py — strict Pydantic v2 contracts prompts/ .j2 — prompts treated as versioned code app/main.py — orchestration prompt assembly, LLM call, validation, scoring demo/streamlit app.py — thin UI layer The business logic is completely separated from the interface. You can call it from Streamlit, CLI, or any other service. The model is non-deterministic. I treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score. For brand-book violations I use a binary 0 or 10 score. There is no “slightly violated”. This removes a lot of model subjectivity on the highest-risk criterion. Prompts live in separate .j2 files. This makes them versionable, reviewable, and easy to A/B test without touching application code. Everything runs locally. Switching between qwen2.5:7b, qwen2.5:14b or llama3.2 is a single config change. Managers rarely provide clean JSON. They paste chat fragments and rough descriptions. So the pipeline first uses an LLM to extract structured data, shows the result to the user for correction, and only then runs the evaluation. Each creative receives three scores: brand alignment 1–10 constraint compliance 0 or 10 message clarity 1–10 total = brand × 0.4 + compliance × 0.3 + clarity × 0.3 The verdict is derived from the total score and critical failures. I wrote 38 unit tests. The external LLM is fully mocked, so tests are deterministic and run in under 2 seconds. They cover schema boundaries, malformed responses, connection failures, and scoring logic. The biggest wins: First-pass review time dropped from tens of minutes to seconds Output became consistent and machine-readable Critical brand violations are much harder to miss The hardest part wasn’t the LLM call — it was designing the contracts and failure modes so the system stays reliable when the model behaves badly. Multimodal support evaluate the actual layouts, not only text Model benchmarks accuracy vs speed REST API for integration into existing workflows Evaluation history and campaign analytics If you’re building production LLM pipelines, I’d love to hear how you handle structured output and hallucination control. How do you currently deal with unreliable LLM responses in your projects?