{"slug": "how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict", "title": "How I Built a Reliable LLM Pipeline for Ad Creative Evaluation (with Strict Pydantic Contracts)", "summary": "A developer built CreativeAudit, a production-oriented pipeline that evaluates ad creatives against a brief using a local LLM and strict Pydantic contracts. The system returns a PASS, NEEDS_REVISION, or FAIL verdict with weighted scores, reducing first-pass review time from tens of minutes to seconds. The developer emphasizes that the hardest part was designing contracts and failure modes to ensure reliability when the model behaves badly.", "body_md": "Art directors routinely spend 20–40 minutes per creative just checking brand guidelines, mandatory elements, and forbidden techniques.\n\nI wanted to automate the first-pass review — without turning it into another unreliable ChatGPT wrapper.\n\nSo I built **CreativeAudit**: a production-oriented pipeline that evaluates ad creatives against a brief and returns a clear `PASS / NEEDS_REVISION / FAIL`\n\nverdict.\n\n**Automated ad creative evaluation against a brief using a local LLM.**\n\nCreativeAudit takes a brief + creatives, builds a precise prompt, gets a structured evaluation from a language model, strictly validates the response, and returns a clear verdict (**PASS / NEEDS_REVISION / FAIL**), weighted score, and explanation.\n\nThis is not \"just another AI chat\". It is a production-oriented first-pass review pipeline: the machine catches routine and critical violations so humans only need to look at the borderline cases.\n\n*Campaign summary metrics, creative cards with verdict, detailed scores and model explanation. Smart-input mode converts free-form text into structured JSON with human review before evaluation starts.*\n\nManual first-pass creative review is slow, expensive and inconsistent:\n\nA plain chat with an LLM has several fatal flaws for this use case:\n\nFor any real workflow this is unacceptable.\n\nI set a few hard requirements:\n\n```\nBrief + Creatives → Jinja2 Prompt → Local LLM (Ollama)\n                                      ↓\n                               Pydantic Validation\n                                      ↓\n                          Score + Verdict + Feedback\n```\n\nArchitecture diagram showing the flow from Brief and Creatives through Jinja2 Prompt and Ollama, followed by Pydantic Validation, resulting in Score, Verdict, and Feedback\n\nKey components:\n\napp/schemas.py — strict Pydantic v2 contracts\n\nprompts/*.j2 — prompts treated as versioned code\n\napp/main.py — orchestration (prompt assembly, LLM call, validation, scoring)\n\ndemo/streamlit_app.py — thin UI layer\n\nThe business logic is completely separated from the interface. You can call it from Streamlit, CLI, or any other service.\n\nThe model is non-deterministic.\n\nI treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score.\n\nFor brand-book violations I use a binary 0 or 10 score.\n\nThere is no “slightly violated”. This removes a lot of model subjectivity on the highest-risk criterion.\n\nPrompts live in separate .j2 files.\n\nThis makes them versionable, reviewable, and easy to A/B test without touching application code.\n\nEverything runs locally.\n\nSwitching between qwen2.5:7b, qwen2.5:14b or llama3.2 is a single config change.\n\nManagers rarely provide clean JSON.\n\nThey paste chat fragments and rough descriptions.\n\nSo the pipeline first uses an LLM to extract structured data, shows the result to the user for correction, and only then runs the evaluation.\n\nEach creative receives three scores:\n\nbrand_alignment (1–10)\n\nconstraint_compliance (0 or 10)\n\nmessage_clarity (1–10)\n\ntotal = brand × 0.4 + compliance × 0.3 + clarity × 0.3\n\nThe verdict is derived from the total score and critical failures.\n\nI wrote 38 unit tests.\n\nThe external LLM is fully mocked, so tests are deterministic and run in under 2 seconds.\n\nThey cover schema boundaries, malformed responses, connection failures, and scoring logic.\n\nThe biggest wins:\n\nFirst-pass review time dropped from tens of minutes to seconds\n\nOutput became consistent and machine-readable\n\nCritical brand violations are much harder to miss\n\nThe hardest part wasn’t the LLM call — it was designing the contracts and failure modes so the system stays reliable when the model behaves badly.\n\nMultimodal support (evaluate the actual layouts, not only text)\n\nModel benchmarks (accuracy vs speed)\n\nREST API for integration into existing workflows\n\nEvaluation history and campaign analytics\n\nIf you’re building production LLM pipelines, I’d love to hear how you handle structured output and hallucination control.\n\nHow do you currently deal with unreliable LLM responses in your projects?", "url": "https://wpnews.pro/news/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict", "canonical_source": "https://dev.to/strelok25dev/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict-pydantic-contracts-376j", "published_at": "2026-08-25 14:02:13+00:00", "updated_at": "2026-08-25 14:15:07.293770+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "developer-tools"], "entities": ["CreativeAudit", "Ollama", "Pydantic", "Streamlit", "qwen2.5:7b", "qwen2.5:14b", "llama3.2"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict", "markdown": "https://wpnews.pro/news/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict.md", "text": "https://wpnews.pro/news/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-reliable-llm-pipeline-for-ad-creative-evaluation-with-strict.jsonld"}}