5‑Second Ad Review with Strict Pydantic Contracts

A developer built CreativeAudit, a local LLM pipeline using Ollama and Pydantic v2 to enforce strict JSON output contracts for 5-second ad reviews, scoring creatives on brand alignment, constraint compliance, and message clarity with a weighted total formula. The system runs fully on-premise to prevent client data leaks, uses Jinja2 prompt templates as versioned code, and includes 38 deterministic unit tests. It aims to replace manual ad review with a fast, deterministic process that respects data privacy.

5‑Second Ad Review with Strict Pydantic Contracts A plain chat with an LLM falls apart for this task: the output jumps formats, the model forgets constraints, you get free‑form text, and client data leaks to the cloud. For any real workflow that’s a deal‑breaker. Design goals I forced myself to meet 1. Strict output contracts – the schema is a hard wall; if the LLM doesn’t obey it, the call fails, no fake scores appear. 2. Local inference – everything runs on‑premise with Ollama, so no data ever leaves the machine. 3. Prompts as code – Jinja2 files live in a versioned folder, making them editable without touching the app. 4. Binary compliance for any brand‑book violation; a 0 or 10 score removes subjectivity on the highest‑risk rule. 5. Human‑in‑the‑loop mode that first extracts structured data from messy input, shows it to the user for correction, then runs the evaluation. Architecture at a glance Brief + Creatives → Jinja2 Prompt → Local LLM Ollama ↓ Pydantic Validation ↓ Score + Verdict + Feedback Key components: app/schemas.py – strict Pydantic v2 contracts that define the exact JSON shape. prompts/ .j2 – versioned prompt templates, treated like source code. app/main.py – orchestrates prompt assembly, LLM call, validation, scoring. demo/streamlit app.py – thin UI layer for quick testing. Engineering decisions that matter Pydantic as the contract – the model is inherently nondeterministic; treating the schema as absolute guarantees that malformed responses never masquerade as scores. Binary 0/10 for critical rules – a creative either fully complies with the brand book or it doesn’t; no “slightly violated” ambiguity. Jinja2 prompts – because they’re separate files, you can A/B test wording, review diffs, and keep the app code clean. Local Ollama inference – swapping between qwen2.5:7b, qwen2.5:14b or llama3.2 is a one‑line config change, keeping deployment lightweight. Smart input mode – managers often paste chat fragments; an initial LLM step normalizes that into structured JSON, which the user can edit before the final audit runs. Scoring model Each creative gets three numeric scores: brand alignment 1‑10 , constraint compliance 0‑10 , and message clarity 1‑10 . The total combines them with weights: total = brand × 0.4 + compliance × 0.3 + clarity × 0.3 Verdicts are then derived from the total and any critical failures flagged by the binary check. Testing strategy I wrote 38 deterministic unit tests, mocking the external LLM so they run in under two seconds. Tests cover schema edge cases, malformed responses, connection hiccups, and the scoring logic, giving a reliable complete guide for anyone who wants to replicate the pipeline from scratch. The whole system lives in a practical tutorial ‑style repo, with clear step‑by‑step instructions for deployment, making it beginner‑friendly while still being a deep dive into real‑world LLM agent workflow for prompt engineering. You are an ad compliance analyst. Given a creative description and a brand brief, output ONLY a JSON object that matches the schema below. Do not add any extra text. { "verdict": "PASS|NEEDS REVISION|FAIL", "brand alignment": 1-10, "constraint compliance": 0-10, "message clarity": 1-10, "feedback": "short sentence explaining why" } CreativeAudit shows that a reliable LLM pipeline can replace hours of manual review with a fast, deterministic process that respects data privacy and enforces strict contracts. If you’re building any AI workflow that needs guaranteed output, this step‑by‑step setup is worth a look. Next Stop trusting raw benchmark scores without looking at the harness → /en/threads/7530/