A plain chat with an LLM falls apart for this task: the output jumps formats, the model forgets constraints, you get free‑form text, and client data leaks to the cloud. For any real workflow that’s a deal‑breaker.
Design goals I forced myself to meet
-
Strict output contracts – the schema is a hard wall; if the LLM doesn’t obey it, the call fails, no fake scores appear.
-
Local inference – everything runs on‑premise with Ollama, so no data ever leaves the machine.
-
Prompts as code – Jinja2 files live in a versioned folder, making them editable without touching the app.
-
Binary compliance for any brand‑book violation; a 0 or 10 score removes subjectivity on the highest‑risk rule.
-
Human‑in‑the‑loop mode that first extracts structured data from messy input, shows it to the user for correction, then runs the evaluation.
Architecture at a glance
Brief + Creatives → Jinja2 Prompt → Local LLM (Ollama)
↓
Pydantic Validation
↓
Score + Verdict + Feedback
Key components:
app/schemas.py
– strict Pydantic v2 contracts that define the exact JSON shape.prompts/*.j2
– versioned prompt templates, treated like source code.app/main.py
– orchestrates prompt assembly, LLM call, validation, scoring.demo/streamlit_app.py
– thin UI layer for quick testing.
Engineering decisions that matter
Pydantic as the contract– the model is inherently nondeterministic; treating the schema as absolute guarantees that malformed responses never masquerade as scores.Binary 0/10 for critical rules– a creative either fully complies with the brand book or it doesn’t; no “slightly violated” ambiguity.** Jinja2 prompts**– because they’re separate files, you can A/B test wording, review diffs, and keep the app code clean.** Local Ollama inference**– swapping between qwen2.5:7b, qwen2.5:14b or llama3.2 is a one‑line config change, keeping deployment lightweight.** Smart input mode**– managers often paste chat fragments; an initial LLM step normalizes that into structured JSON, which the user can edit before the final audit runs.
Scoring model
Each creative gets three numeric scores: brand_alignment (1‑10), constraint_compliance (0‑10), and message_clarity (1‑10). The total combines them with weights:
total = brand × 0.4 + compliance × 0.3 + clarity × 0.3
Verdicts are then derived from the total and any critical failures flagged by the binary check.
Testing strategy
I wrote 38 deterministic unit tests, mocking the external LLM so they run in under two seconds. Tests cover schema edge cases, malformed responses, connection hiccups, and the scoring logic, giving a reliable complete guide for anyone who wants to replicate the pipeline from scratch.
The whole system lives in a practical tutorial‑style repo, with clear step‑by‑step instructions for deployment, making it beginner‑friendly while still being a deep dive into real‑world LLM agent workflow for prompt engineering.
You are an ad compliance analyst. Given a creative description and a brand brief, output ONLY a JSON object that matches the schema below. Do not add any extra text.
{
"verdict": "PASS|NEEDS_REVISION|FAIL",
"brand_alignment": 1-10,
"constraint_compliance": 0-10,
"message_clarity": 1-10,
"feedback": "short sentence explaining why"
}
CreativeAudit shows that a reliable LLM pipeline can replace hours of manual review with a fast, deterministic process that respects data privacy and enforces strict contracts. If you’re building any AI workflow that needs guaranteed output, this step‑by‑step setup is worth a look.
Next Stop trusting raw benchmark scores without looking at the harness →