Art directors routinely spend 20–40 minutes per creative just checking brand guidelines, mandatory elements, and forbidden techniques.
I wanted to automate the first-pass review — without turning it into another unreliable ChatGPT wrapper.
So I built CreativeAudit: a production-oriented pipeline that evaluates ad creatives against a brief and returns a clear PASS / NEEDS_REVISION / FAIL
verdict.
Automated ad creative evaluation against a brief using a local LLM.
CreativeAudit takes a brief + creatives, builds a precise prompt, gets a structured evaluation from a language model, strictly validates the response, and returns a clear verdict (PASS / NEEDS_REVISION / FAIL), weighted score, and explanation.
This is not "just another AI chat". It is a production-oriented first-pass review pipeline: the machine catches routine and critical violations so humans only need to look at the borderline cases.
Campaign summary metrics, creative cards with verdict, detailed scores and model explanation. Smart-input mode converts free-form text into structured JSON with human review before evaluation starts.
Manual first-pass creative review is slow, expensive and inconsistent:
A plain chat with an LLM has several fatal flaws for this use case:
For any real workflow this is unacceptable.
I set a few hard requirements:
Brief + Creatives → Jinja2 Prompt → Local LLM (Ollama)
↓
Pydantic Validation
↓
Score + Verdict + Feedback
Architecture diagram showing the flow from Brief and Creatives through Jinja2 Prompt and Ollama, followed by Pydantic Validation, resulting in Score, Verdict, and Feedback
Key components:
app/schemas.py — strict Pydantic v2 contracts
prompts/*.j2 — prompts treated as versioned code
app/main.py — orchestration (prompt assembly, LLM call, validation, scoring)
demo/streamlit_app.py — thin UI layer
The business logic is completely separated from the interface. You can call it from Streamlit, CLI, or any other service.
The model is non-deterministic.
I treat the Pydantic schema as a hard boundary: if the response doesn’t match the schema, it is rejected as an error — it never becomes a fake score.
For brand-book violations I use a binary 0 or 10 score.
There is no “slightly violated”. This removes a lot of model subjectivity on the highest-risk criterion.
Prompts live in separate .j2 files.
This makes them versionable, reviewable, and easy to A/B test without touching application code.
Everything runs locally.
Switching between qwen2.5:7b, qwen2.5:14b or llama3.2 is a single config change.
Managers rarely provide clean JSON.
They paste chat fragments and rough descriptions.
So the pipeline first uses an LLM to extract structured data, shows the result to the user for correction, and only then runs the evaluation.
Each creative receives three scores:
brand_alignment (1–10)
constraint_compliance (0 or 10)
message_clarity (1–10)
total = brand × 0.4 + compliance × 0.3 + clarity × 0.3
The verdict is derived from the total score and critical failures.
I wrote 38 unit tests.
The external LLM is fully mocked, so tests are deterministic and run in under 2 seconds.
They cover schema boundaries, malformed responses, connection failures, and scoring logic.
The biggest wins:
First-pass review time dropped from tens of minutes to seconds
Output became consistent and machine-readable
Critical brand violations are much harder to miss
The hardest part wasn’t the LLM call — it was designing the contracts and failure modes so the system stays reliable when the model behaves badly.
Multimodal support (evaluate the actual layouts, not only text)
Model benchmarks (accuracy vs speed)
REST API for integration into existing workflows
Evaluation history and campaign analytics
If you’re building production LLM pipelines, I’d love to hear how you handle structured output and hallucination control.
How do you currently deal with unreliable LLM responses in your projects?