Every quarter, we benchmark every major image generation model against real production workloads from our platform. Not synthetic tests, actual jobs from customers generating AI headshots at scale.
This quarter, we tested 8 models across 12,000 inference jobs, scoring each on quality (FID, CLIP, human eval), cost per image, and p95 latency. Here’s the full breakdown.
Most model comparisons use academic datasets, ImageNet, LAION, curated prompt sets. That’s useful for research, but it tells you nothing about how a model performs on your workload.
At Runflow, we route tens of thousands of real inference jobs per day. We see exactly how models perform on corporate headshots, e-commerce product photos, and creative portraits, the actual use cases customers care about.
Our Sentinel evaluation engine scores every output automatically across three dimensions:
We tested the following models, all running on our multi-cloud orchestration layer to normalize for infrastructure differences:
| Model | Version | Type | Provider |
|---|---|---|---|
| Flux.2 [dev] | v2.0.1 | Open Source | Self-hosted |
| Flux.2 [schnell] | v2.0.1 | Open Source | Self-hosted |
| SDXL Lightning | 4-step | Open Source | Self-hosted | | SDXL Turbo | 1-step | Open Source | Self-hosted | | Proprietary A | - | Closed Source | API | | Proprietary B | - | Closed Source | API |
The composite quality score combines FID (40%), CLIP alignment (30%), and human evaluation (30%). All scores are normalized to a 0–100 scale.
The headline: Flux.2 [dev] scored 95, matching or exceeding proprietary models across all three evaluation dimensions. For the first time in our benchmarks, an open-source model leads the portrait generation category outright.
Cost calculations include GPU compute, orchestration overhead, and our platform fee. All models were run on equivalent hardware (A100 80GB) through our multi-cloud orchestration layer.
Latency was measured end-to-end from API request to image delivery, including model (cold start) and network transfer. All measurements are p95 across the full 12K job dataset.
All benchmark results are reproducible. We publish our evaluation pipeline, reference datasets, and scoring rubrics in our open benchmark repository. If you find discrepancies, we want to know—open an issue or reach out directly.
Models labeled “Proprietary A” and “Proprietary B” are anonymized per our testing agreements. We’ll name them explicitly once we have permission from the providers.
Q2 benchmarks will expand to include video generation models (Wan2.6, Kling 2.1, Seedance) and our new virtual try-on pipeline. We’re also adding latency-under-load testing to simulate real production traffic patterns.
Want to run these benchmarks on your own workload? Talk to our team — we’ll set up a custom evaluation against your production data.
Test
Originally published on Runflow.