{"slug": "we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this", "title": "We Ran an Email Deliverability Bake-Off Across 3 Vendors and 500,000+ Emails This Week. Our AI Agent Ran All of It.", "summary": "SaaStr's AI agent, 10K, ran an email deliverability bake-off across three vendors and more than 500,000 emails, finding a challenger vendor delivered 50.5% unique opens versus 21.0% for the incumbent in a matched split of about 2,300 recipients per side on the AI Day campaign. 10K also found the incumbent's flow reported \"Completed\" on a Daily send while step-level records showed 80,708 attempts and 80,708 errors, meaning zero emails went out, and four single-variable diagnostic sends all failed the same way, pointing to the vendor's transport layer. 10K scored the vendors on 7 criteria and recommended moving core newsletter sends to the winning vendor, ramping over days.", "body_md": "We’ve run vendor bake-offs at SaaStr for as long as we’ve bought software. Everyone has.\n\nThey used to look about the same every time. A few demos. A two-week trial. Someone on the team sends a few hundred test emails, checks the dashboards, and builds a spreadsheet. The decision then mostly comes down to which rep was most responsive and what the team already knew how to use.\n\nA few hundred sends, judged on each vendor’s own reporting, by people with other jobs, doesn’t tell you much.\n\nThis week we compared email deliverability across three vendors and more than 500,000 emails. 10K, our AI VP of Marketing, ran the whole thing. It caught every issue, ran the multivariate diagnostics, and wrote a recommendation clear enough that we acted on it the same day.\n\n## What a Bake-Off Used to Cost Us\n\nA proper deliverability comparison takes a lot of people time:\n\n- Building matched audience groups so the comparison is fair\n- Sending identical content through each vendor at the same time\n- Pulling delivery, open, click, bounce, complaint and unsubscribe data out of three dashboards that each define those terms differently\n- Checking whether “delivered” actually means delivered\n- Filtering security-scanner clicks out of human clicks\n- Repeating all of it at enough volume to trust the result\n\nAt our size, no human team was going to do that across 500,000 emails. We’d have done a few hundred test sends, picked a winner, and lived with it for three years.\n\n## Same Campaign, Matched Groups: 50.5% Opens vs. 21.0%\n\nThe cleanest comparison was a split 10K set up on our AI Day campaign: two matched groups of about 2,300 people each, identical content, sent the same morning. One went through our incumbent sender and the other through a challenger.\n\n| Metric | Challenger | Incumbent | \n|---|---|---|\n| Unique opens | 1,168 (50.5%) | 485 (21.0%) | \n| Unique clickers | 281 (12.2%) | 96 (4.2%) | \n| Recorded deliveries | 2,311 | 2,308 | \n\nThe challenger got 2.4x the opens and 2.9x the clicks. With the same audience, email and send time, the gap comes from how each vendor delivered the email.\n\n10K listed the limits before we could get excited. It’s one campaign at about 2,300 per side. Opens are noisy. The raw click counts include security scanners. Clicks moved in the same direction as opens, which made the gap hard to explain away.\n\nFor our sponsors, this mattered a lot. On days when everything was working, every newsletter send through the incumbent had been reaching them with less than half the engagement.\n\n## The Incumbent Reported “Completed” on 80,708 Failed Sends\n\nOn one Daily send, the incumbent’s flow showed “Completed.” 10K queried the step-level records underneath that label and found 80,708 attempts and 80,708 errors. Zero emails went out. The vendor’s status page showed green the whole time.\n\n10K then ran four diagnostic sends, changing one variable each time:\n\n- A different sender\n- Tracking turned off\n- Different subscription settings\n- Minimal plain-text content\n\nAll four failed the same way. That ruled out our content, configuration and list, and put the failure on the vendor’s transport layer.\n\nIn a two-week trial, someone on our team would have seen “Completed” on the dashboard and moved on.\n\n## 10K Scored the Vendors on 7 Criteria\n\n10K wouldn’t pick a winner on opens. Its scorecard:\n\n1. Completed delivery, confirmed at the step level\n2. Inbox placement\n3. Spam complaints\n4. Unsubscribes\n5. Human clicks, with scanner traffic filtered out\n6. Sponsor reporting: whether we can prove to a sponsor that their placement went out and got engagement\n7. Operator effort: how much human time each vendor takes in consent reconciliation, audience prep, schedule checks and reporting\n\nWe had never put operator effort in a bake-off spreadsheet. It turned out to be one of the biggest differences between the vendors.\n\n## The Recommendation Fit on One Page\n\n10K’s plan:\n\n- **Move core newsletter sends to the winning vendor now, ramping over days.** Roughly 20K, then 40K, then full volume, stepping up only if bounces, complaints and unsubscribes hold.\n- **Confirm unsubscribe sync in both directions before scaling.** Unsubscribes from the new sender write back to our CRM consent records, and CRM opt-outs suppress in the new sender. A gap here turns a sending switch into a compliance problem.\n- **Map every other workflow that depends on the incumbent** before canceling anything.\n- **Report to sponsors from the new provider’s own logs.**\n- **Set acceptance criteria before the incumbent sends anything sponsor-backed again:** a root cause, an accounting of every outstanding attempt, and an operator-only test confirmed at the step level.\n- **Don’t renew.**\n\nI wanted to move all 80,000+ subscribers over immediately. 10K pushed back: a clean 5,000-person test doesn’t show a sender can carry the full list, and moving everyone at once would put deliverability at risk. We went with the ramp.\n\n## A Quarter-Long Project Took a Few Days\n\nEvery technique here is standard practice for a good deliverability team: matched splits, step-level verification, isolating one variable at a time, scoring on more than opens.\n\nWe have three humans at SaaStr and no deliverability team. A test at this depth would have meant hiring a consultant for a quarter. 10K ran it in a few days alongside its other work.\n\nThat changes how we buy software:\n\n- **We can keep vendors in a running comparison** instead of re-evaluating every three years.\n- **We test at production volume.** 500,000 emails surfaces problems a few hundred test sends never will.\n- **We check vendor reporting against the underlying records.** Status labels, dashboards and status pages all got cross-checked.\n- **Switching costs less.** The agent that ran the test can stand up the alternative and run the migration ramp.\n\n## Every Core Vendor in Our Stack Is Now in a Running Bake-Off\n\nIf you sell B2B software, your customers’ agents can now run the evaluation your sales team used to steer. They’ll test at real volume, query your step-level data, and score you on how much of their team’s time you take. Incumbency protects you far less when a challenger can be tested against you every week.\n\nIf you’re buying, have your agent run your next bake-off, and give it the scorecard up front: completed delivery confirmed at the source, human engagement, and operator effort.\n\nWe’ve always done vendor bake-offs at SaaStr. This week was the first one where I trusted the result completely.", "url": "https://wpnews.pro/news/we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this", "canonical_source": "https://www.saastr.com/we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this-week-our-ai-agent-ran-all-of-it/", "published_at": "2026-11-05 15:00:24+00:00", "updated_at": "2026-11-05 15:52:16.663603+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools"], "entities": ["SaaStr", "10K"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this", "markdown": "https://wpnews.pro/news/we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this.md", "text": "https://wpnews.pro/news/we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this.txt", "jsonld": "https://wpnews.pro/news/we-ran-an-email-deliverability-bake-off-across-3-vendors-and-500000-emails-this.jsonld"}}