Pages of Our Latency

An engineer who has encountered the same pagination bug at three of their last four companies describes a Friday-afternoon incident in which data quietly went missing for several of the largest ad-tech clients, tracing the cause to page-based pagination whose count query scales linearly with table size. The engineer argues that cursor-based pagination should be the default at scale, a point that matters more now that AI agents consume APIs and require stable, deterministic cursors. The fix, they write, is growth-based observability to catch the degradation before it becomes an outage.

The Missing Adgroups There are many categories of scaling failures, and chief among them are those that begin in unnoticeable silence. I've encountered the same pagination bug at three of my last four companies. I've deduced the underlying reason to be nearly identical in all cases. If you're working at scale, the default pagination method should be cursor based. In the age of A.I., this rings more true now than ever. Here's the incident that made the cost concrete, the architectural reason it happens, why it matters more now that agents consume APIs too, and how to catch it before it becomes an outage. 🏗️ What this post covers: What happened: a Friday-afternoon incident where data quietly went missing for a few of our largest clients Why it happens: page-based pagination's count query scales linearly with table size Why it's the default anyway: ORMs optimize for zero-to-one speed, not structural assumptions Why it matters more now: AI agents consuming APIs want stable, deterministic cursors How to catch it early: growth-based observability, before it becomes an outage I originally wrote this finding as a journal entry some 4+ years ago. The Buyer Cloud Beeswax at the time of the original post models the concept of Adgroups. An Adgroup contains the necessary domain structure to transact on bid requests downstream. Almost every ad-tech company has some concept of this object, and it typically contains entities such as an Advertiser or Brand, Line Items, and references to Creatives. For live bidding, this data must be synced from the UI/API to serving infrastructure in a reasonable amount of time. The faster the better — seconds are great, minutes are acceptable, hours can result in disaster and costly Make Goods. Performance really matters. What works for clients with a few hundred line items doesn't work for those with millions. The default pagination strategy in most ORMs is usually page based, because the strategy requires no assumptions about the underlying table structure. Most frameworks are designed to take an app from zero to one quickly. An application can unevenly grow out of this strategy over time. That's where the story begins. On a Friday afternoon, exactly when things should have been winding down, a DM from customer support landed in my inbox. That late in the day, it could only mean an unspoken problem had been brewing for a day or two. I could taste the hysteria through Slack. The Problem: Data was missing for some but not all clients. I tend to look at problems from multiple angles. This particular issue was known to recur randomly a word used when engineering hasn't quite figured out the triggering conditions . Not only was it random , but the frequency was per client and could potentially auto-resolve. It appeared in various forms once or twice a month, and because clients used the platform in behaviorally different ways, the escalation priority could be assigned different values e.g. P1 to P4 — more on that in a future post . Even though I'd go on to help resolve hundreds of escalations over the years, that first pupil-widening surge — the sympathetic nervous system snapping into action — was an unavoidable part of the job. A pattern emerged after a thorough review of endpoint latency in Datadog and a cursory review of the adgroup data. Degradation had crept up over the course of the month, then spiked sharply — but only for a few of our largest clients. After years of experience, you go through the usual suspects: Was there a recent change? No. Did the client behavior change? Did they add a substantial amount of data recently? Nope. The affected clients hadn't grown suddenly; they had simply accumulated enough data for an old architectural assumption to become expensive. Conversing with customer support and one of the only stateside engineers available our team was split between the U.S. and Europe , I was led by my experience to our pagination strategy. AWS's RDS dashboard includes a quick panel for long-running queries. These are often sizable queries generated by Django's ORM, ranging from 200 to 1,000+ lines. Re-running them sometimes produced result sets of 15 million records or more, but that wasn't the real problem. The culprit was the count query required to compute the page index. Page-based iteration scales linearly while cursor based is done in constant time. Taking a step back, the larger context is that page-based strategies are rooted in an older web experience think Web 2.0 : a human user searching for specific records within a relatively small dataset. It predates advanced UI components and libraries that support sophisticated search. Parsing large datasets was typically left to ETL processes — transferring gigabytes or terabytes of data over HTTP wasn't common. However, in the age of A.I. — where agents may be the dominant consumers of APIs — cursor-based pagination will likely become the norm. If you're modernizing your API for agent consumption, this is an area worth prioritizing. One advantage is stable cursors, which help ensure records aren't skipped or duplicated. Deterministic ordering also lets agents resume after being paused, and checkpointing allows a cursor to be saved and reused later. While I haven't seen "Spot Agents" akin to EC2 spot instances , it's probably only a matter of time before Anthropic or OpenAI pitch the idea to enterprise organizations as a creative way to cut costs. The default pagination strategy is on the cusp of being a technical atavism. Another contributor to this kind of failure is observability. When it comes to scaling, this nonfunctional requirement seldom gets the attention it deserves. As far as I know, there are no native, out-of-the-box database solutions for monitoring table size. However, both MySQL and Postgres provide exporters. Pair that with Prometheus to store the data in a time-series database and Grafana to query it, and you get a very useful alert. delta mysql info schema table size{table="users"} 7d // or to see the percentage increase over the last week delta mysql info schema table size{table="users"} 7d / mysql info schema table size{table="users"} offset 7d I've always said that Datadog pays for itself the first time it helps you avert an outage. If you have access to it, leveraging Database Monitoring DBM is a no-brainer. DBM became available in 2021/2022 — but with so many features in the product, it may easily go unnoticed. Once enabled, setting up Change Alerts to monitor the growth rate is the surest path to reducing an outage of this nature. While observability is key to identifying potential future escalations, several non-functional capabilities also benefit from first assessing whether your platform should adopt a cursor-based strategy. Taking a systems-thinking approach, performance improves through constant-time lookups. As a result, reliability increases because another threat vector has been reduced, and availability improves as well. If your database is performance-throttled, dependent systems will likely start returning 5XX HTTP errors. The user experience will look broken. That Zendesk ticket will get created, and your customer support team will DM you the same way this post begins. Cursor-based pagination is one of the rare scaling choices that improves performance, reliability, and availability at the same time. Don't wait for missing records — or a weekend escalation — to reveal an unsafe default. Find your fastest-growing tables and move them to cursor-based pagination before scale makes the decision for you. Originally published at Skyscrapers On Sand https://notion.skyscrapersonsand.com/Pages-of-Our-Latency-b8857e14fb2c4367b6d7832775812dc9 .