This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.
A workflow I ran every morning pulled new records from a provider API, processed them, and wrote a summary. The summary looked fine. The counts looked plausible. Then a customer asked why an order from last Tuesday was missing, and I found it sitting on page three of the provider's response — a page my workflow had never requested.
Nothing had errored. The HTTP node returned 200, the loop ran, the summary was written. The workflow had processed exactly one page and reported success, because nobody had told it that one page was not the whole list.
A paginated API does not fail when you stop early. It just hands you a partial list that looks complete, and a workflow that treats the first page as the answer will silently drop everything after it.
Every provider paginates differently, and the defaults are there to trip you up. Some cap a page at 20 records, some at 100. Some return a total that counts something other than what you asked for. Some return no total at all — just a page and a cursor.
So the first thing a list workflow must do is read the pagination contract before it reads any record: what is the page size, what field carries the next cursor or offset, and what does the last page actually look like? An empty page is not always the end — some APIs return an empty page in the middle when a filter shifts — and a page that returns fewer records than the page size is not always the end either, if the provider pads or rounds.
If your workflow has no explicit answer to "how do I know I am done?", it does not have pagination. It has a first page with confidence.
The loop itself should be driven by the provider's own continuation signal, not by a number you guessed. Fetch a page, record its cursor, fetch the next page with that cursor, and stop only when the provider says there is no next page — a null cursor, an absent next link, an offset that stops advancing.
Do not loop "five times and hope". Page counts change as data grows: the list that fit in three pages last month needs six this month, and a fixed loop will start dropping records the week the business grows. And do not loop "until the page comes back empty" unless the provider documents that as its end signal — otherwise a transient empty page ends your run early and you will not notice until someone counts.
The loop condition belongs in one place, written against the provider's documented end signal, and tested with a list that spans at least three pages. A two-page test cannot catch an off-by-one in the cursor handoff, because the second page is where most cursor bugs hide.
Once you are actually fetching every page, a second problem appears: pages are not always disjoint. A record created mid-listing can shift offsets, so offset pagination can return the same record twice or skip one entirely. A cursor replayed after a transient error can return a page you already processed.
So the merge step needs its own dedup, keyed on the record's natural id — the order id, the event id, the record id the provider itself assigns. Collect ids as you go, skip any record whose id you have already seen, and never trust "I already processed page two" as a substitute for "I already processed this record". Page numbers are positions, not identities, and positions move.
This is the same discipline as everywhere else in this series: exactly-once is a property of the receiving system. The provider promises you pages, not a clean set. Your job is to turn pages into a set.
A long list is a long run, and long runs get interrupted. If your workflow crashes on page eight of twelve and restarts from page one, every side effect on pages one to seven runs twice — unless those side effects are idempotent, which is a separate promise you have to keep.
So record progress as you go: the last cursor fetched, the last record processed, the count so far. On restart, resume from the recorded cursor, not from the beginning. A listing that cannot resume is a listing that will either duplicate or drop records the first time it is interrupted, and in production, interruption is not an edge case — it is Tuesday.
The checkpoint also gives you something to alert on. A listing that stops advancing — same cursor three runs in a row, page count frozen while the provider's total grows — is a failure, even though no node turned red. Stuck pagination looks exactly like a small, quiet day.
Finally, put a ceiling on the loop. A provider bug, a filter that silently stopped filtering, or a cursor that cycles can turn "fetch all pages" into an infinite loop that burns API quota and execution time all night. A sane cap — a maximum page count well above any legitimate list, with an alert when it is hit — turns an infinite loop into a visible incident.
The cap is not a substitute for the end signal. It is the guard that fires when the end signal never comes, and hitting it should page you, not silently succeed with a partial list. A partial list reported as complete is the original bug; a partial list reported as truncated is at least honest.
It is unglamorous work, and no part of it makes an API faster. But it is the difference between a workflow that processes records and a workflow that processes the first page and calls it a day — and in production, that difference shows up as a missing order, on the day someone finally counts.
I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.