Almost every real n8n workflow eventually hits the same wall: an API that returns its data 100 rows at a time, throttles you after 60 requests a minute, and returns a 429 Too Many Requests exactly when your batch job matters most. If you have ever watched a workflow grab “page 1” of a CRM export and silently stop, this guide is for you.
This is a hands-on walkthrough of three patterns that turn fragile API calls into production-grade ingestion in n8n: cursor and offset pagination, batching with Loop Over Items, and rate-limit backoff that respects Retry-After headers. Each section includes the actual node configuration and JSON you can paste into your own canvas. We assume you already know what an HTTP request, a header, and JSON are — but not necessarily how n8n handles items, expressions, and looping internally.
Why naive HTTP nodes break at scale
A single HTTP Request node pointed at /api/contacts works perfectly in a demo with 40 records. In production with 40,000 records it fails in three predictable ways. First, the API paginates and you only ever receive the first page. Second, you fan out one request per item and the provider rate-limits you halfway through. Third, a transient 503 kills the whole execution because there is no retry.
n8n gives you native tools for all three, but they are easy to misconfigure. The HTTP Request node has a built-in Pagination tab, the Loop Over Items node (formerly “Split In Batches”) controls concurrency, and expressions let you read response headers to time your backoff. Wire them together correctly and a 40,000-row pull becomes boring — which, for an ops engineer, is the goal.
Pattern 1: Pagination with the HTTP Request node
n8n’s HTTP Request node (v4.2+) has a dedicated Pagination section under Options. You pick a mode and n8n keeps calling the endpoint until your stop condition is met, merging every page into the output items automatically. The two modes you will use most are offset/page-based and cursor-based.
Offset and page-based pagination
For an API that accepts ?page=N&per_page=100, configure the Pagination options like this:
{
"options": {
"pagination": {
"paginationMode": "updateAParameterInEachRequest",
"parameters": {
"parameters": [
{ "type": "qs", "name": "page", "value": "={{ $pageCount + 1 }}" },
{ "type": "qs", "name": "per_page", "value": "100" }
]
},
"paginationCompleteWhen": "responseIsEmpty",
"limitPagesFetched": true,
"maxRequests": 200
}
}
}The $pageCount built-in starts at 0, so $pageCount + 1 requests page 1, then 2, and so on. Setting paginationCompleteWhen to responseIsEmpty stops the loop the moment the API returns an empty array. The maxRequests guard is the single most important field on this tab: it is your circuit breaker against an off-by-one stop condition that would otherwise loop forever and burn your API quota.
Cursor-based pagination
Modern APIs — Stripe, Notion, the Google Search Console API, and most GraphQL endpoints — return a cursor or next_page_token instead of a page number. Here you switch the completion rule to “other” and read the cursor straight out of the previous response:
{
"pagination": {
"paginationMode": "updateAParameterInEachRequest",
"parameters": {
"parameters": [
{ "type": "qs", "name": "starting_after",
"value": "={{ $response.body.data.last().id }}" }
]
},
"paginationCompleteWhen": "other",
"completeExpression": "={{ $response.body.has_more === false }}"
}
}The expression {{ $response.body.data.last().id }} reads the last record’s ID from the page you just received and feeds it back as the cursor for the next call. The loop ends when the API tells you has_more is false — no guessing, no empty-page heuristics. This is the most reliable pagination style and worth migrating to whenever a provider offers it.
Pattern 2: Batching writes with Loop Over Items
Pagination solves reading. The opposite problem is writing thousands of items into a downstream system that only tolerates a few requests per second. Pointing a raw HTTP node at 5,000 items makes n8n fire 5,000 near-simultaneous requests — a reliable way to get blocked.
The Loop Over Items node throttles this. Set a batch size and n8n processes the items in controlled chunks, routing each chunk through the loop body before fetching the next one. A batch size of 10 with a short wait between iterations keeps you comfortably under most rate limits:
Loop Over Items
Batch Size: 10
(loop body)
-> HTTP Request (POST /api/records)
-> Wait (1 second)
-> [back to Loop Over Items "done"? no -> next batch]Two details matter. First, the Wait node belongs inside the loop, after the HTTP call, so the pause happens between batches rather than once at the end. Second, enable Retry On Fail on the HTTP node itself (Settings tab → “Retry On Fail”, 3 attempts, 5000 ms between tries). That handles the occasional blip without you building anything. For failures that survive all retries, route the error output to a dead-letter store instead of aborting — the same philosophy we use when building a global error workflow with retries and a dead-letter queue.
Pattern 3: Honoring Retry-After with adaptive backoff
Fixed one-second waits are fine until a provider tightens its limits during peak hours. The professional move is to let the API tell you how long to wait. When a server returns 429, it usually includes a Retry-After header (in seconds). Read it and pause exactly that long.
Build this with the HTTP node configured to not throw on error responses (Settings → “Never Error”), followed by an IF node and a Wait node driven by an expression:
// IF node condition
{{ $json.statusCode === 429 }}
// Wait node — "Wait Amount" expression (seconds)
{{ Number($json.headers['retry-after'] || 2)
* Math.pow(2, $runIndex) }}This combines server guidance with exponential backoff: the base wait comes from Retry-After (defaulting to 2 seconds if the header is absent), multiplied by 2 ^ runIndex so each successive retry waits longer — 2s, 4s, 8s, 16s. After the Wait, loop the item back into the HTTP node. Cap the retries with a counter in workflow static data so a permanently throttled endpoint eventually surrenders to your error branch rather than spinning forever.
Putting it together: a measured example
We tested these patterns against a paginated public dataset API that caps clients at 60 requests per minute and returns 100 records per page. The target: ingest 42,000 records into Postgres.
The naive single-node version pulled exactly 100 records (one page) and reported “success” — the worst kind of failure, because nothing errored. Adding cursor pagination with maxRequests: 500 retrieved all 420 pages. Layering Loop Over Items (batch size 8) plus the Retry-After backoff brought the run to a steady state of roughly 55 requests per minute with zero 429s over a 13-minute execution. Two transient 503s were absorbed by Retry On Fail without operator involvement. The same workflow now runs nightly on a schedule and has not paged anyone in three weeks.
The headline lesson: throughput is not about going faster, it is about going at exactly the speed the API allows and never dropping data when it briefly says no. For high-volume jobs, run this on a self-hosted instance in queue mode with Redis and Postgres so long-running paginated executions do not block your main process.
Keep workflows modular
Pagination, batching, and backoff logic tend to be reused across every integration you build. Rather than copy-pasting the same five nodes into each workflow, extract them into a reusable child workflow and call it with the Execute Workflow node — the approach we cover in depth in our guide to modular sub-workflows. You pass in the endpoint and credentials, and the child returns clean, fully-paginated items. When a provider changes its pagination scheme, you fix it in one place.
Takeaways
Use the HTTP node’s native Pagination tab with a hard maxRequests ceiling; prefer cursor pagination when the API offers it. Throttle writes with Loop Over Items and an in-loop Wait, and turn on Retry On Fail for free resilience. When you hit 429, read Retry-After and apply exponential backoff instead of a fixed sleep. These three patterns cover the overwhelming majority of API ingestion problems you will meet in n8n — whether you are pulling from a CRM, the GSC API, an LLM provider like the OpenAI or Anthropic APIs, or a data platform such as Bright Data.
Found this useful? Bookmark n8nfuel and check back each week — we publish one working, copy-pasteable n8n recipe every morning, complete with real JSON and measured results. If reliability is your concern this week, read our companion piece on building a resilient workflow with retries and a dead-letter queue next.
Frequently asked questions
What is the difference between Loop Over Items and the HTTP node’s pagination?
Pagination is about reading all pages from one endpoint in a single node, automatically. Loop Over Items is about controlling concurrency when you send many requests — typically writes — so you stay under a rate limit. They solve opposite halves of the same scaling problem and are often used together.
How do I stop n8n pagination from looping forever?
Always set limitPagesFetched with a sane maxRequests value as a hard ceiling, and pair it with a precise completion rule — responseIsEmpty for offset APIs or a completeExpression like {{ $response.body.has_more === false }} for cursor APIs. The ceiling protects you if the stop condition is ever wrong.
Does n8n respect the Retry-After header automatically?
No. The built-in “Retry On Fail” uses a fixed interval you configure. To honor Retry-After you read the header with an expression — {{ $json.headers['retry-after'] }} — and drive a Wait node with it, optionally multiplying by 2 ^ runIndex for exponential backoff.
What batch size should I use for Loop Over Items?
Start small — 5 to 10 — and pair it with a one-second in-loop Wait. Then raise the batch size until you approach the provider’s documented rate limit, watching for the first 429. A steady run that sits just under the limit beats an aggressive one that constantly retries.