n8n at scale: loop, batch, or queue for thousands of records?

Many n8n nodes naturally process each incoming item. Adding Loop Over Items with a batch size of one can increase execution time and overhead without adding reliability. First, pass multiple items directly to the destination node and verify whether the selected operation supports bulk input or iterates over items itself.

A loop is justified for explicit requirements: paginate until no records remain, insert a Wait step between batches to respect a quota, accumulate results across rounds, or preserve ordering. Define what happens when one item fails. Do not enable continue-on-error blindly; failed records need a correlation ID, reason, and retry state so they cannot silently disappear from a successful execution.

Batching limits the number of active items, request size, transaction duration, and pressure on an API or database. When an API provides a bulk endpoint, one call with 100 records may outperform 100 individual calls. When quotas apply per request or payload, the batch must respect both. The n8n HTTP Request node supports item batching and intervals, while n8n's rate-limit guidance describes combining Loop Over Items with Wait.

Start with a defensible small batch based on documented destination limits, then load-test it. Measure throughput, p95 latency, 429/5xx rates, peak memory, and retry duration. Increase batch size until just before latency or errors rise sharply. Give every batch a batch_id, record range, attempt count, and idempotency key. Mark it complete only after downstream confirmation so a checkpoint can resume unfinished chunks instead of replaying the entire dataset.

Queue mode fits when many workflows or chunks can run independently and workers need horizontal scaling. In n8n's architecture, the main instance handles timers and webhooks, creates an execution, and places a job in Redis; a worker runs it and persists results to the database. Per-worker concurrency is configurable, but excessive concurrency merely moves the bottleneck to PostgreSQL, Redis, CPU, memory, or the destination API.

Queue mode is not a business message queue for every record. If a producer loads 50,000 records into one execution, a worker still owns that payload. For safe parallelism, split data into chunks and create smaller executions, or persist jobs in a database/outbox and trigger worker workflows by job ID. Useful states include pending, processing, completed, retryable_failed, and dead_lettered; lease timeouts let another worker recover abandoned jobs.

The producer reads the source using a stable cursor, normalizes identifiers, and writes chunk jobs to a job table or outbox. A unique constraint on (source, record_id, operation_version) prevents duplicate job creation. A dispatcher atomically claims a job, assigns a lease, and calls a worker workflow with job_id instead of passing a large payload through the queue. The worker fetches required data, processes one batch, records the result under an idempotency key, and acknowledges completion.

Retry transient timeouts, 429, and 5xx responses with exponential backoff, jitter, and a bounded attempt count. Send invalid data or permission failures to a DLQ rather than retrying forever. If the destination lacks idempotency support, maintain an operation ledger around the side effect and check it on retries. In Kubernetes, scale workers using queue depth and oldest-job age, with a ceiling tied to database pool capacity and downstream quotas.

Implement four guardrail groups. Correctness: idempotency keys, unique constraints, deterministic cursors, and post-run reconciliation. Load control: batch size, worker concurrency, rate limiting, and backpressure as queue age rises. Recovery: bounded retries, exponential backoff, checkpoints, lease timeouts, and selective DLQ replay. Observability: records per second, success and failure by reason, retry count, queue depth, oldest-job age, p95 job latency, and peak memory.

Avoid carrying binary data or complete payloads through every node; discard unused fields early. n8n recommends splitting workflows and limiting data returned by sub-workflows to reduce memory pressure. Configure execution-data retention and pruning according to audit needs rather than retaining every payload indefinitely. Before production, kill a worker mid-batch, simulate 429, submit the same input twice, and verify that processing resumes without omissions or duplicate side effects.

FAQ

1. At what record count is a queue mandatory?
There is no universal threshold. A queue helps when executions are independent, workers must scale, or retries need separation. Thousands of records may still fit a bulk or batch workflow when payload and SLA allow it.

2. What batch size should we use?
Treat documented API or database limits as the ceiling, start smaller, and load-test throughput, p95 latency, errors, and memory. Do not copy one number across unrelated connectors.

3. Is Loop Over Items with batch size one safer?
Not inherently. It reduces items per iteration but increases calls and runtime. Reliability comes from checkpoints, a retry policy, and persistent failed-record tracking.

4. Does n8n queue mode distribute items automatically across workers?
No. It distributes workflow executions. To use workers for independent chunks, the design must create separate executions or jobs.

5. How do we retry without duplicate writes?
Use an idempotency key or unique constraint derived from record identity and operation version, persist job state, and retry only transient failures.