n8n workflow fails mid-run: how do you retry without duplicate data?

Prefer an identifier supplied by the source system, such as order_id, payment_event_id, message_id, or form_submission_id. If no reliable ID exists, build a key from normalized immutable fields—for example, tenant_id + event_type + external_reference—and hash it with SHA-256. Do not use the n8n execution ID because a new run receives a new ID. A timestamp alone is also unsuitable because the same business request may arrive at different times.

Scope matters. A bare customer_id could block valid future orders from that customer, while a newly generated random UUID on every attempt cannot identify duplicates. Store a payload_hash as well. If the same key arrives with a different payload, stop with a conflict instead of silently applying new data to an old result. Set key retention according to the business replay window; payment webhooks may require a much longer window than a short synchronization task.

1. Receive and normalize: authenticate the webhook, normalize the fields used for the key, and preserve one correlation ID end to end.
2. Register the request: atomically write the idempotency_key, payload_hash, status=PROCESSING, locked_until, and received time. In PostgreSQL, use a unique index and INSERT ... ON CONFLICT; when the destination API supports idempotency, send the key in its designated header.
3. Branch on the stored state: proceed when the row was created. If it is already SUCCEEDED, return the saved response or reference. If it is PROCESSING with a valid lease, finish as an in-flight duplicate. If the lease expired, allow exactly one execution to reclaim it through a conditional update.
4. Perform the side effect: pass the same key downstream. For systems without native support, maintain a key-to-external-ID mapping with a unique constraint in the integration layer.
5. Seal the result: store SUCCEEDED, the external ID, and the minimum response needed for replay. Store RETRYABLE_FAILED for transient errors and PERMANENT_FAILED for invalid data or authorization failures.
6. Respond: a synchronous webhook should acknowledge success only after the required state is sealed. For long-running work, acknowledge acceptance and process asynchronously through a queue or outbox.

Avoid a standalone “SELECT, then INSERT” sequence without a transaction or uniqueness constraint. Two concurrent executions can both observe that no row exists and then both create one.

Retry failures that are likely to clear on their own: network timeouts, brief connection loss, HTTP 429, and selected 502/503/504 responses. Set a maximum attempt count, progressively longer delays, and a small random variation so multiple executions do not retry in lockstep. Honor Retry-After when present. The total retry window must fit the business deadline and the destination's capacity.

Do not automatically retry 400/422 validation errors, 401/403 authentication or authorization failures, business-significant 404 responses, or failed business rules. Fail fast, record a clear reason, and route the item for manual review where appropriate. After a timeout on a create operation, query by idempotency key or external reference before attempting another create.

In n8n, Retry On Fail can be enabled on the appropriate node, while an Error Workflow can notify operators or coordinate error handling. Neither replaces the idempotency store. Avoid uncontrolled retries at the node, reverse proxy, queue, and parent-workflow layers simultaneously; appoint one primary retry owner and calculate the worst-case total call count.

Test at least six scenarios: the same payload twice in sequence; two concurrent executions; a timeout after the destination committed; a failure before the write; a worker stopping while holding a lease; and the same key with a different payload. Acceptance means there is only one side effect, retries receive the same external reference, an expired lease can be recovered, and conflicting payloads are rejected.

Monitor duplicates blocked, retries by error code, time spent in PROCESSING, reclaimed leases, permanent failures, queue depth, and the share of cases needing manual review. Logs should include the correlation ID, a masked idempotency key where sensitive, workflow/execution ID, attempt number, old and new state, and external reference. Do not log credentials, tokens, or complete payloads containing personal data.

Before release, confirm that every side-effect node is inventoried; key construction and scope are documented; unique constraints exist; states and timeouts are defined; retry budgets are bounded; dead-letter or manual review exists; dashboards and alerts are active; and the runbook explains how to replay without changing the key.

Is enabling Retry On Fail in n8n enough to prevent duplicates?

No. It only re-attempts a failed node. Protect the side effect with an idempotency key and an atomic database or destination-API operation.

Can the n8n execution ID be the idempotency key?

Not for full-workflow re-runs, because a new run gets a new execution ID. Use the source event ID or a business key that remains stable across all attempts.

Is an IF node that checks whether a record exists sufficient?

Not under concurrency. Use a unique constraint, upsert, or conditional write so the check-and-write boundary is atomic.

What if the API created the record but n8n timed out?

Do not immediately create it again. Look it up by idempotency key or external reference. If found, store that result and mark the operation SUCCEEDED; retry creation only when there is evidence that the first mutation did not occur.

How long should an idempotency key be retained?

Match the period in which the same business operation might be replayed and the severity of duplication. Define TTL by event type, but retain audit evidence longer for payments, invoices, and orders when required.