Duplicate webhooks: how do you process an event only once?

Prefer the provider's event or delivery ID, such as a Stripe Event ID or X-GitHub-Delivery. Prevent cross-tenant and cross-source collisions by storing a composite such as (provider, endpoint, tenant_id, event_id). Do not use only an email address, a second-level timestamp, or the n8n execution ID: these may change between deliveries or fail to identify the business event.

If the source supplies no event ID, derive a composite from immutable fields, for example (source, event_type, object_id, object_version). Use a payload hash only after canonicalizing JSON, because field order or irrelevant metadata can otherwise create a different hash. For semantic duplicates, such as two distinct events that both say an invoice is paid, a business key like (invoice_id, operation=mark_paid) must protect the effect even when event IDs differ. Store the payload hash separately to flag a reused key whose content has changed unexpectedly, and never put sensitive data directly in the key.

Create a webhook_inbox table with at least provider, tenant_id, event_id, event_type, payload_hash, status, attempt_count, lease_until, received_at, completed_at, result_ref, and last_error. Put a unique constraint on the idempotency-key columns. On receipt, execute one atomic statement:

INSERT INTO webhook_inbox
  (provider, tenant_id, event_id, event_type, payload_hash, status)
VALUES ($1, $2, $3, $4, $5, 'received')
ON CONFLICT (provider, tenant_id, event_id) DO NOTHING
RETURNING id;

A returned id means this execution claimed the event. No returned row means it is a duplicate; read the current state and acknowledge safely without rerunning the effect. PostgreSQL documents that ON CONFLICT uses a unique index as the arbiter and provides an atomic outcome under high concurrency. This is fundamentally safer than a separate SELECT followed by INSERT.

Keep the n8n receiver short: Webhook → validate method and content type → verify HMAC over the raw body according to the provider documentation → validate the replay timestamp → normalize identity → atomic PostgreSQL insert → Respond to Webhook. Return 2xx only after the event has been durably stored. A dispatcher or worker workflow processes a new event by inbox_id; a valid duplicate that already exists is also acknowledged so the provider does not retry unnecessarily.

Do not use static workflow data, an in-memory cache, or execution history as the source of truth when multiple replicas or workers are active. Queue mode distributes n8n executions but does not make business operations idempotent. At low volume, the receiver may invoke a sub-workflow after claiming; for higher volume or critical effects, use a durable inbox or job table, lease-based claiming, and bounded concurrency based on database and downstream capacity. Keep sensitive raw payloads out of routine logs.

Suppose the workflow creates an invoice downstream and crashes before updating the inbox to completed. The lease expires, another worker retries, and a second invoice is created. Close this gap by forwarding the same idempotency key when the destination API supports it. Stripe is one example of an API that stores a result by key and returns that result for repeated requests. If you own the destination, enforce a unique business key such as (tenant_id, source_order_id, operation_version).

When an effect cannot be made directly idempotent, use an operation ledger or transactional outbox. Store the business decision and outbox entry in the same transaction; the publisher sends the message, records the external reference, and completes the entry. The consumer still needs idempotency because publishers can retry. For email or notification services without a key, store a notification_key and provider message ID. If a timeout leaves the outcome uncertain, move to unknown for reconciliation instead of immediately sending again.

Use an explicit state machine: received → processing → completed; transient failures become retryable_failed, validation or permission failures become dead_lettered, and an uncertain downstream result becomes unknown. Set lease_until when a worker claims an event. If it dies, another worker may reclaim the event only after the lease expires. Cap attempts and use exponential backoff with jitter. Do not retry invalid 4xx responses indefinitely.

A duplicate received after completed only needs an acknowledgment and may return an internal result_ref. If the event is processing with a valid lease, do not start a second worker. If retryable_failed has reached next_attempt_at, one worker may reclaim it through a conditional update. Retain idempotency records longer than the provider's real redelivery window and the business audit requirement; do not apply an arbitrary short TTL. For late events, compare object versions or retrieve the current source state before updating.

Verify signatures with a secret scoped to each endpoint or tenant, compare in constant time, and enforce the provider's timestamp guidance. Deduplication is not authentication: an attacker can submit a new event ID. Limit payload size, accept only required event types, and keep tokens and raw PII out of logs. If the same key arrives with a different payload hash, flag a conflict instead of treating it as a routine duplicate.

Measure webhook_received_total, duplicate_total, claim_conflict_total, oldest waiting-event age, p95 processing latency, retries by reason, unknown, and DLQ volume. Alert on rates and backlog, not only failed executions. Minimum tests should include sequential duplicates, 20 simultaneous deliveries, timeout after the side effect, worker termination, redelivery after completion, reused key with changed payload, out-of-order events, and invalid or stale signatures. The core acceptance criterion is one business record or notification with a complete audit trail for every delivery.

FAQ

1. Is checking the event ID in the database before processing enough?
No. A separate SELECT and INSERT races under concurrency. Use a unique constraint and one atomic statement so only one execution can claim the event.

2. Does n8n queue mode automatically remove duplicate webhooks?
No. Queue mode distributes executions across workers. You still need an idempotency key, durable event state, and idempotent destination effects.

3. Should the endpoint return 2xx before or after processing?
Verify and durably store the event first, then respond quickly and process asynchronously. Returning success before durable storage can lose the event during a failure.

4. Can I hash the entire payload as the idempotency key?
Only as a fallback. Metadata or JSON ordering can change the hash, while two valid actions may share the same content. Prefer a provider event ID or a versioned business identity.

5. When can deduplication records be deleted?
After the provider's maximum practical redelivery period and the business audit or reconciliation window have both passed. There is no universal safe TTL.