Webhook Reprocessing: What Do You Do When a Delivery Fails

When applications communicate intensively with language models via asynchronous tasks or batch interfaces, webhooks form the primary bridge for status updates and result delivery. In a synchronous API call, the client waits for the response, but for long-running AI generations or background processes, the server sends an HTTP request to a pre-configured endpoint once the task is complete. In theory, this is an elegant pattern that saves resources. In the practice of distributed systems, however, webhook deliveries fail constantly due to temporary network errors, database locks on the receiving end, restarting microservices, or sudden volume spikes.

A robust webhook system can't simply rely on a successful first attempt. Anyone who wants to maintain a stable data infrastructure must explicitly design for failure and recovery. To understand the broader framework of fault-tolerant network architectures, it's worthwhile to first look at how robust integrations with fallbacks and retries are fundamentally set up. In this article, we analyze the complete cycle of webhook failure: from initial error detection and exponential retry schedules to idempotent processing, Dead Letter Queues, and manual audit replays.

Anatomy of a failed webhook: classifying error conditions

Not every error requires the same response. To set up an effective reprocessing mechanism, we first need to distinguish between transient errors, permanent application errors, and ambiguous timeouts. A thoughtless retry on a malformed payload wastes precious server capacity, while halting retries during a temporary network spike leads to data loss.

Transient errors typically arise at the transport layer or from brief overload. Examples include DNS resolution failures, TCP connection timeouts, broken TLS handshakes, and HTTP status codes such as 408 Request Timeout, 429 Too Many Requests, 502 Bad Gateway, 503 Service Unavailable and 504 Gateway Timeout. In all these scenarios, the receiving endpoint has often not processed the payload, and a retry after a short wait is the appropriate response.

Permanent errors, on the other hand, point to a structural configuration or validation problem. Think of HTTP 400 Bad Request (an invalid JSON schema), 401 Unauthorized or 403 Forbidden (invalid signature or expired token), and 404 Not Found (a deleted or misconfigured endpoint). Blindly retrying these calls only leads to an endless cycle of error messages. Such payloads must be split off immediately to an exception queue for inspection.

A particularly treacherous category is the so-called ambiguous timeout. Here, the sending service transmits the webhook but breaks the HTTP connection after, say, 5,000 milliseconds without having received response headers. In this state, it's impossible to know for certain whether the receiving server had already stored and executed the payload before the network interruption, or whether the request never arrived at all. This specific failure behavior mandates that every receiving system be built to be strictly idempotent.

Retry schedules: exponential backoff and decorrelated jitter

When a webhook is not successfully delivered, the sending aggregator or gateway must not immediately resend the payload. If a receiving server is buckling under heavy load and returns 503 Service Unavailable , an immediate flood of hundreds of synchronous retries will take the server down for good. This phenomenon is known as a retry storm or the thundering herd problem.

The standard mitigation is exponential backoff, where the time interval between attempts progressively increases. A classic schedule, for example, uses intervals of 5 seconds, 25 seconds, 2 minutes, 10 minutes, 1 hour, and 6 hours. However, purely deterministic exponential wait times don't fully solve the synchronization problem: thousands of failed messages that started at the same time will all come back in at exactly the same moment after 25 seconds.

To spread these spikes out over time, adding random noise — so-called jitter — is essential. The implementation below shows a proven decorrelated jitter function in TypeScript/Node.js, which dynamically calculates the wait time within safe operational bounds:

interface RetryConfig {
  baseIntervalMs: number;
  maxIntervalMs: number;
  maxAttempts: number;
}

function calculateBackoffWithJitter(
  attempt: number,
  previousIntervalMs: number,
  config: RetryConfig
): number {
  if (attempt <= 1) {
    return config.baseIntervalMs;
  }
  
  // Decorrelated jitter: kies willekeurig tussen basisinterval en 3x vorig interval
  const sleepCeiling = Math.min(config.maxIntervalMs, previousIntervalMs * 3);
  const randomFactor = Math.random();
  const sleepMs = config.baseIntervalMs + randomFactor * (sleepCeiling - config.baseIntervalMs);
  
  return Math.floor(Math.min(config.maxIntervalMs, sleepMs));
}

// Voorbeeld van configuratie:
const webhookPolicy: RetryConfig = {
  baseIntervalMs: 5000,     // Start op 5 seconden
  maxIntervalMs: 86400000,  // Maximaal 24 uur
  maxAttempts: 8            // 8 pogingen over circa 36 uur
};

This algorithm ensures that retry attempts are evenly spread across the time window. This gives the receiving party breathing room to clear queues, restore database connections, and warm up caches without being repeatedly swamped by synchronized requests.

Idempotency on the receiving end: eliminating duplicate processing

Because retries inherently carry the risk that a payload gets delivered more than once (at-least-once delivery), a watertight idempotency mechanism is an absolute requirement. Without idempotency, a resent webhook for a completed LLM summary might, for example, cause tokens to be billed twice, two emails to be sent to an end user, or duplicate rows to end up in a relational database.

Idempotency is implemented by giving each webhook a unique identifier in the payload or HTTP headers, such as X-Webhook-ID or X-Event-ID combined with a X-Idempotency-Key. The receiver checks on arrival whether this key is already known in a central status store (such as Redis or a Postgres idempotency table). Anyone who wants to study the deeper concepts around transactions and deduplication can consult the specialized overview on idempotency in LLM API calls to see how unique keys prevent duplicate processing.

The status lifecycle of a webhook event ideally follows a three-part path:

1. Pending / Processing: The payload has been received and the key is locked with a short time-to-live (TTL). If a second request with the same key arrives within this lease, the server returns 409 Conflict or a 202 Accepted without further action.
2. Completed: Processing has finished successfully. The key is kept for at least 72 hours along with the corresponding HTTP response status. A repeated call with this key immediately gets back a 200 OK without re-executing the underlying business logic.
3. Failed: If processing failed locally before any database mutations, the lock is released so that a later legitimate retry from the sender can be attempted again.

async function handleIncomingWebhook(req: Request, res: Response): Promise<void> {
  const eventId = req.headers['x-event-id'] as string;
  if (!eventId) {
    res.status(400).json({ error: 'Ontbrekende X-Event-ID header' });
    return;
  }

  // Atomair controleren en vastleggen in de cache/database
  const lockAcquired = await redisClient.set(
    `webhook:lock:${eventId}`,
    'PROCESSING',
    'NX',
    'EX',
    60
  );

  if (!lockAcquired) {
    const existingStatus = await redisClient.get(`webhook:status:${eventId}`);
    if (existingStatus === 'COMPLETED') {
      // Reeds verwerkt: retourneer direct succes
      res.status(200).json({ status: 'already_processed', eventId });
      return;
    }
    // Lopende verwerking elders: verzoek afwijzen of parkeren
    res.status(409).json({ error: 'Event wordt momenteel reeds verwerkt' });
    return;
  }

  try {
    await processLLMWebhookPayload(req.body);
    await redisClient.set(`webhook:status:${eventId}`, 'COMPLETED', 'EX', 259200); // 3 dagen
    res.status(200).json({ status: 'success' });
  } catch (err) {
    await redisClient.del(`webhook:lock:${eventId}`);
    res.status(500).json({ error: 'Interne verwerkingsfout' });
  }
}

Dead Letter Queues (DLQ) and poison message isolation

When a webhook still can't be delivered after the maximum configured number of retries (for example, 8 attempts), the message must not be silently discarded. At the same time, a structurally broken payload — a so-called poison message that repeatedly crashes the webhook consumer due to a memory leak or invalid characters — must not block the processing of valid messages.

The solution for this is a Dead Letter Queue (DLQ). Messages that have exhausted their retry budget, or that return a non-recoverable HTTP status code (such as 400 or 422), are automatically removed from the active delivery pipeline and moved to persistent storage. Alongside the original payload, this storage also holds extensive metadata: the number of attempts made, exact timestamps, the received HTTP status codes, and the raw network error messages.

The DLQ acts as a safety buffer. Once an engineering team has fixed a bug in the receiving API or adjusted the firewall rules, the messages can be resubmitted from the Dead Letter Queue to the processing service via a controlled replay script. This prevents valuable asynchronous AI results from being irrevocably lost during prolonged outages.

Verification and security during reprocessing: HMAC and key rotation

A webhook system that communicates over the public internet must be able to guarantee that incoming payloads originate from the legitimate source and haven't been tampered with in transit. This is done by default via an HMAC signature (for example sha256) in a header such as X-Hub-Signature-256.

When designing a reprocessing system, this security layer introduces a number of specific pitfalls. When a webhook is resubmitted after 12 hours, the sender's secret token may have been rotated in the meantime. Moreover, many vendors add a timestamp to the signed payload to prevent replay attacks by malicious actors (for example, a maximum time difference of 5 minutes between signature and receipt).

Structurally securing such tokens and certificates requires tight procedures. To get insight into how to securely store, inject, and rotate API keys and webhook secrets without downtime, see the overview on secure management of API keys in production environments. During a legitimate reprocessing by the source system, the sending gateway must generate the signature with the current payload and a refreshed timestamp on every attempt, or the receiving verification layer must support audit headers specifically authorized for DLQ replays.

import crypto from 'crypto';

function verifyWebhookSignature(
  payloadRaw: string,
  signatureHeader: string,
  secret: string,
  toleranceSeconds: number = 300
): boolean {
  // Voorbeeld header formaat: t=1755273600,v1=6a2b3c...
  const parts = signatureHeader.split(',');
  const timestampPart = parts.find(p => p.startsWith('t='));
  const signaturePart = parts.find(p => p.startsWith('v1='));

  if (!timestampPart || !signaturePart) {
    return false;
  }

  const timestamp = parseInt(timestampPart.split('=')[1], 10);
  const receivedSignature = signaturePart.split('=')[1];
  const currentTime = Math.floor(Date.now() / 1000);

  // Replay-aanval preventie: weiger berichten ouder dan tolerantie
  if (Math.abs(currentTime - timestamp) > toleranceSeconds) {
    return false;
  }

  const signedPayload = `${timestamp}.${payloadRaw}`;
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(signedPayload, 'utf8')
    .digest('hex');

  // Constante-tijd vergelijking om timing attacks te voorkomen
  return crypto.timingSafeEqual(
    Buffer.from(receivedSignature, 'hex'),
    Buffer.from(expectedSignature, 'hex')
  );
}

Interaction with LLM aggregators and gateways on webhook events

In modern AI architectures, requests to models are rarely sent directly to a single provider; there's almost always a routing layer or gateway in between. When an asynchronous batch task or document processing job is spread across multiple providers, the gateway also acts as a webhook relay or webhook aggregator.

This creates an extra abstraction layer: the upstream AI provider sends a webhook to your gateway, and your gateway must then translate, normalize, and forward this status to your organization's internal microservices. If you want to understand how such a central hub operates between various providers and clients, read the detailed guide on how an LLM API aggregator works.

In this model, the gateway acts as a shock absorber. If an internal service is temporarily unreachable, the gateway catches the incoming webhooks from the external model provider, immediately validates them with a 200 OK back to the provider (so it stops its retry counter), and buffers the event in an internal Apache Kafka or RabbitMQ queue. From that point on, the gateway itself manages reprocessing toward the internal backends, fully tuned to the capacity and rate limits of its own application landscape.

Content integrity and fact-checking after automatic reprocessing

Beyond purely infrastructural errors, asynchronous processing with language models carries a specific content-related risk. When a batch task or background job is re-triggered after a network failure, a model, lacking deterministic parameters (such as a fixed seed and temperature: 0), may generate a different answer than during the initial interrupted attempt.

If a webhook arrives after multiple retries and is written directly to a publishing platform or customer record, automatic validation is crucial. To ensure that asynchronously generated text doesn't contain hallucinations or incorrect entities after a restart, the article on fact-checking AI answers offers practical tools for automated and semi-automated verification steps.

We should also clearly delineate the functional scope of asynchronous task patterns. This article specifically covers error handling and reprocessing of failed webhook deliveries; for the initial design of asynchronous event-driven architectures and status polling, we refer to the article on webhooks and asynchronous tasks in API integrations. Combining data integrity at the application level with robust network retries creates a reliable chain.

Monitoring, observability, and manual replay mechanisms

A reprocessing pipeline can't function effectively without deep observability. Blindly relying on automated retries without timely alerting means structural failures are only noticed after hours or days, by which point the Dead Letter Queue has already filled up.

For a healthy webhook infrastructure, at minimum the following metrics should be continuously monitored via dashboards and alerts:

• Webhook Delivery Success Rate: The percentage of webhooks successfully delivered (HTTP 2xx) on the first attempt (target: > 99.5%).
• Retry Queue Depth: The total number of messages currently waiting for a retry attempt. A sudden exponential rise points to an outage at the receiving endpoint.
• DLQ Ingestion Rate: The number of messages per minute that permanently fail and get written to the Dead Letter Queue. Any value greater than zero requires an operational notification.
• Delivery Latency (p95 & p99): The time between the completion of the asynchronous LLM task and the final successful processing by the receiving client.

Once an outage has been resolved, the operations team must have a CLI tool or admin interface for performing controlled replays. Such a replay mechanism must support bulk operations with configurable rate limiting, so that draining a DLQ of 50,000 backlogged messages doesn't immediately overload the just-recovered database all over again.

Operational implementation checklist

To validate a webhook delivery and reprocessing system in production, the checklist below can serve as a guide:

By separating transport errors from logical bugs, enforcing strict idempotency on the receiving end, and spreading retries with jitter, a fragile webhook mechanism turns into a resilient, production-ready data connection that can withstand large-scale network and application outages.