Skip to content
NLEN
Illustration: Hedged requests for lower LLM tail latency

Using hedged requests to lower LLM tail latency

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

In interactive systems and user-facing applications, the average response time of a Large Language Model is rarely the biggest bottleneck. While a user on average receives the first token within 800 milliseconds, outliers in the tail distribution (the so-called tail latency such as p95 and p99) cause unpredictable wait times of sometimes tens of seconds. In microservice architectures, the pattern of hedged requests has been known for decades to absorb variability in network traffic and disk operations. Within LLM integrations, this pattern is rapidly gaining ground to work around slow inference clusters and temporary queues at providers.

This article fits within the foundation for reliability and error handling (pillar A1); consult the anchor article on building robust integrations to see how hedged requests relate to standard retries and fallbacks. Where a normal retry only starts after a call explicitly fails or hits a hard timeout, a hedged request proactively sends a second copy of the request to a parallel instance or provider when the initial response is taking longer than expected. Below we look at the theory, implementation patterns, cancellation mechanisms, and trade-offs around double token costs.

The anatomy of LLM tail latency

The response time of an LLM API call consists of two fundamentally different phases: the Time to First Token (TTFT) and the Inter-Token Latency (ITL). TTFT is determined by network latency, queuing time on the provider's inference cluster, and the so-called prefillphase, in which the entire prompt is processed in parallel across the GPU cores. Once the first token is generated, the autoregressive decodephase starts, in which tokens are generated sequentially. Variance in the p99 region almost always arises during the prefill or queuing period due to noisy neighboreffects, cluster rebalancing, or sudden spikes in the provider's batch scheduling.

For a deeper mathematical analysis of response times, see measuring latency with percentiles instead of averages to get insight into how skewed LLM response times actually are. If a p50 is at 600 milliseconds but the p99 rises to 14,000 milliseconds, this means one in a hundred interactions produces an unacceptable user experience. In complex agentic chains with ten consecutive LLM calls, the cumulative p90 of the total system quickly degrades to the level of the individual p99.

How hedged requests work versus standard retries

The concept of hedged requests was widely popularized by Google in the publication The Tail at Scale (Dean & Barroso, 2013). The principle is simple: send a request to instance A. If, after a pre-calculated threshold (for example the historical p90 or p95 of the TTFT), no data packet or SSE token has arrived yet, send the exact same request to instance B without immediately aborting instance A. Whichever responds first with a valid payload or token stream delivers the final answer; the slower connection is cancelled immediately.

Property Standard retry Hedged request Race / speculative request
Start moment After hard timeout or error code (429/5xx) After dynamic threshold (e.g. p90 TTFT) Directly in parallel at t = 0
Extra load / cost Low (only on hard failures) Controlled (only for the slowest 5–10%) Very high (+100% tokens and calls)
Impact on p99 latency No effect on slow, successful calls Drastic reduction (to ~p90 level) Maximum reduction at the expense of budget
Complexity Low (sequential) Medium (async timers & cancellation) Medium (parallel promises)

Selecting hedging thresholds: static vs. adaptive

The effectiveness of hedging stands or falls with the moment the backup call is fired. If we fire too early (for example at the median p50), token usage and the bill double for half of all traffic. If we fire too late (well past the p99), hedging offers hardly any benefit over a normal timeout. The optimal threshold balances extra API costs against the gain in response time.

Two main approaches are used in production for determining the hedging delay:

When designing timeouts and deadlines, it is essential to fit hedging into the overall deadline budget; see the guide on timeouts, cancellation, and deadline budgets to prevent nested timers from working against each other.

Streaming responses and the Time to First Token (TTFT) signal

In streaming scenarios via Server-Sent Events (SSE), receiving the very first chunk provides the ideal signal to decide the hedging race. As soon as instance A data: {"choices": [{"delta": {"content": "..."}}]} sends the first chunk, we know the prefill phase has completed successfully and the model is actually generating tokens. At that exact moment, the gateway can immediately cancel any scheduled hedged timers.

If the hedged call to instance B has already been sent because instance A exceeded the TTFT threshold, a first-chunk-wins mechanism applies. The connection that returns a valid SSE chunk first gets the streaming pipeline assigned to the client. The other connection is cut off immediately via an HTTP/2 RST_STREAM frame or by closing the TCP socket, to minimize unnecessary network throughput and server load.

Architecture patterns for hedged routing

Where do we send the hedged request? Simply repeating the same request to the same provider endpoint solves local network issues but doesn't help if the provider's entire data center is overloaded. Three architectural routes are possible:

1. Intra-provider (different region or API key): The request is sent to the same provider, but via a different geographic region (for example us-east versus eu-west) or via a separate organization tier. This absorbs local cluster saturation without differences in model behavior.

2. Cross-provider with equivalent models: A request that initially goes to Provider A (for example Claude 3.5 Sonnet via AWS Bedrock) is hedged to Provider B (the same model directly via the Anthropic API, or a comparable alternative). This protects against provider-wide outages.

3. Hybrid tier degradation: The initial request goes to a heavy reasoning model. If the response takes too long, a hedged call starts to a lighter, faster sub-model that generates a more concise but acceptable answer within a few hundred milliseconds.

When a central component manages this traffic, it can be housed directly in a gateway layer; read more about the architecture in the guide on self-hosting an LLM gateway with failover and configuration.

Implementation example in TypeScript

Below is a robust TypeScript implementation of a hedged request runner that supports streaming TTFT, including AbortController and explicit error handling.

interface RequestConfig {
  url: string;
  headers: Record<string, string>;
  body: string;
}

async function fetchWithTTFTHedge(
  primary: RequestConfig,
  hedge: RequestConfig,
  hedgeDelayMs: number,
  hardTimeoutMs: number
): Promise<ReadableStream<Uint8Array>> {
  const primaryController = new AbortController();
  const hedgeController = new AbortController();
  const globalTimeoutController = new AbortController();

  const timeoutId = setTimeout(() => {
    globalTimeoutController.abort(new Error("Global deadline exceeded"));
    primaryController.abort();
    hedgeController.abort();
  }, hardTimeoutMs);

  let winnerResolved = false;

  const executeCall = async (
    config: RequestConfig,
    controller: AbortController,
    isHedge: boolean
  ): Promise<ReadableStream<Uint8Array>> => {
    try {
      const response = await fetch(config.url, {
        method: "POST",
        headers: config.headers,
        body: config.body,
        signal: controller.signal,
      });

      if (!response.ok || !response.body) {
        throw new Error(`HTTP fout: ${response.status}`);
      }

      const reader = response.body.getReader();
      const firstChunk = await reader.read();

      if (firstChunk.done) {
        throw new Error("Lege stream ontvangen");
      }

      if (winnerResolved) {
        reader.cancel();
        controller.abort();
        throw new Error("Andere call won de race");
      }

      winnerResolved = true;

      if (isHedge) {
        primaryController.abort();
      } else {
        hedgeController.abort();
      }

      return new ReadableStream({
        async start(streamController) {
          streamController.enqueue(firstChunk.value);
          try {
            while (true) {
              const { done, value } = await reader.read();
              if (done) {
                streamController.close();
                break;
              }
              streamController.enqueue(value);
            }
          } catch (err) {
            streamController.error(err);
          } finally {
            clearTimeout(timeoutId);
          }
        },
        cancel() {
          reader.cancel();
        }
      });
    } catch (err) {
      if (!winnerResolved && isHedge) {
        throw err;
      }
      throw err;
    }
  };

  return new Promise((resolve, reject) => {
    let primaryFailed = false;
    let hedgeStarted = false;

    executeCall(primary, primaryController, false)
      .then((stream) => {
        clearTimeout(hedgeTimerId);
        resolve(stream);
      })
      .catch((err) => {
        primaryFailed = true;
        if (!hedgeStarted) {
          executeHedge();
        }
      });

    const executeHedge = () => {
      if (winnerResolved || hedgeStarted) return;
      hedgeStarted = true;
      executeCall(hedge, hedgeController, true)
        .then(resolve)
        .catch((err) => {
          if (primaryFailed) {
            clearTimeout(timeoutId);
            reject(new Error("Zowel primaire als hedged aanroep gefaald"));
          }
        });
    };

    const hedgeTimerId = setTimeout(executeHedge, hedgeDelayMs);
  });
}

The downside: double token costs and billing

The biggest drawback of hedging for LLM APIs compared to traditional RPC calls is the financial billing model. LLM providers bill based on processed input tokens (prefill) and generated output tokens (decode). Once a hedged call reaches a provider's server and it starts the prefill phase, those input tokens are billed — even if the connection is closed 50 milliseconds later with a client-side abort.

The financial impact is easy to model: with a hedging threshold set at the p90, a second call is started for 10% of all incoming requests. If the prompt size is 4,000 tokens, the total prefill costs for the system rise by roughly 10%. For tasks with huge context windows (such as RAG with 50,000+ tokens), this can lead to hundreds of euros in wasted prefill costs per day. To prevent unexpected hedging surges from exhausting the monthly budget, you can enforce hard cost limits via a budget cap or kill switch in the routing layer.

Three strategies are applied to keep these costs under control:

Idempotency and side effects in non-streaming tasks

When LLM calls are used for function calling or automated mutations in databases (such as creating records or sending emails), hedging introduces a serious risk of duplicate execution. If two instances generate a tool call in parallel with identical parameters, the backend must not execute that action twice.

For non-idempotent interactions, the system must include strict checks; read more about this in the article on idempotency in LLM API calls to prevent duplicate mutations. Every hedged request must be provided with a unique Client-Request-ID or idempotency key. If both LLM responses still come through because cancellation happened just too late, the gateway ensures that only one result is handed off to the application layer and processed.

When should you specifically NOT use hedged requests?

Although hedged requests are a powerful weapon against slow tail response times, there are scenarios where the pattern backfires or is even dangerous for infrastructure stability:

1. During general provider outages (cascading failures): If a provider is experiencing global overload and all responses are delayed, hedging causes the gateway to send 10% to 50% more traffic to that overloaded cluster. This worsens the outage and directly leads to 429 Too Many Requests errors. To stop cascade instability, you can implementing circuit breakers for unstable LLM APIs so that slow endpoints are temporarily isolated.

2. Tight rate limits (TPM / RPM): If your API tier is running close to the tokens-per-minute (TPM) limit, a spike in hedged requests will immediately lead to rate-limit exhaustion for regular traffic.

3. Asynchronous background processing: For data extraction, offline summarization, or overnight evaluations, p99 latency is completely irrelevant; here only cost efficiency per token matters.

Conclusion and checklist for production

Hedged requests transform unpredictable LLM integrations into reliable building blocks with tight SLA boundaries for end users. By firing selectively based on historical p90/p95 TTFT statistics and cancelling immediately on the first incoming streaming chunk, tail latency can be reduced by tens of percent without doubling costs.

Before rolling out to production, always check the following prerequisites: