llm-d makes inference routing sticky only until the warm endpoint saturates

llm-d has changed its default inference-routing configuration from a weighted blend of four signals to a simpler rule: keep a request with the endpoint that already holds its prefix, but break that affinity when the warm endpoint crosses a calibrated saturation limit. The project’s engineering analysis describes the new token-aware default and the benchmarks behind it.

What changed

The earlier default blended prefix-cache match, queue depth, KV-cache utilization and least-recently-used behavior. llm-d’s authors say that combination was difficult to predict and tune, while pure cache affinity could create hot spots by continuing to feed a warm endpoint after a colder endpoint would respond faster.

The replacement separates workloads by their binding resource. For prefill-bound traffic with long prompts, llm-d combines a prefix-cache-affinity filter with a token-load scorer that tracks uncached prefill work in flight. For decode-bound traffic with long outputs, it pairs the same filter with an active-request scorer. If traffic is mixed or its bottleneck is unknown, the project offers a latency-predictor pipeline, but that option adds training and sidecar deployment work.

The matched configurations now ship across llm-d’s optimized-baseline, agentic-serving, multimodal-serving and prefill/decode-disaggregation guides. The project says they are also running in production fleets at Google Cloud Vertex AI, Red Hat and Mistral.

Why operators should care

Cache-aware routing is valuable only while locality costs less than redistribution. A router that always follows the cached prefix can overload one model server; a router that spreads requests indiscriminately gives up reusable KV-cache work.

llm-d turns that trade-off into a saturation valve. Its filter estimates time to first token from each endpoint’s uncached work and calibrated peak prefill throughput. When the warm endpoint’s estimated delay exceeds the best cold endpoint by more than the configured tolerance, the cold endpoint returns to the candidate set.

In the published tests, ten Qwen3-32B model servers ran on H100 GPUs with vLLM and Kubernetes Gateway API InferencePool routing. The bottleneck-matched configurations sustained two to three times the Kubernetes Service round-robin throughput on the tested prefill-bound workloads while maintaining time-to-first-token behavior through the operating range. The authors also report roughly comparable throughput on the decode-bound test, where active stream count was the more useful signal.

Those numbers are evidence for the routing method, not a portable performance promise. Each workload and hardware path needs its own measurement, and the analysis reports only one run for each workload, configuration and load point rather than confidence intervals.

What to do

Teams already using llm-d should first classify their traffic: long-prompt workloads are usually prefill-bound; short-prompt, long-output workloads are usually decode-slot-bound. The project recommends the token-load configuration for the former and active-request scoring for the latter.

Next, calibrate peak prefill throughput for every model, accelerator, inference engine and max-num-batched-tokens combination. llm-d distributes a calibration recipe and a matrix of measured reference values. Operators should choose the permitted time-to-first-token degradation from their own service objective and avoid copying a token threshold from another deployment.

Finally, test the decision under representative traffic rather than a single synthetic shape. The published work shows that workload bottlenecks can shift and that multimodal traffic needs different affinity and timing thresholds. The useful change is not one universal number; it is a routing policy whose escape condition can be explained, measured and revisited.