ZenteiQ’s 4x llm-d result puts routing, not model code, on the critical path

Google Cloud has published a useful production test of the open-source llm-d project: ZenteiQ says replacing ordinary Kubernetes round-robin balancing with queue- and cache-aware routing increased aggregate inference throughput by 4.07 times across its scientific training-data pipeline.

The result matters less as a cloud-provider scorecard than as evidence that the routing layer can become the limiting factor once a fleet serves long, uneven prompts. It also gives platform teams a concrete operational pattern to examine as llm-d moves into production use.

What changed

ZenteiQ generates and evaluates scientific reasoning traces before feeding verified data into foundation-model training. Its worker pods prepare prompts while vLLM instances run generation, evaluation and guardrail models across multi-node accelerator pools.

According to the case study, standard Kubernetes ClusterIP round-robin balancing created two bottlenecks. Requests with long reasoning traces accumulated behind busy engines even when neighboring instances had capacity. At the same time, prompts sharing 70% to 80% of their instructional context were scattered across replicas, forcing repeated prefill computation instead of reusing warm key-value caches.

ZenteiQ placed GKE Inference Gateway, powered by llm-d, in front of those vLLM pools. The gateway used live queue depth and KV-cache utilization to send work toward available engines while keeping prompts with shared prefixes near warm caches.

What the numbers show

ZenteiQ reports that aggregate throughput rose from 16,964 to 69,114 tokens per second for every 1,000 accelerators. Daily output increased from 1.47 billion to 5.97 billion generated tokens, while individual model pools improved between 3.5 and 5.5 times. The company also measured a 14-percentage-point increase in prefix-cache hit rate.

Those figures are vendor- and customer-reported rather than an independent benchmark, and the post does not provide enough configuration detail to generalize the exact multiplier to another fleet. The mechanism, however, is clear: reduce queue imbalance and avoid redundant prefill work before buying more accelerators or rewriting model code.

Who should care

The pattern applies most directly to platform teams serving heterogeneous, batch-heavy inference jobs through vLLM. Workloads with highly variable output lengths and repeated system or task instructions have the same ingredients that made round-robin routing inefficient in ZenteiQ’s deployment.

For teams evaluating llm-d, the useful next step is to measure queue-depth variance, prefix reuse and cache-hit rates under their own traffic. A large fleet-level gain is plausible only when those bottlenecks are present; otherwise, the added routing layer may improve control without reproducing ZenteiQ’s headline number.

The broader lesson is that distributed inference performance is becoming a scheduling problem. Once model servers are competent, the next gains may come from deciding where each request lands and what cached state it can reuse.