Inference control planes are splitting across hardware, dispatch and routing

The enterprise inference control plane is becoming less like a single load balancer and more like a set of cooperating schedulers. Three primary artifacts published over the past week put that change at different layers: an Open Data Hub early-access Batch Gateway release, llm-d’s media-aware routing design, and IBM’s deployment requirements for dense Spyre accelerator drawers.

Together they show a common constraint. Request placement can no longer be separated from workload shape or hardware operating mode.

Hardware is part of the serving contract

IBM’s Spyre for Power documentation, updated August 25, makes the hardware boundary explicit. A 12-card ENZ0 drawer requires Red Hat AI Inference Server 3.5 or later and runs the accelerators at a reduced default power limit. An eight-card drawer can use a higher-power mode, but operators must set two environment variables when the container starts.

That is not a transparent capacity increase. IBM says the lower 12-card power setting can reduce performance compared with eight-card high-power mode, with RAG and entity-extraction workloads generally affected more than embeddings. It also caps a logical partition at 24 cards and directs four-card RAG containers toward x16 slots to improve synchronization when a model is sharded.

The placement consequence is straightforward: card count alone does not describe available inference capacity. The control plane needs to know the drawer mode, software level and workload class before it treats two accelerator pools as interchangeable.

Batch dispatch becomes its own policy layer

Open Data Hub’s Batch Gateway 3.6 early-access release, dated August 27, packages a fork of llm-d’s OpenAI-compatible batch system and adds llm-d-async dispatcher support to its OpenShift AI deployment demo. The upstream gateway separates an API server from processors and storage, while the optional dispatcher adjusts how batch requests enter a shared inference service.

The release is an early-access artifact, not a general-availability statement. Its significance is architectural: offline work gets a dispatch path that can be controlled separately from interactive calls rather than competing through the same undifferentiated queue.

Multimodal routing prices the request before placing it

The llm-d multimodal serving guide, published August 31, moves another decision into the router. Image cost depends on resolution and model configuration rather than payload byte size, while reused media can hit an encoder cache even when the surrounding prompt does not share a text prefix.

llm-d therefore describes model-aware token estimation, separate prefix and encoder-cache affinity signals, and load scoring based on queued token work. For vision-heavy fleets it also documents separate encode, prefill and decode pools. The guide marks encode disaggregation as experimental and recommends aggregated multimodal serving as the starting path for most operators.

What platform teams should separate

These capabilities are at different maturity levels: IBM documents a deployment requirement for supported RHAIIS hardware configurations; Open Data Hub has tagged Batch Gateway for 3.6 early access; llm-d presents aggregated multimodal routing as its well-lit path while keeping encode disaggregation experimental.

Platform teams should reflect those boundaries in capacity models and rollout plans. Hardware pools need mode-aware benchmarks, batch traffic needs an explicit admission and dispatch policy, and multimodal endpoints need model-specific request accounting. A single replica count or requests-per-second target no longer describes the system well enough.