llm-d turns multimodal inference into a routing and topology problem

Multimodal inference is not merely text serving with an image attached. A new llm-d engineering guide lays out the architectural consequences: media changes request sizing, cache identity, scheduling cost and the shape of the serving pipeline.

The practical message is that operators should not expect a router calibrated for text tokens to place image- and video-heavy requests efficiently. llm-d describes a path that combines model-aware token accounting, content-hash affinity and, for demanding workloads, a separate encode tier.

What changes when media enters the request

Text can be tokenized cheaply before it reaches a model server. Media must first pass through a learned encoder, and its cost depends on properties such as resolution and model configuration rather than payload byte size. A compressed image and a large PNG can represent the same visual-token workload, while similarly sized files can impose very different inference costs.

That creates a three-stage path—encode, prefill and decode—instead of the familiar prefill-and-decode sequence. Encoding is compute-bound and sits directly in the time-to-first-token path. On a shared replica, a vision-heavy request can also interrupt decode progress for other work.

The caching model changes too. Text workloads benefit from shared prefixes in the key-value cache. Media assets are atomic and identified by a content hash, while model servers may maintain separate processor and encoder caches. Randomly spreading repeated images across replicas can therefore repeat both vision encoding and prefill work.

How llm-d routes the work

The llm-d Endpoint Picker Processor can inspect inline image dimensions, use configured defaults for image URLs and accept client-supplied video metadata. Operators can then choose exact model processing through a vLLM render endpoint or a lighter estimator configured for the served model family.

That configuration matters. The guide contrasts resolution-derived token counts for Qwen-VL-style models with fixed token allocations for Gemma-style models. Applying the wrong estimator silently misprices requests and distorts placement decisions.

For reuse, llm-d adds separate signals for prefix affinity and encoder-cache affinity. The latter tracks media hashes recently handled by each pod, allowing requests that reuse an image—but not the surrounding prompt—to return to a replica that may already hold its embeddings. Load scoring remains part of the decision so affinity does not simply create hotspots.

When to split out encoding

For vision-heavy fleets, llm-d describes encode/prefill/decode topologies that move the vision encoder onto a separate hardware pool. This can isolate compute-intensive encoding from latency-sensitive decoding and permit different accelerator choices for each stage.

That path is not yet the default. The guide says encode disaggregation remains experimental in llm-d and vLLM, with interfaces and dispatch contracts still moving. It recommends aggregated multimodal serving—a conventional model deployment plus router configuration—as the well-lit starting point for most operators.

What platform teams should do

Teams introducing multimodal endpoints should first measure the resolution, asset-reuse and request-mix patterns of their real traffic. They should align the token estimator with the model processor, monitor encoder-cache hit ratios and validate whether affinity improves work reuse without concentrating load.

Only workloads with sustained encoding pressure should begin with a separate encode tier. For everyone else, the lower-risk move is to retain an aggregated deployment and make the router media-aware before changing the serving topology.