vLLM turns Decision 3.0 into a multimodal routing service

vLLM has merged support for the six-model Decision 3.0 family, giving deployments a purpose-built /v1/systemone endpoint for routing and classification rather than asking a generative model to emit an answer as text. The published recipe covers the 27-billion-parameter d3 checkpoint; a follow-up merge validates the flash, mini, nano, lite and edge sizes.

What changed

Decision 3.0 accepts a text or JSON state, optionally with images or video, followed by typed questions. Those questions can ask the model to choose among options, answer yes or no, or assign a score. The endpoint returns probabilities for the available answers instead of generated prose, according to the original vLLM merge.

That distinction matters for semantic routing. A platform can use a decision model to classify a request, choose a workflow or decide whether a task needs deeper reasoning without parsing free-form output. The same request can contain as many as 64 questions, while a choice question can expose up to 255 options. Images are capped at 1.6 megapixels and video processing at two frames per second with no more than 32 frames, the merge documentation says.

The implementation detects Decision 3.0 checkpoints from their decision_config.json, loads a separate FP32 answer-code readout and serves the family as D3ForSequenceClassification. It also disables chunked prefill and prefix caching because the models use bidirectional full-attention layers, while their Gated DeltaNet layers remain causal, according to PR #61565.

What the measurements show

On one AMD Instinct MI300X, the five smaller models changed zero to two decisions out of 416 compared with the checkpoints’ own runtime; the reported differences were ties or near-ties. At eight and 64 concurrent clients, vLLM delivered two to 3.6 times the requests per second of that reference server on a mixed text, image and video workload. At one request at a time, however, vLLM was 7% to 32% slower, according to the family-wide validation.

The 27B recipe reports 415 matching decisions out of 416 on MI300X and 1.8 times the reference server’s throughput at 64 clients on MI325X. It also documents a practical cost: a cold start can take minutes, and reducing --max-model-len from the 262,144-token default to 32,768 cuts startup time for deployments that do not need the full window.

What operators should do

The support is merged to vLLM main but is not in the latest stable release cited by the recipe. Operators must install a nightly build from October 15 or later, then start a checkpoint with vllm serve vllm-sr/d3 and send typed questions to /v1/systemone.

That makes the feature suitable for evaluation rather than an automatic production upgrade. Teams should test their own routing thresholds, media mix and startup budget, and should not assume the published AMD concurrency gains transfer to NVIDIA hardware or tensor-parallel deployments: the validation explicitly leaves those configurations unmeasured.