Red Hat brings priority and tenant fairness to shared llm-d inference pools

Red Hat has documented a flow-control layer for llm-d that decides when inference requests may advance before the scheduler decides where they should run. The capability is generally available in Red Hat AI Inference 3.5, according to a new engineering guide published August 27.

The distinction matters for platform teams trying to share expensive GPU pools. A router can choose a healthy backend, but routing alone does not prevent a long or low-priority workload from consuming capacity ahead of latency-sensitive traffic. llm-d’s flow control inserts an admission gate in the Endpoint Picker, keeping requests in central policy queues until they are eligible to move to vLLM.

What the policy controls

The Red Hat guide breaks the mechanism into four decisions: a saturation detector decides when queuing begins; priority bands determine which class is considered first; a fairness policy chooses among tenants within a band; and an ordering policy selects the next request from that tenant’s queue.

Higher integer values represent higher priorities. Requests in the same band can be divided by a fairness identifier, with round-robin selection preventing a busy tenant from taking every dispatch turn. Requests below priority zero are considered “sheddable” and can be rejected after configured limits are reached. Queue time-to-live and capacity bounds control how long overload can accumulate.

For utilization-based admission, llm-d combines per-replica queue pressure and key-value-cache pressure into a pool-saturation signal. When a band reaches its configured usage ceiling, requests in that band and lower bands remain queued for that dispatch attempt. An admitted request then proceeds to the normal scheduler, which selects a vLLM replica.

What Red Hat’s tests show

Red Hat’s published benchmark uses gpt-oss-20B on H100 GPUs and tests consolidation, same-priority tenant fairness, admission tuning and reserved capacity. In the reported runs, high-priority Platinum, Gold and Silver traffic stayed clustered below one second for time to first token, while lower-priority Bronze Batch traffic absorbed substantially more latency. Red Hat cautions that the thresholds are environment-specific and should be validated for the deployed model, request distribution, vLLM configuration and GPU capacity.

What platform teams should do

The practical starting point is to define service objectives for at least two traffic streams, map them to priority bands and fairness identifiers, and drive the shared pool to measured saturation. Teams should observe both sides of the admission boundary: Endpoint Picker queues and saturation, plus vLLM running requests, waiting requests, KV-cache use and preemptions.

Red Hat also provides benchmark configurations and a Flow Control Flight Recorder linked from the guide. Those artifacts make the article more than a feature overview: they give operators a way to reproduce the policy behavior before applying it to a production pool. The AI Inference 3.5 release notes are the product-level reference for the generally available release.