Red Hat turns KV-cache mechanics into an LLM serving capacity plan

Red Hat has published a detailed inference guide that reduces a production LLM deployment to the constraints platform teams can measure: model-weight memory, key-value cache growth, request concurrency and the precision used for computation.

The central point of the Aug. 26 engineering article is that an inference server is not a thin wrapper around a model file. It is the component that decides whether GPU memory is reserved wastefully, reused across shared context or continuously reassigned as requests finish.

Context length becomes a memory budget

Autoregressive models generate one token per forward pass. To avoid recomputing attention state for all earlier tokens on every pass, the server stores each layer’s key and value vectors in a KV cache. That cache grows with context length and concurrent requests even after the model weights are loaded.

Red Hat illustrates the effect with gpt-oss-120b. Its architecture works out to roughly 36 KB of KV-cache growth per token after accounting for layers that do not retain the full history. The guide estimates about 300 MB for an 8,000-token request and 4.8 GB at 128,000 tokens. It contrasts that with a dense 70-billion-parameter architecture, where the per-token cache can be substantially larger.

Those figures are architecture-level estimates rather than workload benchmarks, but they expose the right planning question: how much memory remains after loading the weights, and how will the server allocate it among active requests?

Three runtime controls change utilization

The article highlights three vLLM mechanisms. PagedAttention divides the cache into fixed-size blocks so the server does not reserve every request’s possible maximum context in advance. Continuous batching replaces completed requests as capacity becomes available instead of holding a static batch open for its slowest member. Prefix caching reuses attention state when requests share an opening system prompt, retrieved document or repository file.

Red Hat’s worked example contrasts a worst-case reservation scheme that accommodates roughly three maximum-context requests on an 80 GB H100 with block-level allocation that can accommodate 50 or 60 typical 8,000-token requests. That is an illustrative comparison, not a service-level promise: output length, request mix, scheduler settings and model implementation will move the result.

Precision changes both fit and throughput

Quantization attacks the other large memory consumer: the model itself. Red Hat notes that a hypothetical BF16 representation of the 117-billion-parameter gpt-oss model would require about 234 GB for weights, while its shipped MXFP4 mixture-of-experts weights fit within an 80 GB accelerator.

The guide separates weight-only quantization, which reduces memory traffic, from lower-precision weights and activations, which can also increase tensor-core throughput. It cites up to 1.6-times throughput for FP8 with minimal accuracy impact and says Red Hat’s compressed models recover more than 99% of baseline accuracy. Teams should verify both figures against their own task distribution rather than treating aggregate recovery as proof for a specific application.

The practical deployment sequence follows: calculate weight and KV-cache memory for the chosen architecture, benchmark the expected context distribution, enable paging and continuous batching, test prefix reuse, then validate quantization against application-level quality. Inference cost is not determined by parameter count alone; it is determined by how the entire serving stack manages each token.