Red Hat turns LLM quantization into a deployment decision tree

Red Hat has published a practitioner-oriented guide to choosing quantized large language models, framing precision as a deployment decision rather than a last-resort compromise. The guide connects model formats, GPU architecture and inference workload to the tradeoffs operators actually face: whether a model fits, where latency falls and how much accuracy is retained.

Start with the hardware and engine

The article separates weight quantization from activation quantization. Lower-precision weights reduce transfers between high-bandwidth memory and on-chip memory, which helps the memory-bound token-generation phase. Lower-precision activations let supported tensor cores process the compute-bound prompt-prefill phase faster. Red Hat describes W8A8 as the production-oriented choice for Hopper-class or newer GPUs, W8A16 as a memory-saving option without the same tensor-core benefit, and W4A16 as the more aggressive route when fitting a model onto one card is the overriding constraint.

That distinction matters because model packages are not interchangeable. The guide maps GGUF to llama.cpp, Ollama and LM Studio; MLX to Apple silicon; and FP8, NVFP4, AWQ, GPTQ and compressed-tensors to vLLM on data-center GPUs. The practical advice is to choose the inference engine and target hardware before downloading a quantized checkpoint, rather than selecting by bit count alone.

Memory savings change deployment shape

Red Hat uses the 109-billion-parameter Llama 4 Scout as a sizing example. At BF16, it estimates roughly 220 GB of weight memory and three 80 GB GPUs; INT8 or FP8 brings that estimate to about 109 GB and two GPUs, while INT4 or FP4 reduces it to roughly 55 GB and one GPU. The calculation excludes broader serving overhead, but it shows how precision can change the number of accelerators needed for a deployment and the number of models that can be consolidated on a cluster.

The guide also argues that calibrated methods are essential. It points to GPTQ, AWQ and SmoothQuant implementations in the Red Hat-maintained LLM Compressor project, and reports that Red Hat’s NVFP4 model work recovered about 99% of BF16 accuracy for models in the 70B-to-235B range, with lower but still high ranges for smaller models.

Validate under the intended load

A serving benchmark in the article compares FP16 with FP8 weights and activations for Llama 3 70B on two H100 GPUs, using a retrieval-augmented-generation-shaped workload of about 1,024 input tokens and 128 output tokens. Red Hat reports input throughput rising from 158 to 474 tokens per second and time to first token under load falling from more than 30 seconds to about 4.8 seconds.

The operational takeaway is not that every deployment should use the lowest precision. Teams should select a format their serving engine supports, test accuracy against their own tasks and benchmark at realistic concurrency. Red Hat positions its pre-compressed model catalog, AI Inference Server and OpenShift AI as a path from that checkpoint choice to scaled serving, but the guide’s most useful contribution is the decision framework: fit, engine compatibility, accuracy and load behavior all need to be evaluated together.