Red Hat’s self-hosted LLM cost model starts with actual tokens, not peak throughput

Self-hosting a large language model replaces a simple API price with a harder platform question: what did the organization spend, and how many useful tokens did that infrastructure actually serve? A new Red Hat Developer engineering guide reduces that question to one operating metric: total cost divided by tokens processed over the same period.

What the model counts

The guide treats hardware, software, personnel and facility overhead as parts of the numerator. For purchased accelerators, it recommends amortizing capital cost across the hardware’s expected life; for shared clusters, it suggests allocating control-plane and operating costs across the workloads that use them. It also calls out power, cooling, rack space and the time spent operating model-serving infrastructure.

The denominator is actual token volume, not benchmark capacity. Red Hat’s example puts a platform costing $50,000 per month and serving 500 million tokens at 10 cents per million tokens. If the same platform serves one billion tokens, the unit cost falls to 5 cents without a hardware change. The guide’s point is that an impressive peak-throughput test does not improve the economics if production demand stays flat.

Where platform teams have leverage

Red Hat identifies several ways to change either side of the equation. Teams can right-size accelerators for the workload, autoscale replicas and cloud nodes, shift batch processing into off-peak periods, and consolidate internal inference demand onto shared capacity. The article recommends using load tests such as GuideLLM to estimate maximum capacity, while measuring real usage through OpenShift metrics and Prometheus for the cost calculation.

Serving choices matter when they let a team reduce resources or absorb more work. The guide points to vLLM batching and allocation tuning, and to llm-d features including cache-aware routing, KV-cache offload, and prefill/decode disaggregation. It also distinguishes input, cached and output tokens: output generation is sequential and generally consumes more GPU time, while cached context can avoid repeated computation. Model size, quantization and speculative decoding therefore affect cost only when they lower required capacity or raise useful token volume.

What to do next

Platform teams should build a monthly ledger before optimizing a benchmark. Record accelerator and cluster costs, software entitlements, operations labor and facility overhead; pair those numbers with actual input, cached and output token counts. Then compare utilization with tested capacity and identify whether the binding problem is excessive spend, idle hardware or insufficient serving efficiency.

That framing is useful because it separates engineering progress from economic progress. A throughput gain is valuable when it permits fewer replicas, protects a latency target under more traffic or opens capacity for another workload. Otherwise, the cost per token has not changed. Red Hat’s guide gives OpenShift AI operators a straightforward way to test that distinction before buying more GPUs or declaring an optimization successful.