vLLM details how AMD MI355X outpaced B300 on its Kimi K3 AgentX run

The vLLM project says an eight-GPU AMD Instinct MI355X system serving Kimi K3 delivered 7,567 total tokens per second per GPU at 50 tokens per second per user at P90, versus 6,083 for an NVIDIA HGX B300 system. That is a 24.4% throughput lead in the project's AgentX comparison.

The more important part of vLLM's engineering account is not the headline number. It is the map of work needed to reach it: more than 50 merged AMD- and ROCm-related vLLM pull requests covering attention, recurrence, mixture-of-experts execution, quantization, graph execution and cache support.

What changed in the serving stack

Kimi K3 combines mixture-of-experts layers with two attention mechanisms. Its MLA layers retain a compressed history that grows with a conversation; its KDA layers update fixed-dimension recurrent state. vLLM's optimizations attack both the stored state and the repeated work around it.

Decode context parallelism divides MLA cache positions across eight GPUs instead of replicating the full history on every device. CPU offload preserves state that no longer fits in GPU memory. In the MoE path, vLLM removed a projection that all eight GPUs had been calculating redundantly, then reused an existing all-reduce to assemble the result. Other changes fused KDA operations, removed copies whose outputs were never consumed, and exposed more independent workgroups for speculative verification.

The post also shows why an isolated microbenchmark is not enough. One MoE memory reduction produced less than a 1% gain at moderate concurrency but a 28.9% gain in a separate high-concurrency test, where the prefix-cache hit rate increased. Freeing memory mattered because it kept more reusable state close to the accelerator.

Read the benchmark with its caveats

AgentX replays coding-agent request shapes with synthetic tokens and preserves their timing and prefix-sharing patterns. The published comparison measures complete serving stacks, not bare silicon. At the 50-token target, the AMD setup used CPU offload and moved between DCP and speculative-decoding configurations; the NVIDIA setup used Mooncake offload and DCP8. Both used eight GPUs, tensor parallelism, FP4 weights and FP8 KV cache.

The reported 87.4% advantage in tokens per dollar is modeled, not a cloud price quote. It uses SemiAnalysis assumptions of $1.50 per MI355X GPU-hour and $2.26 per B300 GPU-hour; vLLM notes that some capital and utilization assumptions are unpublished. The metric also counts reused input tokens alongside generated output, while interactivity excludes time to first token.

What practitioners should take away

Teams should not transplant the headline ratio into capacity plans. They should copy the method: profile the deployed request shapes, look for duplicated work across ranks, trace buffers that no selected kernel consumes, and count whether the actual heads and token tiles expose enough parallel work. Then test the complete request at the response speed users need.

That makes this result useful beyond Kimi K3 or MI355X. It is evidence that cache topology, collective placement and kernel scheduling can outweigh nominal peak compute in long-running agent workloads — and that those gains require a sustained upstream integration effort, not one magic kernel.