vLLM Speculators 0.9 makes draft-model training lighter and easier to measure

vLLM’s Speculators project has released version 0.9.0, concentrating on the cost and observability of training the small draft models used for speculative decoding. The release notes report an FP8 hidden-state backend that cuts storage requirements by 50% while maintaining speculator quality, alongside a fused loss-kernel optimization that produces up to a 2x faster forward training pass for models around 1 billion parameters.[^1]

Those are not serving-speed claims. They target the pipeline that prepares a draft model to propose tokens for a larger verifier model. For teams iterating on DSpark, DFlash or related approaches, less hidden-state storage and a faster loss path can shorten the experimentation loop before a model reaches a vLLM deployment.[^1]

What changed

Speculators 0.9 also changes how hidden states move through distributed training. Mooncake transfer metadata is now returned directly through kv_transfer_params, and transfer tensors can be packed into a single buffer. The project says the result is simpler packing logic and better queue-transfer performance.[^1]

The evaluation side gets more concrete instrumentation. The benchmark suite now integrates torch.profiler for GPU-kernel and latency breakdowns, adds an InferenceX-style throughput-versus-interactivity benchmark across concurrency levels, and links documentation to an automated live evaluation dashboard instead of static result tables. The release also adds MRCR long-context evaluation and support for extracting GLM 5.3 hidden states.[^1]

Not every change is additive. Python 3.10 is now deprecated, hybrid training mode is deprecated, and hidden-state noise injection has been removed after project experiments found that it slowed training without changing final model quality.[^1]

Who should care

The release is aimed at engineers training or evaluating speculative-decoding drafts rather than operators who only run an existing vLLM model. It is most relevant where hidden-state datasets are large, Mooncake is part of the data path, or teams need to compare throughput against interactivity instead of relying on one aggregate benchmark.[^1]

What to do

Before upgrading, teams should check their Python baseline and remove assumptions around the deprecated hybrid mode. Training pipelines that persist hidden states can evaluate the FP8 backend against their own quality thresholds, while performance work should use the new profiler and concurrency benchmark to separate kernel bottlenecks from end-to-end serving behavior. The release notes are the starting point for the full compatibility and migration review.[^1]

[^1]: Speculators v0.9.0 release notes