Speculators 0.6.0 brings P-EAGLE parallel drafting to vLLM

Speculators 0.6.0 adds an open-source implementation of P-EAGLE, an Amazon-developed speculative-decoding algorithm that predicts several future tokens in parallel rather than drafting them one at a time. Red Hat’s release walkthrough covers data preparation, hidden-state extraction with vLLM, training, evaluation and deployment.

The practical target is drafting overhead. EAGLE-3 uses a small draft model to propose tokens sequentially before the full model verifies them. P-EAGLE adds prediction depths so that the draft model proposes multiple positions in one forward pass; the verifier checks those proposals together and accepts the valid prefix. Red Hat says that reduces sequential work, improves hardware utilization and is especially useful for long reasoning traces.

What the release adds

The P-EAGLE implementation inherits from Speculators’ existing EAGLE-3 model and introduces three main mechanisms.

Conditional Drop-token sampling reduces training memory by keeping progressively fewer positions at deeper prediction depths. The example decay keeps 70% of positions at depth one and 49% at depth two, with a 20% floor so deeper levels continue to receive training data.

P-EAGLE also uses a mask token and a learnable hidden-state tensor for future positions that do not yet have real tokens. Its embedding table is unfrozen so that the placeholder can acquire a useful representation during training. A custom flex-attention mask then lets each prediction see the causal base context and its own rollout chain without attending sideways into other speculative rollouts.

From Qwen3 training to vLLM serving

Red Hat documents a four-step Qwen3-8B workflow: prepare tokenized data, extract internal states from early, middle and late layers through vLLM, train the draft model, and serve the resulting checkpoint directly with vllm serve. The example uses 5,000 samples, four prediction depths and four H100 GPUs; the article reports roughly 50 minutes of end-to-end training.

Published results for a Qwen3 draft model show that acceptance declines with depth, as expected. Across the listed workloads, the first predicted position was accepted 74.9% of the time on average, while the average accepted length was 3.044 tokens. Math reasoning had the longest reported average at 3.820 tokens; summarization was lowest at 2.480.

Those figures are acceptance measurements, not an end-to-end latency guarantee. Workload distribution, target model and serving hardware still determine whether parallel drafting produces a useful production gain. The release matters because it makes that question testable with a reproducible training and deployment path rather than presenting P-EAGLE only as a research result.

Speculators 0.6.0 and pretrained draft models are available now through the open-source project and Red Hat AI’s model collection linked from the article.