Skip to content
TencentPublic

About

A unified, torch-native training framework for MTP and block-parallel speculative decoding.

Topics

Resources

Contributing

Stars

293 stars

Watchers

2 watching

Forks

Repository files navigation

AngelSpec

A unified training framework for MTP and block-parallel speculative decoding

arXiv Documentation Hugging Face License

AngelSpec is developed by the Tencent Hunyuan AI Infra team, which is a torch-native framework for training speculative-decoding draft models, covering both autoregressive MTP drafting and the block-parallel DFlash family. It is the training framework behind the technical report: all drafters in the report — the TTT-trained MTP drafter and the DFly family — are trained and released with it.

Latest News

  • [2026/11/18] We release AngelSpec v0.2.0, adding training for DFly2, the speculative-decoding drafter that ships with Hy4 flash. DFly2 reuses the target's sparse KV cache and top-k indices, keeping drafting cost constant at any context length. Training for more open-source models, on-policy distillation, and the training and inference images are coming soon. 🔥🔥🔥
  • [2026/07/29] We release AngelSpec v0.1.0, supporting MTP and block-parallel speculative decoding training. Check out our technical report and released models.

Key Features

  • 7 draft architectures — DFly2, DFly, DFlash, DFlare, Eagle3, DSpark, MTP — behind one training pipeline; switching is a config change
  • MTP training with TTT — on-policy multi-depth rollout with memory close to a single causal pass; long-context training up to 256k via Ulysses sequence parallelism
  • Acceptance-aligned objectives — CE, top-k KL, LK losses, D-PACE weighting, and end-to-end TV, composable through configuration
  • MLA / DSA drafters — drafters that read the target's MLA latents and DSA candidates
  • Document-aware sequence packing — Megatron-style fixed-length packing with strict cross-document isolation, on both the DFlash and MTP paths
  • Online evaluation — genuine speculative decoding against the latest checkpoint during training, reporting mean accepted length and per-position acceptance as measured by the serving engine

Architecture

AngelSpec Framework

Inference and training run as separate GPU worker groups connected by a Mooncake tensor store, so hidden-state generation and optimization scale independently. This disaggregated foundation comes from TorchSpec; AngelSpec extends it with the architectures, objectives, and training features above.

Draft Architectures

Architecture Method Key Idea
DFly2 Block-parallel Target's own DSA picks and MLA latents as draft context + MTP fusion (Hy4 flash's drafter)
DFly Block-parallel Hybrid target conditioning + hidden-correction AR head
DFlash Block-parallel Anchor sampling + parallel block generation
DFlare Block-parallel DFlash + learnable per-layer target fusion
Eagle3 Autoregressive TTT Test-time training with input fusion
DSpark Hybrid DFlash backbone + EAGLE-style autoregressive head
MTP Single-head TTT Full MoE decoder layer as draft (Hy3-native)

See the draft-model family docs for details and trade-offs.

DFly2

DFly2 is Hy4 flash's speculative-decoding drafter, trained with AngelSpec and built directly on the target model's Top-k Sparse KV: each draft layer shares the KV cache (KV Share) and the compressed indexer's selection of up to 2,048 positions (Index Share) with its corresponding sparse attention layer. This yields three advantages:

  • Constant cost: each draft token attends to at most ~2K keys, regardless of context length.
  • Zero extra KV memory: no separate KV cache is maintained for the drafter.
  • PD-disaggregation ready: not involved in prefill; the prefill-to-decode transfer adds only one last-layer hidden state.

DFly2

Quick Start

# Install AngelSpec + the vLLM backend
pip install -e ".[vllm]"
pip install mooncake-transfer-engine

# Single-node quickstart (8 GPUs: 4 inference + 4 training)
./examples/qwen3-8b-dfly/run.sh

# Override config values from CLI
./examples/qwen3-8b-dfly/run.sh training.learning_rate=5e-5 training.num_train_steps=500

Or set up a conda environment (installs mooncake too):

./tools/build_conda.sh 1 vllm     # or: sglang
micromamba activate angelspec

CUDA 12.x hosts: PyPI's default torch / vllm wheels target CUDA 13 and won't load on a CUDA-12 driver. Install CUDA-matched wheels first — see Installation.

Examples

Multi-node (Hy3 target)
Example Architecture Mode Released Model
hy3-dfly DFly From scratch AngelSlim/Hy3-DFly-Block8
hy3-mtp MTP From scratch AngelSlim/Hy3-MTP-TTT3
Multi-node (Hy4 flash target)
Example Architecture Mode Released Model
hy-v4-flash-dfly2 DFly2 From scratch tencent/Hy4-flash
Single-node (Qwen3-8B target)
Example Architecture Mode Released Model
qwen3-8b-dspark DSpark From scratch —
qwen3-8b-dfly DFly From scratch AngelSlim/Qwen3-8B-DFly-Block8
qwen3-8b-mtp MTP From scratch AngelSlim/Qwen3-8B-MTP-TTT3
qwen3-8b-dfly-cpt DFly CPT (from ckpt) Continue from AngelSlim/Qwen3-8B-DFly-Block8
Available training configs
Config Architecture Features
configs/vllm_qwen3_8b_dfly.yaml DFly Block-parallel + hidden correction
configs/vllm_qwen3_8b_dflare.yaml DFlare Per-layer target fusion
configs/sglang_qwen3_8b_dflash.yaml DFlash SGLang backend
configs/sglang_qwen3_8b_dspark.yaml DSpark Markov head + confidence
configs/vllm_qwen3_8b_mtp_pack_usp_40k.yaml MTP Packing + USP 40k seq

Released Models

Draft models trained with AngelSpec:

Model Architecture Target Mode
AngelSlim/Hy3-DFly-Block8 DFly Hy3 No-think
AngelSlim/Hy3-DFly-Block8-Think-High DFly Hy3 High-think
AngelSlim/Hy3-MTP-TTT3 MTP Hy3 No-think
AngelSlim/Qwen3-8B-DFly-Block8 DFly Qwen3-8B No-think
AngelSlim/Qwen3-8B-MTP-TTT3 MTP Qwen3-8B No-think

Benchmark

Offline Throughput (HY3-295B-A21B, TP=8)

Main throughput results

Output-token throughput (Tok/s) and speedup relative to AR at temperature 1 across concurrency levels. Each cell uses 3 × 120 s windows; Avg. is the arithmetic mean across six datasets.

License

This project is released under LICENSE. AngelSpec is built upon TorchSpec by LightSeek Foundation and uses Mooncake for disaggregated hidden-state transfer.

Citation

If you find AngelSpec useful, please cite:

@article{angelspec2026,
  title   = {AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding},
  author  = {Liu, Hong and Cen, Rui and Shi, Junhan and Qin, Guangshuo and Zhang, Jiebin and Liu, Tianyu and Fan, Runzhi and Zhao, Guoliang and Xie, Ruobing and Zhang, Kai and Liu, Song and Yu, Guanghua and Zhu, Jianchen},
  journal = {arXiv preprint arXiv:2607.25852},
  year    = {2026}
}

Projects in teams

  • AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression.

About

A unified, torch-native training framework for MTP and block-parallel speculative decoding.

Topics

Resources

Contributing

Stars

293 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages