deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeeokay fashions · GitHub
DeepInfer is the high-performance inference framework that powers DeepSeeokay’s large-scale manufacturing companies. This open-source launch gives offline inference with the core options wanted for serving at scale.
Supported fashions:
Supported options:
- Optimized kernels, mannequin and sampler runtime
- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling (zero-overhead batch scheduler)
- Batch-invariant inference
- Deterministic sampling
- Structured outputs
Feature particulars and configuration
- Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and customized TileLang kernels.
- DSpark and confidence-scheduled verification: speed up decoding with draft tokens and use confidence estimates to schedule verification. Set
DEEPINFER_ENABLE_MTP_VARLEN=1;DEEPINFER_USE_NEXTN_DECODEcontrols the utmost draft size (default: 5). - Decoder SWA bounded replay: scale back decoder prefill work by replaying solely a bounded suffix. Set
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compactandDEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140for the default V4.1 settings; the retained suffix should cowl the SWA window plus the utmost backtracking size. - Asynchronous scheduling: overlap CPU batch scheduling with machine execution. Enabled by default.
- Batch-invariant inference: maintain mannequin outcomes equivalent throughout completely different batch compositions. Enable with
DEEPINFER_BATCH_INVARIANT=1(NVIDIA solely). - Deterministic sampling: produce equivalent outputs for equivalent enter tokens and sampling parameters. Set
DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0and cross a setseq_seedtoSamplingParams(NVIDIA solely). - Structured outputs: constrain era to a regex, JSON schema, or EBNF grammar. Pass
so_args={"schema_type": ..., "schema": ...}toSamplingParams; see examples.
Supported {hardware}:
- NVIDIA SM100 GPUs
- Huawei Ascend 950 NPUs
Required for a whole inference service (not included on this launch):
- A KV cache system
- VE/Prefill/Decode disaggregation (most required engine performance is already carried out)
- Load balancing throughout data-parallel replicas for each prefill and decode
- Linux
- GCC >= 13.0
- Python 3.12
- CUDA 13 (NVIDIA solely)
- CANN 9.2 (Ascend solely)
Before constructing DeepInfer, please create and activate an empty Python 3.12 setting.
# Use `ascend` as an alternative for the Ascend platform
# Keep it exported within the setting the place you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./construct.sh
The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is
downloaded into the Hugging Face cache on the primary run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to
run from an area copy as an alternative.
python examples/offline_inference.py
See examples for particulars.
| Prefill | Decode |
|---|---|
 |
Measured on 4 SM100 GPUs for prefill and 64 for decode — every inside one scale-up area. To reproduce the benchmarks, see run_sweep.sh.
See eval/README.md.
DeepInfer is launched below the Apache License 2.0. See LICENSE for particulars.
DeepInfer was initially constructed totally on vLLM, with further inspiration from SGLang and TensorRT-LLM. We thank their builders.
@misc{deepinfer2026,
title = {DeepInfer: High-performance, large-scale inference framework for DeepSeeokay fashions},
writer = {DeepSeek-AI},
12 months = {2026},
writer = {GitHub},
howpublished = {url{https://github.com/deepseek-ai/DeepInfer}},
}
