DeepSeek open-sources DeepInfer, the inference framework behind its production service: runs V4.1 Flash on NVIDIA SM100 and Huawei Ascend 950 under Apache 2.0, and says one cluster serves 155 trillion tokens a day

Why it matters

【Key points】

On Oct 20 (about 09:00 UTC), DeepSeek made DeepInfer, the inference framework behind its production service, public on GitHub under Apache 2.0 (commercial use allowed).

1. Scope: DeepSeek-V4.1-Flash only (V4.1-Flash-1017 by default), on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

2. Features: attention data parallelism, MoE expert parallelism with load balancing, speculative decoding (DSpark), asynchronous scheduling and structured outputs; batch-invariant and deterministic modes are NVIDIA-only. It builds on DeepSeek's open kernels (FlashMLA, DeepGEMM, DeepEP and others).

3. Not included: the KV cache system, prefill/decode disaggregation and cross-replica load balancing, which DeepSeek says are tightly coupled to internal infrastructure.

4. DeepSeek's own figures (deployment Q&A): one cluster serves 155 trillion tokens a day, at $38 revenue per card per hour (off-peak prices); KV cache is kept 72+ hours. Future models' inference code will also land in DeepInfer. The README credits vLLM as its original base.

【Verifiable】

GitHub: https://github.com/deepseek-ai/DeepInfer

Deployment Q&A (PDF, Chinese): https://github.com/deepseek-ai/DeepInfer/tree/main/doc

deepseek-recipe: https://github.com/deepseek-ai/deepseek-recipe

Checkpoint: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017

【Our read】

1. DeepSeek has released its production serving engine as a whole for the first time, not just kernels; self-hosting teams get the official parallelism and deployment ratios.

2. One codebase for NVIDIA and Ascend 950 puts Chinese chips on DeepSeek's main serving path, though some features are NVIDIA-only.

3. The throughput and revenue numbers are unverified, and the cost-critical KV cache and load-balancing parts stay closed.