DeepSeek's DeepInfer: What the Open-Source Inference Engine Includes, and What It Leaves Out

DeepSeek published DeepInfer, the inference framework it says powers its large-scale production services, as open source under the Apache License 2.0. The README is clear about the limits: this release does offline inference for one model family, on two kinds of hardware, and it leaves out pieces you would need for a complete serving stack. I have not built or run it.

I read one primary source: the repository README at github.com/deepseek-ai/DeepInfer. I did not open the benchmark images, the eval README or the source code. The repository page showed 22,092 stars and a creation date of October 16, 2026 when I read it.

What the release covers

Per the README, the supported model is DeepSeek-V4.1-Flash. The default checkpoint is DeepSeek-V4.1-Flash-1017 from Hugging Face. Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Listed features:

• Optimized kernels, model and sampler runtime, using other DeepSeek repositories such as FlashMLA, DeepGEMM and DeepEP, plus custom TileLang kernels.
• Parallelism: attention data parallel with balanced context parallel, and mixture-of-experts expert parallelism with EPLB.
• DSpark and confidence-scheduled verification: speculative decoding with draft tokens, using confidence estimates to schedule verification.
• Decoder SWA bounded replay: replaying only a bounded suffix to reduce decoder prefill work.
• Asynchronous scheduling, which the README says overlaps CPU batch scheduling with device execution and is on by default.
• Batch-invariant inference and deterministic sampling, both marked NVIDIA only.
• Structured outputs constrained by a regex, JSON schema or EBNF grammar.

What is not included

The README lists what a complete inference service needs that this release does not ship:

• A KV cache system.
• Prefill and decode disaggregation. It says most of the required engine functionality is already implemented.
• Load balancing across data-parallel replicas for both prefill and decode.

This matters. It is an engine for offline inference, not a drop-in replacement for a production serving stack. If you plan to run it at scale you would build or integrate those three parts yourself.

Requirements

The README lists Linux, GCC 13 or newer, Python 3.12, CUDA 13 for NVIDIA and CANN 9.2 for Ascend. You build from source with a script after setting the accelerator, then run the offline example. The default model downloads into the Hugging Face cache on the first run, or you can point it at a local copy.

Two features are reproducibility tools. Batch-invariant inference keeps results identical across different batch compositions. Deterministic sampling returns identical outputs for identical input tokens and sampling parameters when a fixed seed is passed. Both need environment variables set, and the README notes the deterministic mode turns off one of the speculative decoding settings.

The benchmarks

The README shows prefill and decode throughput charts, both images. It says they were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to a sweep script to reproduce them. I could not read the charts, so I have no numbers to report. The setup is a large one, so the figures say little about a single-GPU deployment.

Lineage

The README says DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM, and thanks their developers. That is useful context if you already run vLLM.

What I would check

These are my own untested suggestions.

1. Confirm your hardware. SM100 GPUs and Ascend 950 are a narrow list. Check whether yours is supported before anything else.

2. Check the model list. Only DeepSeek-V4.1-Flash is listed. Other models are not supported by this release.

3. Plan the missing parts. Decide how you would supply a KV cache system, disaggregation and load balancing.

4. Reproduce a benchmark. Run the sweep script on a smaller setup before trusting any chart.

5. Test determinism claims. If you need identical outputs, test the fixed-seed path under your own batch sizes.

6. Read the license and dependencies. The project is Apache 2.0, but it pulls in kernels from other repositories. Check theirs.

What I would not conclude

• That DeepSeek's production stack is fully open. The README says key parts are not included.
• That it beats vLLM or SGLang. I found no head-to-head comparison in what I read.
• That it runs on consumer GPUs. The README lists only SM100 and Ascend 950.
• That the star count reflects use. It reflects attention.

Frequently asked questions

What is DeepInfer?

DeepSeek's inference framework for its models, released as open source for offline inference.

Which models does it support?

DeepSeek-V4.1-Flash, per the README.

What hardware does it run on?

NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

What license is it under?

Apache License 2.0.

Can I serve production traffic with it as is?

Not on its own. The README says a KV cache system, prefill and decode disaggregation and load balancing across replicas are not included.

Does it support structured output?

Yes. It can constrain generation to a regex, a JSON schema or an EBNF grammar.

Is it based on vLLM?

The README says it was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.

Sources

• DeepSeek-AI, DeepInfer repository README (read October 25, 2026): supported model, hardware, features, exclusions, requirements and lineage. Benchmark images, eval README and source code were not read.