21 OCT 2026 — DeepSeek has published the inference engine that runs its own chatbot and API, the software layer that turns a trained model into a service people can query. The company released DeepInfer on GitHub under the Apache 2.0 licence, with the repository's first commit dated 20 October 2026. It supports one model, DeepSeek-V4.1-Flash, and runs on both NVIDIA GPUs and Huawei's Ascend chips from the same code.
Developers noticed quickly. By Wednesday morning Singapore time the DeepInfer repository showed more than 12,500 stars and 1,100 forks on GitHub.
What DeepSeek actually released
The README describes DeepInfer as the framework that powers DeepSeek's large-scale production services, and says the public version is limited to offline inference, though it keeps the features DeepSeek relies on at scale. It bundles kernel libraries DeepSeek has already open-sourced, including FlashMLA, DeepGEMM and DeepEP, along with newer ones such as DeepSelect and custom TileLang kernels.
The engine features sit on top: expert parallelism for the mixture-of-experts layers, speculative decoding that DeepSeek calls DSpark, structured outputs constrained by regex, JSON schema or EBNF grammar, and an asynchronous scheduler that overlaps CPU batch planning with GPU work. The default checkpoint is DeepSeek-V4.1-Flash-1017, which DeepSeek put on Hugging Face on 17 October under the MIT licence, the same day it announced the model update for its website, app and API.
The project credits its origins plainly. "DeepInfer was originally built primarily on vLLM," the README says, with further inspiration from SGLang and NVIDIA's TensorRT-LLM.
Two chip families, one codebase
The supported hardware list has two entries: NVIDIA SM100 GPUs, which MarkTechPost glosses as the Blackwell generation, and Huawei Ascend 950 NPUs. Choosing between them is a single environment variable, DEEPINFER_ACCELERATOR, set to nvidia or ascend before the build. The NVIDIA path needs CUDA 13; the Ascend path needs Huawei's CANN 9.2 toolkit.
Parity is not complete. Batch-invariant inference, which keeps results identical however requests are grouped, and deterministic sampling are both marked NVIDIA only. For anyone building agents that need reproducible runs, that is the difference that matters between the two platforms today.
The release follows DeepSeek's publication in September of Ascend versions of two of its kernel libraries, DeepGEMM-Ascend and DeepEP-Ascend, both visible on the company's GitHub organisation.
The parts left out
DeepInfer is not the whole service. The README lists three things a complete deployment needs that this release does not include: a KV cache system, the split of vision encoding, prefill and decode onto separate machines, and load balancing across data-parallel replicas. A deployment Q&A in the repository explains why the cache and balancing code stays private: those components are "tightly coupled to our internal infrastructure," DeepSeek writes.
The Q&A goes some way to compensating. It describes a storage tier of roughly 100 PiB of SSD holding cached context, kept for three days for agent traffic and seven for chatbot traffic, and recommends storage servers with 20 SSDs of 30.73 TB each. It also admits that splitting prefill and decode across data centres carries a reliability cost, noting that fibre between sites is easily cut by construction crews.
Production figures, by DeepSeek's account
"A single cluster currently serves 155 trillion tokens per day," the Q&A says, putting overall revenue at US$38 per GPU-hour at what DeepSeek calls valley pricing. Neither figure can be checked from outside the company, and the published benchmark charts are DeepSeek's own, with prefill measured on four SM100 GPUs and decoding on 64.
The document is candid in places. Asked whether its internal kernels are better than the public ones, DeepSeek says they are broadly the same, though the public FlashMLA lacks one hand-tuned optimisation that makes its decoding kernel 5% to 10% slower, and end-to-end results differ by at most 5%. Its variable-length verification for speculative decoding cuts the draft tokens checked each step by about 30% and adds 4% to 7% throughput at the production operating point, again by its own measurement.
Guarding the token stream
Alongside DeepInfer, DeepSeek points developers to deepseek-recipe, an MIT-licensed component that converts requests in the Chat Completions, Responses and Messages API formats into token sequences and back. DeepSeek says its behaviour matches the official DeepSeek API, so third-party hosts should get the same results.
One design choice matters for security. When control-token strings turn up in what a user types, in a tool's output or in the arguments of a tool call, deepseek-recipe treats them as plain text, the Q&A says, "preventing text from injecting control tokens that would influence model behavior." Readers curious about how text becomes tokens can try our tokenisation visualiser.
Who can actually use it
The audience is narrow. The recommended decode configuration starts at 64-way expert parallelism, which puts the smallest efficient unit at 128 accelerators. That suits cloud providers and large enterprises far more than a team with a single GPU server, and MarkTechPost's write-up of the release likewise stresses that it is offline inference only.
DeepSeek says inference code for future models and further performance work will continue to land in the same repository. For now the value is as much in the documentation as in the code: a detailed account of how DeepSeek runs one of its own models in production, and what it says that service earns.
View author profile → · Editorial policy
