DeepSeek V4 Technical Report Explained: Architecture & Benchmarks

The DeepSeek V4 technical report, “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence,” was submitted to arXiv on April 26, 2026, as paper 2606.19348^[1]. It is a preview paper that documents the two MoE models DeepSeek released two days earlier on April 24: DeepSeek V4 Pro (1.6T total / 49B active parameters) and DeepSeek V4 Flash (284B total / 13B active), both supporting a 1M-token context.

The paper is credited to “DeepSeek-AI and 318 other authors.” The same content is also published as an official PDF inside the V4-Pro Hugging Face repository^[3], and the model cards, the paper page, and DeepSeek's official news post all point to the same April 24 preview release^[6].

The report's stated goal is efficiency at scale, not benchmark bragging. From DeepSeek's launch blog: “The benchmark numbers are competitive, but not SOTA. It doesn't matter. The real innovation is how DeepSeek v4 is designed for efficient large context length support, and hence as one of the best candidates for agentic tasks”^[5].

Chapter 2 of the report names three upgrades that carry most of the efficiency story: Hybrid Attention, Manifold-Constrained Hyper-Connections, and the Muon optimizer.

Against V3, the report lists further changes: attention moved from the MLA family to the CSA/HCA hybrid, several early dense FFN layers became Hash-routing MoE, expert affinity activation switched from Sigmoid to Sqrt(Softplus(·)), and QK-Clip was removed in favor of Muon plus RMSNorm-normalized attention logits.

• Hybrid Attention (CSA + HCA): Compressed Sparse Attention and Heavily Compressed Attention are interleaved across layers, cutting the attention cost that dominates long-context inference^[2].
• mHC (Manifold-Constrained Hyper-Connections): residual mappings are constrained to the doubly stochastic matrix manifold (the Birkhoff polytope), with spectral norm at most 1, to stabilize cross-layer signal propagation.
• Muon: replaces AdamW as the optimizer for most modules, delivering faster convergence and more stable training.

The efficiency numbers are the heart of the paper. In the 1M-token setting, V4-Pro needs only 27% of V3.2's single-token inference FLOPs and 10% of its KV cache. V4-Flash does even better: 10% of the FLOPs and 7% of the KV cache.

Against a BF16 GQA-8 (head dim 128) baseline, the report puts the V4 family's KV cache at 1M context at roughly 2% of that baseline. That is what makes 1M context a practical default instead of a research demo.

Where the savings come from: KV hybrid storage that keeps RoPE dimensions in BF16 and everything else in FP8 (roughly half the KV memory of a pure-BF16 layout), FP4 compute inside the lightning indexer, smaller attention top-k than V3.2, and the compressed attention itself.

One precision note: routed expert parameters use FP4 while most other parameters stay FP8. FP4×FP8 GEMMs already match the peak FLOPs of FP8×FP8 on current hardware, and the paper argues future hardware could be about one-third more efficient.

DeepSeek pretrained V4-Flash on 32T tokens and V4-Pro on 33T tokens. The corpus expands multilingual data to cover cross-cultural long-tail knowledge, with special emphasis on long-document sources such as scientific papers and technical reports.

Post-training follows a two-stage paradigm. Stage one cultivates a dedicated domain expert for each target domain — math, code, agents, instruction following — using SFT followed by GRPO reinforcement learning. Stage two merges the experts into one unified model through on-policy distillation, where a student model learns from multiple teachers using a reverse KL loss.

The V4-Pro model card publishes a Base Model benchmark table comparing V3.2-Base, V4-Flash-Base, and V4-Pro-Base on identical rows. It is the cleanest way to see what the architecture buys before any post-training.

Read it this way: V4-Pro-Base crushes V3.2-Base on world knowledge (Simple-QA 55.2 vs 28.3, FACTS Parametric 62.6 vs 27.1) and on long context (LongBench-V2 51.5 vs 40.2). V4-Flash-Base, with only 13B active parameters, beats V3.2-Base's 37B active on most rows.

These are base-model, pre-instruction scores. The instruct versions add the two-stage post-training described in the previous step, and the model card reports those separately.

Official pricing for the V4 family is per 1M tokens: V4-Flash charges $0.14 input on a cache miss, $0.0028 on a cache hit, and $0.28 output; V4-Pro charges $0.435, $0.003625, and $0.87^[7].

DeepSeek has announced peak/off-peak pricing — 2x during peak hours, defined as 9:00–12:00 and 14:00–18:00 Beijing time — but it is not yet in effect as of August 1, 2026. For the full cost math, including the ~98% context-cache discount, see the V4 Flash pricing guide.

The report and model cards also publish recommended sampling. For V4-Pro local deployment: temperature 1.0, top_p 1.0, and at least 384K of context for Think Max mode. For Flash-0731: temperature 1.0, top_p 0.95 in agent scenarios and 1.0 otherwise; high/max effort suggests a 384K max output^[4]. A minimal call looks like this:

from openai import OpenAI

client = OpenAI(
    api_key="<DeepSeek API Key>",
    base_url="https://api.deepseek.com",
)

response = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=[{"role": "user", "content": "Write a one-paragraph summary of the attached context."}],
    reasoning_effort="high",
    extra_body={"thinking": {"type": "enabled"}},
)
# Chain of thought arrives in reasoning_content; the answer is in content.

Both model cards ship official serving commands. This is the Flash-0731 vLLM command from the model card, built for a single node with 4×GB300 GPUs and DSpark speculative decoding enabled:

DSpark is DeepSeek's speculative-decoding module, bundled in the 0731 release. The Hugging Face UI shows 304B parameters for the 0731 repo precisely because the DSpark draft module is included — the base model itself is still 284B.

Two deployment notes. The 0731 repo has no Jinja chat template; the official encoding/ directory provides encode_messages and parse_message_from_completion_text instead. And for the full local workflow — weights, VRAM, and quantized options — read the HuggingFace download and local run guide.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

The report explains the architecture; the model cards explain how to use it. The key to reading either V4 model card is the mode comparison, because every benchmark is reported per reasoning mode.

DeepSeek defines three reasoning modes: Non-think for fast, intuitive everyday tasks; Think High with explicit <think> reasoning for complex problems and planning; and Think Max with a dedicated system prompt that pushes the model toward the boundary of its reasoning ability. The API maps these onto reasoning_effort values of low / high / max, and thinking is on by default^[8].

Mode choice moves scores. On the Pro card, SWE-bench Verified goes from 73.6 in Non-Think to 80.6 in Max mode; SimpleQA jumps from 45.0 to 57.9; LiveCodeBench reaches 93.5 in Max; Codeforces hits 3206. When comparing DeepSeek against another model, make sure both ran at the same effort setting.

• Separately, the Flash-0731 card reports nine agent benchmarks where the re-post-trained Flash beats the V4-Pro preview across the board — Terminal Bench 2.1 at 82.7 vs 72.1, DeepSWE at 54.4 vs 12.8, Cybergym at 76.7 vs 52.7.
• Those runs used DeepSeek's upcoming agent harness in minimal mode, which is not yet public. For the full table and its caveats, see the Flash benchmark deep dive and the DeepSeek Harness explainer.
• Bottom line: treat every number in the DeepSeek V4 technical report and the model cards as vendor-reported until an independent harness exists. The architecture claims, the efficiency ratios, and the two-stage post-training recipe are all DeepSeek's own statements.