DeepSeek-V4

The DeepSeek-V4 technical report (April 2026), section by section. The architecture: manifold-constrained hyper-connections projected onto the Birkhoff polytope with Sinkhorn-Knopp, the CSA/HCA hybrid attention (token-level KV compression, the lightning indexer and sparse selection, heavily compressed attention, partial RoPE, mixed-precision KV entries) and the Muon optimizer with hybrid Newton-Schulz. The infrastructure: expert-parallel communication overlap, batch-invariant and deterministic kernels, ZeRO for Muon, two-stage context parallelism, tensor-level checkpointing, and the heterogeneous KV cache with on-disk storage. Pre-training: data construction and document packing, the batch-size, context-length and dense-to-sparse schedules, DeepSeekMoE routing, anticipatory routing and SwiGLU clipping, FP4 quantization-aware training, multi-token prediction and the evaluation suite. Post-training: GRPO specialists with reasoning-effort modes and the DSML tool format, multi-teacher on-policy distillation with lazy teacher logits, and the preemptible rollout service for million-token RL. Math cards derive the report's numbers, problems implement each mechanism, and two premium projects build the hybrid attention and the training stack from scratch.