DeepSeek-R1 (671B-A37B MoE) Architecture Parameters, Benchmarks & VRAM
Full 671B MoE reasoning model with MLA KV-cache compression and 37B active parameters per token.
🧬 DeepSeek-R1 (671B-A37B MoE) Architecture & Tensor Specifications
Verified directly against HuggingFace (deepseek-ai/DeepSeek-R1) config.json & Model Card
1. Backbone & Dimensions
deepseek_v3- Total / Active Params
- 671B / 37B
- Hidden Size (d_model)
- 7,168
- FFN Intermediate Size
- 18,432
- MoE Expert Dim
- 2,048
- MoE Experts
- 256 Routed · Top-8 + 1 Shared
- Activation / Norm
- SiLU (SwiGLU) · ε=1e-6
- Native Tensor Dtype
- FP8 Mixed
2. Attention & Layer Layout
MLA- Num Hidden Layers
- 61 Layers
- Hidden Layer Layout
- 61-Layer DeepSeekMoE + MLA (First 3 Dense FFN + 58 MoE Layers)
- Attention Heads (Q / KV)
- 128 Q / 128 KV
- Head Dimension
- 56
- Linear / KV Compression
- MLA Low-Rank KV Compression: kv_lora_rank=512 + qk_rope_head_dim=64
- Vision Encoder / MTP
- MTP 1L
3. Context, RoPE & Tokenizer
Tokenizer- Native Context Length
- 163,840 tokens
- RoPE Configuration
- YaRN Decoupled RoPE (θ=10,000 · qk_rope_dim=64 · 160K Context)
- Vocab Size
- 129,280 (Untied)
- Special Token IDs
- BOS: 0 · EOS: 1
📋 DeepSeek-R1 (671B-A37B MoE) Official API & System Card Specifications
Verified against official DeepSeek developer documentation & pricing
1. Identity & Capabilities
API Spec- API Model ID
- deepseek-r1
- Release Date
- 2025-05-28
- Weight License
- Open-Weight (API Spec)
- Native Reasoning
- ✅ Supported
- Function Calling & JSON Schema
- ✅ 原生支持
- Batch API 异步批处理
- ✅ Supported (50% off)
2. Context, Output & Modalities
Limits- Max Context Window
- 128,000 tokens (128K)
- Max Output Tokens
- 16,000 tokens
- Input Modalities
- Text
- Output Modalities
- Text
- Direct China Access
- ✅ Direct CN + RMB
- Deployment Mode
- Open-Weight + Cloud API
3. Official Pricing & Cache
CNY / 1M- Input / 1M Tokens
- ¥ 4
- Cached Input / 1M
- ¥ 0.8 -80%
- Output / 1M Tokens
- ¥ 16
- Tiered Pricing Notes
- Alibaba Bailian: Hosted on Alibaba Cloud Bailian (CN Direct)
🏆 Official Benchmark Results
Sourced from official System Card / Model Card & technical reports
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains.
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks.
PhD-level Google-proof science reasoning questions
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions.
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions.
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code.
Length-controlled instruction-following win-rate benchmark
🎛️ Official Sampling & Reasoning Parameters
Recommended generation settings & API parameter constraints
💡 Keep temperature in 0.5–0.7 (0.6 best); avoid system prompts and place all instructions in user message.
⚡ DeepSeek-R1 (671B-A37B MoE) Interactive VRAM & Token Speed Calculator
Physical 96 GB · Load 442.7%
Exceeds usable VRAM by 322.45 GB
MLA compresses multi-batch KV footprint
OLLAMA_CONTEXT_LENGTH=16384 ollama run deepseek-r1:671bInsufficient Local VRAM? Ways to Run DeepSeek-R1 (671B-A37B MoE)
Rent an hourly high-VRAM cloud GPU pod, or check the 12-month Buy vs Cloud vs API break-even curve.
📊 DeepSeek-R1 (671B-A37B MoE) Quantization VRAM Breakdown (16K Context)
| Quantization | Bits/Weight | Weights | KV Cache (16K) | Total VRAM | Quality Note |
|---|---|---|---|---|---|
| FP16 / BF16 | 16 bpw | 1342 GB | 1.15 GB | 1370.44 GB | Lossless Full Precision |
| FP8 / Q8_0 | 8.5 bpw | 712.94 GB | 1.15 GB | 728.8 GB | 99.9% Near-Lossless |
| GGUF Q6_K | 6.56 bpw | 550.22 GB | 1.15 GB | 562.83 GB | Ultra High Quality |
| GGUF Q5_K_M | 5.69 bpw | 477.25 GB | 1.15 GB | 488.4 GB | High Balanced Quality |
| Q4_K_M / AWQ | 4.85 bpw | 406.79 GB | 1.15 GB | 416.53 GB | Sweet Spot (Recommended) |
| GGUF IQ4_XS | 4.25 bpw | 356.47 GB | 1.15 GB | 365.2 GB | Compact 4-bit |
| GGUF Q3_K_M | 3.91 bpw | 327.95 GB | 1.15 GB | 336.11 GB | Moderate Perplexity Loss |
| GGUF Q2_K | 3.15 bpw | 264.21 GB | 1.15 GB | 271.09 GB | Extreme Compression |
🖥️ GPU & Apple Silicon Compatibility for DeepSeek-R1 (671B-A37B MoE) (Q4_K_M)
Frequently Asked Questions
What are the exact architecture parameters of DeepSeek-R1 (671B-A37B MoE)?
DeepSeek-R1 (671B-A37B MoE) uses DeepseekV3ForCausalLM with 61 layers (61-Layer DeepSeekMoE + MLA (First 3 Dense FFN + 58 MoE Layers)), hidden_size=7168, 128 Q heads / 128 KV heads (head_dim=56), and vocab_size=129,280.
How much VRAM does DeepSeek-R1 (671B-A37B MoE) require?
At 4-bit quantization (Q4_K_M, 16K context), DeepSeek-R1 (671B-A37B MoE) requires 416.53 GB VRAM. Full FP16 requires 1370.44 GB.