Kimi K3 (2.8T MoE) Architecture Parameters, Benchmarks & VRAM
World’s first 2.8T-parameter (104B active) open-weight multimodal agentic model with KDA + Gated MLA.
🧬 Kimi K3 (2.8T MoE) Architecture & Tensor Specifications
Verified directly against HuggingFace (moonshotai/Kimi-K3) config.json & Model Card
1. Backbone & Dimensions
kimi_k3- Total / Active Params
- 2.8T / 104B
- Hidden Size (d_model)
- 7,168
- FFN Intermediate Size
- 33,792
- MoE Expert Dim
- 3,072
- MoE Experts
- 896 Routed · Top-16 + 2 Shared
- Activation / Norm
- SiTU-GLU · ε=1e-5
- Native Tensor Dtype
- MXFP4 / MXFP8 QAT
2. Attention & Layer Layout
MLA- Num Hidden Layers
- 93 Layers (24 Global KV)
- Hidden Layer Layout
- 93 Layers: 69 Kimi Delta Attention (KDA) + 24 Gated MLA + Attention Residuals
- Attention Heads (Q / KV)
- 96 Q / 96 KV
- Head Dimension
- 64
- Linear / KV Compression
- 896 routed + 2 shared experts (16 selected/token) · MoonViT-V2 (401M) vision encoder
3. Context, RoPE & Tokenizer
Tokenizer- Native Context Length
- 1,048,576 tokens
- RoPE Configuration
- KDA + Gated MLA RoPE (原生 1,048,576 / 1M Context)
- Vocab Size
- 163,840 (Untied)
- Special Token IDs
- BOS: 163584 · EOS: 163586 · PAD: 163839
📋 Kimi K3 (2.8T MoE) Official API & System Card Specifications
Verified against official Moonshot Kimi developer documentation & pricing
1. Identity & Capabilities
API Spec- API Model ID
- kimi-k3
- Release Date
- 2026-07-16
- Weight License
- Open-Weight (API Spec)
- Native Reasoning
- ✅ Supported
- Function Calling & JSON Schema
- ✅ 原生支持
- Batch API 异步批处理
- ✅ Supported (50% off)
2. Context, Output & Modalities
Limits- Max Context Window
- 1,024,000 tokens (1024K)
- Max Output Tokens
- 1,049,000 tokens
- Input Modalities
- Text + Image + Video
- Output Modalities
- Text
- Direct China Access
- ✅ Direct CN + RMB
- Deployment Mode
- Open-Weight + Cloud API
3. Official Pricing & Cache
CNY / 1M- Input / 1M Tokens
- ¥ 20
- Cached Input / 1M
- ¥ 2 -90%
- Output / 1M Tokens
- ¥ 100
- Tiered Pricing Notes
- Moonshot AI: Moonshot AI official CN platform with auto prefix cache
🏆 Official Benchmark Results
Sourced from official System Card / Model Card & technical reports
PhD-level Google-proof science reasoning questions
Full multimodal Humanity's Last Exam with tools
Agent Arena Long Context Reasoning benchmark.
CyberGym is a benchmark for evaluating AI agents on cybersecurity tasks, testing their ability to identify vulnerabilities, perform security analysis, and complete security-related challenges in a controlled environment.
Scientific chart synthesis and quantitative reasoning
🎛️ Official Sampling & Reasoning Parameters
Recommended generation settings & API parameter constraints
💡 Use top_p=0.95 for single-step reasoning and top_p=1.0 for multi-step agentic harnesses.
📊 Kimi K3 (2.8T MoE) Full Benchmark Scorecard
29 public benchmarks · ranks among 154 models · synced 2026-11-20
💻 Coding 10 items
Terminal-Bench 2.1 is an updated release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal.
MCP Atlas is a benchmark for evaluating AI models on scaled tool use capabilities, measuring how well models can coordinate and utilize multiple tools across complex multi-step tasks.
FrontierSWE measures whether an agent can complete open-ended technical projects at the scale of hours to tens of hours, spanning systems optimization, large-scale code construction, and applied ML research. Performance is reported as a dominance score, where higher is better.
Program Bench evaluates code-generation agents by asking them to recreate a program's behavior from only a compiled binary and documentation.
Kimi Code Bench v2 is Moonshot AI's in-house benchmark for evaluating coding agents on realistic software engineering tasks across 10+ mainstream programming languages and a production tech stack spanning backend services, infrastructure, performance engineering, systems…
DeepSWE 1.1 evaluates software engineering agents using the mini-swe-agent harness.
DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no internet access. It measures an agent's ability to autonomously resolve real-world coding issues end to end.
MLS-Bench Lite is the official 30-task subset of MLS-Bench for evaluating whether AI systems can invent generalizable and scalable machine learning methods across LLM pretraining and post-training, robotics, world models, computer vision, reinforcement learning, optimization, ML…
SWE-Marathon is an ultra-long-horizon software engineering benchmark covering tasks such as building compilers, optimizing kernels, and developing production-grade services. It measures whether agents can sustain quality across extremely long engineering trajectories.
PostTrainBench evaluates a model's ability to autonomously post-train base models. Given pretrain-only base models, the agent must complete the full pipeline of data synthesis, training, evaluation, and iteration within a time budget, scored across downstream benchmarks such as…
🤖 Agents & Tools 9 items
DeepSearchQA is a benchmark for evaluating deep search and question-answering capabilities, testing models' ability to perform multi-hop reasoning and information retrieval across complex knowledge domains.
BrowseComp is a benchmark comprising 1,266 questions that challenge AI agents to persistently navigate the internet in search of hard-to-find, entangled information.
Tool Decathlon is a comprehensive benchmark for evaluating AI agents' ability to use multiple tools across diverse task categories. It measures proficiency in tool selection, sequencing, and execution across ten different tool-use scenarios.
OfficeQA Pro evaluates AI models on professional knowledge-work questions and tasks drawn from real office workflows, including document analysis, spreadsheet reasoning, and information synthesis across business domains.
Job Bench evaluates AI agents on realistic professional tasks that require multi-step planning, research, and production of work artifacts.
AA-Briefcase is an Artificial Analysis evaluation of AI systems on professional knowledge-work tasks, reported as an Elo score.
APEX-Agents is a benchmark evaluating AI agents on long horizon professional tasks that require sustained reasoning, planning, and execution across complex multi-step workflows.
SpreadsheetBench 2 evaluates office automation agents on spreadsheet analysis, reasoning, and manipulation tasks.
AutomationBench is a tool-use benchmark that evaluates AI agents on automating real-world workflows, testing their ability to orchestrate tools and complete multi-step automation tasks.
📐 Math 2 items
MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities.
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions.
👁️ Multimodal 7 items
CharXiv-R is the reasoning component of the CharXiv benchmark, focusing on complex reasoning questions that require synthesizing information across visual chart elements.
A benchmark for early-stage visual reasoning and perception on child-like vision tasks.
MMMU-Pro variant evaluated with tool access enabled.
A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings.
PerceptionBench is Moonshot AI's internal benchmark for evaluating atomic visual perception capabilities.
WorldVQA is a benchmark designed to evaluate atomic vision-centric world knowledge. It assesses models' ability to understand and reason about visual elements representing real-world knowledge.
ZEROBench is a challenging vision benchmark designed to test models on zero-shot visual understanding tasks.
Source: ZeroEval / llm-stats.com (✓ = third-party verified, others vendor-reported).
⚡ Kimi K3 (2.8T MoE) Interactive VRAM & Token Speed Calculator
Physical 96 GB · Load 1842.3%
Exceeds usable VRAM by 1639.12 GB
MLA compresses multi-batch KV footprint
OLLAMA_CONTEXT_LENGTH=16384 ollama run kimi-k3:latestInsufficient Local VRAM? Ways to Run Kimi K3 (2.8T MoE)
Rent an hourly high-VRAM cloud GPU pod, or check the 12-month Buy vs Cloud vs API break-even curve.
📊 Kimi K3 (2.8T MoE) Quantization VRAM Breakdown (16K Context)
| Quantization | Bits/Weight | Weights | KV Cache (16K) | Total VRAM | Quality Note |
|---|---|---|---|---|---|
| FP16 / BF16 | 16 bpw | 5600 GB | 0.45 GB | 5713.75 GB | Lossless Full Precision |
| FP8 / Q8_0 | 8.5 bpw | 2975 GB | 0.45 GB | 3036.25 GB | 99.9% Near-Lossless |
| GGUF Q6_K | 6.56 bpw | 2296 GB | 0.45 GB | 2343.67 GB | Ultra High Quality |
| GGUF Q5_K_M | 5.69 bpw | 1991.5 GB | 0.45 GB | 2033.08 GB | High Balanced Quality |
| Q4_K_M / AWQ | 4.85 bpw | 1697.5 GB | 0.45 GB | 1733.2 GB | Sweet Spot (Recommended) |
| GGUF IQ4_XS | 4.25 bpw | 1487.5 GB | 0.45 GB | 1519 GB | Compact 4-bit |
| GGUF Q3_K_M | 3.91 bpw | 1368.5 GB | 0.45 GB | 1397.62 GB | Moderate Perplexity Loss |
| GGUF Q2_K | 3.15 bpw | 1102.5 GB | 0.45 GB | 1126.3 GB | Extreme Compression |
🖥️ GPU & Apple Silicon Compatibility for Kimi K3 (2.8T MoE) (Q4_K_M)
Frequently Asked Questions
What are the exact architecture parameters of Kimi K3 (2.8T MoE)?
Kimi K3 (2.8T MoE) uses KimiK3ForConditionalGeneration with 93 layers (93 Layers: 69 Kimi Delta Attention (KDA) + 24 Gated MLA + Attention Residuals), hidden_size=7168, 96 Q heads / 96 KV heads (head_dim=64), and vocab_size=163,840.
How much VRAM does Kimi K3 (2.8T MoE) require?
At 4-bit quantization (Q4_K_M, 16K context), Kimi K3 (2.8T MoE) requires 1733.2 GB VRAM. Full FP16 requires 5713.75 GB.