Qwen3-14B Architecture Parameters, Benchmarks & VRAM
Balanced 14.8B dense model tailored for 16GB consumer GPUs and 24GB Macs.
🧬 Qwen3-14B Architecture & Tensor Specifications
Verified directly against HuggingFace (Qwen/Qwen3-14B) config.json & Model Card
1. Backbone & Dimensions
qwen3- Total / Active Params
- 14.8B (Dense)
- Non-Embedding Params
- 13.2B
- Hidden Size (d_model)
- 5,120
- FFN Intermediate Size
- 17,408
- Activation / Norm
- SiLU (SwiGLU) · ε=1e-6
- Native Tensor Dtype
- BF16
2. Attention & Layer Layout
GQA- Num Hidden Layers
- 40 Layers
- Hidden Layer Layout
- 40-Layer Dense Transformer (GQA + SwiGLU + QK-Norm)
- Attention Heads (Q / KV)
- 40 Q / 8 KV (5:1 GQA)
- Head Dimension
- 128
3. Context, RoPE & Tokenizer
Tokenizer- Native Context Length
- 32,768 tokens
- Max Extended Context
- 131,072 tokens
- RoPE Configuration
- RoPE (θ=1,000,000 · 原生 32K · YaRN 4x→128K)
- Vocab Size
- 151,936 (Untied)
- Special Token IDs
- BOS: 151643 · EOS: [151645,151643] · PAD: 151643
- Language Support
- 119 languages & dialects
📋 Qwen3-14B Official API & System Card Specifications
Verified against official Alibaba Qwen developer documentation & pricing
1. Identity & Capabilities
API Spec- API Model ID
- Qwen/Qwen3-14B
- Release Date
- 2025-04-29
- Weight License
- Open-Weight (API Spec)
- Native Reasoning
- ✅ Supported
- Function Calling & JSON Schema
- ✅ 原生支持
- Batch API 异步批处理
- ✅ Supported (50% off)
2. Context, Output & Modalities
Limits- Max Context Window
- 128,000 tokens (128K)
- Max Output Tokens
- 131,000 tokens
- Input Modalities
- Text
- Output Modalities
- Text
- Direct China Access
- ✅ Direct CN + RMB
- Deployment Mode
- Open-Weight + Cloud API
3. Official Pricing & Cache
CNY / 1M- Input / 1M Tokens
- ¥ 0.5
- Cached Input / 1M
- ¥ 0.5
- Output / 1M Tokens
- ¥ 2
- Tiered Pricing Notes
- SiliconFlow: Open-weight cloud hosting on SiliconFlow
🏆 Official Benchmark Results
Sourced from official System Card / Model Card & technical reports
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context.
PhD-level Google-proof science reasoning questions
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions.
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions.
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999.
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code.
Berkeley Function Calling Leaderboard v3 (BFCL-v3) is an advanced benchmark that evaluates large language models' function calling capabilities through multi-turn and multi-step interactions.
🎛️ Official Sampling & Reasoning Parameters
Recommended generation settings & API parameter constraints
💡 Do NOT use greedy decoding (temp=0); use 38,912 max tokens for competition math/code.
💡 Recommended for fast general chat; presence_penalty 0~2 reduces repetition.
📊 Qwen3-14B Full Benchmark Scorecard
17 public benchmarks · ranks among 154 models · synced 2026-11-20
🤖 Agents & Tools 1 items
📐 Math 7 items
MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions.
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
AlignBench is a comprehensive multi-dimensional benchmark for evaluating Chinese alignment of Large Language Models.
American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions.
LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses.
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999.
A competitive programming benchmark using problems from the CodeForces platform. The benchmark evaluates code generation capabilities of LLMs on algorithmic problems with difficulty ratings ranging from 800 to 2400.
🧠 Reasoning & Science 5 items
Arena-Hard-Auto is an automatic evaluation benchmark for instruction-tuned LLMs consisting of 500 challenging real-world prompts curated by BenchBuilder. It includes open-ended software engineering problems, mathematical questions, and creative writing tasks.
AutoLogi is an automated method for synthesizing open-ended logic puzzles to evaluate reasoning abilities of Large Language Models.
ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs).
C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context.
Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each.
📚 Knowledge & General 2 items
Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints.
EQ-Bench Creative Writing v3 is an LLM-judged creative writing benchmark that evaluates models across 32 writing prompts with 3 iterations per prompt. Uses a hybrid scoring system combining rubric assessment and Elo ratings through pairwise comparisons.
Source: ZeroEval / llm-stats.com (✓ = third-party verified, others vendor-reported).
⚡ Qwen3-14B Interactive VRAM & Token Speed Calculator
Physical 32 GB · Load 39.2%
Actual speed varies ~±25% by runtime
Enable Q8/Q4 KV quantization to scale batch concurrency
OLLAMA_CONTEXT_LENGTH=16384 ollama run qwen3:14bRTX 5090 Fits Qwen3-14B · Cost & TCO
Approx. $2,199 (~$69/GB VRAM) · 575W TDP
📊 Qwen3-14B Quantization VRAM Breakdown (16K Context)
| Quantization | Bits/Weight | Weights | KV Cache (16K) | Total VRAM | Quality Note |
|---|---|---|---|---|---|
| FP16 / BF16 | 16 bpw | 29.6 GB | 2.68 GB | 33.33 GB | Lossless Full Precision |
| FP8 / Q8_0 | 8.5 bpw | 15.73 GB | 2.68 GB | 19.17 GB | 99.9% Near-Lossless |
| GGUF Q6_K | 6.56 bpw | 12.14 GB | 2.68 GB | 15.51 GB | Ultra High Quality |
| GGUF Q5_K_M | 5.69 bpw | 10.53 GB | 2.68 GB | 13.87 GB | High Balanced Quality |
| Q4_K_M / AWQ | 4.85 bpw | 8.97 GB | 2.68 GB | 12.29 GB | Sweet Spot (Recommended) |
| GGUF IQ4_XS | 4.25 bpw | 7.86 GB | 2.68 GB | 11.15 GB | Compact 4-bit |
| GGUF Q3_K_M | 3.91 bpw | 7.23 GB | 2.68 GB | 10.51 GB | Moderate Perplexity Loss |
| GGUF Q2_K | 3.15 bpw | 5.83 GB | 2.68 GB | 9.08 GB | Extreme Compression |
🖥️ GPU & Apple Silicon Compatibility for Qwen3-14B (Q4_K_M)
Frequently Asked Questions
What are the exact architecture parameters of Qwen3-14B?
Qwen3-14B uses Qwen3ForCausalLM with 40 layers (40-Layer Dense Transformer (GQA + SwiGLU + QK-Norm)), hidden_size=5120, 40 Q heads / 8 KV heads (head_dim=128), and vocab_size=151,936.
How much VRAM does Qwen3-14B require?
At 4-bit quantization (Q4_K_M, 16K context), Qwen3-14B requires 12.29 GB VRAM. Full FP16 requires 33.33 GB.