Llama-4-Scout-109B-A17B (MoE) Architecture Parameters, Benchmarks & VRAM
Meta 109B total / 17B active multimodal MoE tailored for 128GB unified memory workstations.
🧬 Llama-4-Scout-109B-A17B (MoE) Architecture & Tensor Specifications
Verified directly against HuggingFace (meta-llama/Llama-4-Scout-17B-16E-Instruct) config.json & Model Card
1. Backbone & Dimensions
llama4- Total / Active Params
- 109B / 17B
- Hidden Size (d_model)
- 5,120
- FFN Intermediate Size
- 8,192
- MoE Experts
- 16 Routed · Top-1 + 1 Shared
- Activation / Norm
- SiLU (SwiGLU) · ε=1e-5
- Native Tensor Dtype
- BF16 / FP8
2. Attention & Layer Layout
GQA- Num Hidden Layers
- 48 Layers
- Hidden Layer Layout
- 48-Layer iRoPE Interleaved Multimodal MoE (16 Routed + 1 Shared Expert)
- Attention Heads (Q / KV)
- 40 Q / 8 KV (5:1 GQA)
- Head Dimension
- 128
- Linear / KV Compression
- iRoPE (interleaved RoPE + NoPE temperature scaling) supporting up to 10M context
- Vision Encoder / MTP
- ViT 34L
3. Context, RoPE & Tokenizer
Tokenizer- Native Context Length
- 10,485,760 tokens
- RoPE Configuration
- iRoPE (θ=500,000 · max_position_embeddings=10,485,760 / 10M)
- Vocab Size
- 202,048 (Untied)
- Special Token IDs
- BOS: 200000 · EOS: [200001,200007,200008] · PAD: 200018
📋 Llama-4-Scout-109B-A17B (MoE) Official API & System Card Specifications
Verified against official Meta developer documentation & pricing
1. Identity & Capabilities
API Spec- API Model ID
- meta-llama/llama-4-scout
- Release Date
- 2025-04-05
- Weight License
- Open-Weight (API Spec)
- Native Reasoning
- —
- Function Calling & JSON Schema
- ✅ 原生支持
- Batch API 异步批处理
- ✅ Supported (50% off)
2. Context, Output & Modalities
Limits- Max Context Window
- 1,024,000 tokens (1024K)
- Max Output Tokens
- 16,000 tokens
- Input Modalities
- Text + Image
- Output Modalities
- Text
- Direct China Access
- ✅ Direct CN + RMB
- Deployment Mode
- Open-Weight + Cloud API
3. Official Pricing & Cache
USD / 1M- Input / 1M Tokens
- $ 0.1
- Cached Input / 1M
- $ 0.1
- Output / 1M Tokens
- $ 0.3
- Tiered Pricing Notes
- OpenRouter: Unified multi-provider routing via OpenRouter
🏆 Official Benchmark Results
Sourced from official System Card / Model Card & technical reports
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains.
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks.
PhD-level Google-proof science reasoning questions
MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions.
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and…
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning.
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code.
🎛️ Official Sampling & Reasoning Parameters
Recommended generation settings & API parameter constraints
📊 Llama-4-Scout-109B-A17B (MoE) Full Benchmark Scorecard
11 public benchmarks · ranks among 154 models · synced 2026-11-20
💻 Coding 1 items
📐 Math 5 items
MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili…
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains.
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks.
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and…
MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions.
👁️ Multimodal 3 items
A dataset for Visual Question Answering on document images containing 50,000 questions defined on 12,000+ document images.
ChartQA is a large-scale benchmark comprising 9.6K human-written questions and 23.1K questions generated from human-written chart summaries, designed to evaluate models' abilities in visual and logical reasoning over charts.
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning.
🧠 Reasoning & Science 2 items
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers.
Source: ZeroEval / llm-stats.com (✓ = third-party verified, others vendor-reported).
⚡ Llama-4-Scout-109B-A17B (MoE) Interactive VRAM & Token Speed Calculator
Physical 128 GB · Load 66.1%
MoE: reads only 10.31 GB active weights/step
Enable Q8/Q4 KV quantization to scale batch concurrency
OLLAMA_CONTEXT_LENGTH=16384 ollama run llama4:scoutMac Studio M5 Max 128G Fits Llama-4-Scout-109B-A17B (MoE) · Cost & TCO
Approx. $3,499 (~$27/GB VRAM) · 120W TDP
📊 Llama-4-Scout-109B-A17B (MoE) Quantization VRAM Breakdown (16K Context)
| Quantization | Bits/Weight | Weights | KV Cache (16K) | Total VRAM | Quality Note |
|---|---|---|---|---|---|
| FP16 / BF16 | 16 bpw | 218 GB | 3.22 GB | 226.88 GB | Lossless Full Precision |
| FP8 / Q8_0 | 8.5 bpw | 115.81 GB | 3.22 GB | 122.65 GB | 99.9% Near-Lossless |
| GGUF Q6_K | 6.56 bpw | 89.38 GB | 3.22 GB | 95.69 GB | Ultra High Quality |
| GGUF Q5_K_M | 5.69 bpw | 77.53 GB | 3.22 GB | 83.6 GB | High Balanced Quality |
| Q4_K_M / AWQ | 4.85 bpw | 66.08 GB | 3.22 GB | 71.92 GB | Sweet Spot (Recommended) |
| GGUF IQ4_XS | 4.25 bpw | 57.91 GB | 3.22 GB | 63.59 GB | Compact 4-bit |
| GGUF Q3_K_M | 3.91 bpw | 53.27 GB | 3.22 GB | 58.86 GB | Moderate Perplexity Loss |
| GGUF Q2_K | 3.15 bpw | 42.92 GB | 3.22 GB | 48.3 GB | Extreme Compression |
🖥️ GPU & Apple Silicon Compatibility for Llama-4-Scout-109B-A17B (MoE) (Q4_K_M)
Frequently Asked Questions
What are the exact architecture parameters of Llama-4-Scout-109B-A17B (MoE)?
Llama-4-Scout-109B-A17B (MoE) uses Llama4ForConditionalGeneration with 48 layers (48-Layer iRoPE Interleaved Multimodal MoE (16 Routed + 1 Shared Expert)), hidden_size=5120, 40 Q heads / 8 KV heads (head_dim=128), and vocab_size=202,048.
How much VRAM does Llama-4-Scout-109B-A17B (MoE) require?
At 4-bit quantization (Q4_K_M, 16K context), Llama-4-Scout-109B-A17B (MoE) requires 71.92 GB VRAM. Full FP16 requires 226.88 GB.