Gemma-4-26B-A4B-IT (MoE) Architecture Parameters, Benchmarks & VRAM
Google's 26B total / 4B active multimodal MoE; fits easily inside 24GB VRAM with blazing token generation speed.
🧬 Gemma-4-26B-A4B-IT (MoE) Architecture & Tensor Specifications
Verified directly against HuggingFace (google/gemma-4-26B-A4B-it) config.json & Model Card
1. Backbone & Dimensions
gemma4- Total / Active Params
- 26B / 4B
- Hidden Size (d_model)
- 2,816
- FFN Intermediate Size
- 2,112
- MoE Expert Dim
- 704
- MoE Experts
- 128 Routed · Top-8
- Activation / Norm
- GELU-Tanh (GeGLU) · ε=1e-6
- Native Tensor Dtype
- BF16
2. Attention & Layer Layout
HYBRID- Num Hidden Layers
- 30 Layers (5 Global KV)
- Hidden Layer Layout
- 30 Hybrid MoE Layers: 25 Sliding (w=1024) + 5 Global Full Attention, 128 Experts
- Attention Heads (Q / KV)
- 16 Q / 2 KV (8:1 GQA)
- Head Dimension
- 512
- Linear / KV Compression
- Only 5 global layers scale with 256K context; 25 layers capped at 1024 window
- Vision Encoder / MTP
- ViT 27L
3. Context, RoPE & Tokenizer
Tokenizer- Native Context Length
- 262,144 tokens
- RoPE Configuration
- Dual RoPE (Global Proportional θ=1,000,000 partial=0.25 · Sliding θ=10,000)
- Vocab Size
- 262,144 (Tied)
- Special Token IDs
- BOS: 2 · EOS: [1,106,50] · PAD: 0
- Language Support
- 140+ languages pre-trained
📋 Gemma-4-26B-A4B-IT (MoE) Official API & System Card Specifications
Verified against official Google DeepMind developer documentation & pricing
1. Identity & Capabilities
API Spec- API Model ID
- google/gemma-4-26b-a4b-it
- Release Date
- 2026-04-02
- Weight License
- Open-Weight (API Spec)
- Native Reasoning
- ✅ Supported
- Function Calling & JSON Schema
- ✅ 原生支持
- Batch API 异步批处理
- ✅ Supported (50% off)
2. Context, Output & Modalities
Limits- Max Context Window
- 256,000 tokens (256K)
- Max Output Tokens
- 236,000 tokens
- Input Modalities
- Text + Image + Video
- Output Modalities
- Text
- Direct China Access
- ✅ Direct CN + RMB
- Deployment Mode
- Open-Weight + Cloud API
3. Official Pricing & Cache
USD / 1M- Input / 1M Tokens
- $ 0.09
- Cached Input / 1M
- $ 0.05 -44%
- Output / 1M Tokens
- $ 0.3
- Tiered Pricing Notes
- OpenRouter: Unified multi-provider routing via OpenRouter
🏆 Official Benchmark Results
Sourced from official System Card / Model Card & technical reports
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks.
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili…
PhD-level Google-proof science reasoning questions
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code.
Dual-control conversational tool-use benchmark
A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings.
🎛️ Official Sampling & Reasoning Parameters
Recommended generation settings & API parameter constraints
💡 Supports interleaved multimodal input, built-in thinking mode, and native function calling.
📊 Gemma-4-26B-A4B-IT (MoE) Full Benchmark Scorecard
13 public benchmarks · ranks among 154 models · synced 2026-11-20
📐 Math 5 items
All 30 problems from the 2026 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999.
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili…
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks.
MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities.
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions.
👁️ Multimodal 2 items
A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings.
A comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning, featuring 4,460 questions spanning 17 specialties and 11 body systems.
📜 Long Context 2 items
MRCR v2 (8-needle) is a variant of the Multi-Round Coreference Resolution benchmark that includes 8 needle items to retrieve from long contexts. This tests models' ability to simultaneously track and reason about multiple pieces of information across extended conversations.
MRCR v2 (Multi-Round Coreference Resolution version 2) is an enhanced version of the synthetic long-context reasoning task.
🧠 Reasoning & Science 3 items
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code.
BIG-Bench Extra Hard (BBEH) is a challenging benchmark that replaces each task in BIG-Bench Hard with a novel task that probes similar reasoning capabilities but exhibits significantly increased difficulty.
Source: ZeroEval / llm-stats.com (✓ = third-party verified, others vendor-reported).
⚡ Gemma-4-26B-A4B-IT (MoE) Interactive VRAM & Token Speed Calculator
Physical 32 GB · Load 57.2%
MoE: reads only 2.42 GB active weights/step
Enable Q8/Q4 KV quantization to scale batch concurrency
OLLAMA_CONTEXT_LENGTH=16384 ollama run gemma4:26b-a4bRTX 5090 Fits Gemma-4-26B-A4B-IT (MoE) · Cost & TCO
Approx. $2,199 (~$69/GB VRAM) · 575W TDP
📊 Gemma-4-26B-A4B-IT (MoE) Quantization VRAM Breakdown (16K Context)
| Quantization | Bits/Weight | Weights | KV Cache (16K) | Total VRAM | Quality Note |
|---|---|---|---|---|---|
| FP16 / BF16 | 16 bpw | 52 GB | 0.55 GB | 54.89 GB | Lossless Full Precision |
| FP8 / Q8_0 | 8.5 bpw | 27.63 GB | 0.55 GB | 30.02 GB | 99.9% Near-Lossless |
| GGUF Q6_K | 6.56 bpw | 21.32 GB | 0.55 GB | 23.59 GB | Ultra High Quality |
| GGUF Q5_K_M | 5.69 bpw | 18.49 GB | 0.55 GB | 20.71 GB | High Balanced Quality |
| Q4_K_M / AWQ | 4.85 bpw | 15.76 GB | 0.55 GB | 17.92 GB | Sweet Spot (Recommended) |
| GGUF IQ4_XS | 4.25 bpw | 13.81 GB | 0.55 GB | 15.93 GB | Compact 4-bit |
| GGUF Q3_K_M | 3.91 bpw | 12.71 GB | 0.55 GB | 14.81 GB | Moderate Perplexity Loss |
| GGUF Q2_K | 3.15 bpw | 10.24 GB | 0.55 GB | 12.29 GB | Extreme Compression |
🖥️ GPU & Apple Silicon Compatibility for Gemma-4-26B-A4B-IT (MoE) (Q4_K_M)
Frequently Asked Questions
What are the exact architecture parameters of Gemma-4-26B-A4B-IT (MoE)?
Gemma-4-26B-A4B-IT (MoE) uses Gemma4ForConditionalGeneration with 30 layers (30 Hybrid MoE Layers: 25 Sliding (w=1024) + 5 Global Full Attention, 128 Experts), hidden_size=2816, 16 Q heads / 2 KV heads (head_dim=512), and vocab_size=262,144.
How much VRAM does Gemma-4-26B-A4B-IT (MoE) require?
At 4-bit quantization (Q4_K_M, 16K context), Gemma-4-26B-A4B-IT (MoE) requires 17.92 GB VRAM. Full FP16 requires 54.89 GB.