Skip to content

feat: add Gemma 4 model support (E2B, E4B, 26B A4B, 31B) - #889

Closed
ghost wants to merge 1 commit into
devfrom
unknown repository
Closed

ghost wants to merge 1 commit into
devfrom
unknown repository

Conversation

@ghost

@ghost ghost commented Apr 3, 2026

Copy link
Copy Markdown

Summary

Adds initial support for all 4 Gemma 4 model variants to the C++ inference engine.

Model Configs

  • Gemma 4 E2B: 35 layers, 1536 dim, 8 heads, 1 KV head, 128K context
  • Gemma 4 E4B: 42 layers, 2560 dim, 8 heads, 2 KV heads, 128K context
  • Gemma 4 26B A4B: 30 layers, 2816 dim, 16 heads, 8 KV heads, 128 experts (8 active), 256K context
  • Gemma 4 31B: 60 layers, 5376 dim, 32 heads, 16 KV heads, 256K context

Architecture Changes

  • Per-layer qkv_dim: sliding layers use 256, full attention layers use 512
  • Per-layer kv_heads: full attention layers use reduced KV heads (num_global_kv_heads)
  • Dual RoPE frequencies: 10K for sliding attention, 1M for full attention (configurable via rope_theta_sliding / rope_theta_full)
  • Partial rotary RoPE: full attention layers apply RoPE to only 25% of head dimension (PostQKType::PartialRope)
  • Per-layer KV scaling: attn_kv_scale tensor applied to K/V before cache write
  • Logit soft capping: configurable final_logit_softcapping (30.0 for Gemma 4)

MoE Implementation

  • Top-k expert routing with softmax-weighted combination
  • Router computes logits over all experts, selects top-k, normalizes via softmax
  • Gather-compute-scatter pattern: tokens weighted by routing probability, processed through assigned experts, results accumulated
  • Supports 128 experts with 8 active per token (26B A4B config)

Files Changed

  • gemma/configs.h — new model enums, config fields, IsGemma4() helper
  • gemma/configs.cc — complete model configurations, DeduceModel updates
  • gemma/weights.h — MoE weight pointers, ForEachTensor updates
  • gemma/gemma-inl.h — FFWMoE() with top-k routing
  • gemma/gemma.cc — MoE dispatch, logit soft capping
  • gemma/attention.cc — PartialRope handling, KV scaling
  • gemma/activations.h — configurable RoPE theta values
  • gemma/tensor_info.cc — MoE tensor registration

Notes

  • PLE (Per-Layer Embedding) config fields are present for E2B/E4B but the embedding pathway is not yet wired up
  • KV cache sharing config fields exist but cache indexing logic doesn't use them yet
  • These are optimization features that don't affect correctness

@google-cla

google-cla Bot commented Apr 3, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@ghost ghost closed this Apr 3, 2026
@ghost
ghost deleted the gemma-4-support branch April 3, 2026 15:49
@jan-wassenberg

Copy link
Copy Markdown
Member

Hi Tushar,
impressive work. I'm curious why the pull request is closed - are you concerned about the CLA?

FYI we do have work underway to add Gemma4 (see also #887), but happy to discuss collaborating.

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant