bridge.recipes.gemma.h100.gemma4#

Gemma 4 Dense (E4B) pre-training recipe.

Module Contents#

Functions#

_gemma4_text_conversion_mode

gemma4_e4b_pretrain_2gpu_h100_bf16_config

Return a pre-training config for Gemma 4 E4B (Dense, ~3.8B parameters).

Data#

_GEMMA4_E4B_HF_PATH

__all__

API#

bridge.recipes.gemma.h100.gemma4._GEMMA4_E4B_HF_PATH#

‘google/gemma-4-E4B-it’

bridge.recipes.gemma.h100.gemma4._gemma4_text_conversion_mode()#
bridge.recipes.gemma.h100.gemma4.gemma4_e4b_pretrain_2gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer#

Return a pre-training config for Gemma 4 E4B (Dense, ~3.8B parameters).

Architecture (Gemma 4 E4B):

  • 42 layers, hidden_size=2560, ffn_hidden_size=10240

  • 8 attention heads, 2 KV heads (sliding), 2 KV heads (global, head_dim=512)

  • Sliding-window / global attention interleaved (skip_freq=6)

  • Dual RoPE: sliding θ=10 000, global θ=1 000 000 with 0.25 partial rotation

  • Per-Layer Embeddings (PLE, vocab=262144, dim=256)

  • Shared KV cache across the last 18 layers

  • Local (non-TE) transformer spec via get_gemma4_layer_spec

Default parallelism: TP=2, PP=1, seq_length=4096. Override at launch time with Hydra-style args, e.g.::

checkpoint.pretrained_checkpoint=/path/to/megatron-ckpt
checkpoint.save=/path/to/save
train.train_iters=1000
model.seq_length=4096
bridge.recipes.gemma.h100.gemma4.__all__#

[‘gemma4_e4b_pretrain_2gpu_h100_bf16_config’]