Skip to main content
Ctrl+K
Megatron Bridge - Home Megatron Bridge - Home

Megatron Bridge

  • GitHub
Megatron Bridge - Home Megatron Bridge - Home

Megatron Bridge
  • GitHub

Table of Contents

Performance

  • Performance
  • Performance Archive

Guides

  • Parallelisms Guide
  • Performance Tuning Guide
  • Using Recipes
  • NeMo 2.0 to Megatron Bridge Migration Guide
  • Megatron-LM to Megatron Bridge Guide

Bridge with 🤗 Hugging Face

  • Get Started with 🤗 Hugging Face Conversion
  • Megatron Bridge Conversion Technical Details

Supported Models

  • Supported Models
  • BAGEL-7B-MoT
  • Bailing
    • Ling 2.0
  • BERT
    • Megatron-Style BERT
  • DeepSeek
    • DeepSeek V3
    • DeepSeek V4
  • ERNIE 4.5
  • EXAONE
  • Falcon
    • Falcon H1
  • Gemma
    • Gemma 3
    • Gemma 3 VL (Vision-Language)
    • Gemma 4 VL (26B-A4B MoE)
    • DiffusionGemma
  • GLM
    • GLM 4.5
    • GLM-4.5V
    • GLM-4.7 and GLM-4.7-Flash
    • GLM-5
    • GLM-5.2
  • GPT-OSS
    • GPT OSS 20B
    • GPT OSS 120B
  • HY V3
  • Kimi
    • Kimi K2
    • Kimi-K2.5-VL
    • Kimi K3
  • Llama
    • Llama 3
  • MiniMax
    • MiniMax-M2
    • MiniMax-M3
  • Mistral
    • Ministral 3
  • Xiaomi-MiMo
    • Xiaomi-MiMo
  • Moonlight
    • Moonlight
  • Muse Glimmer 30B
  • Nemotron
    • Nemotron 3 Nano
    • Nemotron 3 Nano 4B
    • Nemotron-3 Nano Omni
    • Nemotron 3 Super
    • Nemotron 3 Ultra
    • Nemotron 3.5 Lightning
  • OLMoE
    • OLMoE
  • Qwen
    • Qwen3-235B-A22B
    • Qwen3-30B-A3B
    • Qwen3-8B
    • Qwen3-MoE
    • Qwen3-Next
    • Qwen2.5-VL
    • Qwen3-VL
    • Qwen3.6-35B-A3B
    • Qwen3.8-27B
    • Qwen2-Audio
    • Qwen2.5-Omni
    • Qwen3-Omni
    • Qwen3-ASR
  • Sarvam
    • Sarvam
  • StepFun
    • Step-3.5-Flash

Training and Customization

  • Configuration Overview
  • Training Entry Points
  • Data Preparation
  • Training Loop Configuration
  • Optimizer and Scheduler Configuration
  • Logging and Monitoring
  • Profiling
  • Theoretical Memory Estimator
  • Checkpointing
  • Megatron FSDP
  • Resiliency
  • Mixed Precision Training
  • CUDA Graphs
  • Hierarchical Context Parallel
  • Dynamic Context Parallel
  • Communication Overlap
  • Attention Optimizations
  • Activation Recomputation
  • CPU Offloading
  • MoE Training Optimization
  • Parameter-Efficient Fine-Tuning (PEFT)
  • Packed Sequences
  • MegatronMIMO Intra-Microbatch Reordering
  • Multi-Token Prediction (MTP)
  • DSpark Speculative Decoding (Draft Training) Design
  • Callbacks

Model Optimization with ModelOpt

  • Quantization
  • Knowledge Distillation
  • Pruning

Development

  • Contribute a New Model to Megatron Bridge
  • Adapting Megatron Bridge in Reinforcement Learning Frameworks
  • Documentation Development
  • API Reference
    • bridge
      • bridge.data
        • bridge.data.sources
        • bridge.data.packing
        • bridge.data.datasets
        • bridge.data.energon
        • bridge.data.megatron_mimo
        • bridge.data.builders
        • bridge.data.collators
        • bridge.data.base
        • bridge.data.token_utils
        • bridge.data.utils
        • bridge.data.iterator_utils
        • bridge.data.conversation_processing
        • bridge.data.loaders
        • bridge.data.batch_utils
        • bridge.data.samplers
        • bridge.data.vlm_processing
        • bridge.data.sft_processing
      • bridge.diffusion
        • bridge.diffusion.data
        • bridge.diffusion.conversion
        • bridge.diffusion.base
        • bridge.diffusion.models
        • bridge.diffusion.common
        • bridge.diffusion.recipes
      • bridge.perf_recipes
        • bridge.perf_recipes.deepseek
        • bridge.perf_recipes.qwen_vl
        • bridge.perf_recipes.glm_moe_dsa
        • bridge.perf_recipes.muse_glimmer
        • bridge.perf_recipes.llama
        • bridge.perf_recipes.kimi
        • bridge.perf_recipes.gpt_oss
        • bridge.perf_recipes.wan
        • bridge.perf_recipes.qwen
        • bridge.perf_recipes.nemotronh
        • bridge.perf_recipes._common
        • bridge.perf_recipes.environment
      • bridge.models
        • bridge.models.deepseek
        • bridge.models.nemotron_vl
        • bridge.models.hf_pretrained
        • bridge.models.stepfun
        • bridge.models.diffusion_gemma
        • bridge.models.mistral
        • bridge.models.gpt
        • bridge.models.glm
        • bridge.models.qwen_vl
        • bridge.models.glm_vl
        • bridge.models.glm_moe_dsa
        • bridge.models.conversion
        • bridge.models.falcon_h1
        • bridge.models.mimo_v2_flash
        • bridge.models.mimo
        • bridge.models.muse_glimmer
        • bridge.models.minimax_m2
        • bridge.models.nemotron_omni
        • bridge.models.exaone
        • bridge.models.olmoe
        • bridge.models.gemma
        • bridge.models.qwen_audio
        • bridge.models.llama
        • bridge.models.mamba
        • bridge.models.qwen3_asr
        • bridge.models.qwen_omni
        • bridge.models.hybrid
        • bridge.models.ernie_vl
        • bridge.models.hy_v3
        • bridge.models.kimi
        • bridge.models.sarvam
        • bridge.models.common
        • bridge.models.gemma_vl
        • bridge.models.minimax_m3
        • bridge.models.gpt_oss
        • bridge.models.bagel
        • bridge.models.bailing
        • bridge.models.decorators
        • bridge.models.ministral3
        • bridge.models.qwen
        • bridge.models.ernie
        • bridge.models.megatron_mimo
        • bridge.models.bert
        • bridge.models.kimi_vl
        • bridge.models.nemotronh
        • bridge.models.config
        • bridge.models.gpt_provider
        • bridge.models.model_provider
        • bridge.models.gpt_full_te_layer_autocast_spec
        • bridge.models.transformer_config
        • bridge.models.t5_provider
        • bridge.models.mla_provider
        • bridge.models.logit_dtype
        • bridge.models.distillation_provider
      • bridge.recipes
        • bridge.recipes.deepseek
        • bridge.recipes.stepfun
        • bridge.recipes.diffusion_gemma
        • bridge.recipes.gemma3_vl
        • bridge.recipes.gpt
        • bridge.recipes.glm
        • bridge.recipes.moonlight
        • bridge.recipes.qwen_vl
        • bridge.recipes.flux
        • bridge.recipes.glm_vl
        • bridge.recipes.nemotronh_multimodal
        • bridge.recipes.gemma4_vl
        • bridge.recipes.muse_glimmer
        • bridge.recipes.qwen2_audio
        • bridge.recipes.minimax
        • bridge.recipes.nemotron_omni
        • bridge.recipes.exaone
        • bridge.recipes.olmoe
        • bridge.recipes.gemma
        • bridge.recipes.llama
        • bridge.recipes.qwen_omni
        • bridge.recipes.kimi
        • bridge.recipes.gpt_oss
        • bridge.recipes.bagel
        • bridge.recipes.ministral3
        • bridge.recipes.wan
        • bridge.recipes.qwen
        • bridge.recipes.utils
        • bridge.recipes.kimi_vl
        • bridge.recipes.nemotronh
        • bridge.recipes.common
        • bridge.recipes.run_plugins
      • bridge.training
        • bridge.training.post_training
        • bridge.training.mlm_compat
        • bridge.training.utils
        • bridge.training.tokenizers
        • bridge.training.eval_context_parallel_rebinding
        • bridge.training.config
        • bridge.training.nvrx_straggler
        • bridge.training.distill
        • bridge.training.megatron_mimo_step
        • bridge.training.train
        • bridge.training.initialize
        • bridge.training.megatron_mimo_parallel_utils
        • bridge.training.gtp
        • bridge.training.gpt_step
        • bridge.training.diffusion_gemma_step
        • bridge.training.eval
        • bridge.training.llava_step
        • bridge.training.forward_step_func_types
        • bridge.training.optim
        • bridge.training.audio_lm_step
        • bridge.training.losses
        • bridge.training.tensor_inspect
        • bridge.training.profiling
        • bridge.training.global_batch_packing
        • bridge.training.pretrain
        • bridge.training.state
        • bridge.training.model_load_save
        • bridge.training.flex_dispatcher_backend
        • bridge.training.callbacks
        • bridge.training.pretrain_megatron_mimo
        • bridge.training.setup
        • bridge.training.comm_overlap
        • bridge.training.vlm_step
        • bridge.training.nemotron_omni_step
        • bridge.training.inprocess_restart
        • bridge.training.train_megatron_mimo
        • bridge.training.checkpointing
        • bridge.training.setup_megatron_mimo
        • bridge.training.mixed_precision
        • bridge.training.fault_tolerance
        • bridge.training.finetune
      • bridge.utils
        • bridge.utils.instantiate_utils
        • bridge.utils.slurm_utils
        • bridge.utils.safe_pickle
        • bridge.utils.vocab_utils
        • bridge.utils.fusions
        • bridge.utils.gdn_utils
        • bridge.utils.activation_map
        • bridge.utils.yaml_utils
        • bridge.utils.safe_url
        • bridge.utils.cuda_graph
        • bridge.utils.import_utils
        • bridge.utils.decorators
        • bridge.utils.common_utils
      • bridge.peft
        • bridge.peft.base
        • bridge.peft.recompute
        • bridge.peft.adapter_wrapper
        • bridge.peft.multi_lora_layers
        • bridge.peft.lora_merge
        • bridge.peft.dora
        • bridge.peft.utils
        • bridge.peft.canonical_lora
        • bridge.peft.multi_lora
        • bridge.peft.lora_layers
        • bridge.peft.module_matcher
        • bridge.peft.lora
        • bridge.peft.walk_utils
        • bridge.peft.dora_layers
      • bridge.package_info

Releases

  • Release Developer Guide
  • Cherrypick Policy
  • Software Component Versions
  • Changelog
  • Known Issues

Agent Skills

  • Agent Skills Reference
    • Build and Dependency
    • Bump Dependency
    • Testing
    • CI/CD
    • Review PR
    • MLM vs Bridge Training
    • Auto Recipe — Recipe Index & Recommendation
    • Adding New Model Support in Megatron-Bridge
    • LLM Bridge Patterns
    • VLM Bridge Patterns
    • Recipe Patterns
    • Test and Example Patterns
    • Create Model Verification Card
    • verl E2E Testing
    • NeMo-RL E2E Testing
    • CPU Offloading
    • MoE Long-Context Training
    • MoE Training Optimization Workflow
    • Nsight Systems Analysis for Megatron Bridge
    • MoE VLM Training
    • Memory Tuning
    • MoE Dispatcher Selection Guide
    • MoE Hardware Configuration Reference
    • Parallelism Strategy Selection Skill
    • CUDA Graphs
    • TP / DP / PP Communication Overlap Skill
    • Megatron FSDP Skill
    • Sequence Packing Skill
    • Hierarchical Context Parallel Skill
    • Vision DP Over CP Skill
    • MoE Expert-Parallel Overlap Skill
    • MoE Communication Overlap
    • Activation Recompute
    • Recipe Index (Library & Benchmark)
    • Nsight Systems analysis pitfalls
    • Nsight Systems SQLite recipes
    • Multi-Node Slurm
    • Multi-Node Slurm Templates
    • Memory Snapshot Analysis
    • Resiliency

Directory Readme Files

  • Megatron Bridge Documentation
  • Releases
  • Training and Customization
  • Model Optimization
  • API Reference
  • bridge
  • bridge.models
  • bridge.models.bert

bridge.models.bert#

Submodules#

  • bridge.models.bert.bert_bridge
  • bridge.models.bert.bert_provider

Package Contents#

Data#

__all__

API#

bridge.models.bert.__all__#

[‘BertModelProvider’]

previous

bridge.models.megatron_mimo.build_model

next

bridge.models.bert.bert_bridge
On this page
  • Submodules
  • Package Contents
    • Data
    • API
      • __all__
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.