title: moe_comm_overlap
validated_on: "2026-07-25"
summary: >
  Megatron-Bridge supports MoE expert-parallel communication overlap through
  overlap_moe_expert_parallel_comm, with optional delayed expert wgrad
  scheduling. A controlled Qwen3 30B-A3B HybridEP run on 16 H100 GPUs measured
  a 15.059% step-time reduction and 17.729% throughput increase from plain EP
  overlap with delayed wgrad disabled.
validation_status:
  moe_overlap_validation:
    - code_verified
  flex_dispatcher_activation:
    - code_verified
  deepep_hybridep_helper_behavior:
    - code_verified
  end_to_end_recipe_smoke:
    - measured
  qwen3_30b_hybridep_h100:
    - measured
  matched_nsys_ab:
    - measured
feature_meaning:
  moe_overlap: >
    Overlap of expert-parallel token dispatch communication with expert compute.
  delay_wgrad_compute: >
    Delayed expert weight-gradient scheduling layered on top of MoE overlap.
  flex_dispatcher: >
    Dispatcher mode used for DeepEP or HybridEP style backends.
recommended_path:
  comm_overlap.overlap_moe_expert_parallel_comm: true_for_moe_tuning
  comm_overlap.delay_wgrad_compute: false_for_first_isolation
  model.moe_shared_expert_overlap: false_when_overlap_is_enabled
measured_result:
  model: Qwen3 30B-A3B
  hardware: "16×H100"
  precision: BF16
  shape: "seq4096 TP1 PP1 CP1 EP16 MBS1 GBS1024"
  dispatcher: HybridEP
  routing: force_balance
  cuda_graph_scopes: "moe_router, moe_preprocess"
  baseline_step_time_ms: 24713.800
  baseline_model_tflops_per_gpu: 244.039
  overlap_step_time_ms: 20992.030
  overlap_model_tflops_per_gpu: 287.305
  step_time_reduction_percent: 15.059
  throughput_gain_percent: 17.729
  rank0_peak_allocated_gib: 62.166
  skipped_iterations: 0
  nan_iterations: 0
profile_result:
  captured_kernels_per_case: 463348
  comm_compute_overlap_off_ms: 9.079
  comm_compute_overlap_on_ms: 3958.997
  comm_hidden_off_percent: 0.11
  comm_hidden_on_percent: 36.55
  gpu_active_union_off_ms: 22820.667
  gpu_active_union_on_ms: 21221.363
known_constraints:
  - expert_model_parallel_size must be greater than 1.
  - num_moe_experts must be greater than 1.
  - moe_token_dispatcher_type must be alltoall or flex.
  - Precision must be BF16 or FP16.
  - If pipeline parallelism is used, virtual pipeline parallelism is required for the overlap path.
known_limitations:
  - Setting moe_flex_dispatcher_backend alone does not activate flex dispatch.
  - Public recipes are often conservative and leave MoE overlap disabled by default.
  - The measured gain is workload-specific and should not be generalized across model, dispatcher, topology, or batch changes.
  - Summed per-stream kernel time is not a wall-time metric when kernels overlap.
evidence:
  - docs/training/communication-overlap.md
  - examples/model_verification_cards/qwen3-30b-a3b/card.yaml
  - src/megatron/bridge/perf_recipes/qwen/h100/qwen3_moe.py
  - docs/parallelisms.md
  - src/megatron/bridge/training/comm_overlap.py
  - src/megatron/bridge/training/flex_dispatcher_backend.py
  - src/megatron/bridge/training/config.py
  - tests/unit_tests/training/test_comm_overlap.py
  - tests/unit_tests/training/test_deepep.py
follow_up_validation:
  - Add a positive Bridge functional smoke for overlap_moe_expert_parallel_comm.
  - Repeat the matched profile method on another model family and topology.
