Skip to navigation

MegatronBridge

View as Markdown

This workload (test_template_name is MegatronBridge) submits training and finetuning tasks based on Megatron-Bridge framework.

Megatron-Bridge and NeMo Run generate the training script in dry-run mode on the submit host. CloudAI retains and parses their sbatch, then writes cloudai_sbatch_script.sh with its standard environment setup, metadata, rank mapping, and pre/post tests. These tests run inside the training allocation. Without the installed MBridge launcher or NeMo Run Python environment, cloudai dry-run produces a placeholder instead of the training commands and logs a warning. The placeholder exits with an error if executed. Install the dependencies and repeat the dry-run to generate the complete training script.

CloudAI owns the job name, output/error paths, and node placement. Configured CloudAI allocation directives override matching NeMo Run directives; other NeMo Run directives and training artifact paths are retained. Set scenario num_nodes and system GPU/task counts to match the training configuration. enable_vboost uses CloudAI’s standard VBoost setup in both normal and single-sbatch execution. An explicitly configured value takes precedence over the ENABLE_VBOOST environment setting. The rendezvous address uses NeMo’s selected head-node hostname, which must resolve inside the training containers.

Additional launcher options can be passed through [extra_cmd_args]. For example:

[extra_cmd_args]
"--custom_srun_args" = "--cpu-bind=none,--kill-on-bad-exit=1"

CloudAI combines -cs and --custom_srun_args from this table, then appends system and test-run extra_srun_args and pre-test-provided arguments. The result is passed through Megatron-Bridge’s native -cs option without deduplication.

For Nsight profiling, use Megatron-Bridge’s cmd_args.enable_nsys and profiling_start_step, profiling_stop_step, and profiling_ranks options. The generic CloudAI [nsys] configuration is not applied to this workload.

To enable native post-training evaluation with Megatron-Bridge v0.3.0 through v0.7.0, explicitly disable detach in the test configuration:

[extra_cmd_args]
"--detach" = "false"

This workload has a hard requirement for the HuggingFace Hub token. There are two options:

  • (recommended) define HF_TOKEN environment variable
  • set cmd_args.hf_token either in Test or Scenario config

Usage Examples

Test TOML example:

name = "megatron_bridge_qwen_30b"
description = "Megatron-Bridge run via CloudAI SlurmSystem for Qwen3 30B A3B"
test_template_name = "MegatronBridge"
[[git_repos]]
url = "https://github.com/NVIDIA-NeMo/Megatron-Bridge.git"
commit = "v0.3.0"
mount_as = "/opt/Megatron-Bridge"
[cmd_args]
gpu_type = "gb200"
gpus_per_node = 8
num_gpus = 8
# Container can be an NGC/enroot URL (nvcr.io#...) or a local .sqsh path.
container_image = "nvcr.io#nvidia/nemo:26.02.00"
model_family_name = "qwen"
model_recipe_name = "qwen3_30b_a3b"
task = "pretrain"
domain = "llm"
compute_dtype = "fp8_mx"

Test Scenario example:

name = "megatron_bridge_qwen_30b"
[[Tests]]
id = "megatron_bridge_qwen_30b"
test_name = "megatron_bridge_qwen_30b"
num_nodes = "2"

Test-in-Scenario example:

name = "megatron-bridge-test"
[[Tests]]
id = "mbridge.1"
num_nodes = 2
time_limit = "00:30:00"
name = "megatron_bridge_qwen_30b"
description = "Megatron-Bridge run via CloudAI SlurmSystem for Qwen3 30B A3B"
test_template_name = "MegatronBridge"
[[Tests.git_repos]]
url = "https://github.com/NVIDIA-NeMo/Megatron-Bridge.git"
commit = "v0.3.0"
mount_as = "/opt/Megatron-Bridge"
[Tests.cmd_args]
container_image = "nvcr.io#nvidia/nemo:26.02.01"
model_family_name = "qwen"
model_recipe_name = "qwen3_30b_a3b"
gpu_type = "gb200"
gpus_per_node = 8
num_gpus = 8
task = "pretrain"
domain = "llm"
compute_dtype = "fp8_mx"

Executor Environment Variables

CloudAI applies global_env_vars and extra_env_vars through its standard job environment setup. These values override matching exports in the generated NeMo Run script. CloudAI’s env_vars.sh is also sourced inside each training task, so configured values can reference task-local Slurm variables.

Existing cmd_args.executor_env_vars entries are also accepted as job environment variables:

[cmd_args.executor_env_vars]
GPU_METRICS_NODES = "0,1"

global_env_vars and extra_env_vars take precedence over these entries. This does not require Megatron-Bridge’s -E/--env support.

Chakra / Kineto Tracing

Disabling nsys

The existing MegatronBridge TOMLs in conf/staging/ have enable_nsys = true. To switch to Kineto/Chakra profiling, either:

  • Omit enable_nsys from the TOML (it defaults to false)
  • Or explicitly set enable_nsys = false

Both are equivalent. Do not set enable_nsys = true alongside Kineto flags — the two profilers conflict and Megatron-Bridge will raise an assertion error.

Enabling Kineto Trace

Add the following to [extra_cmd_args] in your Test TOML. These are Hydra-style overrides passed directly to the Megatron-Bridge training script:

[extra_cmd_args]
"profiling.use_pytorch_profiler" = "true"
"profiling.profile_step_start" = "45"
"profiling.profile_step_end" = "50"
"profiling.profile_ranks" = "[0]"

This produces the following under the NemoRun experiment directory (<cloudai_output>/experiments/<experiment_name>/<run_id>/<experiment_name>/):

  • torch_profile/rank-0.json.gz — Chrome-format kineto trace (always produced)
  • tb_logs/ — TensorBoard events

Enabling Chakra Trace

The Chakra trace is a PyTorch ExecutionTraceObserver recording — a separate file from the kineto trace. Add pytorch_profiler_collect_chakra = true:

[extra_cmd_args]
"profiling.use_pytorch_profiler" = "true"
"profiling.profile_step_start" = "45"
"profiling.profile_step_end" = "50"
"profiling.profile_ranks" = "[0]"
"profiling.pytorch_profiler_collect_chakra" = "true"

This produces an additional chakra/rank-0.json.gz in the same experiment directory. The kineto trace (torch_profile/rank-0.json.gz) is always produced regardless of this flag.

Reference TOML

A ready-to-use TOML is provided at:

conf/experimental/megatron_bridge/test/gb200/megatron_bridge_nemotron_nano_pytorch_profiler.toml

This runs Nemotron 3 Nano 3B (8 GPUs, 60 steps) with Kineto and Chakra profiling at steps 45–50 and is suitable for quickly validating the profiling setup.

Profiling Parameter Reference

ParameterTypeDescription
profiling.use_pytorch_profilerboolEnable PyTorch kineto profiler. Required for all Kineto/Chakra tracing.
profiling.profile_step_startintTraining step at which profiling begins.
profiling.profile_step_endintTraining step at which profiling ends and trace is exported.
profiling.profile_rankslist[int]GPU ranks to profile (e.g. [0] for rank 0 only).
profiling.record_shapesboolRecord tensor shapes in the kineto trace. Expensive — off by default.
profiling.record_memory_historyboolRecord memory allocation history (produces memory_profile.pickle). Expensive — off by default.
profiling.memory_snapshot_pathstrFilename for the memory snapshot pickle file.
profiling.pytorch_profiler_collect_chakraboolCollect Chakra ExecutionTraceObserver trace (chakra/rank-N.json.gz). The kineto trace is always produced; this adds an additional file.
profiling.pytorch_profiler_collect_callstackboolInclude Python call stack in the kineto trace (increases trace size).
enable_nsysboolEnable Nsight Systems profiling. Cannot be used with Kineto. Default: false.

API Documentation

Command Arguments

MegatronBridgeCmdArgs

Megatron-Bridge launcher arguments (translated into setup_experiment.py flags).

FieldTypeDefaultDescription
gpu_typestr'gb200'
log_dirstr''
container_imagestr''
num_gpusint8
enable_vboostbool | NoneFalse
dryrunbool | NoneFalse
enable_nsysbool | NoneFalse
executor_env_varsdict[str, str]Factory: dictAdditional job environment variables. CloudAI global_env_vars and extra_env_vars take precedence.
domainstr | NoneNoneDomain: llm, vlm, or qwen3vl (default llm).
hidden_sizeint | NoneNoneOverride hidden size for experiment.
num_layersint | NoneNoneOverride number of layers.
pipeline_model_parallel_layoutstr | NoneNone
first_k_dense_replaceint | NoneNoneNumber of MoE layers to convert to dense.
model_family_namestr''
model_recipe_namestr''
use_recipesbool | NoneNone
taskstr'pretrain'
compute_dtypestr'bf16'
fp8_recipestr | NoneNone
hf_tokenstr''
nemo_homestr | NoneNone
wandb_keystr | NoneNone
wandb_project_namestr | NoneNone
wandb_entity_namestr | NoneNone
wandb_experiment_namestr | NoneNone
wandb_save_dirstr | NoneNone
wandb_versionstr'0.28.1'W&B version installed in the launcher environment.
numpy_versionstr'1.26.4'NumPy version installed in the launcher environment.
nemorun_versionstr'v0.10.0'NeMo Run Git commit, tag, or branch used by the launcher environment.
max_retriesint | None1
use_tokendropbool | List[bool] | NoneNone
use_megatron_fsdpbool | List[bool] | NoneNone
cuda_graph_implstr | List[str] | NoneNone
cuda_graph_scopestr | List[str] | NoneNone
tpint | List[int] | NoneNone
ppint | List[int] | NoneNone
cpint | List[int] | NoneNone
vpint | List[int] | NoneNone
epint | List[int] | NoneNone
etint | List[int] | NoneNone
mbint | List[int] | NoneNone
gbint | List[int] | NoneNone
seq_lengthint | List[int] | NoneNone
lrfloat | List[float] | NoneNone
min_lrfloat | List[float] | NoneNone
warmup_itersint | List[int] | NoneNone
pretrained_checkpointstr | NoneNone
save_dirstr | NoneNone
load_dirstr | NoneNone
save_intervalint | NoneNone
most_recent_kint | NoneNone
save_config_filepathstr | None'/nemo_run/configs/ConfigContainer.yaml'
datastr | NoneNone
dataset_pathsstr | List[str] | NoneNone
dataset_rootstr | NoneNone
index_mapping_dirstr | NoneNone
dataset_namestr | NoneNone
packed_sequencebool | NoneNone
head_onlybool | NoneNone
tokenizer_typestr | NoneNone
tokenizer_modelstr | NoneNone
vocab_sizeint | NoneNone
pytorch_profilerbool | NoneNone
profiling_start_stepint | NoneNone
profiling_stop_stepint | NoneNone
record_memory_historybool | NoneNone
profiling_gpu_metricsbool | NoneNone
profiling_ranksint | str | List[int] | NoneNoneRank ID, comma-separated rank IDs as a string (e.g. ‘0,4,8’), or list of rank IDs.
nsys_tracestr | List[str] | NoneNoneComma-separated nsys trace events (e.g. cuda,nvtx).
nsys_extra_argsstr | List[str] | NoneNoneComma-separated extra arguments for nsys.
nccl_ubbool | List[bool] | NoneNone
moe_a2a_overlapbool | List[bool] | NoneNone
moe_flex_dispatcher_backendstr | List[str] | NoneNoneMoE flex dispatcher backend: deepep, hybridep, ncclep, or the literal string ‘None’ to request the alltoall dispatcher. Omit the field entirely to keep the recipe’s backend.
max_stepsint | None10
recompute_num_layersint | List[int] | NoneNone
activation_offload_layersint | List[int] | NoneNone
recompute_modulesstr | List[str] | NoneNone
num_distributed_optimizer_instancesint | NoneNone
config_variantstr | NoneNoneConfig variant (e.g. v1, v2). Launcher default is v2.
list_config_variantsbool | NoneNoneIf true, list config variants and interactively select (—list_config_variants).

Test Definition

MegatronBridgeTestDefinition

Megatron-Bridge test definition (CloudAI-managed install + Slurm submission via launcher).

FieldTypeDefaultDescription
namestrRequired
descriptionstrRequired
test_template_namestrRequired
cmd_argscloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeCmdArgsRequired
dse_excluded_argslist[str]Factory: list
extra_env_varsdict[str, str | typing.List[str]]{}
extra_cmd_argsdict[str, str]{}
extra_container_mountslist[str][]
git_reposlist[cloudai._core.installables.git_repo.GitRepo][]
nsyscloudai.models.workload.NsysConfiguration | NoneNone
predictorcloudai.models.workload.PredictorConfig | NoneNone
training_reportcloudai.models.workload.TrainingReportConfig | NoneNone
agentstr'grid_search'
agent_stepsint1
agent_metricslist[str]['default']
agent_reward_functionstr'inverse'
agent_configdict[str, Any] | NoneNoneAgent configuration.
env_paramsdict[str, cloudai.configurator.env_params.EnvParamSpec]Factory: dictEnvironment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key.