MegatronBridge
This workload (test_template_name is MegatronBridge) submits training and finetuning tasks based on Megatron-Bridge framework.
Note
This workload has a hard requirement for the HuggingFace Hub token. There are two options:
-
(recommended) define
HF_TOKENenvironment variable -
set
cmd_args.hf_tokeneither in Test or Scenario config
Usage Examples
Test TOML example:
Test Scenario example:
Test-in-Scenario example:
Executor Environment Variables
By default, CloudAI forwards extra_env_vars as -cb`` ``export`` ``KEY=VALUE commands. This preserves compatibility with Megatron-Bridge releases such as v0.3.0 that do not support -E/--env and allows values to reference job-shell variables such as $SLURM_PROCID.
For a newer Megatron-Bridge version, variables that must be registered in the NeMo-Run executor’s env_vars and container_env can be explicitly set under cmd_args.executor_env_vars:
Each entry is passed to Megatron-Bridge as a repeated -E`` ``KEY=VALUE argument. Values are forwarded literally and may contain commas. Do not use this option with a Megatron-Bridge version that predates -E/--env.
Chakra / Kineto Tracing
Disabling nsys
The existing MegatronBridge TOMLs in conf/staging/ have enable_nsys`` ``=`` ``true. To switch to Kineto/Chakra profiling, either:
-
Omit
enable_nsysfrom the TOML (it defaults tofalse) -
Or explicitly set
enable_nsys`` ``=`` ``false
Both are equivalent. Do not set enable_nsys`` ``=`` ``true alongside Kineto flags — the two profilers conflict and Megatron-Bridge will raise an assertion error.
Enabling Kineto Trace
Add the following to [extra_cmd_args] in your Test TOML. These are Hydra-style overrides passed directly to the Megatron-Bridge training script:
This produces the following under the NemoRun experiment directory (<cloudai_output>/experiments/<experiment_name>/<run_id>/<experiment_name>/):
-
torch_profile/rank-0.json.gz— Chrome-format kineto trace (always produced) -
tb_logs/— TensorBoard events
Enabling Chakra Trace
The Chakra trace is a PyTorch ExecutionTraceObserver recording — a separate file from the kineto trace. Add pytorch_profiler_collect_chakra`` ``=`` ``true:
This produces an additional chakra/rank-0.json.gz in the same experiment directory. The kineto trace (torch_profile/rank-0.json.gz) is always produced regardless of this flag.
Reference TOML
A ready-to-use TOML is provided at:
This runs Nemotron 3 Nano 3B (8 GPUs, 60 steps) with Kineto and Chakra profiling at steps 45–50 and is suitable for quickly validating the profiling setup.
Profiling Parameter Reference
API Documentation
Command Arguments
class cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeCmdArgs(
*,
gpu_type: str = ‘gb200’,
log_dir: str = ”,
container_image: str = ”,
num_gpus: int = 8,
enable_vboost: bool | None = False,
dryrun: bool | None = False,
enable_nsys: bool | None = False,
executor_env_vars: dict[str,
str] = <factory>,
domain: str | None = None,
hidden_size: int | None = None,
num_layers: int | None = None,
pipeline_model_parallel_layout: str | None = None,
first_k_dense_replace: int | None = None,
model_family_name: str = ”,
model_recipe_name: str = ”,
use_recipes: bool | None = None,
task: str = ‘pretrain’,
compute_dtype: str = ‘bf16’,
fp8_recipe: str | None = None,
hf_token: str = ”,
nemo_home: str | None = None,
wandb_key: str | None = None,
wandb_project_name: str | None = None,
wandb_entity_name: str | None = None,
wandb_experiment_name: str | None = None,
wandb_save_dir: str | None = None,
wandb_version: str = ‘0.28.1’,
numpy_version: str = ‘1.26.4’,
nemorun_version: str = ‘v0.10.0’,
max_retries: int | None = 1,
use_tokendrop: bool | ~typing.List[bool] | None = None,
use_megatron_fsdp: bool | ~typing.List[bool] | None = None,
cuda_graph_impl: str | ~typing.List[str] | None = None,
cuda_graph_scope: str | ~typing.List[str] | None = None,
tp: int | ~typing.List[int] | None = None,
pp: int | ~typing.List[int] | None = None,
cp: int | ~typing.List[int] | None = None,
vp: int | ~typing.List[int] | None = None,
ep: int | ~typing.List[int] | None = None,
et: int | ~typing.List[int] | None = None,
mb: int | ~typing.List[int] | None = None,
gb: int | ~typing.List[int] | None = None,
seq_length: int | ~typing.List[int] | None = None,
lr: float | ~typing.List[float] | None = None,
min_lr: float | ~typing.List[float] | None = None,
warmup_iters: int | ~typing.List[int] | None = None,
pretrained_checkpoint: str | None = None,
save_dir: str | None = None,
load_dir: str | None = None,
save_interval: int | None = None,
most_recent_k: int | None = None,
save_config_filepath: str | None = ‘/nemo_run/configs/ConfigContainer.yaml’,
data: str | None = None,
dataset_paths: str | ~typing.List[str] | None = None,
dataset_root: str | None = None,
index_mapping_dir: str | None = None,
dataset_name: str | None = None,
packed_sequence: bool | None = None,
head_only: bool | None = None,
tokenizer_type: str | None = None,
tokenizer_model: str | None = None,
vocab_size: int | None = None,
pytorch_profiler: bool | None = None,
profiling_start_step: int | None = None,
profiling_stop_step: int | None = None,
record_memory_history: bool | None = None,
profiling_gpu_metrics: bool | None = None,
profiling_ranks: int | str | ~typing.List[int] | None = None,
nsys_trace: str | ~typing.List[str] | None = None,
nsys_extra_args: str | ~typing.List[str] | None = None,
nccl_ub: bool | ~typing.List[bool] | None = None,
moe_a2a_overlap: bool | ~typing.List[bool] | None = None,
moe_flex_dispatcher_backend: str | ~typing.List[str] | None = None,
max_steps: int | None = 10,
recompute_num_layers: int | ~typing.List[int] | None = None,
activation_offload_layers: int | ~typing.List[int] | None = None,
recompute_modules: str | ~typing.List[str] | None = None,
num_distributed_optimizer_instances: int | None = None,
config_variant: str | None = None,
list_config_variants: bool | None = None,
**extra_data: ~typing.Any,
)
Bases: CmdArgs
Megatron-Bridge launcher arguments (translated into setup_experiment.py flags).
Test Definition
class cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeCmdArgs, dse_excluded_args: list[str] = <factory>, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.git_repo.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, training_report: ~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = ‘grid_search’, agent_steps: int = 1, agent_metrics: list[str] = [‘default’], agent_reward_function: str = ‘inverse’, agent_config: dict[str, ~typing.Any] | None = None, env_params: dict[str, ~cloudai.configurator.env_params.EnvParamSpec] = <factory>)
Bases: TestDefinition
Megatron-Bridge test definition (CloudAI-managed install + Slurm submission via launcher).
property nemo_run_repo: GitRepo**
Return the NeMo Run repository selected by cmd_args.nemorun_version.
validator validate_git_repos_has_megatron_bridge_repo » * git_repos*
MegatronBridge requires users to pin the Megatron-Bridge repo version via [[git_repos]].
was_run_successful(
tr: TestRun,
) → JobStatusResult
Ensure that the MBridge script finished correctly.
The current state of Megatron-Bridge performance scripts makes us running their tool until the very specific
failure. Right before the failure the M-Bridge script saves output metrics as JSON.
-
At the point of failure the script asks for reference golden values, that we don’t have
-
Then the script will perform convergence test between provided golden and actual golden - we don’t need it
property is_domain_randomization_enabled: bool**
at least one env_params annotation.
Type:
Whether the config declares domain randomization
is_dse_excluded_arg(path: str) → bool
Return whether a dot-separated cmd_args path should be ignored by DSE.
is_env_sampled(
cmd_args_path: str,
) → bool
Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).
metric_observations(
system: System,
tr: TestRun,
) → list[MetricObservation]
Return canonical metric observations produced by this test run, if supported.
validator validate_env_params » * all fields*
Validate env_params annotations against cmd_args.
env_params is an annotation: each key names a cmd_args field whose value is the candidate set (the single source of truth), and the entry carries only how to sample. So each key must name a real cmd_args field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When weights are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in CloudAIGymEnv; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.