> This page is for version 1.8.1 (default).
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

This workload (test_template_name is `MegatronBridge`) submits training and finetuning tasks based on Megatron-Bridge framework.

Note

This workload has a hard requirement for the HuggingFace Hub token. There are two options:

* (recommended) define `HF_TOKEN` environment variable

* set `cmd_args.hf_token` either in Test or Scenario config

## Usage Examples

Test TOML example:

```toml
name = "megatron_bridge_qwen_30b"
description = "Megatron-Bridge run via CloudAI SlurmSystem for Qwen3 30B A3B"
test_template_name = "MegatronBridge"

[[git_repos]]
url = "https://github.com/NVIDIA-NeMo/Megatron-Bridge.git"
commit = "v0.3.0"
mount_as = "/opt/Megatron-Bridge"

[cmd_args]
gpu_type = "gb200"
gpus_per_node = 8
num_gpus = 8
# Container can be an NGC/enroot URL (nvcr.io#...) or a local .sqsh path.
container_image = "nvcr.io#nvidia/nemo:26.02.00"

model_family_name = "qwen"
model_recipe_name = "qwen3_30b_a3b"
task = "pretrain"
domain = "llm"
compute_dtype = "fp8_mx"
```

Test Scenario example:

```toml
name = "megatron_bridge_qwen_30b"

[[Tests]]
id = "megatron_bridge_qwen_30b"
test_name = "megatron_bridge_qwen_30b"
num_nodes = "2"
```

Test-in-Scenario example:

```toml
name = "megatron-bridge-test"

[[Tests]]
id = "mbridge.1"
num_nodes = 2
time_limit = "00:30:00"

name = "megatron_bridge_qwen_30b"
description = "Megatron-Bridge run via CloudAI SlurmSystem for Qwen3 30B A3B"
test_template_name = "MegatronBridge"

  [[Tests.git_repos]]
  url = "https://github.com/NVIDIA-NeMo/Megatron-Bridge.git"
  commit = "v0.3.0"
  mount_as = "/opt/Megatron-Bridge"

  [Tests.cmd_args]
  container_image = "nvcr.io#nvidia/nemo:26.02.01"
  model_family_name = "qwen"
  model_recipe_name = "qwen3_30b_a3b"

  gpu_type = "gb200"
  gpus_per_node = 8
  num_gpus = 8

  task = "pretrain"
  domain = "llm"
  compute_dtype = "fp8_mx"
```

## Executor Environment Variables

By default, CloudAI forwards `extra_env_vars` as `-cb`` ``export`` ``KEY=VALUE` commands. This preserves compatibility with Megatron-Bridge releases such as v0.3.0 that do not support `-E/--env` and allows values to reference job-shell variables such as `$SLURM_PROCID`.

For a newer Megatron-Bridge version, variables that must be registered in the NeMo-Run executor’s `env_vars` and `container_env` can be explicitly set under `cmd_args.executor_env_vars`:

```toml
[cmd_args.executor_env_vars]
GPU_METRICS_NODES = "0,1"
```

Each entry is passed to Megatron-Bridge as a repeated `-E`` ``KEY=VALUE` argument. Values are forwarded literally and may contain commas. Do not use this option with a Megatron-Bridge version that predates `-E/--env`.

## Chakra / Kineto Tracing

### Disabling nsys

The existing MegatronBridge TOMLs in `conf/staging/` have `enable_nsys`` ``=`` ``true`. To switch to Kineto/Chakra profiling, either:

* **Omit** `enable_nsys` from the TOML (it defaults to `false`)

* **Or** explicitly set `enable_nsys`` ``=`` ``false`

Both are equivalent. Do not set `enable_nsys`` ``=`` ``true` alongside Kineto flags — the two profilers conflict and Megatron-Bridge will raise an assertion error.

### Enabling Kineto Trace

Add the following to `[extra_cmd_args]` in your Test TOML. These are Hydra-style overrides passed directly to the Megatron-Bridge training script:

```toml
[extra_cmd_args]
"profiling.use_pytorch_profiler" = "true"
"profiling.profile_step_start" = "45"
"profiling.profile_step_end" = "50"
"profiling.profile_ranks" = "[0]"
```

This produces the following under the NemoRun experiment directory (`<cloudai_output>/experiments/<experiment_name>/<run_id>/<experiment_name>/`):

* `torch_profile/rank-0.json.gz` — Chrome-format kineto trace (always produced)

* `tb_logs/` — TensorBoard events

### Enabling Chakra Trace

The Chakra trace is a PyTorch `ExecutionTraceObserver` recording — a separate file from the kineto trace. Add `pytorch_profiler_collect_chakra`` ``=`` ``true`:

```toml
[extra_cmd_args]
"profiling.use_pytorch_profiler" = "true"
"profiling.profile_step_start" = "45"
"profiling.profile_step_end" = "50"
"profiling.profile_ranks" = "[0]"
"profiling.pytorch_profiler_collect_chakra" = "true"
```

This produces an additional `chakra/rank-0.json.gz` in the same experiment directory. The kineto trace (`torch_profile/rank-0.json.gz`) is always produced regardless of this flag.

### Reference TOML

A ready-to-use TOML is provided at:

```text
conf/experimental/megatron_bridge/test/gb200/megatron_bridge_nemotron_nano_pytorch_profiler.toml
```

This runs Nemotron 3 Nano 3B (8 GPUs, 60 steps) with Kineto and Chakra profiling at steps 45–50 and is suitable for quickly validating the profiling setup.

### Profiling Parameter Reference

| Parameter                                      | Type       | Description                                                                                                                               |
| ---------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `profiling.use_pytorch_profiler`               | bool       | Enable PyTorch kineto profiler. Required for all Kineto/Chakra tracing.                                                                   |
| `profiling.profile_step_start`                 | int        | Training step at which profiling begins.                                                                                                  |
| `profiling.profile_step_end`                   | int        | Training step at which profiling ends and trace is exported.                                                                              |
| `profiling.profile_ranks`                      | list\[int] | GPU ranks to profile (e.g. `[0]` for rank 0 only).                                                                                        |
| `profiling.record_shapes`                      | bool       | Record tensor shapes in the kineto trace. Expensive — off by default.                                                                     |
| `profiling.record_memory_history`              | bool       | Record memory allocation history (produces `memory_profile.pickle`). Expensive — off by default.                                          |
| `profiling.memory_snapshot_path`               | str        | Filename for the memory snapshot pickle file.                                                                                             |
| `profiling.pytorch_profiler_collect_chakra`    | bool       | Collect Chakra ExecutionTraceObserver trace (`chakra/rank-N.json.gz`). The kineto trace is always produced; this adds an additional file. |
| `profiling.pytorch_profiler_collect_callstack` | bool       | Include Python call stack in the kineto trace (increases trace size).                                                                     |
| `enable_nsys`                                  | bool       | Enable Nsight Systems profiling. **Cannot be used with Kineto.** Default: `false`.                                                        |

## API Documentation

### Command Arguments

*class* cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeCmdArgs(

*\**,

*gpu_type: str = 'gb200'*,

*log_dir: str = ''*,

*container_image: str = ''*,

*num_gpus: int = 8*,

*enable_vboost: bool | None = False*,

*dryrun: bool | None = False*,

*enable_nsys: bool | None = False*,

*executor_env_vars: dict\[str*,

*str] = \<factory>*,

*domain: str | None = None*,

*hidden_size: int | None = None*,

*num_layers: int | None = None*,

*pipeline_model_parallel_layout: str | None = None*,

*first_k_dense_replace: int | None = None*,

*model_family_name: str = ''*,

*model_recipe_name: str = ''*,

*use_recipes: bool | None = None*,

*task: str = 'pretrain'*,

*compute_dtype: str = 'bf16'*,

*fp8_recipe: str | None = None*,

*hf_token: str = ''*,

*nemo_home: str | None = None*,

*wandb_key: str | None = None*,

*wandb_project_name: str | None = None*,

*wandb_entity_name: str | None = None*,

*wandb_experiment_name: str | None = None*,

*wandb_save_dir: str | None = None*,

*wandb_version: str = '0.28.1'*,

*numpy_version: str = '1.26.4'*,

*nemorun_version: str = 'v0.10.0'*,

*max_retries: int | None = 1*,

*use_tokendrop: bool | \~typing.List\[bool] | None = None*,

*use_megatron_fsdp: bool | \~typing.List\[bool] | None = None*,

*cuda_graph_impl: str | \~typing.List\[str] | None = None*,

*cuda_graph_scope: str | \~typing.List\[str] | None = None*,

*tp: int | \~typing.List\[int] | None = None*,

*pp: int | \~typing.List\[int] | None = None*,

*cp: int | \~typing.List\[int] | None = None*,

*vp: int | \~typing.List\[int] | None = None*,

*ep: int | \~typing.List\[int] | None = None*,

*et: int | \~typing.List\[int] | None = None*,

*mb: int | \~typing.List\[int] | None = None*,

*gb: int | \~typing.List\[int] | None = None*,

*seq_length: int | \~typing.List\[int] | None = None*,

*lr: float | \~typing.List\[float] | None = None*,

*min_lr: float | \~typing.List\[float] | None = None*,

*warmup_iters: int | \~typing.List\[int] | None = None*,

*pretrained_checkpoint: str | None = None*,

*save_dir: str | None = None*,

*load_dir: str | None = None*,

*save_interval: int | None = None*,

*most_recent_k: int | None = None*,

*save_config_filepath: str | None = '/nemo_run/configs/ConfigContainer.yaml'*,

*data: str | None = None*,

*dataset_paths: str | \~typing.List\[str] | None = None*,

*dataset_root: str | None = None*,

*index_mapping_dir: str | None = None*,

*dataset_name: str | None = None*,

*packed_sequence: bool | None = None*,

*head_only: bool | None = None*,

*tokenizer_type: str | None = None*,

*tokenizer_model: str | None = None*,

*vocab_size: int | None = None*,

*pytorch_profiler: bool | None = None*,

*profiling_start_step: int | None = None*,

*profiling_stop_step: int | None = None*,

*record_memory_history: bool | None = None*,

*profiling_gpu_metrics: bool | None = None*,

*profiling_ranks: int | str | \~typing.List\[int] | None = None*,

*nsys_trace: str | \~typing.List\[str] | None = None*,

*nsys_extra_args: str | \~typing.List\[str] | None = None*,

*nccl_ub: bool | \~typing.List\[bool] | None = None*,

*moe_a2a_overlap: bool | \~typing.List\[bool] | None = None*,

*moe_flex_dispatcher_backend: str | \~typing.List\[str] | None = None*,

*max_steps: int | None = 10*,

*recompute_num_layers: int | \~typing.List\[int] | None = None*,

*activation_offload_layers: int | \~typing.List\[int] | None = None*,

*recompute_modules: str | \~typing.List\[str] | None = None*,

*num_distributed_optimizer_instances: int | None = None*,

*config_variant: str | None = None*,

*list_config_variants: bool | None = None*,

*\*\*extra_data: \~typing.Any*,

)

Bases: `CmdArgs`

Megatron-Bridge launcher arguments (translated into setup_experiment.py flags).

### Test Definition

*class* cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeTestDefinition(*\*, name: str, description: str, test_template_name: str, cmd_args: \~cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeCmdArgs, dse_excluded_args: list\[str] = \<factory>, extra_env_vars: dict\[str, str | \~typing.List\[str]] = \{}, extra_cmd_args: dict\[str, str] = \{}, extra_container_mounts: list\[str] = \[], git_repos: list\[\~cloudai.\_core.installables.git_repo.GitRepo] = \[], nsys: \~cloudai.models.workload.NsysConfiguration | None = None, predictor: \~cloudai.models.workload.PredictorConfig | None = None, training_report: \~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list\[str] = \['default'], agent_reward_function: str = 'inverse', agent_config: dict\[str, \~typing.Any] | None = None, env_params: dict\[str, \~cloudai.configurator.env_params.EnvParamSpec] = \<factory>*)

Bases: `TestDefinition`

Megatron-Bridge test definition (CloudAI-managed install + Slurm submission via launcher).

**&#x20;*property* nemo_run_repo*: GitRepo*\*\*

Return the NeMo Run repository selected by `cmd_args.nemorun_version`.

**&#x20;*validator* validate_git_repos_has_megatron_bridge_repo*  » \* *git_repos**\*

MegatronBridge requires users to pin the Megatron-Bridge repo version via \[\[git_repos]].

was_run_successful(

*tr: TestRun*,

) → JobStatusResult

Ensure that the MBridge script finished correctly.

**The current state of Megatron-Bridge performance scripts makes us running their tool until the very specific**

failure. Right before the failure the M-Bridge script saves output metrics as JSON.

* At the point of failure the script asks for reference golden values, that we don’t have

* Then the script will perform convergence test between provided golden and actual golden - we don’t need it

**&#x20;*property* is_domain_randomization_enabled*: bool*\*\*

at least one `env_params` annotation.

**Type:**

Whether the config declares domain randomization

**&#x20;is_dse_excluded_arg(*path: str*) → bool**

Return whether a dot-separated cmd_args path should be ignored by DSE.

is_env_sampled(

*cmd_args_path: str*,

) → bool

Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

metric_observations(

*system: System*,

*tr: TestRun*,

) → list\[MetricObservation]

Return canonical metric observations produced by this test run, if supported.

**&#x20;*validator* validate_env_params*  » \* *all fields**\*

Validate env_params annotations against cmd_args.

`env_params` is an annotation: each key names a `cmd_args` field whose value is the candidate set (the single source of truth), and the entry carries only *how* to sample. So each key must name a real `cmd_args` field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When `weights` are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in `CloudAIGymEnv`; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.