> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

This workload (`test_template_name` is `MegatronBridge`) submits training and finetuning tasks based on Megatron-Bridge framework.

Megatron-Bridge and NeMo Run generate the training script in dry-run mode on the submit host. CloudAI retains and parses their sbatch, then writes `cloudai_sbatch_script.sh` with its standard environment setup, metadata, rank mapping, and pre/post tests. These tests run inside the training allocation. Without the installed MBridge launcher or NeMo Run Python environment, `cloudai dry-run` produces a placeholder instead of the training commands and logs a warning. The placeholder exits with an error if executed. Install the dependencies and repeat the dry-run to generate the complete training script.

CloudAI owns the job name, output/error paths, and node placement. Configured CloudAI allocation directives override matching NeMo Run directives; other NeMo Run directives and training artifact paths are retained. Set scenario `num_nodes` and system GPU/task counts to match the training configuration. `enable_vboost` uses CloudAI's standard VBoost setup in both normal and single-sbatch execution. An explicitly configured value takes precedence over the `ENABLE_VBOOST` environment setting. The rendezvous address uses NeMo's selected head-node hostname, which must resolve inside the training containers.

Additional launcher options can be passed through `[extra_cmd_args]`. For example:

```toml
[extra_cmd_args]
"--custom_srun_args" = "--cpu-bind=none,--kill-on-bad-exit=1"
```

CloudAI combines `-cs` and `--custom_srun_args` from this table, then appends system and test-run `extra_srun_args` and pre-test-provided arguments. The result is passed through Megatron-Bridge's native `-cs` option without deduplication.

For Nsight profiling, use Megatron-Bridge's `cmd_args.enable_nsys` and `profiling_start_step`, `profiling_stop_step`, and `profiling_ranks` options. The generic CloudAI `[nsys]` configuration is not applied to this workload.

To enable native post-training evaluation with Megatron-Bridge v0.3.0 through v0.7.0, explicitly disable detach in the test configuration:

```toml
[extra_cmd_args]
"--detach" = "false"
```

> **Note**
>
> This workload has a hard requirement for the HuggingFace Hub token. There are two options:
>
> * (recommended) define `HF_TOKEN` environment variable
> * set `cmd_args.hf_token` either in Test or Scenario config

## Usage Examples

Test TOML example:

```toml
name = "megatron_bridge_qwen_30b"
description = "Megatron-Bridge run via CloudAI SlurmSystem for Qwen3 30B A3B"
test_template_name = "MegatronBridge"

[[git_repos]]
url = "https://github.com/NVIDIA-NeMo/Megatron-Bridge.git"
commit = "v0.3.0"
mount_as = "/opt/Megatron-Bridge"

[cmd_args]
gpu_type = "gb200"
gpus_per_node = 8
num_gpus = 8
# Container can be an NGC/enroot URL (nvcr.io#...) or a local .sqsh path.
container_image = "nvcr.io#nvidia/nemo:26.02.00"

model_family_name = "qwen"
model_recipe_name = "qwen3_30b_a3b"
task = "pretrain"
domain = "llm"
compute_dtype = "fp8_mx"
```

Test Scenario example:

```toml
name = "megatron_bridge_qwen_30b"

[[Tests]]
id = "megatron_bridge_qwen_30b"
test_name = "megatron_bridge_qwen_30b"
num_nodes = "2"
```

Test-in-Scenario example:

```toml
name = "megatron-bridge-test"

[[Tests]]
id = "mbridge.1"
num_nodes = 2
time_limit = "00:30:00"

name = "megatron_bridge_qwen_30b"
description = "Megatron-Bridge run via CloudAI SlurmSystem for Qwen3 30B A3B"
test_template_name = "MegatronBridge"

  [[Tests.git_repos]]
  url = "https://github.com/NVIDIA-NeMo/Megatron-Bridge.git"
  commit = "v0.3.0"
  mount_as = "/opt/Megatron-Bridge"

  [Tests.cmd_args]
  container_image = "nvcr.io#nvidia/nemo:26.02.01"
  model_family_name = "qwen"
  model_recipe_name = "qwen3_30b_a3b"

  gpu_type = "gb200"
  gpus_per_node = 8
  num_gpus = 8

  task = "pretrain"
  domain = "llm"
  compute_dtype = "fp8_mx"
```

## Executor Environment Variables

CloudAI applies `global_env_vars` and `extra_env_vars` through its standard job environment setup. These values override matching exports in the generated NeMo Run script. CloudAI's `env_vars.sh` is also sourced inside each training task, so configured values can reference task-local Slurm variables.

Existing `cmd_args.executor_env_vars` entries are also accepted as job environment variables:

```toml
[cmd_args.executor_env_vars]
GPU_METRICS_NODES = "0,1"
```

`global_env_vars` and `extra_env_vars` take precedence over these entries. This does not require Megatron-Bridge's `-E/--env` support.

## Chakra / Kineto Tracing

### Disabling nsys

The existing MegatronBridge TOMLs in `conf/staging/` have `enable_nsys = true`. To switch to Kineto/Chakra profiling, either:

* **Omit** `enable_nsys` from the TOML (it defaults to `false`)
* **Or** explicitly set `enable_nsys = false`

Both are equivalent. Do not set `enable_nsys = true` alongside Kineto flags — the two profilers conflict and Megatron-Bridge will raise an assertion error.

### Enabling Kineto Trace

Add the following to `[extra_cmd_args]` in your Test TOML. These are Hydra-style overrides passed directly to the Megatron-Bridge training script:

```toml
[extra_cmd_args]
"profiling.use_pytorch_profiler" = "true"
"profiling.profile_step_start" = "45"
"profiling.profile_step_end" = "50"
"profiling.profile_ranks" = "[0]"
```

This produces the following under the NemoRun experiment directory (`<cloudai_output>/experiments/<experiment_name>/<run_id>/<experiment_name>/`):

* `torch_profile/rank-0.json.gz` — Chrome-format kineto trace (always produced)
* `tb_logs/` — TensorBoard events

### Enabling Chakra Trace

The Chakra trace is a PyTorch `ExecutionTraceObserver` recording — a separate file from the kineto trace. Add `pytorch_profiler_collect_chakra = true`:

```toml
[extra_cmd_args]
"profiling.use_pytorch_profiler" = "true"
"profiling.profile_step_start" = "45"
"profiling.profile_step_end" = "50"
"profiling.profile_ranks" = "[0]"
"profiling.pytorch_profiler_collect_chakra" = "true"
```

This produces an additional `chakra/rank-0.json.gz` in the same experiment directory. The kineto trace (`torch_profile/rank-0.json.gz`) is always produced regardless of this flag.

### Reference TOML

A ready-to-use TOML is provided at:

```text
conf/experimental/megatron_bridge/test/gb200/megatron_bridge_nemotron_nano_pytorch_profiler.toml
```

This runs Nemotron 3 Nano 3B (8 GPUs, 60 steps) with Kineto and Chakra profiling at steps 45–50 and is suitable for quickly validating the profiling setup.

### Profiling Parameter Reference

| Parameter                                      | Type       | Description                                                                                                                               |
| ---------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `profiling.use_pytorch_profiler`               | bool       | Enable PyTorch kineto profiler. Required for all Kineto/Chakra tracing.                                                                   |
| `profiling.profile_step_start`                 | int        | Training step at which profiling begins.                                                                                                  |
| `profiling.profile_step_end`                   | int        | Training step at which profiling ends and trace is exported.                                                                              |
| `profiling.profile_ranks`                      | list\[int] | GPU ranks to profile (e.g. `[0]` for rank 0 only).                                                                                        |
| `profiling.record_shapes`                      | bool       | Record tensor shapes in the kineto trace. Expensive — off by default.                                                                     |
| `profiling.record_memory_history`              | bool       | Record memory allocation history (produces `memory_profile.pickle`). Expensive — off by default.                                          |
| `profiling.memory_snapshot_path`               | str        | Filename for the memory snapshot pickle file.                                                                                             |
| `profiling.pytorch_profiler_collect_chakra`    | bool       | Collect Chakra ExecutionTraceObserver trace (`chakra/rank-N.json.gz`). The kineto trace is always produced; this adds an additional file. |
| `profiling.pytorch_profiler_collect_callstack` | bool       | Include Python call stack in the kineto trace (increases trace size).                                                                     |
| `enable_nsys`                                  | bool       | Enable Nsight Systems profiling. **Cannot be used with Kineto.** Default: `false`.                                                        |

## API Documentation

### Command Arguments

### `MegatronBridgeCmdArgs`

Megatron-Bridge launcher arguments (translated into `setup_experiment.py` flags).

| Field                                 | Type                              | Default                                    | Description                                                                                                                                                                   |
| ------------------------------------- | --------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gpu_type`                            | `str`                             | `'gb200'`                                  |                                                                                                                                                                               |
| `log_dir`                             | `str`                             | `''`                                       |                                                                                                                                                                               |
| `container_image`                     | `str`                             | `''`                                       |                                                                                                                                                                               |
| `num_gpus`                            | `int`                             | `8`                                        |                                                                                                                                                                               |
| `enable_vboost`                       | `bool \| None`                    | `False`                                    |                                                                                                                                                                               |
| `dryrun`                              | `bool \| None`                    | `False`                                    |                                                                                                                                                                               |
| `enable_nsys`                         | `bool \| None`                    | `False`                                    |                                                                                                                                                                               |
| `executor_env_vars`                   | `dict[str, str]`                  | Factory: `dict`                            | Additional job environment variables. CloudAI global_env_vars and extra_env_vars take precedence.                                                                             |
| `domain`                              | `str \| None`                     | `None`                                     | Domain: llm, vlm, or qwen3vl (default llm).                                                                                                                                   |
| `hidden_size`                         | `int \| None`                     | `None`                                     | Override hidden size for experiment.                                                                                                                                          |
| `num_layers`                          | `int \| None`                     | `None`                                     | Override number of layers.                                                                                                                                                    |
| `pipeline_model_parallel_layout`      | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `first_k_dense_replace`               | `int \| None`                     | `None`                                     | Number of MoE layers to convert to dense.                                                                                                                                     |
| `model_family_name`                   | `str`                             | `''`                                       |                                                                                                                                                                               |
| `model_recipe_name`                   | `str`                             | `''`                                       |                                                                                                                                                                               |
| `use_recipes`                         | `bool \| None`                    | `None`                                     |                                                                                                                                                                               |
| `task`                                | `str`                             | `'pretrain'`                               |                                                                                                                                                                               |
| `compute_dtype`                       | `str`                             | `'bf16'`                                   |                                                                                                                                                                               |
| `fp8_recipe`                          | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `hf_token`                            | `str`                             | `''`                                       |                                                                                                                                                                               |
| `nemo_home`                           | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `wandb_key`                           | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `wandb_project_name`                  | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `wandb_entity_name`                   | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `wandb_experiment_name`               | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `wandb_save_dir`                      | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `wandb_version`                       | `str`                             | `'0.28.1'`                                 | W\&B version installed in the launcher environment.                                                                                                                           |
| `numpy_version`                       | `str`                             | `'1.26.4'`                                 | NumPy version installed in the launcher environment.                                                                                                                          |
| `nemorun_version`                     | `str`                             | `'v0.10.0'`                                | NeMo Run Git commit, tag, or branch used by the launcher environment.                                                                                                         |
| `max_retries`                         | `int \| None`                     | `1`                                        |                                                                                                                                                                               |
| `use_tokendrop`                       | `bool \| List[bool] \| None`      | `None`                                     |                                                                                                                                                                               |
| `use_megatron_fsdp`                   | `bool \| List[bool] \| None`      | `None`                                     |                                                                                                                                                                               |
| `cuda_graph_impl`                     | `str \| List[str] \| None`        | `None`                                     |                                                                                                                                                                               |
| `cuda_graph_scope`                    | `str \| List[str] \| None`        | `None`                                     |                                                                                                                                                                               |
| `tp`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `pp`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `cp`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `vp`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `ep`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `et`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `mb`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `gb`                                  | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `seq_length`                          | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `lr`                                  | `float \| List[float] \| None`    | `None`                                     |                                                                                                                                                                               |
| `min_lr`                              | `float \| List[float] \| None`    | `None`                                     |                                                                                                                                                                               |
| `warmup_iters`                        | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `pretrained_checkpoint`               | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `save_dir`                            | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `load_dir`                            | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `save_interval`                       | `int \| None`                     | `None`                                     |                                                                                                                                                                               |
| `most_recent_k`                       | `int \| None`                     | `None`                                     |                                                                                                                                                                               |
| `save_config_filepath`                | `str \| None`                     | `'/nemo_run/configs/ConfigContainer.yaml'` |                                                                                                                                                                               |
| `data`                                | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `dataset_paths`                       | `str \| List[str] \| None`        | `None`                                     |                                                                                                                                                                               |
| `dataset_root`                        | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `index_mapping_dir`                   | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `dataset_name`                        | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `packed_sequence`                     | `bool \| None`                    | `None`                                     |                                                                                                                                                                               |
| `head_only`                           | `bool \| None`                    | `None`                                     |                                                                                                                                                                               |
| `tokenizer_type`                      | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `tokenizer_model`                     | `str \| None`                     | `None`                                     |                                                                                                                                                                               |
| `vocab_size`                          | `int \| None`                     | `None`                                     |                                                                                                                                                                               |
| `pytorch_profiler`                    | `bool \| None`                    | `None`                                     |                                                                                                                                                                               |
| `profiling_start_step`                | `int \| None`                     | `None`                                     |                                                                                                                                                                               |
| `profiling_stop_step`                 | `int \| None`                     | `None`                                     |                                                                                                                                                                               |
| `record_memory_history`               | `bool \| None`                    | `None`                                     |                                                                                                                                                                               |
| `profiling_gpu_metrics`               | `bool \| None`                    | `None`                                     |                                                                                                                                                                               |
| `profiling_ranks`                     | `int \| str \| List[int] \| None` | `None`                                     | Rank ID, comma-separated rank IDs as a string (e.g. '0,4,8'), or list of rank IDs.                                                                                            |
| `nsys_trace`                          | `str \| List[str] \| None`        | `None`                                     | Comma-separated nsys trace events (e.g. cuda,nvtx).                                                                                                                           |
| `nsys_extra_args`                     | `str \| List[str] \| None`        | `None`                                     | Comma-separated extra arguments for nsys.                                                                                                                                     |
| `nccl_ub`                             | `bool \| List[bool] \| None`      | `None`                                     |                                                                                                                                                                               |
| `moe_a2a_overlap`                     | `bool \| List[bool] \| None`      | `None`                                     |                                                                                                                                                                               |
| `moe_flex_dispatcher_backend`         | `str \| List[str] \| None`        | `None`                                     | MoE flex dispatcher backend: deepep, hybridep, ncclep, or the literal string 'None' to request the alltoall dispatcher. Omit the field entirely to keep the recipe's backend. |
| `max_steps`                           | `int \| None`                     | `10`                                       |                                                                                                                                                                               |
| `recompute_num_layers`                | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `activation_offload_layers`           | `int \| List[int] \| None`        | `None`                                     |                                                                                                                                                                               |
| `recompute_modules`                   | `str \| List[str] \| None`        | `None`                                     |                                                                                                                                                                               |
| `num_distributed_optimizer_instances` | `int \| None`                     | `None`                                     |                                                                                                                                                                               |
| `config_variant`                      | `str \| None`                     | `None`                                     | Config variant (e.g. v1, v2). Launcher default is v2.                                                                                                                         |
| `list_config_variants`                | `bool \| None`                    | `None`                                     | If true, list config variants and interactively select (--list_config_variants).                                                                                              |

### Test Definition

### `MegatronBridgeTestDefinition`

Megatron-Bridge test definition (CloudAI-managed install + Slurm submission via launcher).

| Field                    | Type                                                                      | Default         | Description                                                                                                                                                                                                                   |
| ------------------------ | ------------------------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                                     | Required        |                                                                                                                                                                                                                               |
| `description`            | `str`                                                                     | Required        |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                                     | Required        |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.megatron_bridge.megatron_bridge.MegatronBridgeCmdArgs` | Required        |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                                               | Factory: `list` |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                                      | `{}`            |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                                          | `{}`            |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                                               | `[]`            |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`                       | `[]`            |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`                       | `None`          |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`                         | `None`          |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`                    | `None`          |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                                     | `'grid_search'` |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                                     | `1`             |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                                               | `['default']`   |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                                     | `'inverse'`     |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                                  | `None`          | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]`                 | Factory: `dict` | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |