MegatronBridge
This workload (test_template_name is MegatronBridge) submits training and finetuning tasks based on Megatron-Bridge framework.
Megatron-Bridge and NeMo Run generate the training script in dry-run mode on the submit host. CloudAI retains and parses their sbatch, then writes cloudai_sbatch_script.sh with its standard environment setup, metadata, rank mapping, and pre/post tests. These tests run inside the training allocation. Without the installed MBridge launcher or NeMo Run Python environment, cloudai dry-run produces a placeholder instead of the training commands and logs a warning. The placeholder exits with an error if executed. Install the dependencies and repeat the dry-run to generate the complete training script.
CloudAI owns the job name, output/error paths, and node placement. Configured CloudAI allocation directives override matching NeMo Run directives; other NeMo Run directives and training artifact paths are retained. Set scenario num_nodes and system GPU/task counts to match the training configuration. enable_vboost uses CloudAI’s standard VBoost setup in both normal and single-sbatch execution. An explicitly configured value takes precedence over the ENABLE_VBOOST environment setting. The rendezvous address uses NeMo’s selected head-node hostname, which must resolve inside the training containers.
Additional launcher options can be passed through [extra_cmd_args]. For example:
CloudAI combines -cs and --custom_srun_args from this table, then appends system and test-run extra_srun_args and pre-test-provided arguments. The result is passed through Megatron-Bridge’s native -cs option without deduplication.
For Nsight profiling, use Megatron-Bridge’s cmd_args.enable_nsys and profiling_start_step, profiling_stop_step, and profiling_ranks options. The generic CloudAI [nsys] configuration is not applied to this workload.
To enable native post-training evaluation with Megatron-Bridge v0.3.0 through v0.7.0, explicitly disable detach in the test configuration:
This workload has a hard requirement for the HuggingFace Hub token. There are two options:
- (recommended) define
HF_TOKENenvironment variable - set
cmd_args.hf_tokeneither in Test or Scenario config
Usage Examples
Test TOML example:
Test Scenario example:
Test-in-Scenario example:
Executor Environment Variables
CloudAI applies global_env_vars and extra_env_vars through its standard job environment setup. These values override matching exports in the generated NeMo Run script. CloudAI’s env_vars.sh is also sourced inside each training task, so configured values can reference task-local Slurm variables.
Existing cmd_args.executor_env_vars entries are also accepted as job environment variables:
global_env_vars and extra_env_vars take precedence over these entries. This does not require Megatron-Bridge’s -E/--env support.
Chakra / Kineto Tracing
Disabling nsys
The existing MegatronBridge TOMLs in conf/staging/ have enable_nsys = true. To switch to Kineto/Chakra profiling, either:
- Omit
enable_nsysfrom the TOML (it defaults tofalse) - Or explicitly set
enable_nsys = false
Both are equivalent. Do not set enable_nsys = true alongside Kineto flags — the two profilers conflict and Megatron-Bridge will raise an assertion error.
Enabling Kineto Trace
Add the following to [extra_cmd_args] in your Test TOML. These are Hydra-style overrides passed directly to the Megatron-Bridge training script:
This produces the following under the NemoRun experiment directory (<cloudai_output>/experiments/<experiment_name>/<run_id>/<experiment_name>/):
torch_profile/rank-0.json.gz— Chrome-format kineto trace (always produced)tb_logs/— TensorBoard events
Enabling Chakra Trace
The Chakra trace is a PyTorch ExecutionTraceObserver recording — a separate file from the kineto trace. Add pytorch_profiler_collect_chakra = true:
This produces an additional chakra/rank-0.json.gz in the same experiment directory. The kineto trace (torch_profile/rank-0.json.gz) is always produced regardless of this flag.
Reference TOML
A ready-to-use TOML is provided at:
This runs Nemotron 3 Nano 3B (8 GPUs, 60 steps) with Kineto and Chakra profiling at steps 45–50 and is suitable for quickly validating the profiling setup.
Profiling Parameter Reference
API Documentation
Command Arguments
MegatronBridgeCmdArgs
Megatron-Bridge launcher arguments (translated into setup_experiment.py flags).
Test Definition
MegatronBridgeTestDefinition
Megatron-Bridge test definition (CloudAI-managed install + Slurm submission via launcher).