> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

This workload (`test_template_name` is `NemoRun`) executes NeMo training and fine-tuning tasks using the NeMo Run framework.

## Usage Examples

Test TOML example:

```toml
name = "my_nemo_test"
description = "Example NeMo Run test"
test_template_name = "NemoRun"

[cmd_args]
recipe = "llama3_8b"
task = "pretrain"
```

Test Scenario example:

```toml
name = "nemo-run-test"

[[Tests]]
id = "nemo.1"
num_nodes = 4
time_limit = "02:00:00"

test_name = "my_nemo_test"
```

Test-in-Scenario example:

```toml
name = "nemo-run-test"

[[Tests]]
id = "nemo.1"
num_nodes = 4
time_limit = "02:00:00"

name = "my_nemo_test"
description = "Example NeMo Run test"
test_template_name = "NemoRun"

  [Tests.cmd_args]
  recipe = "llama3_8b"
  task = "pretrain"
```

## API Documentation

### Command Arguments

### `NeMoRunCmdArgs`

NeMoRun test command arguments.

| Field              | Type                                          | Default            | Description |
| ------------------ | --------------------------------------------- | ------------------ | ----------- |
| `docker_image_url` | `str`                                         | Required           |             |
| `task`             | `str`                                         | Required           |             |
| `recipe_name`      | `str`                                         | Required           |             |
| `num_layers`       | `int \| None`                                 | `None`             |             |
| `trainer`          | `cloudai.workloads.nemo_run.nemo_run.Trainer` | Factory: `Trainer` |             |
| `log`              | `cloudai.workloads.nemo_run.nemo_run.Log`     | Factory: `Log`     |             |
| `data`             | `cloudai.workloads.nemo_run.nemo_run.Data`    | Factory: `Data`    |             |
| `optim`            | `cloudai.workloads.nemo_run.nemo_run.Optim`   | Factory: `Optim`   |             |

### Test Definition

### `NeMoRunTestDefinition`

NeMoRun test definition.

| Field                    | Type                                                      | Default                                                                    | Description                                                                                                                                                                                                                   |
| ------------------------ | --------------------------------------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                     | Required                                                                   |                                                                                                                                                                                                                               |
| `description`            | `str`                                                     | Required                                                                   |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                     | Required                                                                   |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.nemo_run.nemo_run.NeMoRunCmdArgs`      | Required                                                                   |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                               | Factory: `list`                                                            |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                      | `{}`                                                                       |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                          | `{}`                                                                       |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                               | `[]`                                                                       |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`       | `[]`                                                                       |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`       | `None`                                                                     |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`         | `None`                                                                     |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`    | `None`                                                                     |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                     | `'grid_search'`                                                            |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                     | `1`                                                                        |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                               | `['default']`                                                              |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                     | `'inverse'`                                                                |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                  | `None`                                                                     | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]` | Factory: `dict`                                                            | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |
| `script`                 | `cloudai._core.installables.file.File`                    | `File(src=PosixPath('src/cloudai/workloads/nemo_run/cloudai_nemorun.py'))` |                                                                                                                                                                                                                               |