> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# YAML Configuration

NeMo AutoModel recipes are configured with YAML. Under the hood, YAML is parsed into a `ConfigNode` which:

* Translates common scalar strings into typed Python values (e.g., `"10"` → `10`).
* Resolves `_target_` (and `*_fn`) into Python callables/classes.
* Supports environment variable interpolation inside YAML strings.
* Tries to make config printing safe by preserving original placeholders (to avoid leaking secrets).

## Load Model and Dataset Configs

Most recipes load YAML configs by using `nemo_automodel.components.config.loader.load_yaml_config()`, which returns a `ConfigNode`.

A `ConfigNode` handles values as follows:

* Nested dicts become nested `ConfigNode` objects
* Lists are recursively wrapped
* Scalars are translated with `translate_value()` when they are YAML strings

### Typed Scalar Translation (`translate_value`)

Only **strings** are translated:

* `"123"` → `123`
* `"3.14"` → `3.14`
* `"true"` / `"false"` → `True` / `False`
* `"None"` / `"none"` → `None`

YAML-native types (such as `step_size: 10` without quotes) are already typed by the YAML parser and remain unchanged.

## Use `_target_` for Instantiation

Any mapping containing a `_target_` key can be instantiated using `ConfigNode.instantiate()`:

```yaml
model:
  _target_: nemo_automodel.NeMoAutoModelForCausalLM.from_pretrained
  pretrained_model_name_or_path: meta-llama/Llama-3.2-1B
```

`ConfigNode` can also resolve callables from these sources:

* **Dotted paths**: `pkg.module.symbol`
* **Local file paths**: `/abs/path/to/file.py:symbol`

### Target Resolution and Code Execution

Target resolution is intentionally permissive and should be treated as code execution:

* A dotted target can import any package discoverable through Python's `sys.path` or reuse a package already present in `sys.modules`. It is not limited to a safe-prefix allowlist.
* An existing `file.py:symbol` target loads that file directly without requiring an environment-variable opt-in. Loading either target form executes the top-level code of the imported module.
* For `file.py:symbol`, private or dunder symbols (names beginning with `_` or containing `__`) are rejected. For dotted targets, private attribute traversal is rejected by default and can be enabled with `NEMO_ENABLE_USER_MODULES=1` or `set_enable_user_modules(True)`.

Only load configuration files and `_target_` values from trusted sources. A YAML file that names an importable package or Python file can execute arbitrary Python code during target resolution.

## Distributed Section (Strategy-Based)

The `distributed:` section is **not** instantiated using `_target_`. Recipes parse it with a fixed schema. Use `strategy: fsdp2`, `strategy: ddp`, or `strategy: megatron_fsdp`. You can also configure parallelism sizes, such as `dp_size`, `tp_size`, and `pp_size`, and strategy-specific options. When pipeline parallelism is enabled (`pp_size > 1`), add a `pipeline:` subsection with options such as `pp_schedule`, `pp_microbatch_size`, and `layers_per_stage`. For examples, see the [Pipeline Parallelism with AutoPipeline](/development/pipeline-parallelism) guide and the recipe configs.

## Prewarm One-Time CUDA Initialization

The LLM training/fine-tuning and VLM fine-tuning recipes accept an optional `prewarm:` section:

```yaml
prewarm:
  cublas_backward: true
  fla_gdn_autotune: true
  comm_groups: true
```

All options default to `false`. Enable only the warmup associated with an observed first-step failure:

* `cublas_backward` initializes cuBLAS backward workspaces during setup.
* `fla_gdn_autotune` autotunes flash-linear-attention gated-delta-net Triton kernels during setup.
* `comm_groups` initializes the process groups used by gradient-norm collectives.

Prewarming is most useful for long-context or otherwise memory-constrained runs. A run can have enough steady-state memory and still fail on its first training step. Activations and gradients are live while cuBLAS workspaces, Triton autotuning buffers, and NCCL communicators are initialized for the first time. Temporary autotuning allocations are released afterward. Some memory, including NCCL scratch space, is outside PyTorch's caching allocator, so `torch.cuda.memory_allocated()` does not show the complete transient peak.

Adding GPUs helps only when it sufficiently reduces the relevant per-rank tensors. Each rank still initializes its own resources, and additional parallel dimensions can introduce more communication groups. Shorter contexts can leave enough memory for lazy initialization, while longer contexts might require the same initialization during setup. Long-context hybrid-attention MoE workloads are one common case. Enable each prewarm based on the observed first-step failure rather than a model name. Prewarming does not reduce steady-state memory or imply support for a particular context length.

## Interpolate Environment Variables in YAML

NeMo AutoModel supports env var interpolation inside YAML **string values**.

### Supported Forms

* **Braced**:
  * `${VAR}`
  * `${VAR,default}`
  * `${var.dot.var}` (dots are treated as part of the env var name)
* **Dollar**:
  * `$VAR`
  * `$var.dot.var`
* **Back-compat**:
  * `${oc.env:VAR}`
  * `${oc.env:VAR,default}`

### Interpolation Behavior

* Interpolation occurs when values are wrapped into a `ConfigNode`.
* If a referenced env var is **missing** and **no default** is provided, config loading raises a `KeyError`.
* Defaults are supported only for braced forms using the first comma: `${VAR,default_value}`.

### Example (Databricks Delta)

```yaml
dataset:
  _target_: nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset
  path_or_dataset_id: delta://catalog.schema.training_data
  delta_storage_options:
    DATABRICKS_HOST: ${DATABRICKS_HOST}
    DATABRICKS_TOKEN: ${DATABRICKS_TOKEN}
    DATABRICKS_HTTP_PATH: ${DATABRICKS_HTTP_PATH}
```

## Prevent Secret Leakage in Logs

When an env var placeholder is resolved, the config keeps the original placeholder in an internal `._orig_value` field for **safe printing**:

* `str(cfg)` / `repr(cfg)` prints placeholders (e.g. `${DATABRICKS_TOKEN}`), not resolved secrets.
* `cfg.to_yaml_dict(use_orig_values=True, redact_sensitive=True)` is the recommended way to produce a loggable YAML dict.

Printing a **leaf value** (for example, `print(cfg.dataset.delta_storage_options.DATABRICKS_TOKEN)`) outputs the resolved secret. Instead, print the full config or use a redacted YAML dict.

## Configure Slurm

Slurm jobs are submitted directly with `sbatch`. No YAML section is required.
Copy the reference script, edit `CONFIG` and the cluster settings, and then submit the job:

```bash
cp slurm.sub my_cluster.sub
vim my_cluster.sub
sbatch my_cluster.sub
```

All cluster-specific configuration, including SBATCH directives, the container image, mounts, secrets, and environment variables, resides in your `sbatch` script. See [Run on a Cluster](/job-launchers/slurm-cluster) for complete examples.