> This page is for version 0.7.0 · 26.10. > For other versions, use one of these documentation indexes: > - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt > - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt > - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt > - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt > - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # YAML Configuration > Configure NeMo AutoModel recipes with YAML, including target instantiation, distributed settings, prewarm options, and environment variable interpolation. NeMo AutoModel recipes use YAML configs. The YAML parser creates a `ConfigNode` that performs the following actions: * Translates common scalar strings into typed Python values (for example, `"10"` → `10`) * Resolves `_target_` and `*_fn` into Python callables and classes * Supports environment variable interpolation inside YAML strings * Makes config printing safer by preserving original placeholders to avoid leaking secrets ## Load Model and Dataset Configs Most recipes load YAML configs using `nemo_automodel.components.config.loader.load_yaml_config()`, which returns a `ConfigNode`. A `ConfigNode` handles values as follows: * Nested dicts become nested `ConfigNode` objects * Lists are recursively wrapped * Scalars are translated with `translate_value()` when they are YAML strings ### Typed Scalar Translation (`translate_value`) Only **strings** are translated: * `"123"` → `123` * `"3.14"` → `3.14` * `"true"` / `"false"` → `True` / `False` * `"None"` / `"none"` → `None` YAML-native types (such as `step_size: 10` without quotes) are already typed by the YAML parser and remain unchanged. ## Use `_target_` for Instantiation Any mapping containing a `_target_` key can be instantiated using `ConfigNode.instantiate()`. ```yaml model: _target_: nemo_automodel.NeMoAutoModelForCausalLM.from_pretrained pretrained_model_name_or_path: meta-llama/Llama-3.2-1B ``` `ConfigNode` can also resolve callables from these sources: * **Dotted paths**: `pkg.module.symbol` * **Local file paths**: `/abs/path/to/file.py:symbol` ### Load a Hugging Face Model Configuration For a nested model configuration, use the explicit NeMo target: ```yaml model: _target_: nemo_automodel.NeMoAutoModelForCausalLM.from_config config: _target_: nemo_automodel.NeMoAutoConfig.from_pretrained pretrained_model_name_or_path: openai-community/gpt2 n_layer: 6 ``` `NeMoAutoConfig` retains the commit selected by the Hub download so concurrent updates to a shared cache's branch reference cannot change the config lookup. It preserves normal online freshness checks and the Transformers options `revision`, `cache_dir`, `subfolder`, `token`, `local_files_only`, `force_download`, and `trust_remote_code`. Nested model overrides continue to work, including CLI overrides such as `--model.config.n_layer 6`. The NeMo model API uses the same resolution internally and keeps subsequent config and pretrained-weight reads on the selected commit. To make separate jobs or independently starting ranks use a predetermined version, set `revision` to an immutable Hub commit in the model-loading arguments. `ConfigNode` invokes the configured callable without applying library-specific cache policy. Existing `transformers.AutoConfig.from_pretrained` targets remain valid and retain upstream behavior; change their target to `NeMoAutoConfig` to use the shared-cache fix. ### Target Resolution and Code Execution Target resolution is intentionally permissive and should be treated as code execution: * A dotted target can import any package discoverable through Python's `sys.path` or reuse a package already present in `sys.modules`. It is not limited to a safe-prefix allowlist. * An existing `file.py:symbol` target loads that file directly without requiring an environment-variable opt-in. Loading either target form executes the top-level code of the imported module. * For `file.py:symbol`, private or dunder symbols (names beginning with `_` or containing `__`) are rejected. For dotted targets, private attribute traversal is rejected by default and can be enabled with `NEMO_ENABLE_USER_MODULES=1` or `set_enable_user_modules(True)`. > **Warning** > > Only load configuration files and `_target_` values from trusted sources. A YAML file that names an importable package or Python file can execute arbitrary Python code during target resolution. ## Distributed Section (Strategy-Based) The `distributed:` section is **not** instantiated using `_target_`. Recipes parse it with a fixed schema. Use `strategy: fsdp2`, `strategy: ddp`, or `strategy: megatron_fsdp`. You can also configure parallelism sizes, such as `dp_size`, `tp_size`, and `pp_size`, and strategy-specific options. When pipeline parallelism is enabled (`pp_size > 1`), add a `pipeline:` subsection with options such as `pp_schedule`, `pp_microbatch_size`, and `layers_per_stage`. For examples, see the [Pipeline Parallelism with AutoPipeline](/nemo/automodel/v0.7/development/pipeline-parallelism) guide and the recipe configs. ## Select Trainable Modules The LLM and VLM fine-tuning recipes accept a `freeze_config:` section. Use `freeze_modules` and `unfreeze_modules` with typed selectors to control which modules are trainable: ```yaml freeze_config: freeze_modules: - path: vision_tower # exact canonical module path - path: audio_tower # repeat the key to select more exact module paths - glob: "*.speech_encoder" # case-sensitive shell-style glob on the full module path unfreeze_modules: - path: multi_modal_projector ``` Each selector is a mapping with exactly one key. `path` matches one module by its exact fully qualified name, while `glob` matches module names with `fnmatch`-style wildcards (`*` crosses `.` separators, so `*_proj` matches projection modules at any depth). List entries are additive: repeat `path` or `glob` entries to select several modules, and a single entry combining both keys is rejected. Matching is recursive: a selected module's entire subtree is frozen or unfrozen. Bare strings are rejected; unknown options and selectors that match no parameters raise an error before training starts. Trainability is resolved in a fixed order. Full fine-tuning preserves the model's existing `requires_grad` state, including intentionally frozen model-owned parameters. PEFT establishes a LoRA-trainable, base-frozen baseline. `freeze_modules` selectors then freeze their modules, and `unfreeze_modules` selectors unfreeze theirs, winning on overlap. This makes combinations such as LoRA plus a fully trainable multimodal projector a two-line configuration. Freezing only controls `requires_grad`; it never changes a parameter's storage dtype. The policy is validated on the complete model, re-resolved after tensor/expert parallel and activation-checkpointing surgery immediately before DDP/FSDP construction, and resolved once more after checkpoint loading before optimizer construction. The modality-specific booleans `freeze_vision_tower`, `freeze_audio_tower`, `freeze_language_model`, and `freeze_video_embedder` remain supported and keep their established attribute and substring matching, applied before `unfreeze_modules`: ```yaml freeze_config: freeze_vision_tower: true freeze_audio_tower: true ``` Legacy-only configurations retain the implicit `freeze_vision_tower: true` default. A configuration that declares `freeze_modules` or `unfreeze_modules`, even as an empty list, uses explicit-selector semantics and does not implicitly freeze vision modules; add a vision selector (for example, `glob: "*vision*"`) when such a configuration should also freeze the vision tower. Omit both selector fields to retain legacy behavior; `null`, `false`, and numeric values are rejected. ## Prewarm One-Time CUDA Initialization The LLM training, LLM fine-tuning, and VLM fine-tuning recipes accept an optional `prewarm:` section. ```yaml prewarm: cublas_backward: true fla_gdn_autotune: true mamba_ssd_autotune: true comm_groups: true ``` All prewarm options default to `false`. For the four initialization options below, enable only the warmup associated with an observed first-step failure. To reduce pipeline-parallel startup time without a first-step failure, see [Reduce Pipeline-Parallel Startup Time](#reduce-pipeline-parallel-startup-time). * `cublas_backward` initializes cuBLAS backward workspaces during setup. * `fla_gdn_autotune` autotunes flash-linear-attention gated-delta-net Triton kernels during setup. * `mamba_ssd_autotune` autotunes Mamba SSD Triton kernels during setup. * `comm_groups` initializes the process groups used by gradient-norm collectives. > **Info** > > Prewarming is most useful for long-context or otherwise memory-constrained runs. A run can have enough steady-state memory and still fail on its first training step. Activations and gradients are live while cuBLAS workspaces, Triton autotuning buffers, and NCCL communicators are initialized for the first time. Temporary autotuning allocations are released afterward. Some memory, including NCCL scratch space, is outside PyTorch's caching allocator, so `torch.cuda.memory_allocated()` does not show the complete transient peak. > > Adding GPUs helps only when it sufficiently reduces the relevant per-rank tensors. Each rank still initializes its own resources, and additional parallel dimensions can introduce more communication groups. Shorter contexts can leave enough memory for lazy initialization, while longer contexts might require the same initialization during setup. Long-context hybrid-attention MoE workloads are one common case. Enable the initialization prewarms above based on the observed first-step failure rather than a model name. Prewarming does not reduce steady-state memory or imply support for a particular context length. > > Triton reuses an autotuned configuration according to each kernel's explicit cache key. The Mamba SSD warmup therefore matches the real head, state, group, chunk, and dtype geometry but uses batch size 1 and two chunks. Batch size and sequence length change the launch grid and activation footprint, but they are not cache keys in the pinned Mamba SSD kernels. Keeping those dimensions small is intentional: using the full training batch and sequence would recreate the peak-memory condition that setup prewarming is meant to avoid. ### Reduce Pipeline-Parallel Startup Time For LLM training and fine-tuning with pipeline parallelism (`distributed.pp_size > 1`), enable concurrent stage prewarming when compilation dominates the first iteration: ```yaml prewarm: pipeline_stage_compile: true ``` Normally, a stage first compiles and autotunes its kernels when it receives the first microbatch from the preceding stage. These one-time costs accumulate along the pipeline. With `pipeline_stage_compile`, each stage instead runs one synthetic forward and backward microbatch during setup, allowing different stages to compile concurrently. Inputs match the stage's static input shapes; FSDP and expert-parallel communication within the stage still runs normally. Use the same setting on all ranks. The warmup does not perform an optimizer step. On completion, it discards its gradients and restores the stage's train/eval flag and the CPU and device random-number-generator states. It requires memory for a stage microbatch's forward and backward passes and does not reduce steady-state memory. The option defaults to `false`; without pipeline stages, or for a stage without static input metadata, the warmup is skipped. The separate VLM fine-tuning recipe does not supply pipeline stages to this warmup. Check the `Pipeline-stage compile prewarm` log entries for per-stage forward/backward times and total warmup time. Compare **setup plus the first training iteration** across runs: moving work into setup makes iteration 0 shorter, but setup time still counts. This option targets startup latency; it does not change the kernels used in subsequent training steps or promise higher steady-state throughput or MFU. See [Pipeline Parallelism](/nemo/automodel/v0.7/development/pipeline-parallelism) for the pipeline configuration. ### Reuse HybridEP JIT Kernels Across Jobs When using the `hybridep` MoE dispatcher, set `NEMO_HYBRIDEP_JIT_CACHE` before launching the training processes to retain compiled HybridEP kernels across jobs: ```bash export NEMO_HYBRIDEP_JIT_CACHE=/mnt/shared/automodel/hybridep-jit ``` Replace the example path with a persistent, writable directory. For reuse across nodes, make the same shared directory accessible to all training processes. In container or Slurm launches, pass the environment variable into every worker and mount the cache directory at the configured path inside the container. See [Run on a Cluster](/nemo/automodel/v0.7/job-launchers/slurm-cluster) for launch and mount examples. The first run with an empty cache still compiles kernels. Later runs copy cached `.so` files into each process's HybridEP JIT directory and load them when creating the HybridEP buffer. Newly compiled kernels are saved back by local rank 0 on each node after the first backward combine and again at process exit. Cache subdirectories include the installed DeepEP version, PyTorch's CUDA version, and the GPU compute capability. `HYBRID_EP_CACHE_DIR` controls the separate per-process JIT location; setting it alone does not enable this shared cache. The shared cache is disabled when `NEMO_HYBRIDEP_JIT_CACHE` is unset and can be enabled independently of `pipeline_stage_compile`, including without pipeline parallelism. It applies only to HybridEP kernels, not to all `torch.compile` or Triton caches. Look for `HybridEP JIT cache: warm-started` and `stored` messages to confirm reuse and store-back. A populated cache reduces compilation work on later launches; it does not increase steady-state training throughput. ## Configure the MFU Reference Peak Recipes that report model FLOP utilization automatically detect the current device and use its dense-BF16 peak from AutoMFU's device table. Ordinary BF16 training does not need an `mfu:` section. Use an explicit override when the hardware is not in the table or when the training precision has a different theoretical peak, such as dense FP8: ```yaml mfu: device: h100 peak_tflops: 1979.0 ``` `peak_tflops` takes precedence over `device`. A device name alone selects an entry from AutoMFU's table; omitting both fields uses the current CUDA device. Hardware detection cannot determine the effective training precision, so FP8 and structured-sparse runs must provide the appropriate peak explicitly. An unknown device emits a warning and reports 0% MFU until you configure a supported `device` value or an explicit `peak_tflops` value. ## Repair Near-Zero Input Embeddings The LLM training and fine-tuning recipe can repair a small number of damaged input-embedding rows after loading the base checkpoint and before creating the optimizer. ```yaml embedding_row_repair: min_norm: 1.0e-4 max_rows: 256 ``` Rows with a non-finite L2 norm or a norm at or below `min_norm` are replaced with the corresponding output-embedding direction, scaled to the RMS norm of healthy input rows. Setup logs the affected token IDs. `max_rows` is a safety limit: setup aborts instead of rewriting a broadly damaged or mismatched checkpoint. This option is intended for diagnosed checkpoint defects where rare token IDs produce extreme input-embedding gradients. It is disabled when the section is absent and is not currently supported with pipeline parallelism. For diagnosis, safety behavior, verification steps, and performance impact, see [Repair Damaged Input-Embedding Rows](/nemo/automodel/v0.7/model-coverage/troubleshooting#repair-damaged-input-embedding-rows). ## Frozen Multimodal FSDP Policy FSDP2 uses the correctness-first `root` policy for fully frozen vision/audio towers and multimodal projectors. ```yaml distributed: strategy: fsdp2 multimodal: frozen_sharding: root ``` The supported policies are: * `root` (default): The always-run outer FSDP root owns frozen multimodal parameters. This keeps collective ordering aligned when some ranks execute a modality branch and others skip it. * `per_layer`: Layers inside the frozen multimodal module get their normal FSDP units. This can reduce peak unshard memory, but it is an expert option: every rank in the FSDP group must execute or skip those units the same number of times and in the same order on every microbatch. * `replicate`: Frozen multimodal parameters are excluded from FSDP and copied on every rank. This removes their collectives at the cost of higher per-rank parameter memory. This setting does not change modules that contain any trainable parameters. For nested MoE or VLM models, `root` requires `distributed.moe.wrap_outer_model: true` when a fully frozen multimodal module is outside the inner text-model FSDP root. Use `per_layer` or `replicate` if the outer model must remain unwrapped. ## Interpolate Environment Variables in YAML NeMo AutoModel supports env var interpolation inside YAML **string values**. ### Supported Forms * **Braced**: * `${VAR}` * `${VAR,default}` * `${var.dot.var}` (dots are treated as part of the env var name) * **Dollar**: * `$VAR` * `$var.dot.var` * **Back-compat**: * `${oc.env:VAR}` * `${oc.env:VAR,default}` ### Interpolation Behavior * Interpolation occurs when values are wrapped into a `ConfigNode`. * If a referenced env var is **missing** and **no default** is provided, config loading raises a `KeyError`. * Defaults are supported only for braced forms using the first comma: `${VAR,default_value}`. ### Example (Databricks Delta) The following configuration shows an example of environment variable interpolation for a Databricks Delta dataset. ```yaml dataset: _target_: nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset path_or_dataset_id: delta://catalog.schema.training_data delta_storage_options: DATABRICKS_HOST: ${DATABRICKS_HOST} DATABRICKS_TOKEN: ${DATABRICKS_TOKEN} DATABRICKS_HTTP_PATH: ${DATABRICKS_HTTP_PATH} ``` ## Prevent Secret Leakage in Logs When an env var placeholder is resolved, the config keeps the original placeholder in an internal `._orig_value` field for **safe printing**. * `str(cfg)` or `repr(cfg)` prints placeholders (for example, `${DATABRICKS_TOKEN}`), not resolved secrets. * `cfg.to_yaml_dict(use_orig_values=True, redact_sensitive=True)` is the recommended way to produce a loggable YAML dict. > **Info** > > Printing a **leaf value** (for example, `print(cfg.dataset.delta_storage_options.DATABRICKS_TOKEN)`) outputs the resolved secret. Instead, print the full config or use a redacted YAML dict. ## Configure Slurm Slurm jobs are submitted directly with `sbatch`. No YAML section is required. Copy the reference script, edit `CONFIG` and the cluster settings, and then submit the job: ```bash cp slurm.sub my_cluster.sub vim my_cluster.sub sbatch my_cluster.sub ``` All cluster-specific configuration, including SBATCH directives, the container image, mounts, secrets, and environment variables, resides in your `sbatch` script. See [Run on a Cluster](/nemo/automodel/v0.7/job-launchers/slurm-cluster) for complete examples.