> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/fastgen/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/fastgen/_mcp/server.

# Troubleshooting

> Resolve common FastGen installation, configuration, memory, and checkpoint problems.

## A config cannot be imported

Launch from the repository root and pass a config path below that directory. Overrides require the `-` separator. Use `--dryrun` to inspect the resolved config before training:

```bash
python train.py --config=fastgen/configs/experiments/EDM/config_dmd2_fast.py --dryrun
```

See [command-line arguments](/fastgen/configuration#command-line-arguments) for override syntax.

## CUDA runs out of memory

* Reduce `dataloader_train.batch_size` and set `trainer.batch_size_global` for automatic gradient accumulation. For manual accumulation, set `trainer.grad_accum_rounds` and clear the global batch override with `trainer.batch_size_global=null`.
* Enable [FSDP](/fastgen/training#use-multiple-gpus) with `trainer.fsdp=True`, `trainer.ddp=False`, and `trainer.fsdp_cpu_offload=True`. Offloading saves GPU memory but uses more host RAM.
* To shard and offload supported preprocessors too, append their names (e.g., `"text_encoder"`, `"vae"`) to `model.extra_fsdp_modules`, preserving existing entries. Preprocessing and decoding calls must run in the same order across ranks in each shard group.
* Use [image](/fastgen/datasets#imagelatentloaderconfig) or [video](/fastgen/datasets#videolatentloaderconfig) latent loaders for pre-encoded latents and conditions. Set `model.enable_preprocessors=False` when all required inputs are precomputed. Sample decoding may still load a VAE.

## CPU runs out of memory

Loading multiple models or preprocessors, including CPU-offloaded parameters, can exhaust host RAM. Pre-encoded inputs help avoid unnecessary preprocessors.

* **Loading:** FSDP/DCP loads into existing shards; monolithic warm starts load full weights per rank before wrapping. DCP under DDP still needs full replicas.
* **Startup:** `model.meta_init=True` reduces replicated initialization where supported, but rank 0 still constructs full models and temporarily deep-copies state during FSDP synchronization. DCP does not remove this startup peak.
* **EMA:** Each `model.use_ema` entry adds a copy before wrapping, then FSDP shards/offloads it too. Keep only needed variants, or disable EMA with `model.use_ema=False`.

Data-loader workers also buffer and prefetch data. For WebDataset loaders, try smaller shuffle buffers and fewer workers; reduce these further if needed:

```python
config.dataloader_train.shuffle_size = 100
config.dataloader_train.num_workers = 2
```

## Loss or gradients become NaN

Check inputs, cached latents/conditions, and checkpoint weights for NaN/Inf.

| Setting                                                                       | Controls                                                           |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `model.precision`                                                             | Method/data arithmetic.                                            |
| `model.precision_model`                                                       | Network dtype and frozen weights.                                  |
| `model.precision_opt`                                                         | Optimized/EMA weights and optimizer state.                         |
| `model.precision_amp`, `model.precision_amp_enc`, `model.precision_amp_infer` | Autocast for training, preprocessing, and inference, respectively. |

Align `model.precision_model` with the precision used during pretraining. Some networks override component precision to match their reference implementation.

* Prefer `model.precision=float32` and `model.precision_opt=float32` for training stability; low-precision EMA updates can round away. Outside FSDP, differing input/weight dtypes need compatible AMP or network-specific handling.
* Under FSDP, normally use `model.precision_amp=null`; try `model.precision_fsdp_reduce=float32` for unstable gradient reductions. For [FP16 training](https://docs.pytorch.org/docs/2.10/amp.html#gradient-scaling), use `model.grad_scaler_enabled=True` with FP32 optimized weights; scaling cannot fix forward-pass NaNs.
* [Attention backends](https://docs.pytorch.org/docs/2.10/generated/torch.nn.functional.scaled_dot_product_attention.html) can differ in numerical stability. Compare alternatives supported by the network; some recipes disable cuDNN SDPA to avoid non-finite gradients.
* Lower `model.net_optimizer.lr` and increase `model.net_scheduler.warm_up_steps` (a list for the default scheduler). Check auxiliary optimizers too; full resume restores optimizer/scheduler state.
* Enable [gradient clipping](/fastgen/callbacks#usage) per optimizer; `GradClip_CALLBACK` targets `net`. Watch the logged non-finite count: this callback replaces NaN/Inf before clipping.

## Model or dataset files are missing

Prepare the [dataset](/fastgen/datasets) and review the selected model's [checkpoint requirements](/fastgen/networks#pretrained-models). Some Hugging Face models require access approval and `HF_TOKEN`. Most non-CIFAR/ImageNet training recipes need their placeholder data paths replaced with your data.

Check `DATA_ROOT_DIR`, `CKPT_ROOT_DIR`, `HF_HOME`, and local-only settings in the [configuration reference](/fastgen/configuration#environment-variables). Avoid enabling offline operation until the required files are cached.

## A new run resumes an earlier experiment

Resume is enabled by default. Use a different `log_config.name` for an independent run. To restart in an existing directory, set `trainer.resume=False` and disable any auto-resume integration with `trainer.auto_resume=null`.

## Distributed workers run independently

Set `trainer.ddp=True` or `trainer.fsdp=True`; `torchrun` alone does not enable distributed training. Verify the resolved configuration and startup logs. See [training](/fastgen/training#use-multiple-gpus).

## A checkpoint fails to load

Choose the intended loading mechanism:

| Load               | Setting and behavior                                                                                                        |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| Resume a run       | `trainer.resume=True` restores the latest run checkpoint, including optimizer, scheduler, scaler, callbacks, and iteration. |
| Auto-resume        | `trainer.auto_resume`: optional integration supplying `save_path` for full resume, even with `trainer.resume=False`.        |
| FastGen warm start | `trainer.checkpointer.pretrained_ckpt_path`: mapped weights, without optimizer/scheduler state.                             |
| External weights   | `model.pretrained_model_path` / `model.pretrained_student_net_path`: teacher/student initialization.                        |

Order: external weights → warm start → resume; later loads replace matching weights. Use a new run directory for warm starts. `trainer.checkpointer.pretrained_ckpt_key_map` defaults to `net` only; map EMA explicitly too. Warm-start iteration metadata still offsets EMA and consistency-training callbacks.

FSDP resume requires DCP; monolithic checkpoints support FSDP warm starts only. Keep the DCP `.pth` marker and shard directories together and accessible to every rank. See [checkpoint storage](/fastgen/configuration#distributed-checkpoint-storage).

## Report a problem

Open a [GitHub issue](https://github.com/NVlabs/FastGen/issues) with the code revision, config and overrides, hardware, and relevant error output. Remove tokens and other credentials from logs before attaching them.