Troubleshooting

A config cannot be imported

Launch from the repository root and pass a config path below that directory. Overrides require the - separator. Use --dryrun to inspect the resolved config before training:

python train.py --config=fastgen/configs/experiments/EDM/config_dmd2_fast.py --dryrun

See command-line arguments for override syntax.

CUDA runs out of memory

• Reduce dataloader_train.batch_size and set trainer.batch_size_global for automatic gradient accumulation. For manual accumulation, set trainer.grad_accum_rounds and clear the global batch override with trainer.batch_size_global=null.
• Enable FSDP with trainer.fsdp=True, trainer.ddp=False, and trainer.fsdp_cpu_offload=True. Offloading saves GPU memory but uses more host RAM.
• To shard and offload supported preprocessors too, append their names (e.g., "text_encoder", "vae") to model.extra_fsdp_modules, preserving existing entries. Preprocessing and decoding calls must run in the same order across ranks in each shard group.
• Use image or video latent loaders for pre-encoded latents and conditions. Set model.enable_preprocessors=False when all required inputs are precomputed. Sample decoding may still load a VAE.

CPU runs out of memory

Loading multiple models or preprocessors, including CPU-offloaded parameters, can exhaust host RAM. Pre-encoded inputs help avoid unnecessary preprocessors.

• Loading: FSDP/DCP loads into existing shards; monolithic warm starts load full weights per rank before wrapping. DCP under DDP still needs full replicas.
• Startup: model.meta_init=True reduces replicated initialization where supported, but rank 0 still constructs full models and temporarily deep-copies state during FSDP synchronization. DCP does not remove this startup peak.
• EMA: Each model.use_ema entry adds a copy before wrapping, then FSDP shards/offloads it too. Keep only needed variants, or disable EMA with model.use_ema=False.

Data-loader workers also buffer and prefetch data. For WebDataset loaders, try smaller shuffle buffers and fewer workers; reduce these further if needed:

config.dataloader_train.shuffle_size = 100
config.dataloader_train.num_workers = 2

Loss or gradients become NaN

Check inputs, cached latents/conditions, and checkpoint weights for NaN/Inf.

Align model.precision_model with the precision used during pretraining. Some networks override component precision to match their reference implementation.

• Prefer model.precision=float32 and model.precision_opt=float32 for training stability; low-precision EMA updates can round away. Outside FSDP, differing input/weight dtypes need compatible AMP or network-specific handling.
• Under FSDP, normally use model.precision_amp=null; try model.precision_fsdp_reduce=float32 for unstable gradient reductions. For FP16 training, use model.grad_scaler_enabled=True with FP32 optimized weights; scaling cannot fix forward-pass NaNs.
• Attention backends can differ in numerical stability. Compare alternatives supported by the network; some recipes disable cuDNN SDPA to avoid non-finite gradients.
• Lower model.net_optimizer.lr and increase model.net_scheduler.warm_up_steps (a list for the default scheduler). Check auxiliary optimizers too; full resume restores optimizer/scheduler state.
• Enable gradient clipping per optimizer; GradClip_CALLBACK targets net. Watch the logged non-finite count: this callback replaces NaN/Inf before clipping.

Model or dataset files are missing

Prepare the dataset and review the selected model’s checkpoint requirements. Some Hugging Face models require access approval and HF_TOKEN. Most non-CIFAR/ImageNet training recipes need their placeholder data paths replaced with your data.

Check DATA_ROOT_DIR, CKPT_ROOT_DIR, HF_HOME, and local-only settings in the configuration reference. Avoid enabling offline operation until the required files are cached.

A new run resumes an earlier experiment

Resume is enabled by default. Use a different log_config.name for an independent run. To restart in an existing directory, set trainer.resume=False and disable any auto-resume integration with trainer.auto_resume=null.

Distributed workers run independently

Set trainer.ddp=True or trainer.fsdp=True; torchrun alone does not enable distributed training. Verify the resolved configuration and startup logs. See training.

A checkpoint fails to load

Choose the intended loading mechanism:

Order: external weights → warm start → resume; later loads replace matching weights. Use a new run directory for warm starts. trainer.checkpointer.pretrained_ckpt_key_map defaults to net only; map EMA explicitly too. Warm-start iteration metadata still offsets EMA and consistency-training callbacks.

FSDP resume requires DCP; monolithic checkpoints support FSDP warm starts only. Keep the DCP .pth marker and shard directories together and accessible to every rank. See checkpoint storage.

Report a problem

Open a GitHub issue with the code revision, config and overrides, hardware, and relevant error output. Remove tokens and other credentials from logs before attaching them.