nemo_automodel.components.models.deepseek_v41.fsdp
nemo_automodel.components.models.deepseek_v41.fsdp
FSDP policy for V4.1’s shared V4 vision tower.
Module Contents
Classes
Functions
Data
API
Bases: ModelParallelizer
Keep DeepSeek-V4.1 vision parameters in dtype-safe FSDP units.
Bind PP-local Engram ownership before the shared parallelizer captures parameters.
Parameters:
The DeepseekV41ForCausalLM to distribute, or one of its pipeline stages.
Meshes, parallelism sizes and strategy of the run.
Reuse V4’s vision norm policy while preserving ordinary language FSDP.
Parameters:
Module to shard in place; vision towers/blocks retain FP32 norms.
Runtime FSDP device mesh.
Mixed-precision policy for the module’s non-norm parameters.
Parameter/gradient offload policy.
PyTorch FSDP resharding setting.
Parameter tensors of arbitrary shapes already managed outside this FSDP unit, retaining their existing DTensor placements.
Returns: nn.Module
The input module with FSDP applied.