core.distributed.fsdp.src.megatron_fsdp.experimental

Experimental Megatron-FSDP implementation.

Submodules

• core.distributed.fsdp.src.megatron_fsdp.experimental.parameter_group
• core.distributed.fsdp.src.megatron_fsdp.experimental.optimizer
• core.distributed.fsdp.src.megatron_fsdp.experimental.fully_shard
• core.distributed.fsdp.src.megatron_fsdp.experimental.module
• core.distributed.fsdp.src.megatron_fsdp.experimental.dbuffer
• core.distributed.fsdp.src.megatron_fsdp.experimental.layout
• core.distributed.fsdp.src.megatron_fsdp.experimental.placement
• core.distributed.fsdp.src.megatron_fsdp.experimental.indexed_order

Package Contents

Data

__all__

API

core.distributed.fsdp.src.megatron_fsdp.experimental.__all__

[‘DBuffer’, ‘Flat’, ‘Partial’, ‘Placement’, ‘Placements’, ‘Replicate’, ‘fully_shard’, ‘fully_shard_c…