`core.dist_checkpointing.core`#

Module for managing distributed checkpoints metadata.

Module Contents#

Documents backends used in the checkpoint.

`check_is_distributed_checkpoint`	Checks if `metadata.json` exists in the checkpoint and is a valid config.
`maybe_load_config`	Returns checkpoint config if `checkpoint_dir` is a distributed checkpoint and None otherwise
`save_config`	Save given config to checkpoint directory.

exception core.dist_checkpointing.core.CheckpointingException#

Bases: Exception

Base checkpointing related exception

Initialization

Initialize self. See help(type(self)) for accurate signature.

class core.dist_checkpointing.core.CheckpointingConfig#

Documents backends used in the checkpoint.

Checkpoint config keeps track of formats used for storing the sharded tensors (sharded_backend) and other objects (common_backend).

Note that versioning is not for the checkpoint content (which is application specific), but for the checkpoint format itself.

core.dist_checkpointing.core.check_is_distributed_checkpoint(checkpoint_dir)#

Checks if metadata.json exists in the checkpoint and is a valid config.

Parameters:: checkpoint_dir – checkpoint directory
Returns:: True if metadata.json exists in the checkpoint and is a valid config.
Return type:: bool

core.dist_checkpointing.core.maybe_load_config( checkpoint_dir: str, ) → Optional[core.dist_checkpointing.core.CheckpointingConfig]#

Returns checkpoint config if checkpoint_dir is a distributed checkpoint and None otherwise

core.dist_checkpointing.core.save_config( config: core.dist_checkpointing.core.CheckpointingConfig, checkpoint_dir: str, )#

Save given config to checkpoint directory.

Parameters:

Returns:

None