`nemo_rl.algorithms.single_controller_utils.rollout_checkpoint`#

Filesystem contract for frequent Single Controller rollout snapshots.

Module Contents#

Classes#

`RolloutSnapshotManifest`	Identity binding one rollout-state cut to reconstructable trainer state.
`ResolvedRolloutCheckpoint`	A committed rollout snapshot selected for startup recovery.
`BootstrapCompatibilityIdentity`	Rollout-semantic inputs that must match a trainer-version-zero cut.

Functions#

`_path_matches`	Return whether one segmented dotpath pattern matches a concrete path.
`_drop_excluded_paths`	Recursively remove denylisted mapping paths from a JSON config dump.
`_fsync_file`	Flush one completed regular file to its backing filesystem.
`_fsync_directory`	Flush directory-entry updates such as rename and replace.
`_fsync_tree`	Flush every snapshot payload before publishing its directory.
`_snapshot_sequence`
`bootstrap_compatibility_identity`	Remove explicitly operational paths from the fail-closed run identity.
`bootstrap_fingerprint`	Hash rollout-semantic inputs needed to reuse a bootstrap snapshot.
`_format_changed_value`	Format one redacted compatibility value without flooding an error.
`_compatibility_differences`	Describe changed compatibility leaves using user-facing dotpaths.
`_bootstrap_anchor_manifest`	Build the bootstrap manifest from one self-consistent identity.
`prune_bootstrap_snapshots`	Remove trainer-version-zero snapshots once a trainer checkpoint exists.
`ensure_bootstrap_anchor`	Create or validate the lightweight trainer-version-zero anchor.
`validate_bootstrap_anchor`	Validate a bootstrap anchor without modifying checkpoint state.
`prepare_snapshot_paths`	Allocate the next temporary/final snapshot directory pair.
`commit_snapshot`	Atomically publish one validated snapshot and retain recent fallbacks.
`resolve_latest_snapshot`	Select the newest complete snapshot compatible with its trainer anchor.

Data#

`ROLLOUT_SNAPSHOT_SCHEMA_VERSION`
`BOOTSTRAP_COMPATIBILITY_SCHEMA_VERSION`
`BOOTSTRAP_DIRNAME`
`BOOTSTRAP_MANIFEST_FILENAME`
`ROLLOUT_SNAPSHOTS_DIRNAME`
`ROLLOUT_SNAPSHOT_MANIFEST_FILENAME`
`RolloutCheckpointAttemptOutcome`
`ROLLOUT_CHECKPOINT_ATTEMPT_OUTCOMES`
`RolloutCheckpointAttemptReason`
`ROLLOUT_CHECKPOINT_ATTEMPT_REASONS`
`_SNAPSHOT_RE`
`_TMP_SNAPSHOT_RE`
`_TRASH_SNAPSHOT_RE`
`_BOOTSTRAP_FINGERPRINT_EXCLUDED_PATHS`
`_MISSING`

API#

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ROLLOUT_SNAPSHOT_SCHEMA_VERSION#: 3

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BOOTSTRAP_COMPATIBILITY_SCHEMA_VERSION#: 7

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BOOTSTRAP_DIRNAME#: ‘bootstrap’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BOOTSTRAP_MANIFEST_FILENAME#: ‘manifest.json’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ROLLOUT_SNAPSHOTS_DIRNAME#: ‘rollout_snapshots’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ROLLOUT_SNAPSHOT_MANIFEST_FILENAME#: ‘manifest.json’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutCheckpointAttemptOutcome#: None

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ROLLOUT_CHECKPOINT_ATTEMPT_OUTCOMES: tuple[nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutCheckpointAttemptOutcome, ...]#: ‘get_args(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutCheckpointAttemptReason#: None

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ROLLOUT_CHECKPOINT_ATTEMPT_REASONS: tuple[nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutCheckpointAttemptReason, ...]#: ‘get_args(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._SNAPSHOT_RE#: ‘compile(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._TMP_SNAPSHOT_RE#: ‘compile(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._TRASH_SNAPSHOT_RE#: ‘compile(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._BOOTSTRAP_FINGERPRINT_EXCLUDED_PATHS#: ‘frozenset(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._path_matches(pattern: tuple[str, ...], path: tuple[str, ...]) → bool#: Return whether one segmented dotpath pattern matches a concrete path.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._drop_excluded_paths( value: Any, *, excluded_paths: tuple[tuple[str, ...], ...], path: tuple[str, ...] = (), ) → Any#: Recursively remove denylisted mapping paths from a JSON config dump.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._fsync_file(path: pathlib.Path) → None#: Flush one completed regular file to its backing filesystem.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._fsync_directory(path: pathlib.Path) → None#: Flush directory-entry updates such as rename and replace.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._fsync_tree(root: pathlib.Path) → None#: Flush every snapshot payload before publishing its directory.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._snapshot_sequence(path: pathlib.Path) → int#

class nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutSnapshotManifest#

Identity binding one rollout-state cut to reconstructable trainer state.

schema_version: int#: None

base_train_step: int#: None

trainer_version: int#: None

current_epoch: int#: None

sampler_dispatch_index: int#: None

mutation_version: int#: None

rolled_back_train_group_count: int#: None

bootstrap_fingerprint: Optional[str]#: None

classmethod from_mapping( raw: Mapping[str, Any], ) → nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutSnapshotManifest#: Parse and validate a committed snapshot manifest.

to_dict() → dict[str, Any]#

class nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ResolvedRolloutCheckpoint#

A committed rollout snapshot selected for startup recovery.

path: pathlib.Path#: None

manifest: nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.RolloutSnapshotManifest#: None

class nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BootstrapCompatibilityIdentity#

Rollout-semantic inputs that must match a trainer-version-zero cut.

schema_version: int#: None

excluded_paths: tuple[str, ...]#: None

config: Mapping[str, Any]#: None

to_dict() → dict[str, Any]#

fingerprint() → str#: Return the canonical digest stored in snapshot manifests.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.bootstrap_compatibility_identity( master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig, ) → nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BootstrapCompatibilityIdentity#: Remove explicitly operational paths from the fail-closed run identity.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.bootstrap_fingerprint( master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig, ) → str#

Hash rollout-semantic inputs needed to reuse a bootstrap snapshot.

This is a compatibility guard, not a hash of the full training recipe. Operational settings are deliberately excluded so a restart may use a different cluster shape, checkpoint interval, or logging destination.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._MISSING#: ‘object(…)’

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._format_changed_value(value: Any) → str#: Format one redacted compatibility value without flooding an error.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._compatibility_differences( checkpoint: Any, expected: Any, *, path: tuple[str, ...] = (), ) → list[str]#: Describe changed compatibility leaves using user-facing dotpaths.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint._bootstrap_anchor_manifest( identity: nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BootstrapCompatibilityIdentity, ) → dict[str, Any]#: Build the bootstrap manifest from one self-consistent identity.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.prune_bootstrap_snapshots( checkpoint_dir: pathlib.Path, *, durable_trainer_checkpoint: pathlib.Path, ) → bool#: Remove trainer-version-zero snapshots once a trainer checkpoint exists.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ensure_bootstrap_anchor( checkpoint_dir: pathlib.Path, *, identity: nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BootstrapCompatibilityIdentity, ) → pathlib.Path#: Create or validate the lightweight trainer-version-zero anchor.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.validate_bootstrap_anchor( anchor: pathlib.Path, *, identity: nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.BootstrapCompatibilityIdentity, ) → None#: Validate a bootstrap anchor without modifying checkpoint state.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.prepare_snapshot_paths( anchor: pathlib.Path, ) → tuple[pathlib.Path, pathlib.Path, int]#: Allocate the next temporary/final snapshot directory pair.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.commit_snapshot( tmp_path: pathlib.Path, final_path: pathlib.Path, *, keep_latest_k: int, ) → None#: Atomically publish one validated snapshot and retain recent fallbacks.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.resolve_latest_snapshot( anchor: pathlib.Path, *, expected_train_step: int, expected_trainer_version: int, expected_bootstrap_fingerprint: Optional[str], ) → Optional[nemo_rl.algorithms.single_controller_utils.rollout_checkpoint.ResolvedRolloutCheckpoint]#: Select the newest complete snapshot compatible with its trainer anchor.

nemo_rl.algorithms.single_controller_utils.rollout_checkpoint#