`nemo_rl.algorithms.rollout_checkpointing.manifest`#

Filesystem contract for frequent Single Controller rollout snapshots.

Module Contents#

Classes#

`RolloutSnapshotManifest`	Identity binding one rollout-state cut to reconstructable trainer state.
`ResolvedRolloutCheckpoint`	A committed rollout snapshot selected for startup recovery.
`BootstrapCompatibilityIdentity`	Rollout-semantic inputs that must match a trainer-version-zero cut.

Functions#

`_path_matches`	Return whether one segmented dotpath pattern matches a concrete path.
`_drop_excluded_paths`	Recursively remove denylisted mapping paths from a JSON config dump.
`_fsync_file`	Flush one completed regular file to its backing filesystem.
`_fsync_directory`	Flush directory-entry updates such as rename and replace.
`_fsync_tree`	Flush every snapshot payload before publishing its directory.
`_snapshot_sequence`
`_normalize_capture_reward_config`	Replace capture reward config with the projection sidecar restore compares.
`bootstrap_compatibility_identity`	Remove explicitly operational paths from the fail-closed run identity.
`_format_changed_value`	Format one redacted compatibility value without flooding an error.
`_compatibility_differences`	Describe changed compatibility leaves using user-facing dotpaths.
`_bootstrap_anchor_manifest`	Build the bootstrap manifest from one self-consistent identity.
`prune_bootstrap_snapshots`	Remove trainer-version-zero snapshots once a trainer checkpoint exists.
`ensure_bootstrap_anchor`	Create or validate the lightweight trainer-version-zero anchor.
`validate_bootstrap_anchor`	Validate a bootstrap anchor without modifying checkpoint state.
`prepare_snapshot_paths`	Allocate the next temporary/final snapshot directory pair.
`commit_snapshot`	Atomically publish one validated snapshot and retain recent fallbacks.
`resolve_latest_snapshot`	Select the newest complete snapshot compatible with its trainer anchor.

Data#

`ROLLOUT_SNAPSHOT_SCHEMA_VERSION`
`BOOTSTRAP_COMPATIBILITY_SCHEMA_VERSION`
`BOOTSTRAP_DIRNAME`
`BOOTSTRAP_MANIFEST_FILENAME`
`ROLLOUT_SNAPSHOTS_DIRNAME`
`ROLLOUT_SNAPSHOT_MANIFEST_FILENAME`
`_SNAPSHOT_RE`
`_TMP_SNAPSHOT_RE`
`_TRASH_SNAPSHOT_RE`
`_BOOTSTRAP_FINGERPRINT_EXCLUDED_PATHS`
`_MISSING`

API#

nemo_rl.algorithms.rollout_checkpointing.manifest.ROLLOUT_SNAPSHOT_SCHEMA_VERSION#: 3

nemo_rl.algorithms.rollout_checkpointing.manifest.BOOTSTRAP_COMPATIBILITY_SCHEMA_VERSION#: 7

nemo_rl.algorithms.rollout_checkpointing.manifest.BOOTSTRAP_DIRNAME#: ‘bootstrap’

nemo_rl.algorithms.rollout_checkpointing.manifest.BOOTSTRAP_MANIFEST_FILENAME#: ‘manifest.json’

nemo_rl.algorithms.rollout_checkpointing.manifest.ROLLOUT_SNAPSHOTS_DIRNAME#: ‘rollout_snapshots’

nemo_rl.algorithms.rollout_checkpointing.manifest.ROLLOUT_SNAPSHOT_MANIFEST_FILENAME#: ‘manifest.json’

nemo_rl.algorithms.rollout_checkpointing.manifest._SNAPSHOT_RE#: ‘compile(…)’

nemo_rl.algorithms.rollout_checkpointing.manifest._TMP_SNAPSHOT_RE#: ‘compile(…)’

nemo_rl.algorithms.rollout_checkpointing.manifest._TRASH_SNAPSHOT_RE#: ‘compile(…)’

nemo_rl.algorithms.rollout_checkpointing.manifest._BOOTSTRAP_FINGERPRINT_EXCLUDED_PATHS#: ‘frozenset(…)’

nemo_rl.algorithms.rollout_checkpointing.manifest._path_matches(pattern: tuple[str, ...], path: tuple[str, ...]) → bool#: Return whether one segmented dotpath pattern matches a concrete path.

nemo_rl.algorithms.rollout_checkpointing.manifest._drop_excluded_paths( value: Any, *, excluded_paths: tuple[tuple[str, ...], ...], path: tuple[str, ...] = (), ) → Any#: Recursively remove denylisted mapping paths from a JSON config dump.

nemo_rl.algorithms.rollout_checkpointing.manifest._fsync_file(path: pathlib.Path) → None#: Flush one completed regular file to its backing filesystem.

nemo_rl.algorithms.rollout_checkpointing.manifest._fsync_directory(path: pathlib.Path) → None#: Flush directory-entry updates such as rename and replace.

nemo_rl.algorithms.rollout_checkpointing.manifest._fsync_tree(root: pathlib.Path) → None#: Flush every snapshot payload before publishing its directory.

nemo_rl.algorithms.rollout_checkpointing.manifest._snapshot_sequence(path: pathlib.Path) → int#

class nemo_rl.algorithms.rollout_checkpointing.manifest.RolloutSnapshotManifest#

Identity binding one rollout-state cut to reconstructable trainer state.

schema_version: int#: None

base_train_step: int#: None

trainer_version: int#: None

current_epoch: int#: None

sampler_dispatch_index: int#: None

mutation_version: int#: None

rolled_back_train_group_count: int#: None

bootstrap_fingerprint: Optional[str]#: None

next_nemo_gym_task_index: Optional[int]#: None

classmethod from_mapping( raw: Mapping[str, Any], ) → nemo_rl.algorithms.rollout_checkpointing.manifest.RolloutSnapshotManifest#: Parse and validate a committed snapshot manifest.

to_dict() → dict[str, Any]#

class nemo_rl.algorithms.rollout_checkpointing.manifest.ResolvedRolloutCheckpoint#

A committed rollout snapshot selected for startup recovery.

path: pathlib.Path#: None

manifest: nemo_rl.algorithms.rollout_checkpointing.manifest.RolloutSnapshotManifest#: None

class nemo_rl.algorithms.rollout_checkpointing.manifest.BootstrapCompatibilityIdentity#

Rollout-semantic inputs that must match a trainer-version-zero cut.

schema_version: int#: None

excluded_paths: tuple[str, ...]#: None

config: Mapping[str, Any]#: None

to_dict() → dict[str, Any]#

fingerprint() → str#: Return the canonical digest stored in snapshot manifests.

nemo_rl.algorithms.rollout_checkpointing.manifest._normalize_capture_reward_config( dumped: dict[str, Any], ) → None#

Replace capture reward config with the projection sidecar restore compares.

Otherwise the bootstrap fingerprint would reject harmless reward-config extras before the sidecar’s semantic comparison ever ran.

nemo_rl.algorithms.rollout_checkpointing.manifest.bootstrap_compatibility_identity( master_config: nemo_rl.algorithms.single_controller_utils.config.MasterConfig, ) → nemo_rl.algorithms.rollout_checkpointing.manifest.BootstrapCompatibilityIdentity#: Remove explicitly operational paths from the fail-closed run identity.

nemo_rl.algorithms.rollout_checkpointing.manifest._MISSING#: ‘object(…)’

nemo_rl.algorithms.rollout_checkpointing.manifest._format_changed_value(value: Any) → str#: Format one redacted compatibility value without flooding an error.

nemo_rl.algorithms.rollout_checkpointing.manifest._compatibility_differences( checkpoint: Any, expected: Any, *, path: tuple[str, ...] = (), ) → list[str]#: Describe changed compatibility leaves using user-facing dotpaths.

nemo_rl.algorithms.rollout_checkpointing.manifest._bootstrap_anchor_manifest( identity: nemo_rl.algorithms.rollout_checkpointing.manifest.BootstrapCompatibilityIdentity, ) → dict[str, Any]#: Build the bootstrap manifest from one self-consistent identity.

nemo_rl.algorithms.rollout_checkpointing.manifest.prune_bootstrap_snapshots( checkpoint_dir: pathlib.Path, *, durable_trainer_checkpoint: pathlib.Path, ) → bool#: Remove trainer-version-zero snapshots once a trainer checkpoint exists.

nemo_rl.algorithms.rollout_checkpointing.manifest.ensure_bootstrap_anchor( checkpoint_dir: pathlib.Path, *, identity: nemo_rl.algorithms.rollout_checkpointing.manifest.BootstrapCompatibilityIdentity, ) → pathlib.Path#: Create or validate the lightweight trainer-version-zero anchor.

nemo_rl.algorithms.rollout_checkpointing.manifest.validate_bootstrap_anchor( anchor: pathlib.Path, *, identity: nemo_rl.algorithms.rollout_checkpointing.manifest.BootstrapCompatibilityIdentity, ) → None#: Validate a bootstrap anchor without modifying checkpoint state.

nemo_rl.algorithms.rollout_checkpointing.manifest.prepare_snapshot_paths( anchor: pathlib.Path, ) → tuple[pathlib.Path, pathlib.Path, int]#: Allocate the next temporary/final snapshot directory pair.

nemo_rl.algorithms.rollout_checkpointing.manifest.commit_snapshot( tmp_path: pathlib.Path, final_path: pathlib.Path, *, keep_latest_k: int, ) → None#: Atomically publish one validated snapshot and retain recent fallbacks.

nemo_rl.algorithms.rollout_checkpointing.manifest.resolve_latest_snapshot( anchor: pathlib.Path, *, expected_train_step: int, expected_trainer_version: int, expected_bootstrap_fingerprint: Optional[str], ) → Optional[nemo_rl.algorithms.rollout_checkpointing.manifest.ResolvedRolloutCheckpoint]#: Select the newest complete snapshot compatible with its trainer anchor.

nemo_rl.algorithms.rollout_checkpointing.manifest#