nemo_rl.data_plane.adapters.tq_nixl_checkpoint#
Checkpoint save / restore for the NixlStore backend (NixlStorageManager).
TransferQueue saves the controller first and storage second, and restores
storage first. Storage state is each unit’s live blobs plus, per blob, the
gidx@field keys its row index points at it with. Rows are index-addressed
(row gidx lives on unit unit_ids[gidx % N]), so:
same unit count: unit
kreloadsunit-k.shardinto itself;a different count: the driver re-puts every live key through the normal put path, which routes it to its new owner (slower: all bytes pass the driver).
Layout under <checkpoint_dir>/nixl_store/::
manifest.json {"version": 2, "unit_ids": [...], "shards": [...]}
unit-<id>.shard live blobs of one unit, back to back
unit-<id>.shard.rows msgpack {blob_id: [live gidx@field keys]}
Version 1 checkpoints (locations kept in the TQ controller) are rejected.
Callers must be quiescent: TQ’s checkpoint runs under NeMo-RL’s lifecycle guard, so no put/get/clear is in flight while a shard is written.
Module Contents#
Functions#
Dump every live blob reachable from |
|
Checkpointing is an admin path, so it reaches units through Ray directly. |
|
Restore every live key of the manifest onto its owner unit now; return the manifest. |
|
Same unit table: unit |
|
A different unit table: re-put each shard’s live keys through the client. |
Data#
API#
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint.log#
‘getLogger(…)’
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint.SUBDIR#
‘nixl_store’
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint.MANIFEST#
‘manifest.json’
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint.VERSION#
2
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._root(checkpoint_dir: str | os.PathLike) pathlib.Path#
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._write_json(path: pathlib.Path, obj: Any) None#
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint.save_storage_checkpoint(
- client: Any,
- checkpoint_dir: str | os.PathLike,
Dump every live blob reachable from
client.store; return the manifest.
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._unit_actor(
- store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
- unit_id: int,
Checkpointing is an admin path, so it reaches units through Ray directly.
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._active_units( ) list[int]#
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._save_units(
- store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
- root: pathlib.Path,
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint.load_storage_checkpoint(
- client: Any,
- checkpoint_dir: str | os.PathLike,
Restore every live key of the manifest onto its owner unit now; return the manifest.
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._load_in_place(
- store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
- root: pathlib.Path,
- manifest: dict[str, Any],
Same unit table: unit
kreloads its own shard (RAM first, then its SSD tier).
- nemo_rl.data_plane.adapters.tq_nixl_checkpoint._load_rerouted(
- client: Any,
- root: pathlib.Path,
- manifest: dict[str, Any],
A different unit table: re-put each shard’s live keys through the client.