nemo_rl.data_plane.adapters.tq_nixl_checkpoint#

Checkpoint save / restore for the NixlStore backend (NixlStorageManager).

TransferQueue saves the controller first and storage second, and restores storage first. Storage state is each unit’s live blobs plus, per blob, the gidx@field keys its row index points at it with. Rows are index-addressed (row gidx lives on unit unit_ids[gidx % N]), so:

  • same unit count: unit k reloads unit-k.shard into itself;

  • a different count: the driver re-puts every live key through the normal put path, which routes it to its new owner (slower: all bytes pass the driver).

Layout under <checkpoint_dir>/nixl_store/::

manifest.json        {"version": 2, "unit_ids": [...], "shards": [...]}
unit-<id>.shard      live blobs of one unit, back to back
unit-<id>.shard.rows msgpack {blob_id: [live gidx@field keys]}

Version 1 checkpoints (locations kept in the TQ controller) are rejected.

Callers must be quiescent: TQ’s checkpoint runs under NeMo-RL’s lifecycle guard, so no put/get/clear is in flight while a shard is written.

Module Contents#

Functions#

_root

_write_json

save_storage_checkpoint

Dump every live blob reachable from client.store; return the manifest.

_unit_actor

Checkpointing is an admin path, so it reaches units through Ray directly.

_active_units

_save_units

load_storage_checkpoint

Restore every live key of the manifest onto its owner unit now; return the manifest.

_load_in_place

Same unit table: unit k reloads its own shard (RAM first, then its SSD tier).

_load_rerouted

A different unit table: re-put each shard’s live keys through the client.

Data#

API#

nemo_rl.data_plane.adapters.tq_nixl_checkpoint.log#

‘getLogger(…)’

nemo_rl.data_plane.adapters.tq_nixl_checkpoint.SUBDIR#

‘nixl_store’

nemo_rl.data_plane.adapters.tq_nixl_checkpoint.MANIFEST#

‘manifest.json’

nemo_rl.data_plane.adapters.tq_nixl_checkpoint.VERSION#

2

nemo_rl.data_plane.adapters.tq_nixl_checkpoint._root(checkpoint_dir: str | os.PathLike) → pathlib.Path#
nemo_rl.data_plane.adapters.tq_nixl_checkpoint._write_json(path: pathlib.Path, obj: Any) → None#
nemo_rl.data_plane.adapters.tq_nixl_checkpoint.save_storage_checkpoint(
client: Any,
checkpoint_dir: str | os.PathLike,
) → dict[str, Any]#

Dump every live blob reachable from client.store; return the manifest.

nemo_rl.data_plane.adapters.tq_nixl_checkpoint._unit_actor(
store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
unit_id: int,
)#

Checkpointing is an admin path, so it reaches units through Ray directly.

nemo_rl.data_plane.adapters.tq_nixl_checkpoint._active_units(
store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
) → list[int]#
nemo_rl.data_plane.adapters.tq_nixl_checkpoint._save_units(
store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
root: pathlib.Path,
) → dict[str, Any]#
nemo_rl.data_plane.adapters.tq_nixl_checkpoint.load_storage_checkpoint(
client: Any,
checkpoint_dir: str | os.PathLike,
) → dict[str, Any]#

Restore every live key of the manifest onto its owner unit now; return the manifest.

nemo_rl.data_plane.adapters.tq_nixl_checkpoint._load_in_place(
store: nemo_rl.data_plane.nixl.blobstore.UnitSlabStore,
root: pathlib.Path,
manifest: dict[str, Any],
) → None#

Same unit table: unit k reloads its own shard (RAM first, then its SSD tier).

nemo_rl.data_plane.adapters.tq_nixl_checkpoint._load_rerouted(
client: Any,
root: pathlib.Path,
manifest: dict[str, Any],
) → None#

A different unit table: re-put each shard’s live keys through the client.