nemo_rl.models.megatron.ckpt_parallel_read#

Parallel reads for torch_dist checkpoint loads.

PyTorch’s FileSystemReader.read_data reads a rank’s items one at a time from one stream, and Megatron builds its reader with no load-side thread count, so this patches read_data to read the same byte ranges with N threads (16 MB requests, at most WINDOW_BYTES ahead) while the calling thread loads the items in order with PyTorch’s own code. Peak host memory per rank: WINDOW_BYTES plus ~2x the largest item.

Module Contents#

Functions#

install

Patch FileSystemReader.read_data to read with threads threads (if > 1).

_read_file

_load_oldest

_pread_into

_load_item

Data#

API#

nemo_rl.models.megatron.ckpt_parallel_read.CHUNK_BYTES#

None

nemo_rl.models.megatron.ckpt_parallel_read.WINDOW_BYTES#

None

nemo_rl.models.megatron.ckpt_parallel_read.READ_DATA_SHA256#

‘86ad890ac621f4f563ee5f66a2059ebb3a3ba022a58d91d4e23d757405c691be’

nemo_rl.models.megatron.ckpt_parallel_read._original_read_data#

None

nemo_rl.models.megatron.ckpt_parallel_read.install(threads: int) → None#

Patch FileSystemReader.read_data to read with threads threads (if > 1).

nemo_rl.models.megatron.ckpt_parallel_read._read_file(
reader: Any,
planner: Any,
fd: int,
pool: Any,
reqs: list,
) → None#
nemo_rl.models.megatron.ckpt_parallel_read._load_oldest(
reader: Any,
planner: Any,
pending: collections.deque,
) → int#
nemo_rl.models.megatron.ckpt_parallel_read._pread_into(fd: int, view: memoryview, offset: int) → None#
nemo_rl.models.megatron.ckpt_parallel_read._load_item(
reader: Any,
planner: Any,
req: Any,
buf: bytearray,
) → None#