nemo_rl.models.megatron.ckpt_parallel_read#
Parallel reads for torch_dist checkpoint loads.
PyTorch’s FileSystemReader.read_data reads a rank’s items one at a time from one
stream, and Megatron builds its reader with no load-side thread count, so this patches
read_data to read the same byte ranges with N threads (16 MB requests, at most
WINDOW_BYTES ahead) while the calling thread loads the items in order with
PyTorch’s own code. Peak host memory per rank: WINDOW_BYTES plus ~2x the largest item.
Module Contents#
Functions#
Patch |
|
Data#
API#
- nemo_rl.models.megatron.ckpt_parallel_read.CHUNK_BYTES#
None
- nemo_rl.models.megatron.ckpt_parallel_read.WINDOW_BYTES#
None
- nemo_rl.models.megatron.ckpt_parallel_read.READ_DATA_SHA256#
‘86ad890ac621f4f563ee5f66a2059ebb3a3ba022a58d91d4e23d757405c691be’
- nemo_rl.models.megatron.ckpt_parallel_read._original_read_data#
None
- nemo_rl.models.megatron.ckpt_parallel_read.install(threads: int) None#
Patch
FileSystemReader.read_datato read withthreadsthreads (if > 1).
- nemo_rl.models.megatron.ckpt_parallel_read._read_file(
- reader: Any,
- planner: Any,
- fd: int,
- pool: Any,
- reqs: list,
- nemo_rl.models.megatron.ckpt_parallel_read._load_oldest(
- reader: Any,
- planner: Any,
- pending: collections.deque,
- nemo_rl.models.megatron.ckpt_parallel_read._pread_into(fd: int, view: memoryview, offset: int) None#
- nemo_rl.models.megatron.ckpt_parallel_read._load_item(
- reader: Any,
- planner: Any,
- req: Any,
- buf: bytearray,