nemo_rl.algorithms.single_controller_utils.debug_writer#
Stream untruncated training tensors without retaining a second step batch.
Module Contents#
Classes#
One chunk of per-sample training columns handed to a DebugWriter. |
|
What one add_chunk call stored, handed to the controller for finish_step. |
|
Base class for writers that store a step’s training rows chunk by chunk. |
|
Write each shard’s rows to JSONL part files, merged into one file per step. |
|
Write each chunk as its own Parquet file, listed by a per-step manifest. |
Functions#
Return the Parquet schema of a training dump row. |
|
Move the tensor’s CPU storages, jagged values and offsets included, to shared memory. |
|
Return the chunk with every tensor column in shared memory, ready to pickle. |
|
Build the debug writer that cfg selects for one shard. |
Data#
API#
- class nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriterConfig#
Bases:
pydantic.BaseModel- target: str#
‘parquet’
- class nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk#
One chunk of per-sample training columns handed to a DebugWriter.
.. attribute:: step
Optimizer step the rows belong to.
.. attribute:: sample_ids
One id per sample in the chunk.
.. attribute:: tags
Per-sample metadata dicts, or None for empty metadata.
.. attribute:: input_lengths
Valid token count per sample, used to trim sequence columns.
.. attribute:: sequences
Per-sample columns, dense or jagged, that the writer trims to input_lengths, such as token ids and advantages.
.. attribute:: scalars
Per-sample columns written as given; variable-length values must be jagged tensors rather than padded.
With background writing, add_chunk moves the CPU tensors of the chunk to shared memory, so the caller must not modify them after it returns.
- step: int#
None
- sample_ids: list[str]#
None
- tags: list[dict[str, Any]] | None#
None
- input_lengths: torch.Tensor#
None
- sequences: dict[str, torch.Tensor]#
None
- scalars: dict[str, torch.Tensor]#
None
- class nemo_rl.algorithms.single_controller_utils.debug_writer.ChunkReceipt#
What one add_chunk call stored, handed to the controller for finish_step.
.. attribute:: shard_id
Id of the shard whose writer stored the chunk.
.. attribute:: rows
Number of rows stored.
.. attribute:: ref
Writer-specific location of the stored rows, such as a part file path or an object key.
- shard_id: str#
None
- rows: int#
None
- ref: Any#
None
- class nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriter(
- log_dir: str,
- *,
- shard_id: str = '0',
- background: bool = False,
Bases:
abc.ABCBase class for writers that store a step’s training rows chunk by chunk.
Each shard calls add_chunk on its own instance, and the controller calls finish_step on another once the step completes and close once at shutdown.
A subclass that defines init must accept log_dir, the keyword-only shard_id and background, and pass them to this base. Extra keys of the debug_writer config arrive as further keyword arguments.
Initialization
Store the log directory and shard id and start with no rows written.
- Parameters:
log_dir – Directory the writer stores rows under.
shard_id – Identifier of the shard that owns this writer.
background – Whether _write_chunk hands its writing to a separate process; flush waits for the chunks handed over.
- add_chunk( ) nemo_rl.algorithms.single_controller_utils.debug_writer.ChunkReceipt#
Validate one chunk of per-sample rows and write it to storage.
The first chunk of a step replaces any rows already stored for it; later chunks append to them.
- Parameters:
chunk – The chunk’s step and per-sample columns.
- Returns:
The receipt for the stored rows, to be passed on to finish_step.
- _validate( ) None#
Raise ValueError if the chunk’s columns disagree in length or a sequence is too short.
- finish_step(
- step: int,
- receipts: list[nemo_rl.algorithms.single_controller_utils.debug_writer.ChunkReceipt],
Publish a completed optimizer step and reset the per-step state.
- Parameters:
step – The completed optimizer step.
receipts – Receipts add_chunk returned for this step’s chunks, naming what to publish and how many rows it must hold.
- _submit(fn: Callable[..., None], *args: Any) None#
Run fn(*args) in the background process, with at most two pending.
- flush() None#
Wait until every chunk handed to the background is written.
- close() None#
Finish the pending chunks and stop the background process.
- abstractmethod _write_chunk(
- chunk: nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk,
- *,
- replace: bool,
Write one validated chunk to storage and return the receipt for its rows.
The first chunk of a step comes with replace set, telling the writer to drop any rows already stored for that step.
- abstractmethod _publish_step(
- step: int,
- receipts: list[nemo_rl.algorithms.single_controller_utils.debug_writer.ChunkReceipt],
Publish a completed step’s rows, raising on a row-count mismatch.
- class nemo_rl.algorithms.single_controller_utils.debug_writer.JsonDebugWriter(
- log_dir: str,
- *,
- shard_id: str = '0',
- background: bool = False,
Bases:
nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriterWrite each shard’s rows to JSONL part files, merged into one file per step.
Rows use the legacy train_data singleton-batch shape, and publishing numbers them with idx across the merged file. A failed step leaves part files, never a completed-looking JSONL file. Shards must share a filesystem with the controller.
Initialization
Store the log directory and shard id and start with no rows written.
- Parameters:
log_dir – Directory the writer stores rows under.
shard_id – Identifier of the shard that owns this writer.
background – Whether _write_chunk hands its writing to a separate process; flush waits for the chunks handed over.
- _dump_path(step: int) pathlib.Path#
Return the published JSONL path.
- _shard_path(step: int) pathlib.Path#
Return this shard’s part file path.
- _write_chunk(
- chunk: nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk,
- *,
- replace: bool,
Write the chunk’s rows to this shard’s part file, starting it over if replace is set.
- static _write_part(
- part: pathlib.Path,
- mode: str,
- chunk: nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk,
Write one validated chunk’s rows to its part file (maybe in the background).
- _publish_step(
- step: int,
- receipts: list[nemo_rl.algorithms.single_controller_utils.debug_writer.ChunkReceipt],
Merge the part files the receipts name into one JSONL file and publish it atomically.
- nemo_rl.algorithms.single_controller_utils.debug_writer._parquet_schema() pyarrow.Schema#
Return the Parquet schema of a training dump row.
- class nemo_rl.algorithms.single_controller_utils.debug_writer.ParquetDebugWriter(
- log_dir: str,
- *,
- shard_id: str = '0',
- background: bool = False,
Bases:
nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriterWrite each chunk as its own Parquet file, listed by a per-step manifest.
Files go under train_data_parquet/step-
/ in the log directory. Columns a step does not produce are null. A step is complete only once _manifest.json exists in its directory; read only the files it lists. Shards must share a filesystem with the controller. pyarrow is imported when a chunk is written, so the json writer does not need it. Initialization
Store the log directory and shard id and start with no rows written.
- Parameters:
log_dir – Directory the writer stores rows under.
shard_id – Identifier of the shard that owns this writer.
background – Whether _write_chunk hands its writing to a separate process; flush waits for the chunks handed over.
- _validate( ) None#
Also raise ValueError for a column the Parquet schema lacks, before any background write.
- _step_dir(step: int) pathlib.Path#
Return the directory holding the step’s chunk files and manifest.
- _write_chunk(
- chunk: nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk,
- *,
- replace: bool,
Write the chunk to a new Parquet file; _publish_step drops the files of earlier attempts.
- static _write_file(
- directory: pathlib.Path,
- name: str,
- chunk: nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk,
Write one validated chunk to directory/name atomically (maybe in the background).
- _publish_step(
- step: int,
- receipts: list[nemo_rl.algorithms.single_controller_utils.debug_writer.ChunkReceipt],
Write _manifest.json listing the step’s chunk files, after checking them against the receipts.
Move the tensor’s CPU storages, jagged values and offsets included, to shared memory.
-
) nemo_rl.algorithms.single_controller_utils.debug_writer.TrainChunk#
Return the chunk with every tensor column in shared memory, ready to pickle.
- nemo_rl.algorithms.single_controller_utils.debug_writer.DEBUG_WRITER_REGISTRY: dict[str, Callable[..., nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriter]]#
None
- nemo_rl.algorithms.single_controller_utils.debug_writer.make_debug_writer(
- cfg: nemo_rl.algorithms.single_controller_utils.debug_writer.DebugWriterConfig,
- log_dir: str,
- *,
- shard_id: str = '0',
- background: bool = False,
Build the debug writer that cfg selects for one shard.
A target found in DEBUG_WRITER_REGISTRY picks that writer; any other target containing a dot is imported as a dotted path to a DebugWriter subclass.
- Parameters:
cfg – Debug writer config naming the writer and any constructor keys.
log_dir – Base log directory handed to the writer.
shard_id – Identifier of the shard that owns the returned writer.
background – Write chunks in a separate process; passed to the writer’s constructor.
- Returns:
The constructed writer.