nemo_curator.stages.audio.io.convert

Module Contents

Classes

Name	Description
`AudioToDocumentStage`	Convert AudioTask entries into DocumentBatch DataFrames.

Functions

Name	Description
`_is_tensor`	Check if a value is a torch.Tensor without importing torch at module level.

Data

_NON_SERIALIZABLE_KEYS

API

class nemo_curator.stages.audio.io.convert.AudioToDocumentStage()

Bases: ProcessingStage[AudioTask, DocumentBatch]

Convert AudioTask entries into DocumentBatch DataFrames.

Overrides process_batch to aggregate an entire batch of AudioTask objects into a single multi-row DocumentBatch, avoiding the overhead of many single-row DataFrames. Set batch_size to control how many audio entries land in each DataFrame (default 64).

Non-serializable keys (torch tensors, raw audio arrays) are stripped before building the DataFrame as a safety net, even if upstream stages failed to clean them up.

batch_size

int = 64

name

= 'AudioToDocumentStage'

nemo_curator.stages.audio.io.convert.AudioToDocumentStage._sanitize(
    data: dict
) -> dict

staticmethod

Remove non-serializable keys and any remaining tensor values.

nemo_curator.stages.audio.io.convert.AudioToDocumentStage.process(
    task: nemo_curator.tasks.AudioTask
) -> nemo_curator.tasks.DocumentBatch

nemo_curator.stages.audio.io.convert.AudioToDocumentStage.process_batch(
    tasks: list[nemo_curator.tasks.AudioTask]
) -> list[nemo_curator.tasks.DocumentBatch]

nemo_curator.stages.audio.io.convert._is_tensor(
    v: object
) -> bool

Check if a value is a torch.Tensor without importing torch at module level.

nemo_curator.stages.audio.io.convert._NON_SERIALIZABLE_KEYS = frozenset({'waveform', 'audio', 'audio_data', 'audio_array', 'segments'})