`stages.video.io.clip_writer`#

Module Contents#

Classes#

ClipWriterStage

Stage that writes clips and metadata for clip transcoding.

API#

class stages.video.io.clip_writer.ClipWriterStage#

Bases: nemo_curator.stages.base.ProcessingStage[nemo_curator.tasks.video.VideoTask, nemo_curator.tasks.video.VideoTask]

Stage that writes clips and metadata for clip transcoding.

This class processes video clips through a series of steps including embedding generation, metadata extraction, and writing to storage.

static calculate_sha256(buffer: bytes) → str#: Get sha256 of byte array.

caption_models: list[str] | None#: None

dry_run: bool#: None

embedding_algorithm: str#: ‘cosmos-embed1’

enhanced_caption_models: list[str] | None#: None

generate_captions: bool#: None

generate_embeddings: bool#: None

generate_previews: bool#: None

static get_output_path_ce1_embd(output_path: str) → str#: Get path to store generated clip embeddings of Cosmos-Embed1.

static get_output_path_ce1_embd_parquet(output_path: str) → str#: Get path to store generated clip embeddings of Cosmos-Embed1 in a parquet file.

static get_output_path_clips( output_path: str, *, filtered: bool = False, ) → str#: Get path to store generated clips.

static get_output_path_iv2_embd(output_path: str) → str#: Get path to store generated clips.

static get_output_path_iv2_embd_parquet(output_path: str) → str#: Get path to store generated clip embeddings in a parquet file.

static get_output_path_metas(output_path: str, version: str) → str#: Get path to store clip metadatas.

static get_output_path_previews(output_path: str) → str#: Get path to store generated clips.

static get_output_path_processed_clip_chunks(output_path: str) → str#: Get path to store processed clip chunks.

static get_output_path_processed_videos(output_path: str) → str#: Get path to store processed videos.

input_path: str#: None

inputs() → tuple[list[str], list[str]]#

Define stage input requirements.

Returns (tuple[list[str], list[str]]): Tuple of (required_attributes, required_columns) where: - required_top_level_attributes: List of task attributes that must be present - required_data_attributes: List of attributes within the data that must be present

log_stats: bool#: False

max_workers: int#: 6

output_path: str#: None

outputs() → tuple[list[str], list[str]]#

Define stage output specification.

Returns (tuple[list[str], list[str]]): Tuple of (output_attributes, output_columns) where: - output_top_level_attributes: List of task attributes this stage adds/modifies - output_data_attributes: List of attributes within the data that this stage adds/modifies

process( task: nemo_curator.tasks.video.VideoTask, ) → nemo_curator.tasks.video.VideoTask#: Process a task and return the result. Args: task (X): Input task to process Returns (Y | list[Y]): - Single task: For 1-to-1 transformations - List of tasks: For 1-to-many transformations (e.g., readers) - None: If the task should be filtered out

setup( worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None, ) → None#: Setup method called once before processing begins. Override this method to perform any initialization that should happen once per worker. Args: worker_metadata (WorkerMetadata, optional): Information about the worker (provided by some backends)

upload_clips: bool#: None

verbose: bool#: False

stages.video.io.clip_writer#

Module Contents#

Classes#

API#

`stages.video.io.clip_writer`#