nv_ingest.framework.orchestration.morpheus.stages.transforms package#

Submodules#

nv_ingest.framework.orchestration.morpheus.stages.transforms.embed_text_stage module#

nv_ingest.framework.orchestration.morpheus.stages.transforms.embed_text_stage.generate_text_embed_extractor_stage(
c: Any,
transform_config: Dict[str, Any],
task: str = 'embed',
task_desc: str = 'text_embed_extraction',
pe_count: int = 1,
)[source]#

Generates a multiprocessing stage to perform text embedding extraction from a pandas DataFrame.

Parameters:

  • c (Any) – Global configuration object.

  • transform_config (Dict[str, Any]) – Configuration parameters for the text embedding extractor, validated against EmbedExtractionsSchema.

  • task (str, optional) – The task name for the stage worker function (default is “embed”).

  • task_desc (str, optional) – A descriptor used for latency tracing and logging (default is “text_embed_extraction”).

  • pe_count (int, optional) – Number of process engines to use concurrently (default is 1).

Returns:

A configured stage that processes a pandas DataFrame and returns a tuple of (DataFrame, trace_info dict).

Return type:

MultiProcessingBaseStage

nv_ingest.framework.orchestration.morpheus.stages.transforms.image_caption_extraction module#

nv_ingest.framework.orchestration.morpheus.stages.transforms.image_caption_extraction.generate_caption_extraction_stage(
c: Config,
transform_config: Dict[str, Any],
task: str = 'caption',
task_desc: str = 'caption_extraction',
pe_count: int = 8,
)[source]#

Generates a caption extraction stage with the specified configuration.

Parameters:

  • c (Config) – Morpheus global configuration object.

  • transform_config (dict) – Configuration parameters for caption extraction.

  • task (str, optional) – The task name to match for the stage worker function, by default “caption”.

  • task_desc (str, optional) – A descriptor to be used in latency tracing, by default “caption_extraction”.

  • pe_count (int, optional) – Number of processing elements to use, by default 8.

Returns:

The generated caption extraction stage.

Return type:

MultiProcessingBaseStage

Raises:

ValueError – If an error occurs during stage generation.

Module contents#

nv_ingest.framework.orchestration.morpheus.stages.transforms.generate_caption_extraction_stage(
c: Config,
transform_config: Dict[str, Any],
task: str = 'caption',
task_desc: str = 'caption_extraction',
pe_count: int = 8,
)[source]#

Generates a caption extraction stage with the specified configuration.

Parameters:

  • c (Config) – Morpheus global configuration object.

  • transform_config (dict) – Configuration parameters for caption extraction.

  • task (str, optional) – The task name to match for the stage worker function, by default “caption”.

  • task_desc (str, optional) – A descriptor to be used in latency tracing, by default “caption_extraction”.

  • pe_count (int, optional) – Number of processing elements to use, by default 8.

Returns:

The generated caption extraction stage.

Return type:

MultiProcessingBaseStage

Raises:

ValueError – If an error occurs during stage generation.