nv_ingest.framework.orchestration.ray.stages.transforms package#

Submodules#

nv_ingest.framework.orchestration.ray.stages.transforms.image_caption module#

nv_ingest.framework.orchestration.ray.stages.transforms.text_embed module#

nv_ingest.framework.orchestration.ray.stages.transforms.text_splitter module#

nv_ingest.framework.orchestration.ray.stages.transforms.text_splitter.text_splitter_fn(
control_message: IngestControlMessage,
stage_config: TextSplitterSchema,
) → IngestControlMessage[source]#

Process an incoming IngestControlMessage by splitting and tokenizing its text.

Parameters:

  • control_message (IngestControlMessage) – The incoming message containing the payload DataFrame.

  • stage_config (BaseModel) – The stage level configuration object

Returns:

The updated message with its payload transformed.

Return type:

IngestControlMessage

Module contents#