nv_ingest.stages.storages package#

Submodules#

nv_ingest.stages.storages.embedding_storage_stage module#

nv_ingest.stages.storages.embedding_storage_stage.generate_embedding_storage_stage(
c: Config,
task: str = 'store_embedding',
task_desc: str = 'Store_embeddings_minio',
pe_count: int = 1,
)[source]#

Helper function to generate a multiprocessing stage to perform pdf content extraction.

Parameters:

  • c (Config) – Morpheus global configuration object

  • embedding_storage_config (dict) – Configuration parameters for embedding storage.

  • task (str) – The task name to match for the stage worker function.

  • task_desc (str) – A descriptor to be used in latency tracing.

  • pe_count (int) – Integer for how many process engines to use for pdf content extraction.

Returns:

A Morpheus stage with applied worker function.

Return type:

MultiProcessingBaseStage

nv_ingest.stages.storages.embedding_storage_stage.upload_embeddings(
df: DataFrame,
params: Dict[str, Any],
) → DataFrame[source]#

Identify contents (e.g., images) within a dataframe and uploads the data to MinIO. The image metadata in the metadata column is updated with the URL of the uploaded data.

nv_ingest.stages.storages.image_storage_stage module#

class nv_ingest.stages.storages.image_storage_stage.ImageStorageStage(
config: Config,
module_config: Dict = None,
raise_on_failure: bool = False,
)[source]#

Bases: PassThruTypeMixin, SinglePortStage

Stores images.

Parameters:

config (Config) – Pipeline configuration instance.

accepted_types() → Tuple[source]#

Returns accepted input types for this stage.

Returns:

Accepted input types.

Return type:

Tuple(IngestControlMessage, MultiResponseMessage, MultiMessage)

property name: str#

The name of the stage. Used in logging. Each derived class should override this property with a unique name.

Returns:

Name of a stage.

Return type:

str

supported_execution_modes() → tuple[ExecutionMode][source]#

Returns a tuple of supported execution modes of this stage. By default this returns (ExecutionMode.GPU,). Subclasses can override this method to specify different execution modes.

For most stages the values will be static, and this can be accomplished by making use of either the CpuOnlyMixin or GpuAndCpuMixin mixins.

However, complex stages may choose to make this decision at runtime, in which case this method should be overridden. directly within the stage class.

supports_cpp_node()[source]#

Indicates whether this stage supports a C++ node.

Module contents#