Stages Documentation

Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API morpheus.stages. In addition to this there are several custom stages contained in the Examples and Developer Guides.

  • Doca Stage DocaSourceStage A source stage used to receive raw packet data from a ConnectX-6 Dx NIC. This stage is not compiled by default refer to the Doca Example for details on building this stage.

  • Linear Modules Stage LinearModulesStage Loads an existing, registered, module and wraps it as a Morpheus stage. Refer to Morpheus Modules for details on modules.

  • Monitor Stage MonitorStage Display throughput numbers at a specific point in the pipeline.

  • Multi Port Module Stage MultiPortModulesStage Loads an existing, registered, multi-port module and wraps it as a multi-port Morpheus stage. Refer to Morpheus Modules for details on modules.

  • Trigger Stage TriggerStage Buffer data until the previous stage has completed, useful for testing performance of one stage at a time.

  • AppShield Source Stage AppShieldSourceStage Load Appshield messages from one or more plugins into a dataframe.

  • Azure Source Stage AzureSourceStage Load Azure Active Directory messages.

  • Cloud Trail Source Stage CloudTrailSourceStage Load messages from a Cloudtrail directory.

  • Control Message File Source Stage ControlMessageFileSourceStage Recieves control messages from different sources specified by a list of (fsspec)[https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=open_files#fsspec.open_files] strings.

  • Control Message Kafka Source Stage ControlMessageKafkaSourceStage Load control messages from a Kafka cluster.

  • Databricks Delta Lake Source Stage DataBricksDeltaLakeSourceStage Source stage used to load messages from a DeltaLake table.

  • Duo Source Stage DuoSourceStage Load Duo Authentication messages.

  • File Source Stage FileSourceStage Load messages from a file.

  • HTTP Client Source Stage HttpClientSourceStage Poll a remote HTTP server for incoming data.

  • HTTP Server Source Stage HttpServerSourceStage Start an HTTP server and listens for incoming requests on a specified endpoint.

  • In Memory Source Stage InMemorySourceStage Input source that emits a pre-defined list of dataframes.

  • Kafka Source Stage KafkaSourceStage Load messages from a Kafka cluster.

  • RSS Source Stage RSSSourceStage Load RSS feed items into a pandas DataFrame.

  • LLM Engine Stage LLMEngineStage Execute an LLM engine within a Morpheus pipeline.

  • HTTP Client Sink Stage HttpClientSinkStage Write all messages to an HTTP endpoint.

  • HTTP Server Sink Stage HttpServerSinkStage Start an HTTP server and listens for incoming requests on a specified endpoint.

  • In Memory Sink Stage InMemorySinkStage Collect incoming messages into a list that can be accessed after the pipeline is complete.

  • Databricks Delta Lake Sink Stage DataBricksDeltaLakeSinkStage Write messages to a DeltaLake table.

  • Write To Elastic Search Stage WriteToElasticsearchStage Write the messages as documents to Elasticsearch.

  • Write To File Stage WriteToFileStage Write all messages to a file.

  • Write To Kafka Stage WriteToKafkaStage Write all messages to a Kafka cluster.

  • Write To Vector DB Stage WriteToVectorDBStage Write all messages to a Vector Database.

  • Deserialize Stage DeserializeStage Partition messages based on the pipeline config’s pipeline_batch_size parameter.

  • Drop Null Stage DropNullStage Drop null data entries from a DataFrame.

  • Preprocess AE Stage PreprocessAEStage Prepare Autoencoder input DataFrames for inference.

  • Preprocess FIL Stage PreprocessFILStage Prepare FIL input DataFrames for inference.

  • Preprocess NLP Stage PreprocessNLPStage Prepare NLP input DataFrames for inference.

  • Train AE Stage TrainAEStage Train an Autoencoder model on incoming data.

Previous Typedef PyObject
Next Modules
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.