Stages Documentation

Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API morpheus.stages. In addition to this there are several custom stages contained in the Examples and Developer Guides.

• Doca

• General

• Inference

• Input

• LLM

• Output

• Post-process

• Pre-process

• Doca Stage DocaSourceStage A source stage used to receive raw packet data from a ConnectX-6 Dx NIC. This stage is not compiled by default refer to the Doca Example for details on building this stage.

• Linear Modules Stage LinearModulesStage Loads an existing, registered, module and wraps it as a Morpheus stage. Refer to Morpheus Modules for details on modules.

• Monitor Stage MonitorStage Display throughput numbers at a specific point in the pipeline.

• Multi Port Module Stage MultiPortModulesStage Loads an existing, registered, multi-port module and wraps it as a multi-port Morpheus stage. Refer to Morpheus Modules for details on modules.

• Trigger Stage TriggerStage Buffer data until the previous stage has completed, useful for testing performance of one stage at a time.

• Auto Encoder Inference Stage AutoEncoderInferenceStage PyTorch inference stage used for Auto Encoder pipeline mode.

• PyTorch Inference Stage PyTorchInferenceStage PyTorch inference stage used for most pipeline modes with the exception of Auto Encoder.

• Triton Inference Stage TritonInferenceStage Inference stage which utilizes a Triton Inference Server.

• AppShield Source Stage AppShieldSourceStage Load Appshield messages from one or more plugins into a dataframe.

• Azure Source Stage AzureSourceStage Load Azure Active Directory messages.

• Cloud Trail Source Stage CloudTrailSourceStage Load messages from a Cloudtrail directory.

• Control Message File Source Stage ControlMessageFileSourceStage Recieves control messages from different sources specified by a list of (fsspec)[https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=open_files#fsspec.open_files] strings.

• Control Message Kafka Source Stage ControlMessageKafkaSourceStage Load control messages from a Kafka cluster.

• Databricks Delta Lake Source Stage DataBricksDeltaLakeSourceStage Source stage used to load messages from a DeltaLake table.

• Duo Source Stage DuoSourceStage Load Duo Authentication messages.

• File Source Stage FileSourceStage Load messages from a file.

• HTTP Client Source Stage HttpClientSourceStage Poll a remote HTTP server for incoming data.

• HTTP Server Source Stage HttpServerSourceStage Start an HTTP server and listens for incoming requests on a specified endpoint.

• In Memory Source Stage InMemorySourceStage Input source that emits a pre-defined list of dataframes.

• Kafka Source Stage KafkaSourceStage Load messages from a Kafka cluster.

• RSS Source Stage RSSSourceStage Load RSS feed items into a pandas DataFrame.

• LLM Engine Stage LLMEngineStage Execute an LLM engine within a Morpheus pipeline.

• HTTP Client Sink Stage HttpClientSinkStage Write all messages to an HTTP endpoint.

• HTTP Server Sink Stage HttpServerSinkStage Start an HTTP server and listens for incoming requests on a specified endpoint.

• In Memory Sink Stage InMemorySinkStage Collect incoming messages into a list that can be accessed after the pipeline is complete.

• Databricks Delta Lake Sink Stage DataBricksDeltaLakeSinkStage Write messages to a DeltaLake table.

• Write To Elastic Search Stage WriteToElasticsearchStage Write the messages as documents to Elasticsearch.

• Write To File Stage WriteToFileStage Write all messages to a file.

• Write To Kafka Stage WriteToKafkaStage Write all messages to a Kafka cluster.

• Write To Vector DB Stage WriteToVectorDBStage Write all messages to a Vector Database.

• Add Classifications Stage AddClassificationsStage Add detected classifications to each message.

• Add Scores Stage AddScoresStage Add probability scores to each message.

• Filter Detections Stage FilterDetectionsStage Filter message by a classification threshold.

• Generate Viz Frames Stage GenerateVizFramesStage Write out visualization DataFrames.

• MLflow Drift Stage MLFlowDriftStage Report model drift statistics to MLflow.

• Serialize Stage SerializeStage Include & exclude columns from messages.

• Timeseries Stage TimeSeriesStage Perform time series anomaly detection and add prediction.

• Deserialize Stage DeserializeStage Partition messages based on the pipeline config’s pipeline_batch_size parameter.

• Drop Null Stage DropNullStage Drop null data entries from a DataFrame.

• Preprocess AE Stage PreprocessAEStage Prepare Autoencoder input DataFrames for inference.

• Preprocess FIL Stage PreprocessFILStage Prepare FIL input DataFrames for inference.

• Preprocess NLP Stage PreprocessNLPStage Prepare NLP input DataFrames for inference.

• Train AE Stage TrainAEStage Train an Autoencoder model on incoming data.