Stages Documentation
Stages are the building blocks of Morpheus pipelines. Below is a list of the most commonly used stages. For a full list of stages, refer to the stages API morpheus.stages
. In addition to this there are several custom stages contained in the Examples and Developer Guides.
DOCA Source Stage
DocaSourceStage
A source stage used to receive raw packet data in GPU memory from a ConnectX NIC using DOCA GPUNetIO function within a CUDA kernel to actually receive and process Ethernet network packets. Receive packets information is passed to next pipeline stage in the form of RawPacketMessage. This stage is not compiled by default refer to the DOCA Example for details on building this stage.DOCA Convert Stage
DocaConvertStage
Convert the RawPacketMessage format received by the DOCA Source Stage into a more complex message format MetaMessage. Packets’ info never leave the GPU memory. This stage is not compiled by default refer to the DOCA Example for details on building this stage.
Linear Modules Stage
LinearModulesStage
Loads an existing, registered, module and wraps it as a Morpheus stage. Refer to Morpheus Modules for details on modules.Monitor Stage
MonitorStage
Display throughput numbers at a specific point in the pipeline.Multi Port Module Stage
MultiPortModulesStage
Loads an existing, registered, multi-port module and wraps it as a multi-port Morpheus stage. Refer to Morpheus Modules for details on modules.Trigger Stage
TriggerStage
Buffer data until the previous stage has completed, useful for testing performance of one stage at a time.
PyTorch Inference Stage
PyTorchInferenceStage
PyTorch inference stage used for most pipeline modes with the exception of Auto Encoder.Triton Inference Stage
TritonInferenceStage
Inference stage which utilizes a Triton Inference Server.
App Shield Source Stage
AppShieldSourceStage
Load App Shield messages from one or more plugins into a DataFrame.Control Message File Source Stage
ControlMessageFileSourceStage
Receives control messages from different sources specified by a list of (fsspec)[https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=open_files#fsspec.open_files] strings.Control Message Kafka Source Stage
ControlMessageKafkaSourceStage
Load control messages from a Kafka cluster.Databricks Delta Lake Source Stage
DataBricksDeltaLakeSourceStage
Source stage used to load messages from a DeltaLake table.File Source Stage
FileSourceStage
Load messages from a file.HTTP Client Source Stage
HttpClientSourceStage
Poll a remote HTTP server for incoming data.HTTP Server Source Stage
HttpServerSourceStage
Start an HTTP server and listens for incoming requests on a specified endpoint.In Memory Source Stage
InMemorySourceStage
Input source that emits a pre-defined list of DataFrames.Kafka Source Stage
KafkaSourceStage
Load messages from a Kafka cluster.RSS Source Stage
RSSSourceStage
Load RSS feed items into a pandas DataFrame.
LLM Engine Stage
LLMEngineStage
Execute an LLM engine within a Morpheus pipeline.
HTTP Client Sink Stage
HttpClientSinkStage
Write all messages to an HTTP endpoint.HTTP Server Sink Stage
HttpServerSinkStage
Start an HTTP server and listens for incoming requests on a specified endpoint.In Memory Sink Stage
InMemorySinkStage
Collect incoming messages into a list that can be accessed after the pipeline is complete.Databricks Delta Lake Sink Stage
DataBricksDeltaLakeSinkStage
Write messages to a DeltaLake table.Write To Elastic Search Stage
WriteToElasticsearchStage
Write the messages as documents to Elasticsearch.Write To File Stage
WriteToFileStage
Write all messages to a file.Write To Kafka Stage
WriteToKafkaStage
Write all messages to a Kafka cluster.Write To Vector DB Stage
WriteToVectorDBStage
Write all messages to a Vector Database.
Add Classifications Stage
AddClassificationsStage
Add detected classifications to each message.Add Scores Stage
AddScoresStage
Add probability scores to each message.Filter Detections Stage
FilterDetectionsStage
Filter message by a classification threshold.Generate Viz Frames Stage
GenerateVizFramesStage
Write out visualization DataFrames.MLflow Drift Stage
MLFlowDriftStage
Report model drift statistics to MLflow.Serialize Stage
SerializeStage
Include & exclude columns from messages.Time Series Stage
TimeSeriesStage
Perform time series anomaly detection and add prediction.
Deserialize Stage
DeserializeStage
Partition messages based on thepipeline_batch_size
parameter of the pipeline’smorpheus.config.Config
object.Drop Null Stage
DropNullStage
Drop null data entries from a DataFrame.Preprocess FIL Stage
PreprocessFILStage
Prepare FIL input DataFrames for inference.Preprocess NLP Stage
PreprocessNLPStage
Prepare NLP input DataFrames for inference.