Message classes, which contain data being transfered between pipeline stages
- class InferenceMemory(*args, **kwargs)[source]
This is a base container class for data that will be used for inference stages. This class is designed to hold generic tensor data in cupy arrays.
- class InferenceMemoryAE(*args, **kwargs)[source]
This is a container class for data that needs to be submitted to the inference server for auto encoder usecases.
- Parameters
- inputcupy.ndarray
Inference input.
- seq_idscupy.ndarray
Ids used to index from an inference input to a message. Necessary since there can be more inference inputs than messages (i.e., if some messages get broken into multiple inference requests).
- Attributes
- input
- seq_ids
- class InferenceMemoryFIL(*args, **kwargs)[source]
This is a container class for data that needs to be submitted to the inference server for FIL category usecases.
- Parameters
- input__0cupy.ndarray
Inference input.
- seq_idscupy.ndarray
Ids used to index from an inference input to a message. Necessary since there can be more inference inputs than messages (i.e., if some messages get broken into multiple inference requests).
- Attributes
- input__0
- seq_ids
- class InferenceMemoryNLP(*args, **kwargs)[source]
This is a container class for data that needs to be submitted to the inference server for NLP category usecases.
- Parameters
- input_idscupy.ndarray
The token-ids for each string padded with 0s to max_length.
- input_maskcupy.ndarray
The mask for token-ids result where corresponding positions identify valid token-id values.
- seq_idscupy.ndarray
Ids used to index from an inference input to a message. Necessary since there can be more inference inputs than messages (i.e., if some messages get broken into multiple inference requests).
- Attributes
- input_ids
- input_mask
- seq_ids
- class MessageBase(*args, **kwargs)[source]
Base class for all messages. Returns a C++ implementation if
CppConfig.get_should_use_cpp()
isTrue
and the class has an associated C++ implementation (cpp_class
), returns the Python implementation for all others.
- class MessageMeta(*args, **kwargs)[source]
This is a container class to hold batch deserialized messages metadata.
- Parameters
- dfpandas.DataFrame
Input rows in dataframe.
- Attributes
count
Returns the number of messages in the batch.
- property count: int
Returns the number of messages in the batch.
- Returns
- int
number of messages in the MessageMeta.df.
- class MultiAEMessage(*args, **kwargs)[source]
Subclass of
MultiMessage
specific to the AutoEncoder pipeline, which contains the model.- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_ranges
(ranges[, num_selected_rows])Perform a copy of the current message instance for the given
ranges
of rows.get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_slice
(start, stop)Returns sliced batches based on offsets supplied.
set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- copy_ranges(ranges, num_selected_rows=None)[source]
Perform a copy of the current message instance for the given
ranges
of rows.- Parameters
- rangestyping.List[typing.Tuple[int, int]]
Rows to include in the copy in the form of
[(`start_row
,stop_row
),…]` The final output is exclusive of thestop_row
, i.e.[start_row, stop_row)
. For example to copy rows 1-2 & 5-7ranges=[(1, 3), (5, 8)]
- num_selected_rowstyping.Union[None, int]
Optional specify the number of rows selected by
ranges
, otherwise this is computed by the result.
- Returns
- get_slice(start, stop)[source]
Returns sliced batches based on offsets supplied. Automatically calculates the correct
mess_offset
andmess_count
.- Parameters
- startint
Start offset address.
- stopint
Stop offset address.
- Returns
- morpheus.pipeline.preprocess.autoencoder.MultiAEMessage
A new
MultiAEMessage
with sliced offset and count.
- class MultiInferenceFILMessage(*args, **kwargs)[source]
A stronger typed version of
MultiInferenceMessage
that is used for FIL workloads. Helps ensure the proper inputs are set and eases debugging.- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.input__0
Input to FIL model inference.
inputs
Get inputs stored in the TensorMemory container.
seq_ids
Returns sequence ids, which are used to keep track of messages in a multi-threaded environment.
timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_ranges
(ranges[, num_selected_rows])Perform a copy of the current message instance for the given
ranges
of rows.get_input
(name)Get input stored in the TensorMemory container.
get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_slice
(start, stop)Returns sliced batches based on offsets supplied.
set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- property input__0
Input to FIL model inference.
- Returns
- cupy.ndarray
Input data.
- property seq_ids
Returns sequence ids, which are used to keep track of messages in a multi-threaded environment.
- Returns
- cupy.ndarray
Sequence ids.
- class MultiInferenceMessage(*args, **kwargs)[source]
This is a container class that holds the TensorMemory container and the metadata of the data contained within it. Builds on top of the
MultiMessage
class to add additional data for inferencing.This class requires two separate memory blocks for a batch. One for the message metadata (i.e., start time, IP address, etc.) and another for the raw inference inputs (i.e., input_ids, seq_ids). Since there can be more inference input requests than messages (This happens when some messages get broken into multiple inference requests) this class stores two different offset and count values.
mess_offset
andmess_count
refer to the offset and count in the message metadata batch andoffset
andcount
index into the inference batch data.- Parameters
- memory<bsp-code-inline code="TensorMemory">TensorMemory</bsp-code-inline>
Inference memory.
- offsetint
Message offset in inference memory instance.
- countint
Message count in inference memory instance.
- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.inputs
Get inputs stored in the TensorMemory container.
timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_ranges
(ranges[, num_selected_rows])Perform a copy of the current message instance for the given
ranges
of rows.get_input
(name)Get input stored in the TensorMemory container.
get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_slice
(start, stop)Returns sliced batches based on offsets supplied.
set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- get_input(name)[source]
Get input stored in the TensorMemory container.
- Parameters
- namestr
Input key name.
- Returns
- cupy.ndarray
Inference input.
- get_slice(start, stop)[source]
Returns sliced batches based on offsets supplied. Automatically calculates the correct
mess_offset
andmess_count
.- Parameters
- startint
Start offset address.
- stopint
Stop offset address.
- Returns
MultiInferenceMessage
A new
MultiInferenceMessage
with sliced offset and count.
- property inputs
Get inputs stored in the TensorMemory container.
- Returns
- cupy.ndarray
Inference inputs.
- class MultiInferenceNLPMessage(*args, **kwargs)[source]
A stronger typed version of
MultiInferenceMessage
that is used for NLP workloads. Helps ensure the proper inputs are set and eases debugging.- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.input_ids
Returns token-ids for each string padded with 0s to max_length.
input_mask
Returns mask for token-ids result where corresponding positions identify valid token-id values.
inputs
Get inputs stored in the TensorMemory container.
seq_ids
Returns sequence ids, which are used to keep track of which inference requests belong to each message.
timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_ranges
(ranges[, num_selected_rows])Perform a copy of the current message instance for the given
ranges
of rows.get_input
(name)Get input stored in the TensorMemory container.
get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_slice
(start, stop)Returns sliced batches based on offsets supplied.
set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- property input_ids
Returns token-ids for each string padded with 0s to max_length.
- Returns
- cupy.ndarray
The token-ids for each string padded with 0s to max_length.
- property input_mask
Returns mask for token-ids result where corresponding positions identify valid token-id values.
- Returns
- cupy.ndarray
The mask for token-ids result where corresponding positions identify valid token-id values.
- property seq_ids
Returns sequence ids, which are used to keep track of which inference requests belong to each message.
- Returns
- cupy.ndarray
Ids used to index from an inference input to a message. Necessary since there can be more inference inputs than messages (i.e., if some messages get broken into multiple inference requests).
- class MultiMessage(*args, **kwargs)[source]
This class holds data for multiple messages at a time. To avoid copying data for slicing operations, it holds a reference to a batched metadata object and stores the offset and count into that batch.
- Parameters
- meta<bsp-code-inline code="<a href="#morpheus.messages.MessageMeta">MessageMeta</a>"><a href="#morpheus.messages.MessageMeta">MessageMeta</a></bsp-code-inline>
Deserialized messages metadata for large batch.
- mess_offsetint
Offset into the metadata batch.
- mess_countint
Messages count.
- Attributes
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_ranges
(ranges[, num_selected_rows])Perform a copy of the current message instance for the given
ranges
of rows.get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_slice
(start, stop)Returns sliced batches based on offsets supplied.
set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- copy_meta_ranges(ranges, mask=None)[source]
Perform a copy of the underlying dataframe for the given
ranges
of rows.- Parameters
- rangestyping.List[typing.Tuple[int, int]]
Rows to include in the copy in the form of
[(`start_row
,stop_row
),…]` Thestop_row
isn’t included. For example to copy rows 1-2 & 5-7ranges=[(1, 3), (5, 8)]
- masktyping.Union[None, cupy.ndarray, numpy.ndarray]
Optionally specify rows as a cupy array (when using cudf Dataframes) or a numpy array (when using pandas Dataframes) of booleans. When not-None
ranges
will be ignored. This is useful as an optimization as this avoids needing to generate the mask on it’s own.
- Returns
Dataframe
- copy_ranges(ranges, num_selected_rows=None)[source]
Perform a copy of the current message instance for the given
ranges
of rows.- Parameters
- rangestyping.List[typing.Tuple[int, int]]
Rows to include in the copy in the form of
[(`start_row
,stop_row
),…]` Thestop_row
isn’t included. For example to copy rows 1-2 & 5-7ranges=[(1, 3), (5, 8)]
- num_selected_rowstyping.Union[None, int]
Optional specify the number of rows selected by
ranges
, otherwise this is computed by the result.
- Returns
- get_meta(columns=None)[source]
Return column values from
morpheus.pipeline.messages.MessageMeta.df
.- Parameters
- columnstyping.Union[None, str, typing.List[str]]
Input column names. Returns all columns if
None
is specified. When a string is passed, aSeries
is returned. Otherwise aDataframe
is returned.
- Returns
- Series or Dataframe
Column values from the dataframe.
- get_meta_list(col_name=None)[source]
Return a column values from morpheus.messages.MessageMeta.df as a list.
- Parameters
- col_namestr
Column name in the dataframe.
- Returns
- List[str]
Column values from the dataframe.
- get_slice(start, stop)[source]
Returns sliced batches based on offsets supplied. Automatically calculates the correct
mess_offset
andmess_count
.- Parameters
- startint
Start offset address.
- stopint
Stop offset address.
- Returns
MultiInferenceMessage
A new
MultiInferenceMessage
with sliced offset and count.
- property id: List[int]
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.- Returns
- List[int]
ID column values from the dataframe as list.
- property id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.- Returns
- pandas.Series
ID column values from the dataframe.
- set_meta(columns, value)[source]
Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- Parameters
- columnstyping.Union[None, str, typing.List[str]]
Input column names. Sets the value for the corresponding column names. If
None
is specified, all columns will be used. If the column does not exist, a new one will be created.- valueAny
Value to apply to the specified columns. If a single value is passed, it will be broadcast to all rows. If a
Series
orDataframe
is passed, rows will be matched by index.
- property timestamp: List[int]
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
- Returns
- List[int]
Timestamp column values from the dataframe as list.
- class MultiResponseAEMessage(*args, **kwargs)[source]
A stronger typed version of
MultiResponseProbsMessage
that is used for inference workloads that return a probability array. Helps ensure the proper outputs are set and eases debugging.- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.outputs
Get outputs stored in the TensorMemory container.
probs
Probabilities of prediction.
timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
- user_id
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_output_ranges
(ranges[, mask])Perform a copy of the underlying output tensors for the given
ranges
of rows.copy_ranges
(ranges)Perform a copy of the current message, dataframe and tensors for the given
ranges
of rows.get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_output
(name)Get output stored in the TensorMemory container.
get_slice
(start, stop)Perform a slice of the current message from
start
:stop
(excludingstop
)set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- copy_ranges(ranges)[source]
Perform a copy of the current message, dataframe and tensors for the given
ranges
of rows.- Parameters
- rangestyping.List[typing.Tuple[int, int]]
Rows to include in the copy in the form of
[(`start_row
,stop_row
),…]` Thestop_row
isn’t included. For example to copy rows 1-2 & 5-7ranges=[(1, 3), (5, 8)]
- ——-
- `MultiResponseAEMessage`
- get_slice(start, stop)[source]
Perform a slice of the current message from
start
:stop
(excludingstop
)For example to slice from rows 1-3 use
m.get_slice(1, 4)
. The returnedMultiResponseMessage
will contain references to the same underlying Dataframe and output tensors, and this calling this method is reletively low cost compared toMultiResponseAEMessage.copy_ranges
- Parameters
- startint
Starting row of the slice
- stopint
Stop of the slice
- ——-
- `MultiResponseAEMessage`
- class MultiResponseMessage(*args, **kwargs)[source]
This class contains several inference responses as well as the cooresponding message metadata.
- Parameters
- memory<bsp-code-inline code="TensorMemory">TensorMemory</bsp-code-inline>
This is a response container instance for triton inference requests.
- offsetint
Offset of each response message into the
TensorMemory
block.- countint
Inference results size of all responses.
- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.outputs
Get outputs stored in the TensorMemory container.
timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_output_ranges
(ranges[, mask])Perform a copy of the underlying output tensors for the given
ranges
of rows.copy_ranges
(ranges)Perform a copy of the current message, dataframe and tensors for the given
ranges
of rows.get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_output
(name)Get output stored in the TensorMemory container.
get_slice
(start, stop)Perform a slice of the current message from
start
:stop
(excludingstop
)set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- copy_output_ranges(ranges, mask=None)[source]
Perform a copy of the underlying output tensors for the given
ranges
of rows.- Parameters
- rangestyping.List[typing.Tuple[int, int]]
Rows to include in the copy in the form of
[(`start_row
,stop_row
),…]` Thestop_row
isn’t included. For example to copy rows 1-2 & 5-7ranges=[(1, 3), (5, 8)]
- masktyping.Union[None, cupy.ndarray, numpy.ndarray]
Optionally specify rows as a cupy array (when using cudf Dataframes) or a numpy array (when using pandas Dataframes) of booleans. When not-None
ranges
will be ignored. This is useful as an optimization as this avoids needing to generate the mask on it’s own.
- Returns
- typing.Dict[str, cupy.ndarray]
- copy_ranges(ranges)[source]
Perform a copy of the current message, dataframe and tensors for the given
ranges
of rows.- Parameters
- rangestyping.List[typing.Tuple[int, int]]
Rows to include in the copy in the form of
[(`start_row
,stop_row
),…]` Thestop_row
isn’t included. For example to copy rows 1-2 & 5-7ranges=[(1, 3), (5, 8)]
- ——-
- `MultiResponseMessage`
- get_output(name)[source]
Get output stored in the TensorMemory container.
- Parameters
- namestr
Output key name.
- Returns
- cupy.ndarray
Inference output.
- get_slice(start, stop)[source]
Perform a slice of the current message from
start
:stop
(excludingstop
)For example to slice from rows 1-3 use
m.get_slice(1, 4)
. The returnedMultiResponseMessage
will contain references to the same underlying Dataframe and output tensors, and this calling this method is reletively low cost compared toMultiResponseMessage.copy_ranges
- Parameters
- startint
Starting row of the slice
- stopint
Stop of the slice
- ——-
- `MultiResponseMessage`
- property outputs
Get outputs stored in the TensorMemory container.
- Returns
- cupy.ndarray
Inference outputs.
- class MultiResponseProbsMessage(*args, **kwargs)[source]
A stronger typed version of
MultiResponseMessage
that is used for inference workloads that return a probability array. Helps ensure the proper outputs are set and eases debugging.- Attributes
id
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
as list.id_col
Returns ID column values from
morpheus.pipeline.messages.MessageMeta.df
.outputs
Get outputs stored in the TensorMemory container.
probs
Probabilities of prediction.
timestamp
Returns timestamp column values from morpheus.messages.MessageMeta.df as list.
Methods
copy_meta_ranges
(ranges[, mask])Perform a copy of the underlying dataframe for the given
ranges
of rows.copy_output_ranges
(ranges[, mask])Perform a copy of the underlying output tensors for the given
ranges
of rows.copy_ranges
(ranges)Perform a copy of the current message, dataframe and tensors for the given
ranges
of rows.get_meta
([columns])Return column values from
morpheus.pipeline.messages.MessageMeta.df
.get_meta_list
([col_name])Return a column values from morpheus.messages.MessageMeta.df as a list.
get_output
(name)Get output stored in the TensorMemory container.
get_slice
(start, stop)Perform a slice of the current message from
start
:stop
(excludingstop
)set_meta
(columns, value)Set column values to
morpheus.pipelines.messages.MessageMeta.df
.- property probs
Probabilities of prediction.
- Returns
- cupy.ndarray
probabilities
- class ResponseMemory(*args, **kwargs)[source]
Output memory block holding the results of inference.
Methods
get_output
(name)Return the output tensor specified by
name
.- get_output(name)[source]
Return the output tensor specified by
name
.- Parameters
- namestr
Name of output tensor.
- Returns
- cupy.ndarray
Tensor corresponding to name.
- class ResponseMemoryAE(*args, **kwargs)[source]
Subclass of
ResponseMemory
specific to the AutoEncoder pipeline.- Parameters
- probscupy.ndarray
Probabilities tensor
- user_idstr
User id the inference was performed against.
- explain_dfpd.Dataframe
Explainability Dataframe, for each feature a column will exist with a name in the form of:
{feature}_z_loss
containing the loss z-score along withmax_abs_z
andmean_abs_z
columns
- Attributes
- explain_df
- probs
Methods
get_output
(name)Return the output tensor specified by
name
.
- class ResponseMemoryProbs(*args, **kwargs)[source]
Subclass of
ResponseMemory
containng an output tensor named ‘probs’.- Parameters
- probscupy.ndarray
Probabilities tensor
- Attributes
- probs
Methods
get_output
(name)Return the output tensor specified by
name
.
- class UserMessageMeta(*args, **kwargs)[source]
This class extends MessageMeta to also hold userid corresponding to batched metadata.
- Parameters
- dfpandas.DataFrame
Input rows in dataframe.
- user_idstr
User id.
- Attributes
count
Returns the number of messages in the batch.
Modules