NVIDIA Morpheus (24.06)

morpheus.stages.inference.triton_inference_stage.TritonInferenceNLP

class TritonInferenceNLP(inf_queue, c, model_name, server_url, force_convert_inputs=False, use_shared_memory=False, inout_mapping=None)[source]

Bases: morpheus.stages.inference.triton_inference_stage._TritonInferenceWorker

This class extends TritonInference to deal with scenario-specific NLP models inference requests like building response.

Parameters
inf_queuemorpheus.utils.producer_consumer_queue.ProducerConsumerQueue

Inference queue.

cmorpheus.config.Config

Pipeline configuration instance.

model_namestr

Name of the model specifies which model can handle the inference requests that are sent to Triton inference server.

server_urlstr

Triton server gRPC URL including the port.

force_convert_inputsbool, default = False

Whether or not to convert the inputs to the type specified by Triton. This will happen automatically if no data would be lost in the conversion (i.e., float -> double). Set this to True to convert the input even if data would be lost (i.e., double -> float).

use_shared_memorybool, default = False

Whether or not to use CUDA Shared IPC Memory for transferring data to Triton. Using CUDA IPC reduces network transfer time but requires that Morpheus and Triton are located on the same machine.

inout_mappingtyping.Dict[str, str]

Dictionary used to map pipeline input/output names to Triton input/output names. Use this if the Morpheus names do not match the model.

Methods

build_output_message(x) Create initial inference response message with result values initialized to zero.
calc_output_dims(x) Calculates the dimensions of the inference output message data given an input message.
default_inout_mapping() Returns default dictionary used to map NLP pipeline input/output names to Triton input/output names
init() This function instantiate triton client and memory allocation for inference input and output.
needs_logits() Determines whether a logits calculation is needed for the value returned by the Triton inference response.
process(batch, cb) This function sends batch of events as a requests to Triton inference server using triton client API.
stop() Override this function to stop the inference workers or carry out any additional cleanups.
supports_cpp_node
build_output_message(x)[source]

Create initial inference response message with result values initialized to zero. Results will be set in message as each inference mini-batch is processed.

Parameters
xmorpheus.pipeline.messages.MultiInferenceMessage

Batch of inference messages.

Returns
morpheus.pipeline.messages.MultiResponseMessage

Response message with probabilities calculated from inference results.

calc_output_dims(x)[source]

Calculates the dimensions of the inference output message data given an input message.

Parameters
xmorpheus.pipeline.messages.MultiInferenceMessage

Pipeline inference input batch before splitting into smaller inference batches.

Returns
typing.Tuple

Output dimensions of response.

classmethod default_inout_mapping()[source]

Returns default dictionary used to map NLP pipeline input/output names to Triton input/output names

Returns
default_inout_mappingtyping.Dict[str, str]

Dictionary with default input and output names.

init()[source]

This function instantiate triton client and memory allocation for inference input and output.

classmethod needs_logits()[source]

Determines whether a logits calculation is needed for the value returned by the Triton inference response.

process(batch, cb)[source]

This function sends batch of events as a requests to Triton inference server using triton client API.

Parameters
batchmorpheus.pipeline.messages.MultiInferenceMessage

Mini-batch of inference messages.

cbtyping.Callable[[morpheus.pipeline.messages.TensorMemory], None]

Callback to set the values for the inference response.

stop()[source]

Override this function to stop the inference workers or carry out any additional cleanups.

Previous morpheus.stages.inference.triton_inference_stage.TritonInferenceFIL
Next morpheus.stages.inference.triton_inference_stage.TritonInferenceStage
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.