Class MultiResponseMessage

Base Type

class MultiResponseMessage : public morpheus::DerivedMultiMessage<MultiResponseMessage, MultiTensorMessage>

This class is used to get or set the inference output from message containers derived from ResponseMemory.

Public Functions

MultiResponseMessage(const MultiResponseMessage &other) = default

Default copy constructor.

MultiResponseMessage(std::shared_ptr<MessageMeta> meta, TensorIndex mess_offset = 0, TensorIndex mess_count = -1, std::shared_ptr<TensorMemory> memory = nullptr, TensorIndex offset = 0, TensorIndex count = -1, std::string id_tensor_name = "seq_ids", std::string probs_tensor_name = "probs")

Construct a new Multi Response Message object.

Parameters
  • meta – Holds a data table, in practice a cudf DataFrame, with the ability to return both Python and C++ representations of the table

  • mess_offset – Offset into the metadata batch

  • mess_count – Messages count

  • memory – Shared pointer of a tensor memory

  • offset – Message offset in inference memory instance

  • count – Message count in inference memory instance

  • id_tensor_name – Name of the tensor that correlates tensor rows to message IDs

  • probs_tensor_name – Name of the tensor that holds output probabilities

const TensorObject get_output(const std::string &name) const

Returns the output tensor with the given name.

Parameters

name

Throws

std::runtime_error – If no tensor matching name exists

Returns

const TensorObject

TensorObject get_output(const std::string &name)

Returns the output tensor with the given name.

Parameters

name

Throws

std::runtime_error – If no tensor matching name exists

Returns

TensorObject

void set_output(const std::string &name, const TensorObject &value)

Update the value of a given output tensor. The tensor must already exist, otherwise this will halt on a fatal error.

Parameters
  • name

  • value

TensorObject get_probs_tensor() const

Get the tensor that holds output probabilities. Equivalent to get_tensor(probs_tensor_name)

Returns

const TensorObject

Public Members

std::string probs_tensor_name

Previous Class MultiMessage
Next Class MultiResponseProbsMessage
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.