Class LLMEngine

• Defined in File llm_engine.hpp

Base Type

• public morpheus::llm::LLMNode (Class LLMNode)

class LLMEngine : public morpheus::llm::LLMNode

A subclass of LLMNode that acts as container for components required in processing LLM service requests. For example, a prompt generator and LLM service can each be implemented as an LLMNode and added to the engine. Input mappings are used to match node’s input to outputs of parent or sibling nodes. Task handlers can also be added to the engine to perform additional processing on the outputs stored in the LLMContext of the engine.

Public Functions

LLMEngine()

Construct a new LLMEngine object.

~LLMEngine() override

Destroy the LLMEngine object.

virtual void add_task_handler(user_input_mappings_t inputs, std::shared_ptr<LLMTaskHandler> task_handler)

Add new task handler to this engine.

Parameters

• inputs – input mappings that specifies inputs to new task handler

• task_handler – task handler object

virtual Task<std::vector<std::shared_ptr<ControlMessage>>> run(std::shared_ptr<ControlMessage> input_message)

Execute nodes in this engine and pass outputs to its task handlers. Must pass this a control message with a ‘llm_engine’ task containing ‘task_type and ‘task_dict’ properties required for execution of task(s).

Parameters

input_message – input control message

Returns

Task>>