morpheus_llm.llm.services.nvfoundation_llm_service.NVFoundationLLMClient#

class NVFoundationLLMClient(parent, *, model_name, **model_kwargs)[source]#

Bases: LLMClient

Client for interacting with a specific model in Nemo. This class should be constructed with the NeMoLLMService.get_client method.

Parameters:
parentNVFoundationMService

The parent service for this client.

model_namestr

The name of the model to interact with.

model_kwargsdict[str, typing.Any]

Additional keyword arguments to pass to the model when generating text.

Attributes:
model_kwargs

Methods

generate(**input_dict)

Issue a request to generate a response based on a given prompt.

generate_async(**input_dict)

Issue an asynchronous request to generate a response based on a given prompt.

generate_batch()

Issue a request to generate a list of responses based on a list of prompts.

generate_batch_async()

Issue an asynchronous request to generate a list of responses based on a list of prompts.

get_input_names()

Returns the names of the inputs to the model.
generate(**input_dict)[source]#

Issue a request to generate a response based on a given prompt.

Parameters:
input_dictdict

Input containing prompt data.

async generate_async(**input_dict)[source]#

Issue an asynchronous request to generate a response based on a given prompt.

Parameters:
input_dictdict

Input containing prompt data.

generate_batch(
inputs: dict[str, list],
return_exceptions: Literal[True],
) → list[str | BaseException][source]#
generate_batch(
inputs: dict[str, list],
return_exceptions: Literal[False],
) → list[str]
generate_batch(
inputs: dict[str, list],
return_exceptions: bool = False,
) → list[str] | list[str | BaseException]

Issue a request to generate a list of responses based on a list of prompts.

Parameters:
inputsdict

Inputs containing prompt data.

return_exceptionsbool

Whether to return exceptions in the output list or raise them immediately.

**kwargs

Additional keyword arguments for generate batch.

async generate_batch_async(
inputs: dict[str, list],
return_exceptions: Literal[True],
) → list[str | BaseException][source]#
async generate_batch_async(
inputs: dict[str, list],
return_exceptions: Literal[False],
) → list[str]
async generate_batch_async(
inputs: dict[str, list],
return_exceptions: bool = False,
) → list[str] | list[str | BaseException]

Issue an asynchronous request to generate a list of responses based on a list of prompts.

Parameters:
inputsdict

Inputs containing prompt data.

return_exceptionsbool

Whether to return exceptions in the output list or raise them immediately.

**kwargs

Additional keyword arguments for generate batch async.

get_input_names()[source]#

Returns the names of the inputs to the model.

Returns:
list[str]

List of input names.