Skip to navigation

nemoguardrails.integrations.langchain.providers.trtllm

View as MarkdownOpen in Claude

Submodules

Package Contents

Classes

NameDescription
TRTLLMA custom Langchain LLM class that integrates with TRTLLM triton models.
TritonClientAn abstraction of the connection to a triton inference server.

API

class nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM()

Bases: BaseLLM

A custom Langchain LLM class that integrates with TRTLLM triton models.

Arguments: server_url: (str) The URL of the Triton inference server to use. model_name: (str) The name of the Triton TRT model to use. temperature: (str) Temperature to use for sampling top_p: (float) The top-p value to use for sampling top_k: (float) The top k values use for sampling beam_width: (int) Last n number of tokens to penalize repetition_penalty: (int) Last n number of tokens to penalize length_penalty: (float) The penalty to apply repeated tokens tokens: (int) The maximum number of tokens to generate. client: The client object used to communicate with the inference server

_get_model_default_parameters
Dict[str, Any]
_identifying_params
Dict[str, Any]

Get all the identifying parameters.

_invocation_params
Dict[str, Any]
_llm_type
str
beam_width
Optional[int] = 1
client
Any = Field(default=None, exclude=True)
length_penalty
Optional[float] = 1.0
model_name
str = 'ensemble'
repetition_penalty
Optional[float] = 1.0
server_url
str = Field(None, alias='server_url')
streaming
Optional[bool] = True
temperature
Optional[float] = 1.0
tokens
Optional[int] = 100
top_k
Optional[int] = 1
top_p
Optional[float] = 0
nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM._acall(
args = (),
kwargs = {}
)
async

Async version.

nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM._call(
prompt: str,
stop: typing.Optional[typing.List[str]] = None,
run_manager: typing.Optional[langchain_core.callbacks.manager.CallbackManagerForLLMRun] = None,
kwargs: typing.Any = {}
) -> str

Execute an inference request.

Parameters:

prompt
str

The prompt to pass into the model.

stop
Optional[List[str]]Defaults to None

A list of strings to stop generation when encountered

Returns: str

The string generated by the model

nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM.validate_environment() -> 'TRTLLM'

Validate that python package exists in environment.

class nemoguardrails.integrations.langchain.providers.trtllm.TritonClient(
server_url: str
)

An abstraction of the connection to a triton inference server.

client
= grpcclient.InferenceServerClient(server_url)
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.close_streaming() -> None

Close the streaming connection.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.generate_inputs(
prompt: str,
tokens: int = 32,
temperature: float = 0.5,
top_k: float = 0,
top_p: float = 0.9,
beam_width: int = 1,
repetition_penalty: float = 1,
length_penalty: float = 1.0
) -> typing.List['grpcclient.InferInput']
staticmethod

Create the input for the triton inference server.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.generate_outputs() -> typing.List['grpcclient.InferRequestedOutput']
staticmethod

Generate the expected output structure.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.get_model_concurrency(
model_name: str,
timeout: int = 1000
) -> int

Get the modle concurrency.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.get_model_list() -> typing.List[str]

Get a list of models loaded in the triton server.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.load_model(
model_name: str,
timeout: int = 1000
) -> None

Load a model into the server.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.prepare_tensor(
name: str,
input_data: typing.Any
) -> 'grpcclient.InferInput'
staticmethod

Prepare an input data structure.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.process_result(
result: typing.Dict[str, str]
) -> typing.Dict[str, str]
staticmethod

Post-process the result from the server.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.request_streaming(
model_name: str,
result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]],
params: typing.Any = {}
) -> None

Request a streaming connection.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.send_prompt_streaming(
model_name: str,
request_inputs: typing.Any,
request_outputs: typing.Optional[typing.Any],
result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]]
) -> None

Send the prompt and start streaming the result.

nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.stream_callback(
result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]],
result: typing.Any,
error: str
) -> None

Add streamed result to queue.