> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/llms.txt. # nemoguardrails.integrations.langchain.providers.trtllm ## Submodules * **[`nemoguardrails.integrations.langchain.providers.trtllm.client`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/client)** * **[`nemoguardrails.integrations.langchain.providers.trtllm.llm`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/llm)** ## Package Contents ### Classes | Name | Description | | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | | [`TRTLLM`](#nemoguardrails-integrations-langchain-providers-trtllm-llm-TRTLLM) | A custom Langchain LLM class that integrates with TRTLLM triton models. | | [`TritonClient`](#nemoguardrails-integrations-langchain-providers-trtllm-client-TritonClient) | An abstraction of the connection to a triton inference server. | ### API ```python class nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM() ``` **Bases:** `BaseLLM` A custom Langchain LLM class that integrates with TRTLLM triton models. Arguments: server_url: (str) The URL of the Triton inference server to use. model_name: (str) The name of the Triton TRT model to use. temperature: (str) Temperature to use for sampling top_p: (float) The top-p value to use for sampling top_k: (float) The top k values use for sampling beam_width: (int) Last n number of tokens to penalize repetition_penalty: (int) Last n number of tokens to penalize length_penalty: (float) The penalty to apply repeated tokens tokens: (int) The maximum number of tokens to generate. client: The client object used to communicate with the inference server **`_get_model_default_parameters`** `Dict[str, Any]` --- **`_identifying_params`** `Dict[str, Any]` Get all the identifying parameters. --- **`_invocation_params`** `Dict[str, Any]` --- **`_llm_type`** `str` --- **`beam_width`** `Optional[int] = 1` --- **`client`** `Any = Field(default=None, exclude=True)` --- **`length_penalty`** `Optional[float] = 1.0` --- **`model_name`** `str = 'ensemble'` --- **`repetition_penalty`** `Optional[float] = 1.0` --- **`server_url`** `str = Field(None, alias='server_url')` --- **`streaming`** `Optional[bool] = True` --- **`temperature`** `Optional[float] = 1.0` --- **`tokens`** `Optional[int] = 100` --- **`top_k`** `Optional[int] = 1` --- **`top_p`** `Optional[float] = 0` --- ```python nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM._acall( args = (), kwargs = {} ) ``` async Async version. ```python nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM._call( prompt: str, stop: typing.Optional[typing.List[str]] = None, run_manager: typing.Optional[langchain_core.callbacks.manager.CallbackManagerForLLMRun] = None, kwargs: typing.Any = {} ) -> str ``` Execute an inference request. **Parameters:** **`prompt`** `str` The prompt to pass into the model. --- **`stop`** `Optional[List[str]]` — default: None A list of strings to stop generation when encountered --- **Returns:** `str` The string generated by the model ```python nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM.validate_environment() -> 'TRTLLM' ``` Validate that python package exists in environment. ```python class nemoguardrails.integrations.langchain.providers.trtllm.TritonClient( server_url: str ) ``` An abstraction of the connection to a triton inference server. **`client`** `= grpcclient.InferenceServerClient(server_url)` --- ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.close_streaming() -> None ``` Close the streaming connection. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.generate_inputs( prompt: str, tokens: int = 32, temperature: float = 0.5, top_k: float = 0, top_p: float = 0.9, beam_width: int = 1, repetition_penalty: float = 1, length_penalty: float = 1.0 ) -> typing.List['grpcclient.InferInput'] ``` staticmethod Create the input for the triton inference server. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.generate_outputs() -> typing.List['grpcclient.InferRequestedOutput'] ``` staticmethod Generate the expected output structure. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.get_model_concurrency( model_name: str, timeout: int = 1000 ) -> int ``` Get the modle concurrency. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.get_model_list() -> typing.List[str] ``` Get a list of models loaded in the triton server. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.load_model( model_name: str, timeout: int = 1000 ) -> None ``` Load a model into the server. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.prepare_tensor( name: str, input_data: typing.Any ) -> 'grpcclient.InferInput' ``` staticmethod Prepare an input data structure. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.process_result( result: typing.Dict[str, str] ) -> typing.Dict[str, str] ``` staticmethod Post-process the result from the server. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.request_streaming( model_name: str, result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]], params: typing.Any = {} ) -> None ``` Request a streaming connection. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.send_prompt_streaming( model_name: str, request_inputs: typing.Any, request_outputs: typing.Optional[typing.Any], result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]] ) -> None ``` Send the prompt and start streaming the result. ```python nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.stream_callback( result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]], result: typing.Any, error: str ) -> None ``` Add streamed result to queue. ## Docs - [nemoguardrails.integrations.langchain.providers.trtllm.client](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/client.md) - [nemoguardrails.integrations.langchain.providers.trtllm.llm](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/llm.md)