> This page is for version v0.25.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo/guardrails/latest/llms.txt
> - v0.25.0: https://docs.nvidia.com/nemo/guardrails/v0.25.0/llms.txt
> - v0.24.1: https://docs.nvidia.com/nemo/guardrails/v0.24.1/llms.txt
> - v0.24.0: https://docs.nvidia.com/nemo/guardrails/v0.24.0/llms.txt
> - v0.23.0: https://docs.nvidia.com/nemo/guardrails/v0.23.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/nemo/guardrails/v0.22.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt.

# nemoguardrails.integrations.langchain.providers.trtllm

## Submodules

* **[`nemoguardrails.integrations.langchain.providers.trtllm.client`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/client)**
* **[`nemoguardrails.integrations.langchain.providers.trtllm.llm`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/integrations/langchain/providers/trtllm/llm)**

## Package Contents

### Classes

| Name                                                                                          | Description                                                             |
| --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [`TRTLLM`](#nemoguardrails-integrations-langchain-providers-trtllm-llm-TRTLLM)                | A custom Langchain LLM class that integrates with TRTLLM triton models. |
| [`TritonClient`](#nemoguardrails-integrations-langchain-providers-trtllm-client-TritonClient) | An abstraction of the connection to a triton inference server.          |

### API

```python
class nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM()
```

**Bases:** `BaseLLM`

A custom Langchain LLM class that integrates with TRTLLM triton models.

Arguments:
server_url: (str) The URL of the Triton inference server to use.
model_name: (str) The name of the Triton TRT model to use.
temperature: (str) Temperature to use for sampling
top_p: (float) The top-p value to use for sampling
top_k: (float) The top k values use for sampling
beam_width: (int) Last n number of tokens to penalize
repetition_penalty: (int) Last n number of tokens to penalize
length_penalty: (float) The penalty to apply repeated tokens
tokens: (int) The maximum number of tokens to generate.
client: The client object used to communicate with the inference server

**`_get_model_default_parameters`** `Dict[str, Any]`

---

**`_identifying_params`** `Dict[str, Any]`

Get all the identifying parameters.

---

**`_invocation_params`** `Dict[str, Any]`

---

**`_llm_type`** `str`

---

**`beam_width`** `Optional[int] = 1`

---

**`client`** `Any = Field(default=None, exclude=True)`

---

**`length_penalty`** `Optional[float] = 1.0`

---

**`model_name`** `str = 'ensemble'`

---

**`repetition_penalty`** `Optional[float] = 1.0`

---

**`server_url`** `str = Field(None, alias='server_url')`

---

**`streaming`** `Optional[bool] = True`

---

**`temperature`** `Optional[float] = 1.0`

---

**`tokens`** `Optional[int] = 100`

---

**`top_k`** `Optional[int] = 1`

---

**`top_p`** `Optional[float] = 0`

---

```python
nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM._acall(
    args = (),
    kwargs = {}
)
```

async

Async version.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM._call(
    prompt: str,
    stop: typing.Optional[typing.List[str]] = None,
    run_manager: typing.Optional[langchain_core.callbacks.manager.CallbackManagerForLLMRun] = None,
    kwargs: typing.Any = {}
) -> str
```

Execute an inference request.

**Parameters:**

**`prompt`** `str`

The prompt to pass into the model.

---

**`stop`** `Optional[List[str]]` — default: None

A list of strings to stop generation when encountered

---

**Returns:** `str`

The string generated by the model

```python
nemoguardrails.integrations.langchain.providers.trtllm.TRTLLM.validate_environment() -> 'TRTLLM'
```

Validate that python package exists in environment.

```python
class nemoguardrails.integrations.langchain.providers.trtllm.TritonClient(
    server_url: str
)
```

An abstraction of the connection to a triton inference server.

**`client`** `= grpcclient.InferenceServerClient(server_url)`

---

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.close_streaming() -> None
```

Close the streaming connection.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.generate_inputs(
    prompt: str,
    tokens: int = 32,
    temperature: float = 0.5,
    top_k: float = 0,
    top_p: float = 0.9,
    beam_width: int = 1,
    repetition_penalty: float = 1,
    length_penalty: float = 1.0
) -> typing.List['grpcclient.InferInput']
```

staticmethod

Create the input for the triton inference server.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.generate_outputs() -> typing.List['grpcclient.InferRequestedOutput']
```

staticmethod

Generate the expected output structure.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.get_model_concurrency(
    model_name: str,
    timeout: int = 1000
) -> int
```

Get the modle concurrency.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.get_model_list() -> typing.List[str]
```

Get a list of models loaded in the triton server.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.load_model(
    model_name: str,
    timeout: int = 1000
) -> None
```

Load a model into the server.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.prepare_tensor(
    name: str,
    input_data: typing.Any
) -> 'grpcclient.InferInput'
```

staticmethod

Prepare an input data structure.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.process_result(
    result: typing.Dict[str, str]
) -> typing.Dict[str, str]
```

staticmethod

Post-process the result from the server.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.request_streaming(
    model_name: str,
    result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]],
    params: typing.Any = {}
) -> None
```

Request a streaming connection.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.send_prompt_streaming(
    model_name: str,
    request_inputs: typing.Any,
    request_outputs: typing.Optional[typing.Any],
    result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]]
) -> None
```

Send the prompt and start streaming the result.

```python
nemoguardrails.integrations.langchain.providers.trtllm.TritonClient.stream_callback(
    result_queue: queue.Queue[typing.Union[typing.Optional[typing.Dict[str, str]], str]],
    result: typing.Any,
    error: str
) -> None
```

Add streamed result to queue.