nemoguardrails.integrations.langchain.providers.trtllm
Submodules
nemoguardrails.integrations.langchain.providers.trtllm.clientnemoguardrails.integrations.langchain.providers.trtllm.llm
Package Contents
Classes
API
Bases: BaseLLM
A custom Langchain LLM class that integrates with TRTLLM triton models.
Arguments: server_url: (str) The URL of the Triton inference server to use. model_name: (str) The name of the Triton TRT model to use. temperature: (str) Temperature to use for sampling top_p: (float) The top-p value to use for sampling top_k: (float) The top k values use for sampling beam_width: (int) Last n number of tokens to penalize repetition_penalty: (int) Last n number of tokens to penalize length_penalty: (float) The penalty to apply repeated tokens tokens: (int) The maximum number of tokens to generate. client: The client object used to communicate with the inference server
Get all the identifying parameters.
Async version.
Execute an inference request.
Parameters:
The prompt to pass into the model.
A list of strings to stop generation when encountered
Returns: str
The string generated by the model
Validate that python package exists in environment.
An abstraction of the connection to a triton inference server.
Close the streaming connection.
Create the input for the triton inference server.
Generate the expected output structure.
Get the modle concurrency.
Get a list of models loaded in the triton server.
Load a model into the server.
Prepare an input data structure.
Post-process the result from the server.
Request a streaming connection.
Send the prompt and start streaming the result.
Add streamed result to queue.