> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/ccluster/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/ccluster/_mcp/server.

# Deploying TensorRT-LLM

> Deploy TensorRT-LLM images on CCluster through General Inference

[NVIDIA TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) images run through [General Inference](/ccluster/apps/inference) and expose an OpenAI-compatible API with `trtllm-serve`. This example walks through deploying `TinyLlama/TinyLlama-1.1B-Chat-v1.0` with a validated release image.

## Before you begin

Prepare the following:

* Access to a cluster with a suitable NVIDIA GPU hardware instance
* A Hugging Face token if the model is gated or you need higher Hub rate limits
* A Bearer token or client certificate for endpoint authentication

The image and model versions must be compatible. Review the [TensorRT-LLM support matrix](https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html) before using a different model or image tag.

## Configure the deployment

From **Create deployment**, select **Inference**, then configure the container:

| Field                   | Example value                         |
| ----------------------- | ------------------------------------- |
| **Add or Select Image** | `nvcr.io/nvidia/tensorrt-llm/release` |
| **Container Tag**       | `1.3.0rc22`                           |
| **Container Port**      | `8000`                                |
| **Protocol**            | `HTTP`                                |
| **Health check path**   | `/health`                             |

The image might appear as `nvidia/tensorrt-llm/release` in the prebuilt image list. The tag above is a validated example; use a tag that supports your selected model.

### Add command

The TensorRT-LLM release image does not start an inference server by itself. Under **Optional Details → Add command**, enter:

```bash showLineNumbers={false}
/bin/bash -lc 'trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000 --config /etc/engine/config.yaml'
```

Replace the model identifier with the Hugging Face model or TensorRT-LLM engine path you want to serve. Wrap the full `trtllm-serve` command in `/bin/bash -lc '...'` so the image shell initializes the TensorRT-LLM environment before the server starts.

| Argument                           | Why it matters                                            |
| ---------------------------------- | --------------------------------------------------------- |
| `/bin/bash -lc '...'`              | Loads the image environment before running `trtllm-serve` |
| `trtllm-serve <model>`             | Starts the OpenAI-compatible server                       |
| `--host 0.0.0.0`                   | Required so CCluster can reach the server                 |
| `--port 8000`                      | Must match **Container Port**                             |
| `--config /etc/engine/config.yaml` | Points at the mounted config file                         |

### Config file

Add a config file with these settings:

| Field          | Value         |
| -------------- | ------------- |
| **Filename**   | `config.yaml` |
| **Mount path** | `/etc/engine` |

Use the directory as the mount path, not the full filename. CCluster mounts the file at `/etc/engine/config.yaml`.

For the TinyLlama example, use:

```yaml
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
max_batch_size: 8
max_num_tokens: 2048
max_seq_len: 2048
trust_remote_code: true

kv_cache_config:
  free_gpu_memory_fraction: 0.8
  enable_block_reuse: true

enable_iter_perf_stats: true
```

Tune `max_batch_size`, `max_seq_len`, and `free_gpu_memory_fraction` for your model and GPU memory. For TensorRT-LLM 1.3.x, keep these options at the top level; do not wrap them in `pytorch_backend_config`.

### Model credentials

Under **Optional Details → Environment Variables**, set `HF_TOKEN` to a Hugging Face token when the model is gated or private. You can store the token in a [Vault object](/ccluster/resources/vault) instead of entering it directly.

Every endpoint must also be protected with a Bearer token, an mTLS client certificate, or both. See [Securing Endpoints](/ccluster/resources/private).

From here the deployment follows the standard General Inference flow — select the cluster and hardware, click **Deploy**, and wait until the deployment is ready. Model download and GPU initialization can take several minutes. See [General Inference](/ccluster/apps/inference) for details.

## Test the endpoint

Copy the endpoint URL and Bearer token from the deployment detail page:

```bash showLineNumbers={false}
export ENDPOINT='https://<endpoint_url>'
export CCLUSTER_TOKEN='<your_bearer_token>'

# Check readiness
curl -sS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  "$ENDPOINT/health"

# Find the served model identifier
curl -sS \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  "$ENDPOINT/v1/models"

# Send a chat completion request
curl -sS "$ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 64
  }'
```

Use the exact model `id` returned by `/v1/models` in subsequent requests. The first request can take longer while the runtime finishes warming up.

## Troubleshooting

| Symptom                                                      | Resolution                                                                              |
| ------------------------------------------------------------ | --------------------------------------------------------------------------------------- |
| Deployment does not become ready although the server started | Confirm the command includes `--host 0.0.0.0`.                                          |
| Logs report `Is a directory: '.../config.yaml'`              | Set **Mount path** to `/etc/engine` and **Filename** to `config.yaml`.                  |
| Logs report unsupported `pytorch_backend_config` arguments   | Remove that block and place supported options at the top level of the YAML file.        |
| API returns `401`                                            | Send the deployment's Bearer token in the `Authorization` header.                       |
| API returns `503` or `no healthy upstream`                   | Wait for the deployment to become ready, then check the host and config mount settings. |
| Model download is slow or fails                              | Set `HF_TOKEN` and confirm that it can access the model.                                |

## What's next

#### [General Inference](/ccluster/apps/inference)

Deploy and scale any containerized model on CCluster.

#### [Securing Endpoints](/ccluster/resources/private)

Protect your endpoints with Bearer tokens and mTLS certificates.

#### [Managing Vault Objects](/ccluster/resources/vault)

Store the registry credentials and API keys your deployments use.