Deploying TensorRT-LLM

NVIDIA TensorRT-LLM images run through General Inference and expose an OpenAI-compatible API with trtllm-serve. This example walks through deploying TinyLlama/TinyLlama-1.1B-Chat-v1.0 with a validated release image.

Before you begin

Prepare the following:

• Access to a cluster with a suitable NVIDIA GPU hardware instance
• A Hugging Face token if the model is gated or you need higher Hub rate limits
• A Bearer token or client certificate for endpoint authentication

The image and model versions must be compatible. Review the TensorRT-LLM support matrix before using a different model or image tag.

Configure the deployment

From Create deployment, select Inference, then configure the container:

The image might appear as nvidia/tensorrt-llm/release in the prebuilt image list. The tag above is a validated example; use a tag that supports your selected model.

Add command

The TensorRT-LLM release image does not start an inference server by itself. Under Optional Details → Add command, enter:

/bin/bash -lc 'trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000 --config /etc/engine/config.yaml'

Replace the model identifier with the Hugging Face model or TensorRT-LLM engine path you want to serve. Wrap the full trtllm-serve command in /bin/bash -lc '...' so the image shell initializes the TensorRT-LLM environment before the server starts.

Config file

Add a config file with these settings:

Use the directory as the mount path, not the full filename. CCluster mounts the file at /etc/engine/config.yaml.

For the TinyLlama example, use:

model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
max_batch_size: 8
max_num_tokens: 2048
max_seq_len: 2048
trust_remote_code: true
kv_cache_config:
  free_gpu_memory_fraction: 0.8
  enable_block_reuse: true
enable_iter_perf_stats: true

Tune max_batch_size, max_seq_len, and free_gpu_memory_fraction for your model and GPU memory. For TensorRT-LLM 1.3.x, keep these options at the top level; do not wrap them in pytorch_backend_config.

Model credentials

Under Optional Details → Environment Variables, set HF_TOKEN to a Hugging Face token when the model is gated or private. You can store the token in a Vault object instead of entering it directly.

Every endpoint must also be protected with a Bearer token, an mTLS client certificate, or both. See Securing Endpoints.

From here the deployment follows the standard General Inference flow — select the cluster and hardware, click Deploy, and wait until the deployment is ready. Model download and GPU initialization can take several minutes. See General Inference for details.

Test the endpoint

Copy the endpoint URL and Bearer token from the deployment detail page:

export ENDPOINT='https://<endpoint_url>'
export CCLUSTER_TOKEN='<your_bearer_token>'
# Check readiness
curl -sS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  "$ENDPOINT/health"
# Find the served model identifier
curl -sS \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  "$ENDPOINT/v1/models"
# Send a chat completion request
curl -sS "$ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $CCLUSTER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 64
  }'

Use the exact model id returned by /v1/models in subsequent requests. The first request can take longer while the runtime finishes warming up.

Troubleshooting

What’s next