NVIDIA TensorRT-Cloud Documentation

Important

NVIDIA TensorRT-Cloud is provided as a developer preview in Early Access (EA). Access is restricted and is provided upon request (refer to the Getting TensorRT-Cloud Access section).

TensorRT-Cloud (TRTC) helps developers deploy GenAI models with the best possible inference configurations for their workloads, by offering two key functionalities:

1. TensorRT-LLM configuration sweeping to help you optimize inference across popular OSS LLMs and NVIDIA hardware SKUs.

2. TensorRT engine-building capabilities across diverse NVIDIA GPUs, OS, and library dependencies. The goal is to enable developers to build optimized TensorRT and TensorRT-LLM engines with the convenience of a command line interface (CLI) for the variety of NVIDIA GPUs that applications need to support. This is done through on-demand engine building. This, coupled with the weight refit capabilities of NVIDIA TensorRT 10.0, allows you to integrate TensorRT accelerated inference in your applications without worrying about bloating your application binaries.

The TensorRT-Cloud CLI is the interface through which you interact with TensorRT-Cloud.

Getting Started

• Quick Start Guide
  • Prerequisites
  • Launching a Sweep for an Optimized TensorRT-LLM Engine
  • Building a TensorRT-LLM Engine
  • Building a TensorRT Engine for an ONNX Model
• Release Notes
  • TensorRT-Cloud 0.6.0 Early Access (EA)
• Supported Models, Hardware, and OS
  • Quantization Modes Supported by Model Family

User Guide

• Getting Access
  • Requesting a TensorRT Cloud Enabled NGC Org
  • Installing TensorRT-Cloud CLI
  • Logging into TensorRT-Cloud
• Sweeping for Optimized TensorRT-LLM Engines
  • Launching a Sweep
    • Automatic Sweep Configuration
    • User Created, Custom Sweep Configuration
      • Sweep Configuration Schema
      • Sweep Search Space
      • Sweep Overview
  • Input Sources for Sweeping
    • Hugging Face Repository
    • NGC Private Registry
    • Pre-signed URL
    • Local File Path
  • Input Types
    • Hugging Face Checkpoint
    • TensorRT-LLM Checkpoint
    • Hugging Face Weightless Checkpoint
  • GPU Type and Parallelism
  • Quantization
  • Specifying Benchmark Options
  • Querying Sweep Status
  • Cancelling a Sweep
  • Obtaining Sweep Results
  • Building the Optimal Engine for a Sweep
  • Retrying Trials
  • Listing Sweeps
• Building a TensorRT-LLM Engine
  • Specifying an Engine Build Configuration
  • Building from a Hugging Face Repository
  • Building from a TensorRT-LLM Checkpoint
    • Weight-Stripped Engine Generation
  • Quantized Checkpoint Generation
    • Using Quantized Checkpoints to Build an Engine
  • TensorRT-LLM Engine Build Metrics
  • Custom Build Result Location
    • Uploading Build Results to a Custom NGC Model
  • Running a TensorRT-LLM Engine
• Building an ONNX Engine
  • Specifying an Engine Build Configuration
  • Specifying the ONNX Model
  • Querying the Status of a Build
  • Obtaining the Results of a Build
  • Weightful Engine Generation
    • Use Case
  • Weight-Stripped Engine Generation
    • Use Case
    • URL Inputs with --strip-weights
    • Refit Support
  • Refittable Engine Generation (Weightful or Weight-Stripped)
  • Building with Large ONNX Files
  • Supported trtexec Arguments
  • Custom Build Result Location
  • Running a TensorRT Engine
• Usage Credits
  • GPU-Specific Credit Allocation
  • Consuming Credits

Troubleshooting

• Bug Reporting

Reference

• Software License Agreement
  • NVIDIA CLOUD AGREEMENT
  • SERVICE-SPECIFIC TERMS FOR NVIDIA TensorRT CLOUD SERVICE
  • NVIDIA TensorRT Engine License Agreement