NVIDIA TensorRT-Cloud Documentation

TensorRT-Cloud Overview

• Introduction
• Release Notes
  • TensorRT-Cloud 0.5.0 Early Access (EA)
  • TensorRT-Cloud 0.4.1 Early Access (EA)
  • TensorRT-Cloud 0.4.0 Early Access (EA)
  • TensorRT-Cloud 0.3.0 Early Access (EA)
  • TensorRT-Cloud 0.2.0 Early Access (EA)
  • TensorRT-Cloud 0.1.1 Early Access (EA)
• Support Compatibility
  • Checking Available GPUs
  • Planned GPU Support

Getting Started

• Getting TensorRT-Cloud Access
• Installing TensorRT-Cloud CLI
• Enabling Early Access (EA) Features in TensorRT-Cloud CLI
• Logging into TensorRT-Cloud
• Example: Quick Start
  • Downloading a Pre-built TensorRT Engine
  • Building a TensorRT Engine for the ONNX Model
  • Building a TensorRT-LLM Engine

User Guide

• TensorRT Engines for Community Models
  • Discovering Models
  • Discovering Available Engine Versions
    • Filtering Engines by Configs
  • Downloading a Pre-built Engine
  • Running a Pre-Built Engine
• Building an ONNX Engine
  • Specifying an Engine Build Configuration
  • Specifying the ONNX Model
  • Output Zip File
  • Weightful Engine Generation
    • Use Case
  • Weight-Stripped Engine Generation
    • Use Case
    • URL Inputs with --strip-weights
    • Refit Support
  • Refittable Engine Generation (Weightful or Weight-Stripped)
  • Building with Large ONNX Files
  • Resuming Interrupted Builds
  • Supported trtexec Arguments
  • Running a TensorRT Engine
• Building a TensorRT-LLM Engine
  • Specifying an Engine Build Configuration
  • Building from a Hugging Face Repository
  • Building from a TensorRT-LLM Checkpoint
    • Weight-Stripped Engine Generation
  • Quantized Checkpoint Generation
    • Using Quantized Checkpoints to Build an Engine
  • TensorRT-LLM Engine Build Metrics
    • Running a TensorRT-LLM Engine

Troubleshooting

• Bug Reporting

Reference

• Software License Agreement
  • NVIDIA CLOUD AGREEMENT
  • SERVICE-SPECIFIC TERMS FOR NVIDIA TensorRT CLOUD SERVICE
  • NVIDIA TensorRT Engine License Agreement