Installing TensorRT-Cloud CLI

Prerequisites

  1. Install Linux, Windows, or MacOS.

  2. Install Python 3.8 or above.

  3. The tensorrt or tensorrt-lean Python package must be installed with the version matching the TensorRT engine for refit support through the CLI. The tensorrt-lean package is required for refitting version-compatible engines (those built with the --vc trtexec flag), and tensorrt is required for refitting non-version-compatible engines.

    python3 -m pip install tensorrt==10.0.1 tensorrt-cu12==10.0.1
python3 -m pip install tensorrt-lean==10.0.1 tensorrt_lean-cu12==10.0.1

    For more information, refer to the NVIDIA TensorRT Installation Guide.

  4. If you would like to refit a TensorRT-LLM engine, install it locally. For more information, refer to the NVIDIA TensorRT-LLM Quick Start Guide.

Note

refit is not supported on MacOS as TensorRT does not support MacOS.

Steps

The TensorRT-Cloud CLI tool is distributed as a Python wheel packaged in a tar file. To access the tar, refer to Getting TensorRT-Cloud Access.

  1. Use pip to install.

    python3 -m pip install trt-cloud

  2. Verify the TensorRT-Cloud CLI tool was installed successfully.

    trt-cloud --version

  3. Get an overview of the CLI usage.

    trt-cloud --help