Support Compatibility#

Checking Available GPUs#

The GPUs available on TensorRT-Cloud are subject to change dynamically as we enable more configurations. To figure out what is currently supported, run:

trt-cloud info

This will produce output similar to:

[I] Available runners:
┌─────────┬────────────┬──────────────────────────────────┬────────────────────┬───────────────────────────────┐
│ OS       GPU         TRT Versions (for ONNX builds)    TRT-LLM Versions    Command                       │
├─────────┼────────────┼──────────────────────────────────┼────────────────────┼───────────────────────────────┤
│ Linux    A100        10.0.1, 10.2.0                    0.11.0              --os=linux --gpu=A100         │
│ Linux    H100        10.0.1, 10.2.0                    0.11.0              --os=linux --gpu=H100         │
│ Windows  GTX1660TI   10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=GTX1660TI  │
│ Windows  RTX30508GB  10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTX30508GB │
│ Windows  RTX3070     10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTX3070    │
│ Windows  RTX4000SFF  10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTX4000SFF │
│ Windows  RTX4060TI   10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTX4060TI  │
│ Windows  RTX4070     10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTX4070    │
│ Windows  RTX4090     10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTX4090    │
│ Windows  RTXA5000    10.0.1, 10.2.0                    0.11.0              --os=windows --gpu=RTXA5000   │
└─────────┴────────────┴──────────────────────────────────┴────────────────────┴───────────────────────────────┘

Note

TensorRT-Cloud does not support the full cross-combination of runtime GPUs and OS. For example, in the above output, engines may be built for inference on NVIDIA RTX 4090 on Windows but not NVIDIA RTX 4090 on Linux.

Planned GPU Support#

Over the next few minor releases, we will be adding support for the following platforms:

  • NVIDIA GeForce GTX 1630

  • NVIDIA GTX 1650 Super

  • NVIDIA GTX 1660 Super

  • NVIDIA 2060 SUPER

  • NVIDIA 2080 SUPER

  • NVIDIA Quadro RTX 4000

  • NVIDIA Quadro RTX 6000

  • NVIDIA Quadro RTX 8000

  • NVIDIA RTX A2000

  • NVIDIA RTX A6000

  • NVIDIA 4080 SUPER

  • NVIDIA RTX 4500 Ada Generation

  • NVIDIA RTX 6000 Ada Generation

  • NVIDIA Geforce RTX 2060

  • NVIDIA Geforce RTX 2080 Ti

  • NVIDIA GeForce RTX 3090 Ti

If you require any other SKU or GPUs to be supported, contact tensorrt-cloud-contact@nvidia.com.