Support Compatibility#
Checking Available GPUs#
The GPUs available on TensorRT-Cloud are subject to change dynamically as we enable more configurations. To figure out what is currently supported, run:
trt-cloud info
This will produce output similar to:
[I] Available runners: ┌─────────┬────────────┬──────────────────────────────────┬────────────────────┬───────────────────────────────┐ │ OS │ GPU │ TRT Versions (for ONNX builds) │ TRT-LLM Versions │ Command │ ├─────────┼────────────┼──────────────────────────────────┼────────────────────┼───────────────────────────────┤ │ Linux │ A100 │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=linux --gpu=A100 │ │ Linux │ H100 │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=linux --gpu=H100 │ │ Windows │ GTX1660TI │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=GTX1660TI │ │ Windows │ RTX30508GB │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTX30508GB │ │ Windows │ RTX3070 │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTX3070 │ │ Windows │ RTX4000SFF │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTX4000SFF │ │ Windows │ RTX4060TI │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTX4060TI │ │ Windows │ RTX4070 │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTX4070 │ │ Windows │ RTX4090 │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTX4090 │ │ Windows │ RTXA5000 │ 10.0.1, 10.2.0 │ 0.11.0 │ --os=windows --gpu=RTXA5000 │ └─────────┴────────────┴──────────────────────────────────┴────────────────────┴───────────────────────────────┘
Note
TensorRT-Cloud does not support the full cross-combination of runtime GPUs and OS. For example, in the above output, engines may be built for inference on NVIDIA RTX 4090 on Windows but not NVIDIA RTX 4090 on Linux.
Planned GPU Support#
Over the next few minor releases, we will be adding support for the following platforms:
NVIDIA GeForce GTX 1630
NVIDIA GTX 1650 Super
NVIDIA GTX 1660 Super
NVIDIA 2060 SUPER
NVIDIA 2080 SUPER
NVIDIA Quadro RTX 4000
NVIDIA Quadro RTX 6000
NVIDIA Quadro RTX 8000
NVIDIA RTX A2000
NVIDIA RTX A6000
NVIDIA 4080 SUPER
NVIDIA RTX 4500 Ada Generation
NVIDIA RTX 6000 Ada Generation
NVIDIA Geforce RTX 2060
NVIDIA Geforce RTX 2080 Ti
NVIDIA GeForce RTX 3090 Ti
If you require any other SKU or GPUs to be supported, contact tensorrt-cloud-contact@nvidia.com.