NVIDIA TensorRT Inference Server¶
The NVIDIA TensorRT Inference Server (TRTIS) provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server.
What’s New in 0.9.0 Beta¶
- TRTIS now monitors the model repository for any change and dynamically reloads the model when necessary, without requiring a server restart. It is now possible to add and remove model versions, add/remove entire models, modify the model configuration, and modify the model labels while the server is running.
- Added a model priority parameter to the model configuration. Currently the model priority controls the CPU thread priority when executing the model and for TensorRT models also controls the CUDA stream priority.
- Fixed a bug in GRPC API: changed the model version parameter from string to int. This is a non-backwards compatible change.
- Added –strict-model-config=false option to allow some model configuration properties to be derived automatically. For some model types, this removes the need to specify the config.pbtxt file.
- Improved performance from an asynchronous GRPC frontend.
Features¶
TRTIS provides the following features:
- Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT integrated models.
- Multi-GPU support. TRTIS can distribute inferencing across all system GPUs.
- Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.
- Batching support. For models that support batching, the server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. TRTIS also supports dynamic batching where individual inference requests are dynamically combined together to improve inference throughput. Dynamic batching is transparent to the client requesting inference.
- Model repositories may reside on a locally accessible file system (e.g. NFS) or in Google Cloud Storage.
- Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.
- Metrics indicating GPU utiliization, server throughput, and server latency.
User Guide
Developer Guide