Triton Image Allowlist

The Triton image allowlist is a management feature that controls which Triton images can be used to create Triton pools and bespoke Triton instances. This feature gives administrators some basic controls over which images can and cannot be used when creating Triton instances. If an image is not in the allowlist, it cannot be used to create a new Triton instance. The Triton allowlist service is used to inspect and modify the allowlist.

The list of operations for the Triton Allowlist Service includes the following:

  • TritonAllowlist/Append adds a Triton container image to the list of the allowed Triton container images.

    The server returns success or failure depending on whether the requested image could be added to the allowlist.

    Attempting to add images that are already present in the allowlist does not result in any changes.

  • TritonAllowlist/List returns a list of the allowed Triton container images.

    This RPC begins streaming a response after the request has been received.

    Each response message contains a Triton image that belongs to the list.

  • TritonAllowlist/Remove deletes a Triton container image from the list of the allowed Triton container images.

    The server returns success or failure depending on whether the image could be removed from the allowlist.

Initially, the allowlist only contains the default Triton image, as configured during installation. This can be seen by running tmsctl allowlist list. For example, to verify your installation you can run the command to return the default image:

Copy
Copied!
            

$ tmsctl allowlist list nvcr.io/nvidia/tritonserver:23.09-py3

At this stage, new Triton instances can only be created with nvcr.io/nvidia/tritonserver:23.09-py3 as the Triton image. For example, trying to use nvcr.io/nvidia/tritonserver:23.08-py3 fails.

Copy
Copied!
            

$ tmsctl lease create --triton-image nvcr.io/nvidia/tritonserver:23.08-py3 -m "name=$MODEL_NAME,uri=$MODEL_URI" fatal: Requested Triton container image ("nvcr.io/nvidia/tritonserver:23.08-py3") is unreachable or not provided in a supported format. Unreachable container images either do not exist or require privileges not granted to the server. (triton_options_bespoke.triton.container_image @ Acquire)

New entries can be added using tmsctl allowlist add. For example:

Copy
Copied!
            

$ tmsctl allowlist add nvcr.io/nvidia/tritonserver:23.08-py3 Added nvcr.io/nvidia/tritonserver:23.08-py3 $ tmsctl allowlist list nvcr.io/nvidia/tritonserver:23.08-py3 nvcr.io/nvidia/tritonserver:23.09-py3

After adding images, you can create new bespoke Triton instances and Triton pools specifying nvcr.io/nvidia/tritonserver:23.08-py3 as the image. For example:

Copy
Copied!
            

$ tmsctl lease create --triton-image nvcr.io/nvidia/tritonserver:23.08-py3 -m "name=$MODEL_NAME,uri=$MODEL_URI" Lease da21b2c0e68b49ffa8f0f6db0b030128 State: Valid Expires: 2023-10-18T15:43:53Z Triton: triton-6d8c9d13.tmsns.svc.cluster.local <nvcr.io/nvidia/tritonserver:23.08-py3> Models: Name Url Status <model_name> <model_url> Ready

You can remove entries using tmsctl allowlist rm. For example:

Copy
Copied!
            

$ tmsctl allowlist rm nvcr.io/nvidia/tritonserver:23.10-py3 Removed nvcr.io/nvidia/tritonserver:23.10-py3 $ tmsctl allowlist list nvcr.io/nvidia/tritonserver:23.09-py3

After removing images, any attempts to create new bespoke Tritons or Triton pools specifying the removed image causes a failure.

© Copyright 2023, NVIDIA. Last updated on Dec 11, 2023.