Overview#

We are introducing a new environment variable, NIM_REPOSITORY_OVERRIDE, that enables users to override the location where protocol and repository model files are stored.

When NIM_REPOSITORY_OVERRIDE is set and not empty, it replaces the default repository location and protocol used to access the repository.

Structure#

The value for this environment variable follows the structure:

<repository_type>://<repository_location>

Supported Repository Types#

  • s3

S3 Support#

Using the Feature#

1. Create an S3 Bucket#

Create an S3 bucket (for example: nim_bucket).

2. Download Files#

Download the files listed in the model manifest from NGC to a local file system. You can find the default model manifest in the container at /opt/nim/etc/default/model_manifest.yaml.

3. Upload Files to S3#

Upload the files to the S3 bucket (for example: nim_bucket) using a URL-encoded path as the key for each object.

Example#

  • Source URI:

    ngc://nim/meta/llama3-8b-instruct:hf?file=config.json

  • NIM_REPOSITORY_OVERRIDE:

    s3://nim_bucket/

  • Expected S3 Path:

    s3://nim_bucket/nim%2Fmeta%2Fllama3-8b-instruct%3Ahf%3Ffile%3Dconfig.json

4. Set the Environment Variable#

export NIM_REPOSITORY_OVERRIDE=s3://nim_bucket/

5. Configure AWS Credentials#

Specify your AWS credentials using the following environment variables or in a credentials file:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN  # (if using temporary credentials)

6. Launch NIM#

Example Docker Command#

The following command requires that you have stored a temporary AWS token in /home/usr/.aws and stored your CA certificate in /etc/ssl/certs:

docker run --rm --runtime=nvidia --gpus=all -p 8000:8000 \
    -e NGC_API_KEY=$NGC_API_KEY \
    -v /home/usr/.aws:/tmp/.aws \
    -e AWS_SHARED_CREDENTIALS_FILE=/tmp/.aws/credentials \
    -e AWS_PROFILE=default \
    -e AWS_REGION=us-east-1 \
    -e NIM_REPOSITORY_OVERRIDE=$NIM_REPOSITORY_OVERRIDE \
    -v /etc/ssl/certs:/etc/ssl/certs:ro \
    nim-image
On this page