Skip to navigation

Utilities

View as Markdown

NIM includes a set of utility scripts to assist with NIM operation.

Utilities can be launched by adding the name of the desired utility to the docker run command. For example, you can execute the list-model-profiles utility with the following command:

docker run --rm --runtime=nvidia --gpus=all \
  --entrypoint list-model-profiles \
  nvcr.io/nim/nvidia/genmol:1.0.0

You can get more information about each utility with the -h flag:

docker run --rm --runtime=nvidia --gpus=all \
  --entrypoiint download-to-cache \
  nvcr.io/nim/nvidia/genmol:1.0.0 \
  -h

List Available Model Profiles

Prints to the console the system information detected by NIM, and the list of all profiles for the chosen NIM. Profiles are categorized by whether or not they are compatible with the current system, based on the system information detected.

list-model-profiles

Example

docker run -it --rm --gpus all -e NIM_LOG_LEVEL=WARNING \
  --entrypoint list-model-profiles \
  nvcr.io/nim/nvidia/genmol:1.0.0
SYSTEM INFO
- Free GPUs:
  -  [2230:10de] (0) NVIDIA RTX A6000 [current utilization: 2%]
  -  [2230:10de] (1) NVIDIA RTX A6000 [current utilization: 3%]
MODEL PROFILES
- All profiles:
    - a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036 (type=bert)

Download Model Profiles to NIM Cache

Downloads selected or default model profile(s) to NIM cache. Can be used to pre-cache profiles prior to deployment. Requires NGC_API_KEY and LOCAL_NIM_CACHE in environment. The folder $LOCAL_NIM_CACHE needs to be writable (with mode 777).

download-to-cache
  --all 
        Set this to download all profiles to cache
  --profiles [PROFILES ...], -p [PROFILES ...] 
        Profile(s) to download. If none are provided, 
        the optimal profile is downloaded. Multiple 
        profiles can be specified separated by spaces.

Example

docker run -it --rm --gpus all -e NGC_API_KEY -v $LOCAL_NIM_CACHE:/opt/nim/.cache \
  --entrypoint download-to-cache \
  nvcr.io/nim/nvidia/genmol:1.0.0 \
  -p a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036
"timestamp": "2024-12-17 07:14:22,524", "level": "INFO", "message": "Fetching contents for profile a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036"
"timestamp": "2024-12-17 07:14:22,524", "level": "INFO", "message": "{
  "type": "bert"
}"
"timestamp": "2024-12-17 07:14:22,524", "level": "INFO", "message": "Using the profile specified by the user: a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036"
"timestamp": "2024-12-17 07:14:22,524", "level": "INFO", "message": "Downloading manifest profile: a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036"
tokenizer.json [00:00:00] [█████████████████████████████████████████████████████████████████████████████████] 45.23 KiB/45.23 KiB 2.12 MiB/s (0s)
model.ckpt [00:00:03] [█████████████████████████████████████████████████████████████████████████████████] 338.59 MiB/338.59 MiB 106.88 MiB/s (0s)

Create Model Store

Extracts files from a cached model profile and creates a properly formatted directory. If the profile is not already cached, it will be downloaded to the model cache. Downloading the profile requires NGC_API_KEY in environment.

create-model-store
  --profile <PROFILE>, -p <PROFILE>
        Profile hash to create a model directory of. 
        Will be downloaded if not present.
  --model-store <MODEL_STORE>, -m <MODEL_STORE>
        Directory path where model `--profile` 
        will be extracted and copied to.

Example

docker run -it --rm --gpus all -e NGC_API_KEY -v $LOCAL_NIM_CACHE:/opt/nim/.cache \
  --entrypoint create-model-store \
  nvcr.io/nim/nvidia/genmol:1.0.0 \
  -p a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036 -m /tmp
"timestamp": "2024-12-17 07:18:41,484", "level": "INFO", "message": "Fetching contents for profile a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036"
"timestamp": "2024-12-17 07:18:41,484", "level": "INFO", "message": "Using the default model_cache_path: /opt/nim/workspace"
"timestamp": "2024-12-17 07:18:41,484", "level": "INFO", "message": "Creating model store at /tmp"
model.ckpt [00:00:03] [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 338.59 MiB/338.59 MiB 106.80 MiB/s (0s)tokenizer.json [00:00:00] [█████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 45.23 KiB/45.23 KiB 2.81 MiB/s (0s)"timestamp": "2024-12-17 07:18:45,697", "level": "INFO", "message": "Copying contents for profile a525212dd4373ace3be568a38b5d189a6fb866e007adf56e750ccd11e896b036 to /tmp"

Check NIM Cache

Checks if the NIM cache directory is present and can be written to.

nim-llm-check-cache-env

Example

docker run -it --rm --gpus all -v /bad_path:/opt/nim/.cache \
  --entrypoint nim-llm-check-cache-env \
  nvcr.io/nim/nvidia/genmol:1.0.0
The NIM cache directory /opt/nim/.cache is read-only. Application could fail if the model is not already present in the cache.

Set Cache Environment Variables

Prints commands for setting cache environment variables to console.

nim-llm-set-cache-env

Example

docker run -it --rm --gpus all \
  --entrypoint nim-llm-set-cache-env \
  nvcr.io/nim/nvidia/genmol:1.0.0
export NGC_HOME=/opt/nim/.cache/ngc/hub
export HF_HOME=/opt/nim/.cache/huggingface