Utilities#

NIM includes a set of utility scripts to assist with NIM operation.

Utilities can be launched by adding the name of the desired utility to the docker run command.

See the Supported Models section for setting valid values for CONTAINER_ID in below examples.

List Model Profiles#

nim_list_model_profiles()#

Prints the system information detected by NIM, and the list of all profiles for the chosen NIM to the console. Profiles are categorized by whether or not they are compatible with the current system, based on the system information detected.

This function can also be called using its alias list-model-profiles.

Example#

export CONTAINER_ID=parakeet-0-6b-ctc-en-us
docker run -it --rm --gpus all --entrypoint nim_list_model_profiles \
    nvcr.io/nim/nvidia/$CONTAINER_ID:latest

SYSTEM INFO
- Free GPUs:
-  [2331:10de] (0) NVIDIA H100 PCIe [current utilization: 0%]
MODEL PROFILES
- Compatible with system:
    - 3b65973c2bae708ba3e2f6dc65f7e85c2c12829fdd2970c848e4b7ee00175505 - {'bs': 'default', 'mode': 'all', 'model_type': 'rmir', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - c08781f3bd37aafe92d47d2e1f82dc8705f508d9f8ad11c6fcccd598adcf0fe4 - {'ampereplus': 'enabled', 'bs': 'default', 'gpu_device': '20b2', 'mode': 'all', 'model_type': 'prebuilt', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
- Incompatible with system:
    - 0cc6046cfea9c48e4a3da9dae93047e86fe078382347c2cd7a70ccf0ffb15b3d - {'bs': 'default', 'mode': 'str', 'model_type': 'rmir', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - 2eaec380f4418bcadcb1e3b1a889df2e16e628ae501ebde4b18d48b5212ea759 - {'bs': 'default', 'mode': 'str-thr', 'model_type': 'rmir', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - 332afa6f40cf0743d888b105db8b22ef051789d13cedcc6b99d3c9aaf527693b - {'ampereplus': 'enabled', 'bs': 'default', 'gpu_device': '20b2', 'mode': 'str-thr', 'model_type': 'prebuilt', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - 3715b3b8aac7cfd20ccf875bf28cee9648ce9a2c75c8957d8ff45eed71f8b26a - {'ampereplus': 'enabled', 'bs': '1', 'gpu_device': '20b2', 'mode': 'ofl', 'model_type': 'prebuilt', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - 42021258d0aba0d6367470f3f3339adc0bef7830f6662b5f11d3613d618daa91 - {'ampereplus': 'enabled', 'bs': 'default', 'gpu_device': '20b2', 'mode': 'str', 'model_type': 'prebuilt', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - 575d6c352f9a19cd1fcd24977d5d28421994e6966a5d59be2cc34e818155387e - {'bs': '1', 'mode': 'ofl', 'model_type': 'rmir', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - 9b43f0397877a8c1911e1ac56902d179fc69eb935496b3ffd6abcf6bcf060489 - {'ampereplus': 'enabled', 'bs': '1', 'gpu_device': '20b2', 'mode': 'str', 'model_type': 'prebuilt', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - aa7e6750641ae6b08661ee1bc530752afa8b445d15ee8b51dd3318faf409f496 - {'ampereplus': 'enabled', 'bs': 'default', 'gpu_device': '20b2', 'mode': 'ofl', 'model_type': 'prebuilt', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - cae9f6d064909978e49685b7ff2a1360f06dfea4aefc2f002ea48e91fc9ac43b - {'bs': 'default', 'mode': 'ofl', 'model_type': 'rmir', 'name': 'parakeet-0-6b-ctc-riva-en-us'}
    - d7fc7c3524b2cd1c9c6eedae4349cfc1ef5be8f9bb438322b0b93b5951ebfc9d - {'bs': '1', 'mode': 'str', 'model_type': 'rmir', 'name': 'parakeet-0-6b-ctc-riva-en-us'}

Download Model Profiles to NIM Cache#

nim_download_to_cache()#

Downloads selected or default model profile(s) to NIM cache. Can be used to pre-cache profiles prior to deployment. Requires NGC_API_KEY in environment.

This function can also be called using its alias download-to-cache.

--profiles [PROFILES ...], -p [PROFILES ...]#

Profile hashes to download. If none are provided, the optimal profile is downloaded. Multiple profiles can be specified separated by spaces.

--all#

Set to download all profiles to cache

--lora#

Set this to download default lora profile. This expects --profiles and --all arguments are not specified.

manifest-file <manifest_file>, -m <manifest_file>#

The manifest file path is an optional parameter that users can specify. It allows for the downloading of model profiles.

--model-cache-path <model-cache-path>#

The model cache path is an optional parameter that users can specify. This feature enables the modification of the default model_cache_path.

Example#

export CONTAINER_ID=parakeet-0-6b-ctc-en-us
export LOCAL_NIM_CACHE=$HOME/cache
docker run -it --rm --gpus all -e NGC_API_KEY \
    -v $LOCAL_NIM_CACHE:/opt/nim/.cache --entrypoint nim_download_to_cache \
    nvcr.io/nim/nvidia/$CONTAINER_ID:latest \
    -p 3715b3b8aac7cfd20ccf875bf28cee9648ce9a2c75c8957d8ff45eed71f8b26a

"timestamp": "2025-02-28 07:43:11,420", "level": "INFO", "message": "Fetching contents for profile 3715b3b8aac7cfd20ccf875bf28cee9648ce9a2c75c8957d8ff45eed71f8b26a"
"timestamp": "2025-02-28 07:43:11,420", "level": "INFO", "message": "Fetching contents for profile 3715b3b8aac7cfd20ccf875bf28cee9648ce9a2c75c8957d8ff45eed71f8b26a"
"timestamp": "2025-02-28 07:43:11,420", "level": "INFO", "message": "{
"ampereplus": "enabled",
"bs": "1",
"gpu_device": "20b2",
"mode": "ofl",
"model_type": "prebuilt",
"name": "parakeet-0-6b-ctc-riva-en-us"
}"

Create Model Store#

nim_create_model_store()#

Extracts files from a cached model profile and creates a properly formatted directory. If the profile is not already cached, it will be downloaded to the model cache. Downloading the profile requires NGC_API_KEY in environment.

This function can also be called using its alias create-model-store.

--profile <PROFILE>, -p <PROFILE>#

Profile hash to create a model directory of. Will be downloaded if not present.

--model-store <MODEL_STORE>, -m <MODEL_STORE>#

Directory path where model --profile will be extracted and copied to.

--model-cache-path <model-cache-path>#

The manifest file path is an optional parameter that users can specify. This feature enables the modification of the default model_cache_path.

Example#

export CONTAINER_ID=parakeet-0-6b-ctc-en-us
export LOCAL_NIM_CACHE=$HOME/cache
docker run -it --rm --gpus all -e NGC_API_KEY \
    -v $LOCAL_NIM_CACHE:/opt/nim/.cache --entrypoint nim_create_model_store \
    nvcr.io/nim/nvidia/$CONTAINER_ID:latest \
    -p 3715b3b8aac7cfd20ccf875bf28cee9648ce9a2c75c8957d8ff45eed71f8b26a \
    -m /tmp

...
"timestamp": "2025-02-28 07:45:10,533", "level": "INFO", "message": "Creating model store at /tmp"
...
"timestamp": "2025-02-28 07:45:10,540", "level": "INFO", "message": "Copying contents for profile 3715b3b8aac7cfd20ccf875bf28cee9648ce9a2c75c8957d8ff45eed71f8b26a to /tmp"
...

Check NIM Cache#

nim_check_cache_env()#

Checks if the NIM cache directory is present and can be written to.

This function can also be called using its alias nim-llm-check-cache-env.

Example#

export CONTAINER_ID=parakeet-0-6b-ctc-en-us
export LOCAL_NIM_CACHE=$HOME/cache
docker run -it --rm --gpus all -e NGC_API_KEY \
    -v /bad_path:/opt/nim/.cache --entrypoint nim_create_model_store \
    nvcr.io/nim/nvidia/$CONTAINER_ID:latest \

The NIM cache directory /opt/nim/.cache is read-only. Application may fail if the model is not already present in the cache.
On this page