Caching Non-LLM NIM

About NVIDIA NIM Microservices

NVIDIA NIM microservices are a set of easy-to-use microservices for accelerating the deployment of foundation models on any cloud or data center. These microservices help keep your data secure. NIM microservices have production-grade runtimes and support a wide variety of domains, such as retrieval, vision, speech, biology, and safety and moderation.

For more information, refer to NVIDIA NIM.

Non-LLM NIM Caching

NIM Cache supports non-LLM NIMs with and without model profiles. Choose the caching configuration based on whether the NIM uses model profiles.

Cache NVIDIA NeMo Retriever NIM 2.x+

NVIDIA NeMo Retriever NIM 2.x+ containers introduce a purpose-built runtime. The NIM selects optimized CUDA kernels automatically at startup, so you do not need to list or select model profiles. The NIM downloads model weights on the first run.

You can use a NIM Cache to download the model weights before you deploy the NIM and store the weights and runtime cache on persistent storage. The NIM Operator runs the NIM container in download-only mode, waits for the download to finish, and then uses the same persistent volume for the NIM Service. The Operator mounts the persistent volume at /model for model weights and /opt/cache for cached runtime artifacts, and configures the download-only mode and model path automatically.

Choose a Model Download Provider

The NeMo Retriever NIM 2.x+ containers pull the NIM container image from NGC. Independently, these containers can download model weights from Hugging Face or NGC. The spec.source.ngc.modelPuller field identifies the NIM container image. The field does not select the model download provider.

The spec.source.ngc.authSecret field supplies the credential used by the selected provider. The NIM Service spec.authSecret field supplies the same credential to the running NIM.

Model Download Provider	Configuration	Credential
Hugging Face (default)	No provider setting is required.	HF_TOKEN
NGC	Set NIM_MODEL_DOWNLOAD_PROVIDER=ngc for the NIM Cache and NIM Service.	NGC_API_KEY

In both cases, you need to set ngc-secret to pull the NIM container image from NGC. The Operator also uses this secret to inspect the image before it creates the caching job.

Use Hugging Face as the Model Download Provider

The following sample uses Hugging Face, the default model download provider for NeMo Retriever NIM 2.x+. Before you apply the sample, create the following secrets in the nim-service namespace:

• ngc-secret for pulling the NIM container image from NGC.

• hf-api-secret containing HF_TOKEN for downloading model weights from Hugging Face.

Refer to Image Pull Secrets for instructions.

1. Create a file, such as retriever-embedding.yaml, with the following contents:

   ---
   # NIM Cache for NVIDIA NeMo Retriever Embedding NIM 2.0.0
   apiVersion: apps.nvidia.com/v1alpha1
   kind: NIMCache
   metadata:
     name: llama-nemotron-embed-vl-1b-v2
     namespace: nim-service
   spec:
     source:
       ngc:
         modelPuller: nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:2
         pullSecret: ngc-secret
         authSecret: hf-api-secret
     storage:
       pvc:
         create: true
         storageClass: ""
         size: "50Gi"
         volumeAccessMode: ReadWriteOnce
   
   ---
   # NIM Service for NVIDIA NeMo Retriever Embedding NIM 2.0.0
   apiVersion: apps.nvidia.com/v1alpha1
   kind: NIMService
   metadata:
     name: llama-nemotron-embed-vl-1b-v2
     namespace: nim-service
   spec:
     image:
       repository: nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2
       tag: "2.0.0"
       pullPolicy: IfNotPresent
       pullSecrets:
         - ngc-secret
     authSecret: hf-api-secret
     storage:
       nimCache:
         name: llama-nemotron-embed-vl-1b-v2
     replicas: 1
     resources:
       limits:
         nvidia.com/gpu: 1
     expose:
       service:
         type: ClusterIP
         port: 8000
   

   The spec.source.ngc.model field is intentionally omitted because this NIM does not use model profiles.

2. Create the NIM Cache and NIM Service:

   $ kubectl apply -f retriever-embedding.yaml
   

3. Wait for the NIM Cache and NIM Service to report Ready:

   $ kubectl get nimcache,nimservice -n nim-service
   

   The NIM Cache status does not list model profiles for this NIM.

Use NGC as the Model Download Provider

To download model weights from NGC instead, create ngc-api-secret containing NGC_API_KEY as described in Image Pull Secrets. In the complete sample, replace hf-api-secret with ngc-api-secret and add NIM_MODEL_DOWNLOAD_PROVIDER=ngc to spec.env for both resources:

# NIMCache fields
spec:
  source:
    ngc:
      authSecret: ngc-api-secret
  env:
    - name: NIM_MODEL_DOWNLOAD_PROVIDER
      value: "ngc"
---
# NIMService fields
spec:
  authSecret: ngc-api-secret
  env:
    - name: NIM_MODEL_DOWNLOAD_PROVIDER
      value: "ngc"

For details about model download providers and persistent storage paths, refer to Get Started With NVIDIA NeMo Retriever Embedding NIM.

Cache NIMs That Use Model Profiles

Use one of the following model sources:

NVIDIA NGC Catalog

When you create a NIM Cache resource with the NGC Catalog as the source, the NIM Operator starts a pod that lists the available model profiles. The Operator creates a config map of the model profiles.

To pull models from the NGC Catalog production branch, you must have created Kubernetes secrets to hold your NGC Catalog API key and pass the secret names as spec.source.ngc.pullSecret and spec.source.ngc.authSecret. Refer to Image Pull Secrets for more details on creating these secrets.

The following shows an example of using the NGC catalog as a cache source.

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMCache
metadata:
  name: rerankqa-mistral-4b-v3
  namespace: nim-service
spec:
  source:
    ngc:
      modelPuller: nvcr.io/nim/nvidia/nv-rerankqa-mistral-4b-v3:1.0.2
      pullSecret: ngc-secret
      authSecret: ngc-api-secret
      model:   #Include the model object to describe the model you want to pull from NGC
        engine: tensorrt
        tensorParallelism: "1"
  storage:
    pvc:
      create: true
      storageClass: ''
      size: "50Gi"
      volumeAccessMode: ReadWriteOnce

Note

NVIDIA recommends that you use profile filtering when caching models using source.ngc.model.. Models can have several profiles, and without filtering by one or more parameters, you can download more models than intended, which can increase your storage requirements. For more information on NIM profiles and their model storage requirements, refer to the NIM Models documentation.

Refer to the following table for information about fields for NVIDIA NGC Catalog as a NIM Cache Source:

Field	Description	Default Value
spec.source.ngc.model	Specifies an object of filtering information for the model and profile you want to cache. If you want to cache a Multi-LLM model, use spec.source.ngc.modelEndpoint instead.	None
spec.source.ngc.model.engine	Specifies a model caching constraint based on the engine. Common values are as follows: tensorrt – optimized engine for embedding and reranking onnx – community engine Each NIM microservice determines the supported engines. Refer to the microservice documentation for the latest information. By default, the caching job matches model profiles for all engines.	None
spec.source.ngc.model.profiles	Specifies an array of model profiles to cache. When you specify this field, automatic profile selection is disabled and all other source.ngc.model fields are ignored. The following partial specification requests a specific model profile. spec: source: ngc: modelPuller: nvcr.io/nim/meta/llama3-8b-instruct:1.0.3 model: profiles: - 8835c31... You can determine the model profiles by running the list-model-profiles command. You can specify all to download all model profiles. Use the all parameter with care as some models have many profiles that will take several minutes to download and a large amount of storage space to cache.	None
spec.source.ngc.modelPuller	Specifies the NIM container image used to download model weights.	None

Mirrored Local Model Registries (S3, HTTPS, or JFrog)

To use a NGC Mirrored Local Model Registry as a NIM Cache source, set NIM_REPOSITORY_OVERRIDE as an environment variable for the NIM.

Refer to Repository Override for NVIDIA NIM for LLMs for more detailed instructions.

Refer to NIM for LLMs Environment Variables for more information on the NIM_REPOSITORY_OVERRIDE environment variable.

Note

The NIM Cache fields relevant to Mirrored Local Model Registries are the same as for NVIDIA NGC Catalog as a NIM Cache Source.

The following sample manifests are available in the config/samples/nim/caching/ngc-mirror directory.

S3

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMCache
metadata:
  name: meta-llama3-2-1b-instruct
  namespace: nim-service
spec:
  env:
    - name: NIM_REPOSITORY_OVERRIDE
      value: "s3://nim_bucket/"
    - name: AWS_PROFILE
      value: "default"
    - name: AWS_REGION
      value: "us-east-1"
  source:
    ngc:
      modelPuller: nvcr.io/nim/meta/llama-3.2-1b-instruct:1.8
      pullSecret: ngc-secret
      authSecret: aws-api-secret
      model:
        engine: "tensorrt"
        tensorParallelism: "1"
  storage:
    pvc:
      create: true
      storageClass: ''
      size: "50Gi"
      volumeAccessMode: ReadWriteOnce

Note

You must specify your AWS credentials in the aws-api-secret using the following environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN  # (if using temporary credentials)

For more information, refer to Configure AWS Credentials.

HTTPS

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMCache
metadata:
  name: meta-llama3-2-1b-instruct
  namespace: nim-service
spec:
  env:
    - name: NIM_REPOSITORY_OVERRIDE
      value: "https://<server-name>:<port>/"
  source:
    ngc:
      modelPuller: nvcr.io/nim/meta/llama-3.2-1b-instruct:1.8
      pullSecret: ngc-secret
      authSecret: https-api-secret
      model:
        engine: "tensorrt"
        tensorParallelism: "1"
  storage:
    pvc:
      create: true
      storageClass: ''
      size: "50Gi"
      volumeAccessMode: ReadWriteOnce

JFrog

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMCache
metadata:
  name: meta-llama3-2-1b-instruct
  namespace: nim-service
spec:
  env:
    - name: NIM_REPOSITORY_OVERRIDE
      value: "jfrog://<server-name>:<port>/"
  source:
    ngc:
      modelPuller: nvcr.io/nim/meta/llama-3.2-1b-instruct:1.8
      pullSecret: ngc-secret
      authSecret: jfrog-api-secret
      model:
        engine: "tensorrt"
        tensorParallelism: "1"
  storage:
    pvc:
      create: true
      storageClass: ''
      size: "50Gi"
      volumeAccessMode: ReadWriteOnce