NeMo Guardrails Microservice Deployment Guide#

Find more information on prerequisites and configuring the key parts of the values.yaml file in the NeMo Guardrails Helm chart.

Prerequisites#

A persistent volume provisioner that uses network storage such as NFS, S3, vSAN, and so on. The guardrails configuration is stored in persistent storage.
Optional: Install NVIDIA GPU Operator if your nodes have NVIDIA GPUs. The demonstration configuration shown on this page does not require NVIDIA GPUs.
To use a single model from one locally-deployed NIM for LLMs container, deploy the container and know the service name so you can specify the service name in the NIM_ENDPOINT_URL environment variable when you install the NeMo Guardrails microservice Helm chart.
To use models from multiple locally-deployed NIM for LLMs containers, deploy the following services as prerequisites:

Running the Microservice Container#

Set the environment variables with your NGC API key and NVIDIA API key:

$ export NGC_CLI_API_KEY="M2..."
$ export NVIDIA_API_KEY="nvapi-..."

Create a namespace for the microservice:

$ kubectl create namespace guardrails-ms

Add a Docker registry secret for downloading the container image from NVIDIA NGC:

$ kubectl create secret -n guardrails-ms docker-registry nvcrimagepullsecret \
   --docker-server=nvcr.io \
   --docker-username='$oauthtoken' \
   --docker-password=$NGC_CLI_API_KEY

Search for the NeMo Guardrails Helm chart from the NGC Catalog and select the desired version of the chart. For more information about using Helm charts provided in the NGC Catalog in general, see Helm Charts in the NGC Catalog User Guide.

Download the chart using the helm fetch command with your user name and NGC API key as follows:
```
$ helm fetch "https://helm.ngc.nvidia.com/nvidia/nemo-microservices/charts/nemo-guardrails-25.6.0.tgz" \
   --username='$oauthtoken' \
   --password=$NGC_CLI_API_KEY
```
Save the default chart values in a file:
```
$ helm show values nemo-guardrails-25.6.0.tgz > values.yaml
```
Edit the values.yaml file. For more information about the values, refer to NeMo Guardrails Microservice Helm Chart.

Refer to the following common configuration options:
- Using Locally-Deployed NIM for LLMs
- Using Models from NVIDIA API Catalog Exclusively

Install the chart and then port-forward the service:

$ helm install guardrails-ms nemo-guardrails-25.6.0.tgz \
    --namespace guardrails-ms \
    -f values.yaml

Partial Output

 Get the application URL by running these commands:
   export POD_NAME=...
   export CONTAINER_PORT=...
   echo "Visit http://127.0.0.1:8080 to use your application"
   kubectl port-forward $POD_NAME 8080:$CONTAINER_PORT

Running the export and kubectl commands prints the URL that you can use to interact with the microservice.

Using Locally-Deployed NIM for LLMs#

Update the values.yaml file to specify the service address for NeMo NIM Proxy:

env:
  NIM_ENDPOINT_URL: nemo-nim-proxy:8000
  # Or, a single NIM for LLM service
  # NIM_ENDPOINT_URL: meta-llama3-8b-instruct:8000

Optional: Configure access to models on NVIDIA API Catalog.
1. Add the following secret that populates the NVIDIA_API_KEY environment variable in the container:
```
$ kubectl create secret -n guardrails-ms generic nvidia-api-secret \
    --from-literal=NVIDIA_API_KEY=$NVIDIA_API_KEY
```
2. Update the values.yaml file so that NVIDIA_API_KEY is populated in the container from the secret:
```
guardrails:
  nvcfAPIKeySecretName: nvidia-api-secret
```

Using Models from NVIDIA API Catalog Exclusively#

If you plan to leave the NIM_ENDPOINT_URL environment variable set to the default value https://integrate.api.nvidia.com/v1, add the follow secret that populates the NIM_ENDPOINT_API_KEY environment variable in the container:
```
$ kubectl create secret -n guardrails-ms generic nim-endpoint-api-secret \
    --from-literal=nim-endpoint-api-key=$NVIDIA_API_KEY
```

Edit the values.yaml file with the following changes to enable the demonstration configuration:

env:
  DEMO: True
  NIM_ENDPOINT_API_KEY:
    valueFrom:
      secretKeyRef:
        name: nim-endpoint-api-secret
        key: nim-endpoint-api-key

Using NFS Configuration Store#

To use an NFS location for the configuration store, you need to add the following to your values.yaml file:

configStore:
  nfs:
    enabled: true
    path: "/path/to/nfs/share"
    server: "nfs-server.example.com"
    mountPath: "/config-store"
    storageClass: "standard"

Important

Make sure the configStore.nfs.path key points to the root of the configuration store directory, which is the directory containing subfolders for each individual Guardrails configuration.