System Configuration#

This section provides a step by step instruction on how the blueprints are deployed once the Cluster Software Reference stack has been deployed and configured.

Assumptions#

  • The environment has egress to the internet enabled

Pre-requisites for installing RAG and AI-Q#

  • A Kubernetes Cluster with a supported version installed with NVIDIA GPU Operator and NVIDIA Network Operator installed, which should already be in place if the NVIDIA Software Enterprise RA is followed.

  • Active Subscription to NVAIE and Access to the NGC Enterprise Catalog. Please generate and download your nvcr.io access token in NGC.

  • Account in Tavily and an Access token to Tavily API, this will be needed for AI-Q to run internet search agent

  • Install Helm in the Kubernetes Cluster and download the helm CLI. This should have been installed if the Software Enterprise RA is followed.

  • Access to Kubernetes Clusters config file and kubectl CLI installed, this file is in the Enterprise RA cluster BCMe head node under //.kube/config

  • A default Storage Class is defined to create Persistent Volume Claims by Kubernetes, this can be an NFS or Block-based storage class, the storage class name is default.

  • Ability to create two namespaces in the Kubernetes cluster. rag, aiq

Deploy and Configure RAG Blueprint#

NVIDIA RAG Blueprint can be deployed in multiple ways, for the purpose of this document and respect to the Enterprise Software Reference Architecture we are using Helm as the deployment method. For detailed information on Helm and how to deploy RAG using Helm, refer to the NVIDIA Github Repository here. For Helm-specific deployment steps, refer to the deploy-helm documentation. From the system that has access to the Kubernetes API and has Helm deployed, run the following on the command prompt

Export the NGC API KEY

export NGC_API_KEY="nvapi-<redacted>"

Create a directory for RAG, CD to it and fetch the RAG repo

mkdir rag
cd rag
Git clone https://github.com/NVIDIA-AI-Blueprints/rag.git

Add the appropriate Helm Repos to the cluster

helm repo add nvidia-nim https://helm.ngc.nvidia.com/nim/nvidia/ --username='$oauthtoken' --password=$NGC_API_KEY
helm repo add nim https://helm.ngc.nvidia.com/nim/ --username='$oauthtoken' --password=$NGC_API_KEY
helm repo add nemo-microservices https://helm.ngc.nvidia.com/nvidia/nemo-microservices --username='$oauthtoken' --password=$NGC_API_KEY
helm repo add baidu-nim https://helm.ngc.nvidia.com/nim/baidu --username='$oauthtoken' --password=$NGC_API_KEY
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add otel https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add zipkin https://zipkin.io/zipkin-helm
helm repo add prometheus https://prometheus-community.github.io/helm-charts

Change the directory

cd rag/deploy/helm

create a namespace for RAG

kubectl create namespace rag

Update the default values from RAG Helm chart to work with RTX PRO 6000 BSE GPUs, copy and paste the default values file to a new file name in the rag/deploy/helm folder called values-2gpu-rtxpro6000.yaml. The detailed YAML file for this config is provided in Appendix A, RAG Config, below are key things to update in the file

Ensure under the NIM section, the model is updated with Nemotron 49B v1.5 with the right NIM profile for RTX PRO with fp8 precision

nim-llm:
  enabled: true
  image:
    repository: nvcr.io/nim/nvidia/llama-3.3-nemotron-super-49b-v1.5
    tag: "1.14.1"
  resources:
    limits:
      nvidia.com/gpu: 2
      memory: 128Gi  # Increased from 96Gi
    requests:
      nvidia.com/gpu: 2
      memory: 128Gi  # Increased from 96Gi
  model:
    name: "nvidia/llama-3.3-nemotron-super-49b-v1.5"
  env:
    - name: NIM_MODEL_NAME
      value: "nvidia/llama-3.3-nemotron-super-49b-v1.5"
    - name: NIM_MODEL_PROFILE
      value: "610f006b15f3adbdb072da0b4155d8a772332cf1768fb7389ef92a83c31c26dc"

Also, if needed you can provide a specific node in the cluster to deploy rest of the RAG components

Install RAG using Helm

helm upgrade --install rag -n rag https://helm.ngc.nvidia.com/nvidia/blueprint/charts/nvidia-blueprint-rag-v2.2.0.tgz -f values-2gpu-rtxpro6000.yaml \
--username '$oauthtoken' \
--password "${NGC_API_KEY}" \
--set imagePullSecret.password=$NGC_API_KEY \
--set ngcApiSecret.password=$NGC_API_KEY

Update Helm Dependencies

helm dependency update nvidia-blueprint-rag

At this point RAG has been deployed and configured, the pods for RAG nim-llm-0 will be deployed, these pods will take a few minutes to go into “running” state along with other components and also the RAG server and nim-llm will have services created in Kubernetes with cluster IP. We need to make sure that the RAG server and the NIM LLM have IP addresses that can be reached by the benchmarking RAG tool. For this exercise we used External Loadbalancing IPs for both these services, you can do the same

Edit the NIM-LLM service, RAG server to use Load Balancing IP

kubectl patch service nim-llm -n rag \
  -p '{"spec":{"type":"LoadBalancer","ports":[{"port":8000,"targetPort":8000}]}}'

kubectl patch service rag-server -n rag \
  -p '{"spec":{"type":"LoadBalancer","ports":[{"port":8081,"targetPort":8081}]}}'

At this point the setup should have the RAG pipeline deployed with all its components in Milvus in a standalone fashion.

Deploy and Configure AI-Q Blueprint#

We are deploying the NVIDIA AI-Q blueprint from Github Repo here.

Clone the Git Repo into a folder called aiq

git clone https://github.com/NVIDIA-AI-Blueprints/aiq-research-assistant

Set environment variables, on the system from which you are installing AI-Q

export NGC_API_KEY="nvapi-xxx" # your API key
export TAVILY_API_KEY="yyy" # your Tavily API key, optional for web search

Create a namespace:

kubectl create namespace aira
cd aiq-research-assistant/

Add the AI-Q helm chart

helm repo add nvidia-nim https://helm.ngc.nvidia.com/nim/nvidia/ --username='$oauthtoken' --password=$NGC_API_KEY
helm repo add nim https://helm.ngc.nvidia.com/nim/ --username='$oauthtoken' --password=$NGC_API_KEY

helm repo add nvidia-nim https://helm.ngc.nvidia.com/nim \
  --username='$oauthtoken' \
  --password=$NGC_API_KEY

We need to change the default Helm Values file for our system, copy the values.yaml file in the folder under deploy/helm/aiq-aira to a file called values-rtx-pro-6k.yaml in the deploy/helm folder. We will be editing the file to make some key changes.

Update the nim-llm section to point to the version of the latest version of NIM with the correct profile for RTX PRO 6000 BSE GPUs

nim-llm:
  enabled: true
  service:
    name: "nim-llm"
  image:
      repository: nvcr.io/nim/meta/llama-3.3-70b-instruct
      pullPolicy: IfNotPresent
      tag: "1.12.0"
  resources:
    limits:
      nvidia.com/gpu: 1
    requests:
      nvidia.com/gpu: 1
  model:
    name: "meta/llama-3.3-70b-instruct"
  env:
    - name: NIM_MODEL_NAME
      value: "meta/llama-3.3-70b-instruct"
    - name: NIM_MODEL_PROFILE
      value: "257a3035dbddb2a2cd48f5763ee64f972af4ba0dd5ef92ade1dbb478f6cf5dd3"

Detailed config YAML for AI-Q is provided in Appendix B of this document

Deploy the AI-Q Helm Chart

helm upgrade --install aira -n aiq deploy/helm/aiq-aira -f deploy/helm/aiq-aira/values-rtx-pro-6k.yaml \
--set imagePullSecret.password=$NGC_API_KEY \
--set ngcApiSecret.password=$NGC_API_KEY \
--set tavilyApiSecret.password=$TAVILY_API_KEY

Once AI-Q has been deployed, make sure all the pods are up and running.

We then need to make the frontend and the NIM-llm pod available with an IP so we can reach it from outside the cluster. We also need to expose the Phoenix pod to trace the benchmarking tests as well as the instruct-llm in AI-Q

kubectl patch svc aira-aira-frontend -n aiq -p '{"spec": {"type": "LoadBalancer", "ports": [{"name": "http", "port": 3001, "NodePort": 30001}]}}'

kubectl patch service instruct-llm -n aiq \
  -p '{"spec":{"type":"LoadBalancer","ports":[{"port":8000,"targetPort":8000}]}}'

kubectl patch service aiq-aira-phoenix -n aiq \
  -p '{"spec":{"type":"LoadBalancer","ports":[{"port":60006,"targetPort":6006}]}}'

By now we should have AI-Q deployed and all the services loadbalanced. We are now going to ingest two datasets within the system using NeMo Retriever extraction.

Ingesting Enterprise Data for Research#

RAG and AI-Q blueprint have been configured and deployed, we now need to ingest Enterprise files, data that can be used by AI-Q to conduct deep research. This will be done using NeMo Retriever extraction, an open source library that is part of the RAG pipeline. Individual files can be added to RAG in a single Collection, or multiple collections can be created as well. The AI research assistant demo web application requires two default collections. One collection supports a biomedical research prompt and contains reports on Cystic Fibrosis. The second supports a financial research prompt and contains public financial documents from Alphabet, Meta, and Amazon. To load these default collections, apply the standalone Kubernetes job:

Set the RAG_INGEST_URL environment variable based on your RAG deployment. First we need to port forward the rag ingestor service

kubectl port-forward -n rag service/ingestor-server 8082:8082
export RAG_INGEST_URL="http://localhost:8082"

On the same node where RAG_INGEST_URL was set above, create a Python environment with the correct dependencies:

uv python install 3.12
uv venv --python 3.12 --python-preference managed
uv run pip install -r data/requirements.txt

Copy the Enterprise zip files intended for user research into the current directory for upload. These files will be ingested into the RAG system, enabling AI-Q users to conduct in-depth research.

Note

This folder already has the default Financial and Bio Medical Collection needed for the benchmarking of this folder

cd data
cp files/* .

Run the ingest:

uv run python zip_to_collection.py

By now the RAG and AI-Q have been installed and the data needed for deep research is already embedded and ingested into Milvus.