Kubernetes Environment

ACE Agent provides Kubernetes deployment using NVIDIA Unified Cloud Services (UCS) Tools. NVIDIA Unified Cloud Services Tools (UCS Tools) is a low-code framework for developing cloud-native, real-time, and multimodal AI applications. The NVIDIA ACE Agent releases includes the following UCS microservices:

UCF Microservices

Microservice Name	Version	Description
ucf.svc.ace-agent.chat-engine	4.0.0	Chat Engine Microservice
ucf.svc.ace-agent.nlp-server	4.0.0	NLP Server Microservice
ucf.svc.ace-agent.plugin-server	4.0.0	Plugin Server Microservice
ucf.svc.ace-agent.chat-controller	4.0.0	Chat Controller Microservice for Speech Support
ucf.svc.ace-agent.web-app	4.0.0	Sample Web Application

You can easily create your own custom application Helm chart using ACE Agent microservices with UCS applications. The ACE Agent Quick Start package comes with a number of UCS applications for sample bots which can be found in the ./deploy/ucs_apps/ directory.

Prerequisites

Before you start using NVIDIA ACE Agent, it’s assumed that you meet the following prerequisites. The current version of ACE Agent is only supported on NVIDIA data centers.

1. You have access and are logged into NVIDIA GPU Cloud (NGC). You have installed the NGC CLI tool on the local system and you have logged into the NGC container registry. For more details about NGC, refer to the NGC documentation.

2. You have installed UCS tools along with prerequisite setups such as Helm, Kubernetes, GPU Operator, and so on. Refer to UCS tools developer system and deployment system prerequisite sections for detailed instructions. The latest UCS 2.5 tools require Ubuntu 22.04, alternatively you can build the Docker image for executing UCS commands using the Ubuntu 22.04 base image.

3. You have access to an NVIDIA Volta, NVIDIA Turing, NVIDIA Ampere, NVIDIA Ada Lovelace, or an NVIDIA Hopper Architecture-based GPU.

Setup

1. Install the Local Path Provisioner by running the following command if not already done:

   curl https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.23/deploy/local-path-storage.yaml | sed 's/^  name: local-path$/  name: mdx-local-path/g' | microk8s kubectl apply -f -
   

   Note

   If you miss deploying the local path provisioner Helm chart, you will see the following errors during deployment.

   Warning  FailedMount  4m10s (x2 over 4m11s)  kubelet  MountVolume.SetUp failed for volume "workload-cm-volume" : failed to sync configmap cache: timed out waiting for the condition
   Warning  FailedMount  4m10s (x2 over 4m11s)  kubelet  MountVolume.SetUp failed for volume "scripts-cm-volume" : failed to sync configmap cache: timed out waiting for the condition
   Warning  FailedMount  4m10s (x2 over 4m11s)  kubelet  MountVolume.SetUp failed for volume "configs-volume" : failed to sync configmap cache: timed out waiting for the condition
   

2. Setup the mandatory Kubernetes secrets required for deployment.

   export NGC_CLI_API_KEY=...
   
   kubectl create secret docker-registry ngc-docker-reg-secret --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password="${NGC_CLI_API_KEY}"
   
   kubectl create secret generic ngc-api-key-secret --from-literal=NGC_CLI_API_KEY="${NGC_CLI_API_KEY}"
   

3. Download NVIDIA ACE Agent Quick Start Scripts by cloning the GitHub ACE repository.

   git clone git@github.com:NVIDIA/ACE.git
   cd ACE
   

4. Go to the ACE Agent microservices directory.

   cd microservices/ace_agent
   

Deploying a Bot via UCS Application

We will use the stock bot application specs as an example, which is present in the Quick Start package at ./deploy/ucs_apps/speech_bot.

The Stock bot uses the mixtral-8x7b-instruct-v0.1 model deployed via the NVIDIA API Catalog as the main model.

1. Get the API key from the mixtral-8x7b-instruct-v0.1 for trying out the bot and create the Kubernetes secret for passing NVIDIA_API_KEY before deploying the bot Helm chart.

   cat <<EOF | tee custom-env.txt
   NVIDIA_API_KEY="nvapi-XXX"
   EOF
   kubectl create secret generic custom-env-secrets --from-file=ENV=custom-env.txt
   

2. Generate the Helm Chart using tools.

   ucf_app_builder_cli app build deploy/ucs_apps/speech_bot/stock_bot/app.yaml deploy/ucs_apps/speech_bot/stock_bot/app-params.yaml
   

3. Deploy the Helm Chart.

   helm install ace-agent deploy/ucs_apps/speech_bot/stock_bot/ucf-app-speech-bot-3.6.0/
   

4. Wait for all pods to be ready.

   watch kubectl get pods
   

5. Try out the deployed bot using a web frontend application. Get the nodeport for ace-agent-webapp-deployment-service using kubectl get svc and interact with the bot using the URL http://<workstation IP>:<NodePortWebApp>.

   Note

   For accessing the mic on the browser, we need to either convert http to https endpoint by adding SSL validation or update your chrome://flags/ or edge://flags/ to allow http://<Node_IP>:<Webapp_NodePort> as a secure endpoint.

6. Stop the deployment and remove the persistent volumes.

   helm uninstall ace-agent
   kubectl delete pvc --all
   

Building a Custom Helm Chart using ACE Agent Microservices

In this tutorial, we will showcase how you can create a Helm Chart using ACE Agent Microservices for use cases like text-based bot and speech-based bot and extend it for your custom applications. ACE Agent provides Kubernetes support using Unified Cloud Services (UCS). You can use the tutorials for building UCS applications in the UCS documentation as a reference.

We will use:

• the UCS CLI interface, however, you should be able to execute the same steps via UCS Studio.

• various customizations for the UCS application specs, and

• various UCS microservices in this tutorial.

Building a Chatbot UCS Application

In this tutorial, we will use the stock sample bot present at ./samples/stock_bot as a reference bot for building the chatbot UCS application. We already have a UCS application for the stock chat bot at ./deploy/ucs_apps/chat_bot/stock_bot/ and you can use the same as reference during this tutorial.

1. Create the boilerplate UCS application specs.

   ucf_app_builder_cli app create test-app
   

2. Build the UCS application with the Chat Engine microservice only. Update test-app/app.yaml as follows:

   1. We only need the Chat Engine Microservice, however, we might optionally include the Web App Microservice for the UI to interact with the text-based bot. The dependencies we need are:

      dependencies:
      - ucf.svc.ace-agent.chat-engine:3.6.0
      - ucf.svc.ace-agent.web-app:3.6.0
      

   

   1. Update the components section with the Chat Engine and Web App Microservices.

      components:
      - name: chat-engine
      type: ucf.svc.ace-agent.chat-engine
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: webapp
      type: ucf.svc.ace-agent.web-app
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      

   2. Update the connections between the Chat Engine and the Web App Microservices.

      connections:
        webapp/chat-engine: chat-engine/http-api
      

   3. The Stock bot uses the mixtral-8x7b-instruct-v0.1 model deployed via the NVIDIA API Catalog as the main model. Get the API key from the mixtral-8x7b-instruct-v0.1 for trying out the bot and create the Kubernetes secret for passing NVIDIA_API_KEY before deploying the bot Helm chart.

      cat <<EOF | tee custom-env.txt
      NVIDIA_API_KEY="nvapi-XXX"
      EOF
      kubectl create secret generic custom-env-secrets --from-file=ENV=custom-env.txt
      

   4. Retrieve the NGC CLI API key for downloading NGC resources. We have created ngc-api-key-secret in the Prerequisites section. Remove the vault section and update the secrets section with ngc-api-key-secret and custom-env-secrets.

      secrets:
        k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY:
          k8sSecret:
          secretName: ngc-api-key-secret
          key: NGC_CLI_API_KEY
        k8sSecret/custom-env-secrets:
          k8sSecret:
          secretName: custom-env-secrets
          key: ENV
      

   5. Configure ngc-api-key-secret and custom-env-secrets in the chat-engine section.

      - name: chat-engine
        type: ucf.svc.ace-agent.chat-engine
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          custom-env-secrets: k8sSecret/custom-env-secrets
      

   6. Mount the bot to the Chat Engine Microservice. We can do the same by providing the local directory under the chat-engine component in the application specs.

      - name: chat-engine
      type: ucf.svc.ace-agent.chat-engine
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          openai-key-secret: k8sSecret/openai-key-secret/OPENAI_API_KEY
      files:
          config_dir: ../samples/stock_bot/
      

3. Generate the Helm Chart using UCS tools.

   ucf_app_builder_cli app build test-app/app.yaml
   

4. Deploy the generated Helm Chart and wait for all pods to be ready. Interact with the bot using the Web app at http://<NodeIP>:<NodePort>/. You can get NodePort using kubectl get svc.

   helm install test test-app/test-app-0.0.1/
   

5. As we haven’t deployed the Plugin server microservices, queries related to live stock price will not work. Add an ACE Agent Plugin server in the UCS application. Update test-app/app.yaml.

   1. Update the dependencies section with the Plugin Microservices.

      dependencies:
      - ucf.svc.ace-agent.chat-engine:3.6.0
      - ucf.svc.ace-agent.web-app:3.6.0
      - ucf.svc.ace-agent.plugin-server:3.6.0
      

   

   1. Add the plugin-server in components.

      - name: plugin-server
      type: ucf.svc.ace-agent.plugin-server
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      

   2. Mount the plugin configurations by providing the local directory under the plugin-server component.

      - name: plugin-server
      type: ucf.svc.ace-agent.plugin-server
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      files:
          config_dir: ../samples/stock_bot/
      

   3. Add a connection between the Chat Engine and the Plugin microservices.

      connections:
        webapp/chat-engine: chat-engine/http-api
        chat-engine/plugin-server: plugin-server/http-api
      

   4. Specify parameters for the Plugin microservices. You can either update the parameters section in the plugin-server component, or create a new file test-app/params.yaml and update.

      plugin-server:
        pluginConfigPath: "plugin_config.yaml" # relative path of the plugin_config.yaml
      

6. Generate the updated Helm Chart and deploy the application. You should be able to interact with the updated bot using the Web app at http://<NodeIP>:<NodePort>/.

   ucf_app_builder_cli app build test-app/app.yaml test-app/params.yaml
   
   # Uninstall existing deployment
   helm uninstall test
   kubectl delete pvc --all
   
   # Deploy updated helm chart
   helm install test test-app/test-app-0.0.1/
   

   Try all queries supported by the stock sample bot.

7. The stock sample bot doesn’t use any NLP models, but if you are adding NLP models in your custom bots, we need to add an NLP server microservice.

   1. Add the NLP server microservice in the dependencies component.

      dependencies:
      - ucf.svc.ace-agent.chat-engine:3.6.0
      - ucf.svc.ace-agent.web-app:3.6.0
      - ucf.svc.ace-agent.plugin-server:3.6.0
      - ucf.svc.ace-agent.nlp-server:3.6.0
      

   

   1. Update the components.

      - name: nlp-server
      type: ucf.svc.ace-agent.nlp-server
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      

   2. Update the connections. The Chat Engine utilizes the NLP server.

      connections:
        webapp/chat-engine: chat-engine/http-api
        chat-engine/plugin-server: plugin-server/http-api
        chat-engine/nlp-server: nlp-server/api-server
      

   3. We can mount the model configurations by providing the local directory under the nlp-server component.

      - name: nlp-server
      type: ucf.svc.ace-agent.nlp-server
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      files:
          config_dir: ../samples/stock_bot/
      

   4. Update the NLP Server Microservice parameters and GPU device to use in test-app/params.yaml.

      nlp-server:
        ucfVisibleGpus: [0]
        modelConfigPath: "model_config.yaml"
      

8. Generate the updated Helm Chart and deploy the application. You should be able to interact with the updated bot using the Web app at http://<NodeIP>:<NodePort>/.

   # Build updated helm chart
   ucf_app_builder_cli app build test-app/app.yaml test-app/params.yaml
   
   # Uninstall previous deployment
   helm uninstall test
   kubectl delete pvc --all
   
   # Install updated helm chart
   helm install test test-app/test-app-0.0.1/
   

Building Speech Bot UCS Application

In the previous section, we have created a text based UCS application for stock sample bot and will update the UCS application to support speech to speech conversations. We already have a UCS application for the stock speech bot at ./deploy/ucs_apps/speech_bot/stock_bot/ and you can use the same as reference during this tutorial.

1. Add Speech support using the Chat Controller Microservice.

   1. Add the Chat Controller Microservice in the dependencies section.

      dependencies:
      - ucf.svc.ace-agent.chat-engine:3.6.0
      - ucf.svc.ace-agent.web-app:3.6.0
      - ucf.svc.ace-agent.plugin-server:3.6.0
      - ucf.svc.ace-agent.nlp-server:3.6.0
      - ucf.svc.ace-agent.chat-controller:3.6.0
      

   2. Add the chat-controller to the components section.

      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      

   3. Add the Riva Speech Skills microservice for deploying speech models in the dependencies section.

      dependencies:
      - ucf.svc.ace-agent.chat-engine:3.6.0
      - ucf.svc.ace-agent.web-app:3.6.0
      - ucf.svc.ace-agent.plugin-server:3.6.0
      - ucf.svc.riva.speech-skills:2.15.1
      - ucf.svc.ace-agent.nlp-server:3.6.0
      - ucf.svc.ace-agent.chat-controller:3.6.0
      

   

   1. Add the riva-speech to the components section.

      - name: riva-speech
      type: ucf.svc.riva.speech-skills
      parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      

   2. Add the Chat Controller connections to the Riva server and Chat Engine. Update the webapp connection with Chat Controller.

      connections:
        chat-engine/plugin-server: plugin-server/http-api
        chat-engine/nlp-server: nlp-server/api-server
        nlp-server/riva-triton-grpc: riva-speech/triton-grpc
        nlp-server/riva-api: riva-speech/riva-speech-api
        chat-controller/riva: riva-speech/riva-speech-api
        chat-controller/chat-engine: chat-api/http-api
        webapp/chat-controller: chat-controller/grpc-api
      

   3. Update the Riva server parameters with the ASR and TTS models in test-app/params.yaml.

      riva-speech:
        riva:
          visibleGpus: "0"
        modelRepoGenerator:
          ngcModelConfigs:
          triton0:
              models:
              #> description: List of NGC models for deployment
      
              - nvidia/ucs-ms/rmir_asr_parakeet_1-1b_en_us_str_vad:2.15.0
              - nvidia/riva/rmir_tts_radtts_hifigan_en_us_ipa:2.15.0
      

2. Generate the updated Helm Chart and deploy the application. Wait for all pods to be ready, it might take around 40-50 minutes. You should be able to interact with the updated bot using the Speech Web App at http://<NodeIP>:<NodePort>/.

   # Rebuild the helm chart
   ucf_app_builder_cli app build test-app/app.yaml test-app/params.yaml
   
   # Uninstall previous deployment
   helm uninstall test
   kubectl delete pvc --all
   
   # Install updated helm chart
   helm install test test-app/test-app-0.0.1/
   

Note

For accessing the mic on the browser, we need to have the HTTPS endpoint. You can add SSL validation and convert the endpoint to HTTPS, or update your chrome://flags/, or edge://flags/ to allow http://<Node_IP>:<Webapp_NodePort> as a secure endpoint.

1. For production deployment, use NGC resources for providing bot configurations to various ACE Agent Microservices. This will allow you to keep track of versions and don’t have to depend on the host file systems. We will update the chat-engine Microservice, however, you can update the NLP, Plugin, and Chat Controller Microservices similarly.

   1. Remove files from the chat-engine component in test-app/app.yaml.

      - name: chat-engine
        type: ucf.svc.ace-agent.chat-engine
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          openai-key-secret: k8sSecret/openai-key-secret/OPENAI_API_KEY
      

   2. Add the chat-engine parameters in test-app/params.yaml. If files are provided via a component in test-app/app.yaml, the configNgcPath parameter will not be used. If you have multiple bots in NGC and want to deploy a single bot, use botConfigName to point to the required bot_config.yaml.

      chat-engine:
        configNgcPath: <NGC_RESOURCE_PATH>
        botConfigName: bot_config.yaml
      

   3. Rebuild the Helm Chart and redeploy.

Building LangChain Bot UCS Application

In this tutorial, we will showcase how you can create the UCS application for the sample LangChain based bot present at ./samples/ddg_langchain_bot. This tutorial can be used for creating UCS applications for the other sample bots such as ./samples/rag_bot or your own custom bot.

The DuckDuckGo LangChain sample bot uses OpenAI models and we will need an OPENAI_API_KEY. We already have a UCS application for the DuckDuckGo LangChain bot at ./deploy/ucs_apps/speech_bot/ddg_langchain_bot/ and you can use the same as reference during this tutorial.

1. Create the boilerplate UCS application specs.

   ucf_app_builder_cli app create langchain-app
   

2. Update langchain-app/app.yaml as follows:

   1. LangChain Agent will be deployed as part of the Plugin server microservice. We will need a Chat Controller and Riva Speech Skills for speech support. We might optionally include the Web App microservice for the UI to interact with the bot using text or speech. The dependencies we need are:

      dependencies:
      - ucf.svc.riva.speech-skills:2.15.1
      - ucf.svc.ace-agent.plugin-server:3.6.0
      - ucf.svc.ace-agent.chat-controller:3.6.0
      - ucf.svc.ace-agent.web-app:3.6.0
      

   

   1. Update the components section for the microservices in the dependencies section.

      - name: riva-speech
        type: ucf.svc.riva.speech-skills
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: plugin-server
        type: ucf.svc.ace-agent.plugin-server
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: webapp
        type: ucf.svc.ace-agent.web-app
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      

   2. Update the connections between the microservices.

      connections:
        chat-controller/riva: riva-speech/riva-speech-api
        chat-controller/chat-api: plugin-server/http-api
        webapp/chat-controller: chat-controller/grpc-api
      

   3. Configure the OpenAI API key. To create Kubernetes secret, run:

      export OPENAI_API_KEY=...
      kubectl create secret generic openai-key-secret --from-literal=OPENAI_API_KEY=${OPENAI_API_KEY}
      

   4. Set the NGC CLI API key for downloading NGC resources. We have created ngc-api-key-secret in the Prerequisites section. Remove the vault section and update the secrets with ngc-api-key-secret and openai-key-secret.

      secrets:
        k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY:
          k8sSecret:
            secretName: ngc-api-key-secret
            key: NGC_CLI_API_KEY
        k8sSecret/openai-key-secret/OPENAI_API_KEY:
          k8sSecret:
            secretName: openai-key-secret
            key: OPENAI_API_KEY
      

   5. Configure ngc-api-key-secret and openai-key-secret in the plugin-server, chat-controller, and riva-skills section as needed.

      - name: riva-speech
        type: ucf.svc.riva.speech-skills
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      
      - name: plugin-server
        type: ucf.svc.ace-agent.plugin-server
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          openai-key-secret: k8sSecret/openai-key-secret/OPENAI_API_KEY
      
      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      

   6. Mount the bot to the Plugin server and Chat Controller microservices. We can do the same by providing the local directory under the plugin-server and chat-controller components in the application specs.

      - name: plugin-server
        type: ucf.svc.ace-agent.plugin-server
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          openai-key-secret: k8sSecret/openai-key-secret/OPENAI_API_KEY
        files:
          config_dir: ../samples/ddg_langchain_bot
      
      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
        files:
          config_dir: ../samples/ddg_langchain_bot
      

3. Specify the parameters for the microservices. You can either update the parameters section in the component itself, or create langchain-app/app-params.yaml. Update langchain-app/app-params.yaml as follows:

   1. Update the Riva Speech parameters with the ASR and TTS models in langchain-app/params.yaml.

      riva-speech:
        riva:
          visibleGpus: "0"
        modelRepoGenerator:
          ngcModelConfigs:
          triton0:
              models:
              #> description: List of NGC models for deployment
      
              - nvidia/ucs-ms/rmir_asr_parakeet_1-1b_en_us_str_vad:2.15.0
              - nvidia/riva/rmir_tts_radtts_hifigan_en_us_ipa:2.15.0
      

   2. Update the Chat Controller parameters in langchain-app/params.yaml. We will be using the speech_lite pipeline. The Plugin server will expose the APIs with the LangChain endpoint prefix.

      chat-controller:
        pipeline: speech_lite
        speechConfigPath: "speech_config.yaml"
        chatEndpointPrefix: "langchain"
      

   3. Update the Plugin server parameters.

      plugin-server:
        pluginConfigPath: "plugin_config.yaml"
      

4. The LangChain plugin requires additional packages to be installed in the Plugin server container.

   1. Build a custom Dockerfile by copying the requirements from ddg_langchain_bot/plugins/requirements_dev.txt into deploy/docker/dockerfiles/plugin_server.Dockerfile.

      ##############################
      # Install custom dependencies
      ##############################
      RUN pip3 install \
          langchain==0.1.1 \
          langchain-community==0.0.13 \
          langchain-core==0.1.12 \
          duckduckgo-search==5.3.1b1
      

Note

If you see a crash in the Plugin server or an issue with fetching a response from DuckDuckGo, try using a more recent duckduckgo-search version.

1. Build the container and push to the NGC Docker registry.

   # Set required environment variables for docker-compose.yaml
   source deploy/docker/docker_init.sh
   
   # Build custom plugin server docker image
   docker compose -f deploy/docker/docker-compose.yml build plugin-server
   
   # Retag docker image and push to NGC docker registry
   docker tag docker.io/library/plugin-server:4.0.0 <CUSTOM_DOCKER_IMAGE_PATH>:<VERSION>
   
   docker push <CUSTOM_DOCKER_IMAGE_PATH>:<VERSION>
   

If you want to use a different Docker registry, update imagePullSecrets in langchain-app/app.yaml.

1. Override the Plugin server image using langchain-app/params.yaml.

   plugin-server:
     pluginConfigPath: "plugin_config.yaml"
     applicationSpecs:
       deployment:
       containers:
           container:
           image:
               repository: <CUSTOM_DOCKER_IMAGE_PATH>
               tag: <VERSION>
   

1. Generate the updated Helm Chart and deploy the application. You should be able to interact with the updated bot using the Speech Web App at http://<NodeIP>:<NodePort>/.

   # Build helm chart using UCS tools
   ucf_app_builder_cli app build langchain-app/app.yaml langchain-app/params.yaml
   
   # Uninstall previous deployment
   helm uninstall test
   kubectl delete pvc --all
   
   # Deploy generated helm chart
   helm install test langchain-app/langchain-app-0.0.1/
   

Note

For accessing the mic on the browser, we need to have the HTTPS endpoint. You can add SSL validation and convert the endpoint to HTTPS, or update your chrome://flags/, or edge://flags/ to allow http://<Node_IP>:<Webapp_NodePort> as a secure endpoint.

Building Event Interface-Based Bot UCS Application

In this tutorial, we will showcase how you can create the UCS application for event interface based bot and will use the Colang 2.0 sample bot present at ./samples/colang_2_sample_bot/ as an example.

The Colang 2.0 sample bot uses OpenAI models and we will need an OPENAI_API_KEY. We already have a UCS application for the Colang 2.0 sample bot at ./deploy/ucs_apps/speech_bot/colang2_sample_bot/ and you can use the same as reference during this tutorial.

1. Create the boilerplate UCS application specs.

   ucf_app_builder_cli app create event-app
   

2. Update event-app/app.yaml as follows:

   1. Chat Engine will deploy the event interface bot and Redis microservice used for passing events. We will need a Chat Controller and Riva Speech Skills for Speech Support. The dependencies we need are:

      dependencies:
      - ucf.svc.riva.speech-skills:2.15.1
      - ucf.svc.ace-agent.chat-engine:3.6.0
      - ucf.svc.ace-agent.chat-controller:3.6.0
      - ucf.svc.core.redis-timeseries:0.0.17
      

   

   1. Update the components section for the microservices in the dependencies section.

      - name: riva-speech
        type: ucf.svc.riva.speech-skills
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: chat-engine
        type: ucf.svc.ace-agent.chat-engine
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      - name: redis-timeseries
        type: ucf.svc.core.redis-timeseries
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
      

   2. Update the connections between the microservices.

      connections:
        chat-engine/redis: redis-timeseries/redis
        chat-controller/redis: redis-timeseries/redis
        chat-controller/riva: riva-speech/riva-speech-api
        chat-controller/chat-api: chat-engine/http-api
      

   3. Configure the OpenAI API key. To create Kubernetes secret, run:

      export OPENAI_API_KEY=...
      kubectl create secret generic openai-key-secret --from-literal=OPENAI_API_KEY=${OPENAI_API_KEY}
      

   4. Set the NGC CLI API key for downloading NGC resources. We have created ngc-api-key-secret in the Prerequisites section. Remove the vault section and update the secrets with ngc-api-key-secret and openai-key-secret.

      secrets:
        k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY:
          k8sSecret:
            secretName: ngc-api-key-secret
            key: NGC_CLI_API_KEY
        k8sSecret/openai-key-secret/OPENAI_API_KEY:
          k8sSecret:
            secretName: openai-key-secret
            key: OPENAI_API_KEY
      

   5. Configure ngc-api-key-secret and openai-key-secret in the chat-engine, chat-controller, and riva-skills section as needed.

      - name: riva-speech
        type: ucf.svc.riva.speech-skills
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      - name: chat-engine
        type: ucf.svc.ace-agent.chat-engine
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          openai-key-secret: k8sSecret/openai-key-secret/OPENAI_API_KEY
      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
      

   6. Mount the bot to the Chat Engine and Chat Controller microservices. We can do the same by providing the local directory under the chat-engine and chat-controller components in the application specs.

      - name: chat-engine
        type: ucf.svc.ace-agent.chat-engine
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
          openai-key-secret: k8sSecret/openai-key-secret/OPENAI_API_KEY
        files:
          config_dir: ../samples/colang_2_sample_bot
      - name: chat-controller
        type: ucf.svc.ace-agent.chat-controller
        parameters:
          imagePullSecrets:
          - name: ngc-docker-reg-secret
        secrets:
          ngc-api-key-secret: k8sSecret/ngc-api-key-secret/NGC_CLI_API_KEY
        files:
          config_dir: ../samples/colang_2_sample_bot
      

3. Specify the parameters for the microservices. You can either update the parameters section in the component itself, or create event-app/app-params.yaml. Update event-app/app-params.yaml as follows:

   1. Update the Riva Speech parameters with the ASR and TTS models in event-app/params.yaml.

      riva-speech:
        riva:
          visibleGpus: "0"
        modelRepoGenerator:
          ngcModelConfigs:
          triton0:
              models:
              #> description: List of NGC models for deployment
      
              - nvidia/ucs-ms/rmir_asr_parakeet_1-1b_en_us_str_vad:2.15.0
              - nvidia/riva/rmir_tts_radtts_hifigan_en_us_ipa:2.15.0
      

   2. Update the Chat Controller parameters in event-app/app-params.yaml. We will be using the speech_umim pipeline.

      chat-controller:
        pipeline: speech_umim
        speechConfigPath: "speech_config.yaml"
      

   3. Update the chat-engine to use the event interface.

      chat-engine:
        botConfigName: colang_2_sample_bot_config.yml
        interface: event
      

4. Generate the updated Helm Chart and deploy the application.

   ucf_app_builder_cli app build event-app/app.yaml event-app/app-params.yaml
   helm uninstall test
   kubectl delete pvc --all
   helm install test event-app/event-app-0.0.1/
   

5. Interact with the updated bot using the gRPC Server Sample Client.