Setup Guide#

This document provides a comprehensive guide for deploying Tokkio Workflow on Google Cloud Platform (GCP). While multiple approaches exist for implementing Tokkio Workflow in the GCP environment, this guide focuses on a specific deployment method utilizing Tokkio’s opinionated deployment scripts. These scripts automate the process of setting up Tokkio Workflow along with its required infrastructure components.

Prerequisites#

GCP Prerequisites#

  1. GCP Console access with admin privileges

  2. Service Account setup with Owner role and respective Key file

  3. Cloud Storage bucket for deployment state

  4. Public DNS zone

Hardware#

Controller instance#

  • Ubuntu 22.04 Operating system

  • Generate an SSH key pair

  • Ensure passwordless sudo access

Access#

  • Access to all the artifacts used during the bringing up of Tokkio Pipeline Application. For example, Tokkio Application Helm chart on NGC.

Infrastructure Layout#

Tokkio Workflow setup on GCP requires several cloud resources to be created such as VPC, Instance Groups, Firewall Rules, Service accounts, Cloud DNS, Cloud load balancing etc., Below picture shows the overall layout of components brought up on GCP.

GCP Deployment Architecture

Note

Many of the resources in this setup may not fall in Free tier, you can check GCP billing reference pages for understanding cost implications.

Installation Steps#

  1. Clone the NVIDIA/ACE.git repository and navigate to the gcp deployment scripts directory

git clone https://github.com/NVIDIA/ACE.git
cd ACE/workflows/tokkio/4.1/scripts/one-click/gcp

  1. Prepare a config file, either by copying base config-template.yml or by copying one of the example config files available under config-template-examples folder.

#list of example templates
config-template-examples/
├── llm-ov-3d-rp-6x-streams
│   ├── config-template.yml
│   └── my-config.env
└── llm-ue-3d-coturn-3x-streams
    ├── config-template.yml
    ├── my-config.env
    └── user-helm-override-values.yml

Copy a config template of your choice as base config template for this installation.

cp config-template-examples/llm-ov-3d-rp-6x-streams/config-template.yml ./my-config.yml

  1. Modify the my-config.yml with your specific settings

vi my-config.yml

  1. Set up environment variables.

  • Similar to config-template file, you can copy an example env file from config-template-examples folder

  • Modify environment variables file with your specific settings

cp config-template-examples/llm-ov-3d-rp-6x-streams/my-config.env my-env-file.env
vi my-env-file.env

  1. Source the environment variables file

source my-env-file.env

  1. Run the installation command

./envbuild.sh install --component all --config-file ./my-config.yml

  1. Capture installation results at the end of logs. Output will look like the sample below.

access_urls:
  api_endpoint: "https://<api_sub_domain>.<base_domain>"
  elasticsearch_endpoint: "https://elastic-<name>.<base_domain>"
  grafana_endpoint: "https://grafana-<name>..<base_domain>"
  kibana_endpoint: "https://kibana-<name>..<base_domain>"
  ui_endpoint: "https://<ui_sub_domain>.<base_domain>"
ssh_command:
app:
  bastion: ssh -i /home/my-user/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null <username>@<bastion-instance-ip-address>
  master: ssh -i /home/my-user/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -i /home/my-user/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -W %h:%p <username>@<bastion-instance-ip-address>" <username>@<app-instance-ip-address>
turn:
  master: ssh -i /home/my-user/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null <username>@<turn-instance-ip-address>

  1. Verify installation

  • Check pod status: kubectl get pods \-n \<application-namespace\>

  • Wait for all pods to reach Ready status (may take up to 60 minutes)

  • Once all the pods are healthy you can access the UI using the URL from installation output access_urls.ui_endpoint

  • For the first time, the browser should prompt permissions such as Mic, Speaker or Camera necessary for the UI to operate. Upon accepting the permissions, the UI should load.

Verify Installation

Uninstallation Steps#

Uninstalling Just The Application#

  • Source the correct environment variables file

source my-env-file.env

  • Run the below commands to uninstall just the application components

./envbuild.sh uninstall --component app --config-file ./my-config.yml

  • Clear any persistent-volumes from previous installation by below command

kubectl delete pv -n <application-namespace>

Uninstalling The Whole Setup#

  • Source the correct environment variables file

source my-env-file.env

  • Run the below un-install command

./envbuild.sh uninstall --component all --config-file ./my-config.yml

Caution

This step uninstalls the entire Kubernetes cluster, GCP infrastructure resources that were brought up during the install step. So use this with caution.

Prepare environment variables#

The config template yml file contains several inputs about the infrastructure and application’s needs. For ease of use, some of these are wired to lookup environment variables. For example {{ lookup(‘env’, ‘NGC_CLI_API_KEY’) }} is expanding it to the NGC_CLI_API_KEY environment variable. What this means is, we can set an environment variable for NGC_CLI_API_KEY with its value and OneClick Script can access it automatically.

Prepare a file to hold these environment variables and their values by vi my-env-file.env and populate them with actual values. Example shown below.

cat my-env-file.env
export OPENAI_API_KEY="<replace-with-actual-value>"
export NGC_CLI_API_KEY="<replace-with-actual-value>"
export NVIDIA_API_KEY="<replace-with-actual-value>"
export CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE="<replace-with-absolute-path-where-service-account-key-json-file-stored>"
export CLOUDSDK_CORE_PROJECT="<replace-with-gcp-project-id>"

Use the source command to load these variables into your current shell session. The source command reads and executes commands from the specified file in the current shell environment, making the variables defined in the file available in the shell.

Caution

If you modify your <my-env-file.env> file or start a new shell, you will have to run source <my-env-file.env> again before running ./envbuild.sh command.

Additional Considerations#

Cost#

Many resources in this setup may exceed GCP Free Tier limits. Review the GCP Free Program terms for better understanding. And, use the GCP Pricing Calculator to estimate costs beyond Free Tier limits. Monitor usage through the Cloud Billing console and set up budgets and alerts to avoid unexpected charges.

Security#

The security of Tokkio in production environments is the responsibility of the end users deploying it. When deploying in a production environment, have the security experts review any potential risks and threats; define the trust boundaries, secure the communication channels, integrate AuthN & AuthZ with appropriate access controls, keep the deployment including the containers up to date, and ensure the containers are secure and free of vulnerabilities.

Essential Skills and Background#

Familiarity With Google Cloud Platform (GCP)#

Users should have a basic understanding of Google Cloud Platform, including its core services and billing model. Below are some of the Key areas that Users should be familiar with.

  • Resource Management: Understanding GCP’s resource hierarchy, including organizations, folders, and projects, is crucial for effective cost management and access control.

  • Identity and Access Management (IAM): Familiarity with IAM roles and permissions is essential for securing resources and managing user access.

  • Core Services: Knowledge of fundamental GCP services such as Compute Engine, Cloud Storage, is important.

  • Cloud Console: Familiarity with GCP’s management Graphical interface is required for resource provisioning and management, especially for setting up one-time pre-requisites.

  • Networking: Basic knowledge of Virtual Private Cloud (VPC) and related networking concepts is important.

Familiarity with Command-Line-Interface (CLI)#

  • Basic Commands: Users should be comfortable with basic command-line operations, such as navigating directories, executing scripts, and managing files.

  • Environment Configuration: Understanding how the environment variables and how PATH setup works on Linux will greatly help operating OneClick script.

  • Scripting Basics: Basic scripting knowledge (e.g., shell scripting) is beneficial for understanding how the OneClick script operates and for troubleshooting any issues that may arise.

Familiarity with YAML#

  • YAML Syntax and Structure: YAML is often used for configuration files in cloud-native applications due to its readability and flexibility. The Configuration templates used in deployment script uses YAML format. Users should be familiar with YAML syntax and structure.

Familiarity with Kubernetes eco system#

Tokkio pipeline is a Cloud native application, and uses concepts like Containerization, Kubernetes, helm etc. Users need to be familiar with these to get the best results from using deployment scripts and the app.

  • Kubernetes Basics: Users should have basic understanding of Kubernetes core concepts such as pods, services and deployments

  • kubectl: Familiarity with the kubectl the command line tool used to interact with Kubernetes clusters including querying the status or logs of Running application pods etc.

  • Helm: Understanding Helm package manager for Kubernetes that simplifies application deployment by managing charts (Collections of pre-configured Kubernetes resource definitions). And how to use helm with override values will help configuring the templates appropriately.

On this page