> This page is for version Release 2.3. > For other versions, use one of these documentation indexes: > - Release 3.0 (default): https://docs.nvidia.com/isaaclab/3.0/llms.txt > - Release 2.3: https://docs.nvidia.com/isaaclab/2.3/llms.txt > - Release 2.2: https://docs.nvidia.com/isaaclab/2.2/llms.txt > - Release 2.1: https://docs.nvidia.com/isaaclab/2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/isaaclab/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/isaaclab/_mcp/server. # Deploying CloudXR Teleoperation on Kubernetes This section explains how to deploy CloudXR Teleoperation for Isaac Lab on a Kubernetes (K8s) cluster. ## System Requirements * **Minimum requirement**: Kubernetes cluster with a node that has at least 1 NVIDIA RTX PRO 6000 / L40 GPU or equivalent * **Recommended requirement**: Kubernetes cluster with a node that has at least 2 RTX PRO 6000 / L40 GPUs or equivalent > **Note** > > If you are using DGX Spark, check [DGX Spark Limitations](https://isaac-sim.github.io/IsaacLab/release/2.3.0/source/setup/installation/index.html#dgx-spark-details-and-limitations) for compatibility. ## Software Dependencies * `kubectl` on your host computer * If you use MicroK8s, you already have `microk8s kubectl` * Otherwise follow the [official kubectl installation guide](https://kubernetes.io/docs/tasks/tools/#kubectl) * `helm` on your host computer * If you use MicroK8s, you already have `microk8s helm` * Otherwise follow the [official Helm installation guide](https://helm.sh/docs/intro/install/) * Access to NGC public registry from your Kubernetes cluster, in particular these container images: * `https://catalog.ngc.nvidia.com/orgs/nvidia/containers/isaac-lab` * `https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cloudxr-runtime` * NVIDIA GPU Operator or equivalent installed in your Kubernetes cluster to expose NVIDIA GPUs * NVIDIA Container Toolkit installed on the nodes of your Kubernetes cluster ## Preparation On your host computer, you should have already configured `kubectl` to access your Kubernetes cluster. To validate, run the following command and verify it returns your nodes correctly: ```bash kubectl get node ``` If you are installing this to your own Kubernetes cluster instead of using the setup described in the [Appendix: Setting Up a Local K8s Cluster with MicroK8s](/source/deployment/cloudxr_teleoperation_cluster#appendix-setting-up-a-local-k8s-cluster-with-microk8s), your role in the K8s cluster should have at least the following RBAC permissions: ```yaml rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["apps"] resources: ["deployments", "replicasets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["services"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] ``` ## Installation > **Note** > > The following steps are verified on a MicroK8s cluster with GPU Operator installed (see configurations in the [Appendix: Setting Up a Local K8s Cluster with MicroK8s](/source/deployment/cloudxr_teleoperation_cluster#appendix-setting-up-a-local-k8s-cluster-with-microk8s)). You can configure your own K8s cluster accordingly if you encounter issues. 1. Download the Helm chart from NGC (get your NGC API key based on the [public guide](https://docs.nvidia.com/ngc/ngc-overview/index.html#generating-api-key)): ```bash helm fetch https://helm.ngc.nvidia.com/nvidia/charts/isaac-lab-teleop-2.3.0.tgz \ --username='$oauthtoken' \ --password= ``` 1. Install and run the CloudXR Teleoperation for Isaac Lab pod in the default namespace, consuming all host GPUs: ```bash helm upgrade --install hello-isaac-teleop isaac-lab-teleop-2.3.0.tgz \ --set fullnameOverride=hello-isaac-teleop \ --set hostNetwork="true" ``` > **Note** > > You can remove the need for host network by creating an external LoadBalancer VIP (e.g., with MetalLB), and setting the environment variable `NV_CXR_ENDPOINT_IP` when deploying the Helm chart: > > ```yaml > # local_values.yml file example: > fullnameOverride: hello-isaac-teleop > streamer: > extraEnvs: > - name: NV_CXR_ENDPOINT_IP > value: "" > - name: ACCEPT_EULA > value: "Y" > ``` > > ```bash > # command > helm upgrade --install --values local_values.yml \ > hello-isaac-teleop isaac-lab-teleop-2.3.0.tgz > ``` 1. Verify the deployment is completed: ```bash kubectl wait --for=condition=available --timeout=300s \ deployment/hello-isaac-teleop ``` After the pod is running, it might take approximately 5-8 minutes to complete loading assets and start streaming. ## Uninstallation You can uninstall by simply running: ```bash helm uninstall hello-isaac-teleop ``` ## Appendix: Setting Up a Local K8s Cluster with MicroK8s Your local workstation should have the NVIDIA Container Toolkit and its dependencies installed. Otherwise, the following setup will not work. ### Cleaning Up Existing Installations (Optional) ```bash # Clean up the system to ensure we start fresh sudo snap remove microk8s sudo snap remove helm sudo apt-get remove docker-ce docker-ce-cli containerd.io # If you have snap docker installed, remove it as well sudo snap remove docker ``` ### Installing MicroK8s ```bash sudo snap install microk8s --classic ``` ### Installing NVIDIA GPU Operator ```bash microk8s helm repo add nvidia https://helm.ngc.nvidia.com/nvidia microk8s helm repo update microk8s helm install gpu-operator \ -n gpu-operator \ --create-namespace nvidia/gpu-operator \ --set toolkit.env[0].name=CONTAINERD_CONFIG \ --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd-template.toml \ --set toolkit.env[1].name=CONTAINERD_SOCKET \ --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \ --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \ --set toolkit.env[2].value=nvidia \ --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \ --set-string toolkit.env[3].value=true ``` > **Note** > > If you have configured the GPU operator to use volume mounts for `DEVICE_LIST_STRATEGY` on the device plugin and disabled `ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED` on the toolkit, this configuration is currently unsupported, as there is no method to ensure the assigned GPU resource is consistently shared between containers of the same pod. ### Verifying Installation Run the following command to verify that all pods are running correctly: ```bash microk8s kubectl get pods -n gpu-operator ``` You should see output similar to: ```text NAMESPACE NAME READY STATUS RESTARTS AGE gpu-operator gpu-operator-node-feature-discovery-gc-76dc6664b8-npkdg 1/1 Running 0 77m gpu-operator gpu-operator-node-feature-discovery-master-7d6b448f6d-76fqj 1/1 Running 0 77m gpu-operator gpu-operator-node-feature-discovery-worker-8wr4n 1/1 Running 0 77m gpu-operator gpu-operator-86656466d6-wjqf4 1/1 Running 0 77m gpu-operator nvidia-container-toolkit-daemonset-qffh6 1/1 Running 0 77m gpu-operator nvidia-dcgm-exporter-vcxsf 1/1 Running 0 77m gpu-operator nvidia-cuda-validator-x9qn4 0/1 Completed 0 76m gpu-operator nvidia-device-plugin-daemonset-t4j4k 1/1 Running 0 77m gpu-operator gpu-feature-discovery-8dms9 1/1 Running 0 77m gpu-operator nvidia-operator-validator-gjs9m 1/1 Running 0 77m ``` Once all pods are running, you can proceed to the [Installation](/source/deployment/cloudxr_teleoperation_cluster#installation) section.