Skip to navigation
Installation and ConfigurationOperations

Clusters

View as Markdown

This guide explains the procedure to view and manage Clusters.

The Cluster table provides a quick and easy way to see the status of your cluster.

Clusters Table

The Clusters table can be found under Resources in the NVIDIA Run:ai platform.

The clusters table provides a list of the clusters added to NVIDIA Run:ai platform, along with their status.

The clusters table consists of the following columns:

ColumnDescription
ClusterThe name of the cluster
Kubernetes distributionThe flavor of Kubernetes distribution
Kubernetes versionThe version of Kubernetes installed
StatusThe status of the cluster. For more information see the table below. Hover over the information icon for a short description and links to troubleshooting
Last connected

Indicates the most recent time the cluster successfully connected to the control plane.

  • If the cluster is currently connected, the value is displayed as Now.
  • If the cluster is disconnected or has experienced issues, the exact timestamp of the last successful connection is displayed.
Network topologiesThe network topologies associated with the cluster
Creation timeThe timestamp when the cluster was created
URLThe URL that was given to the cluster
NVIDIA Run:ai cluster versionThe NVIDIA Run:ai version installed on the cluster
NVIDIA Run:ai cluster UUIDThe unique ID of the cluster

Cluster Status

StatusDescription
Waiting to connectThe cluster has never been connected.
DisconnectedThere is no communication from the cluster to the Control plane. This may be due to a network issue. See troubleshooting scenarios.
Missing prerequisitesSome prerequisites are missing from the cluster. As a result, some features may be impacted. If the cluster’s NVIDIA GPU Operator version does not meet the minimum required for GPU fractions, this is shown as a separate reason stating that GPU fractions are not supported. See Prerequisites and Features for the required versions, and troubleshooting scenarios below.
Service issuesAt least one of the services is not working properly. You can view the list of nonfunctioning services for more information. See troubleshooting scenarios.
ConnectedThe NVIDIA Run:ai cluster is connected, and all NVIDIA Run:ai services are running.

Network Topologies Associated with the Cluster

Click one of the values in the Network topologies column to view the list of network topologies and their parameters.

ColumnDescription
TopologyThe name of the topology
LabelsThe ordered set of node label keys that define the topology hierarchy
Node poolsThe node pool(s) the network topology is associated with
Created byThe user who created the network topology
Creation timeThe timestamp of when the network topology was created

Customizing the Table View

  • Filter - Click ADD FILTER, select the column to filter by, and enter the filter values
  • Search - Click SEARCH and type the value to search by
  • Sort - Click each column header to sort by
  • Column selection - Click COLUMNS and select the columns to display in the table
  • Download table - Click MORE and then Click Download as CSV. Export to CSV is limited to 20,000 rows.

Adding a New Cluster

Follow the setup and installation instructions below to get the installation instructions to install the NVIDIA Run:ai cluster.

Setup

  1. In the NVIDIA Run:ai UI, go to Resources -> Clusters and click NEW CLUSTER.
  2. Enter a unique name for your cluster.
  3. Optional: Choose the NVIDIA Run:ai cluster version (latest, by default).
  4. Set the cluster location:
    • Same as control plane - Install the NVIDIA Run:ai cluster on the same Kubernetes cluster as the NVIDIA Run:ai control plane.
    • Remote control plane - Install the NVIDIA Run:ai cluster on a different Kubernetes cluster than the NVIDIA Run:ai control plane. Enter the Cluster URL. For more information, see Fully Qualified Domain Name requirement.
  5. Click CONTINUE

Installation Instructions

  1. Follow the installation instructions and run the commands provided on your Kubernetes cluster. For more information, see Install the Cluster.
  2. Click DONE

The cluster is displayed in the table with the status Waiting to connect. Once installation is complete, the cluster status changes to Connected.

Managing Network Topologies

Network topologies optimize placement and accelerate distributed workloads by keeping pods on nodes that are as close to each other as possible in the network. For more details, see Accelerating workloads with network topology-aware scheduling.

To add topologies that represent the cluster’s network:

  1. Select the cluster you want to add a network topology for
  2. In the top action bar, click NETWORK TOPOLOGIES
  3. In the Network Topologies Associated with <Cluster Name> modal, click + NETWORK TOPOLOGY
  4. Enter a unique name for the topology. If the name already exists, you will be requested to enter a different name.
  5. Click + LABEL to add the node label keys that represent the network hierarchy
    • Order labels from farthest (first) to closest (last)
    • Ensure the labels match the corresponding keys on the nodes. For example: cloud.provider.com/topology-block, cloud.provider.com/topology-rack, kubernetes.io/hostname
    • Drag labels to adjust their order if needed
  6. Click SAVE NETWORK TOPOLOGY

After creating the network topology, the administrator must associate it with the relevant node pool(s). See Node pools for more details:

  • If you are using the default node pool (that is, no additional node pools are defined), attach the topology to the default node pool.
  • If different node pools have different topologies, each node pool must be linked to its corresponding topology.
  • If the entire cluster shares the same topology, link the same topology to all node pools.

Removing a Cluster

  1. Select the cluster you want to remove
  2. Click REMOVE
  3. A dialog appears: Make sure to carefully read the message before removing
  4. Click REMOVE to confirm the removal.

Using the API

Go to the Clusters API reference to view the available actions

Using MCP

The NVIDIA Run:ai MCP server exposes Cluster and Infrastructure tools to inspect clusters via natural language:

  • get_cluster_physical_inventory - see a cluster’s GPU nodes, GPU models, and total, allocatable, allocated, and free GPUs, CPU cores, and memory
  • get_cluster_infrastructure_health - see nodes that are NotReady or Unknown, with their conditions, taints, and GPU counts
  • get_cluster_metrics - track a cluster’s GPU and CPU capacity and utilization over time

See AI Agents for setup.

Troubleshooting

Before starting, make sure you have access to the Kubernetes cluster where NVIDIA Run:ai is deployed with the necessary permissions

Troubleshooting Scenarios

Cluster disconnected

Description: When the cluster’s status is ‘disconnected’, there is no communication from the cluster services reaching the NVIDIA Run:ai Platform. This may be due to networking issues or issues with NVIDIA Run:ai services.

Mitigation:

  1. Check NVIDIA Run:ai’s services status:

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to view pods

    3. Copy and paste the following command to verify that NVIDIA Run:ai’s services are running:

      kubectl get pods -n runai | grep -E 'runai-agent|cluster-sync|assets-sync'

    4. If any of the services are not running, see the ‘cluster has service issues’ scenario.

  2. Check the network connection

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to create pods

    3. Copy and paste the following command to create a connectivity check pod:

      kubectl run control-plane-connectivity-check -n runai --image=wbitt/network-multitool --command -- /bin/sh -c 'curl -sSf <control-plane-endpoint> > /dev/null && echo "Connection Successful" || echo "Failed connecting to the Control Plane"'

    4. Replace <control-plane-endpoint> with the URL of the Control Plane in your environment. If the pod fails to connect to the Control Plane, check for potential network policies

  3. Check and modify the network policies

    1. Open your terminal

    2. Copy and paste the following command to check the existence of network policies:

      kubectl get networkpolicies -n runai

    3. Review the policies to ensure that they allow traffic from the NVIDIA Run:ai namespace to the Control Plane. If necessary, update the policies to allow the required traffic

      Example of allowing traffic:

      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-control-plane-traffic
        namespace: runai
      spec:
        podSelector:
          matchLabels:
            app: runai
        policyTypes:
          - Ingress
          - Egress
        egress:
          - to:
              - ipBlock:
                  cidr: <control-plane-ip-range>
            ports:
              - protocol: TCP
                port: <control-plane-port>
        ingress:
          - from:
              - ipBlock:
                  cidr: <control-plane-ip-range>
            ports:
              - protocol: TCP
                port: <control-plane-port>

    4. Check infrastructure-level configurations:

      • Ensure that firewall rules and security groups allow traffic between your Kubernetes cluster and the Control Plane
      • Verify required ports and protocols:
        • Ensure that the necessary ports and protocols for NVIDIA Run:ai’s services are not blocked by any firewalls or security groups

  4. Check NVIDIA Run:ai services logs

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to view logs

    3. Copy and paste the following commands to view the logs of the NVIDIA Run:ai services:

      kubectl logs deployment/runai-agent -n runai
      kubectl logs deployment/cluster-sync -n runai
      kubectl logs deployment/assets-sync -n runai

    4. Try to identify the problem from the logs. If you cannot resolve the issue, continue to the next step.

  5. Diagnosing internal network issues:
    NVIDIA Run:ai operates on Kubernetes, which uses its internal subnet and DNS services for communication between pods and services. If you find connectivity issues in the logs, the problem might be related to Kubernetes’ internal networking.

    To diagnose DNS or connectivity issues, you can start a debugging Pod with networking utilities:

    1. Copy the following command to your terminal, to start a pod with networking tools:

      kubectl run -i --tty netutils --image=dersimn/netutils -- bash

      This command creates an interactive pod (netutils) where you can use networking commands like ping, curl, nslookup, etc., to troubleshoot network issues.

    2. Use this pod to perform network resolution tests and other diagnostics to identify any DNS or connectivity problems within your Kubernetes Cluster.

Cluster disconnected due to a faulty GPU node

Description: A single faulty GPU node can cause the entire cluster to show a Disconnected status in the NVIDIA Run:ai platform, even when all other nodes are healthy. This typically occurs when the node is stuck in GPU validation and the NVIDIA Run:ai system DaemonSets cannot complete initialization on it. This happens because the NVIDIA Run:ai platform evaluates cluster health based on all system DaemonSet pods being available. A single stuck pod causes the DaemonSet to report as unavailable, which cascades to the cluster-level health status.

Cordoning the node is not sufficient. A common first response is to cordon the faulty node using kubectl cordon. However, cordoning only prevents new pods from being scheduled on the node. It does not evict pods that already tolerate the node.kubernetes.io/unschedulable taint. The NVIDIA Run:ai system DaemonSets tolerate this taint by default. As a result, their pods remain running (or stuck) on the cordoned node, and the cluster continues to report as Disconnected.

Mitigation: These steps require cluster-admin permissions to taint nodes. To isolate the faulty node and allow the cluster to recover:

  1. Add a custom taint to the faulty node to prevent new DaemonSet pods from being scheduled on it:

    kubectl taint node <node-name> runai/gpu-fault=true:NoSchedule

  2. Identify the stuck DaemonSet pod on the faulty node:

    kubectl -n runai get pods --field-selector spec.nodeName=<node-name>

  3. Delete the stuck pod. The taint from step 1 prevents the DaemonSet controller from recreating it, which reduces the expected pod count and allows the DaemonSet to report as fully available:

    kubectl -n runai delete pod <pod-name>

  4. Verify the cluster status returns to Connected in the NVIDIA Run:ai platform UI.

  5. After the hardware issue on the node is resolved, remove the taint:

    kubectl taint node <node-name> runai/gpu-fault=true:NoSchedule-

Note

If the node has no workloads worth preserving on its remaining healthy GPUs, a full drain and reboot (kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data) is an alternative approach. This resolves the underlying hardware fault directly and the taint workaround is not needed.

Cluster has service issues

Description: When a cluster’s status is ‘Has service issues`, it means that one or more NVIDIA Run:ai services running in the cluster are not available.

Mitigation:

  1. Verify non-functioning services

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to view the runaiconfig resource

    3. Copy and paste the following command to determine which services are not functioning:

      kubectl get runaiconfig -n runai runai -ojson | jq -r '.status.conditions | map(select(.type == "Available"))'
  2. Check for Kubernetes events

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to view events

    3. Copy and paste the following command to get all Kubernetes events:

      kubectl get events -A
  3. Inspect resource details

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to describe resources

    3. Copy and paste the following command to check the details of the required resource:

      kubectl describe <resource_type> <name>

Cluster is waiting to connect

Description: When the cluster’s status is ‘waiting to connect’, it means that no communication from the cluster services reaches the NVIDIA Run:ai Platform. This may be due to networking issues or issues with NVIDIA Run:ai services.

Mitigation:

  1. Check NVIDIA Run:ai’s services status

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to view pods

    3. Copy and paste the following command to verify that NVIDIA Run:ai’s services are running:

      kubectl get pods -n runai | grep -E 'runai-agent|cluster-sync|assets-sync'

    4. If any of the services are not running, see the ‘cluster has service issues’ scenario.

  2. Check the network connection

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to create pods

    3. Copy and paste the following command to create a connectivity check pod:

      kubectl run control-plane-connectivity-check -n runai --image=wbitt/network-multitool --command -- /bin/sh -c 'curl -sSf <control-plane-endpoint> > /dev/null && echo "Connection Successful" || echo "Failed connecting to the Control Plane"'

    4. Replace <control-plane-endpoint> with the URL of the Control Plane in your environment. If the pod fails to connect to the Control Plane, check for potential network policies:

  3. Check and modify the network policies

    1. Open your terminal

    2. Copy and paste the following command to check the existence of network policies:

      kubectl get networkpolicies -n runai

    3. Review the policies to ensure that they allow traffic from the NVIDIA Run:ai namespace to the Control Plane. If necessary, update the policies to allow the required traffic

    4. Example of allowing traffic:

      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-control-plane-traffic
        namespace: runai
      spec:
        podSelector:
          matchLabels:
            app: runai
        policyTypes:
          - Ingress
          - Egress
        egress:
          - to:
              - ipBlock:
                  cidr: <control-plane-ip-range>
            ports:
              - protocol: TCP
                port: <control-plane-port>
        ingress:
          - from:
              - ipBlock:
                  cidr: <control-plane-ip-range>
            ports:
              - protocol: TCP
                port: <control-plane-port>

    5. Check infrastructure-level configurations:

    6. Ensure that firewall rules and security groups allow traffic between your Kubernetes cluster and the Control Plane

    7. Verify required ports and protocols:

      • Ensure that the necessary ports and protocols for NVIDIA Run:ai’s services are not blocked by any firewalls or security groups
  4. Check NVIDIA Run:ai services logs

    1. Open your terminal

    2. Make sure you have access to the Kubernetes cluster with permissions to view logs

    3. Copy and paste the following commands to view the logs of the NVIDIA Run:ai services:

      kubectl logs deployment/runai-agent -n runai
      kubectl logs deployment/cluster-sync -n runai
      kubectl logs deployment/assets-sync -n runai

    4. Try to identify the problem from the logs. If you cannot resolve the issue, continue to the next step

Cluster is missing prerequisites

Description: When a cluster’s status displays Missing prerequisites, it indicates that at least one of the Mandatory Prerequisites has not been fulfilled. In such cases, NVIDIA Run:ai services may not function properly.

One specific case is GPU fractions readiness: if the cluster’s NVIDIA GPU Operator version does not meet the minimum required version, or the cluster otherwise does not meet the prerequisites for GPU fractions, the status message names the reason and states that GPU fractions are not supported. This does not block the rest of the cluster from functioning; only GPU fractions are affected. To resolve this, update the installation to meet the GPU fractions prerequisites, for example by upgrading the NVIDIA GPU Operator to the required version. See Prerequisites and Features for the required versions.

Mitigation:

If you have ensured that all prerequisites are installed and the status still shows missing prerequisites, follow these steps:

  1. Check the message in the NVIDIA Run:ai platform for further details regarding the missing prerequisites.
  2. Inspect the runai-public ConfigMap:

    1. Open your terminal. In the terminal, type the following command to list all ConfigMaps in the runai namespace:

      kubectl get configmap runai-public -n runai
  3. Describe the ConfigMap

    1. Locate the ConfigMap named runai-public from the list

    2. To view the detailed contents of this ConfigMap, type the following command:

      kubectl describe configmap runai-public -n runai
  4. Find Missing Prerequisites
    1. In the output displayed, look for a section labeled dependencies.required
    2. This section provides detailed information about any missing resources or prerequisites. Review this information to identify what is needed