Cluster Setup & Management

Cloud Functions admins can install the NVIDIA Cluster Agent to enable existing GPU Clusters to act as deployment targets for NVCF functions. The NVIDIA Cluster Agent is a function deployment orchestrator that communicates with the NVCF control plane. This page describes how to do the following:

• Register a cluster with NVCF using the NVIDIA Cluster Agent.

• Configure the cluster by defining GPU instance types, configurations, region, and authorized NCA (NVIDIA Cloud Account) IDs.

• Verify the cluster setup was successful.

After installing the NVIDIA Cluster Agent on a cluster:

• The registered cluster will show as a deployment option in the GET /v2/nvcf/clusterGroups API response, and Cloud Functions deployment menu.

• Any functions under the cluster’s authorized NCA IDs can now deploy on the cluster.

• Access to a Kubernetes cluster including GPU-enabled nodes (“GPU cluster”)

  • The cluster must have a compatible version of Kubernetes.

  • The cluster must have the NVIDIA GPU Operator installed.

    • If your cloud provider does not support the NVIDIA GPU Operator, Manual Instance Configuration is possible, but not recommended due to lack of maintainability.

• Registering the cluster requires kubectl and helm installed.

• The user registering the cluster must have the cluster-admin role privileges in order to install the NVIDIA Cluster Agent Operator (nvca-operator).

• The user registering the cluster must have the Cloud Functions Admin role within their NGC organization.

Supported Kubernetes Versions

See Kubernetes Release for all support versions of Kubernetes.

Considerations

• The NVIDIA Cluster Agent currently only supports caching if the cluster is enabled with StorageClass configurations. If the “Caching Support” capability is enabled, the agent will make a best effort by attempting to detect storage during deployments, and fall back on non-cached workflows.

• All NVIDIA managed clusters support autoscaling functionality fully for all heuristics. However, clusters registered to NVCF via the agent only support autoscaling via the function queue depth heuristic.

Reach the cluster registration page by navigating to Cloud Functions in the NGC product dropdown, and choosing “Settings” on the left hand menu. You must be a Cloud Functions Admin in order to see this page. Choose “Register Cluster” to begin the registration process.

Configuration

See below for descriptions of all cluster configuration options.

Elevating efficiency for rapid instance spin-up, mandating extra storage configuration and caching support attribute in Advanced cluster setup.

By default, the cluster will be authorized to the NCA ID of the current NGC organization being used during cluster configuration. If you choose to share the cluster with other NGC organizations, you will need to retrieve their corresponding NCA IDs. Sharing the cluster will allow other NVCF accounts to deploy cloud functions on it, with no limitations on how many GPUs within the cluster they deploy on.

Advanced Settings

See below for descriptions of all capability options in the “Advanced Settings” section of the cluster configuration. Note that for customer managed clusters (registered via the Cluster Agent) Dynamic GPU Discovery is enabled by default. For NVIDIA internal clusters, Collect Function Logs is also enabled by default.

Caching Support

Enabling caching for models, resources and containers is recommended for optimal performance. You must create StorageClass configurations for caching within your cluster to fully enable “Caching Support” with the Cluster Agent. See examples below:

StorageClass Configurations in GCP

nvcf-sc.yaml

            

            
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: nvcf-sc
provisioner: pd.csi.storage.gke.io
allowVolumeExpansion: true
volumeBindingMode: Immediate
reclaimPolicy: Retain
parameters:
    type: pd-ssd
    csi.storage.k8s.io/fstype: xfs
        

nvcf-cc-sc.yaml

            

            
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: nvcf-cc-sc
provisioner: pd.csi.storage.gke.io
allowVolumeExpansion: true
volumeBindingMode: Immediate
reclaimPolicy: Retain
parameters:
    type: pd-ssd
    csi.storage.k8s.io/fstype: xfs
        

StorageClass Configurations in Azure

nvcf-sc.yaml

            

            
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: nvcf-sc
provisioner: file.csi.azure.com
allowVolumeExpansion: true
volumeBindingMode: Immediate
reclaimPolicy: Retain
parameters:
    skuName: Standard_LRS
    csi.storage.k8s.io/fstype: xfs
        

nvcf-cc-sc.yaml

            

            
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: nvcf-cc-sc
provisioner: file.csi.azure.com
allowVolumeExpansion: true
volumeBindingMode: Immediate
reclaimPolicy: Retain
parameters:
    skuName: Standard_LRS
    csi.storage.k8s.io/fstype: xfs
        

Apply the StorageClass Configurations

Save the StorageClass template to files nvcf-sc.yaml and nvcf-cc-sc.yaml and apply them as:

            

            
kubectl create -f nvcf-sc.yaml
kubectl create -f nvcf-cc-sc.yaml
        

Install the Cluster Agent

After configuring the cluster, an NGC Cluster Key will be generated for authenticating to NGC, and you will be presented with a command snippet for installing the NVIDIA Cluster Agent Operator. Please refer to this command snippet for the most up-to-date installation instructions.

Once the Cluster Agent Operator installation is complete, the operator will automatically install the desired NVIDIA Cluster Agent version and the Status of the cluster in the Cluster Page will become “Ready”.

Afterwards, you will be able to modify the configuration at any time. Cluster name and SSA client ID (only available for NVIDIA internal clusters) are not reconfigurable. Please refer to any additional installation instructions for reconfiguration in the UI. Once the configuration is updated, the Cluster Agent Operator, which polls for changes every 15 minutes, will apply the new configuration.

Verify Cluster Agent Installation via UI

At any time, you can view the clusters you have begun registering, or registered, along with their status, in the Settings page.

• A status of Ready indicates the Cluster Agent has registered the cluster with NVCF successfully.

• A status of Not Ready indicates the registration command has either just been applied and is in progress, or that registration is failing.

In cases when registration is failing, please use the following command for retrieving additional details:

            

            
kubectl get nvcfbackend -n nvca-operator
        

When a cluster is Not Ready, you can resume registration at any time to finish installation.

The “GPU Utilization” column is based on the number of GPUs occupied over the number of GPUs available within the cluster. The “Last Connected” column indicates when the last status update was received from the Cluster Agent to the NVCF control plane.

Verify Cluster Agent Installation via Terminal

Verify the installation was successful via the following command, you should see a “healthy” response, as in this example:

            

            
> kubectl get nvcfbackend -n nvca-operator
NAME AGE VERSION HEALTH
nvcf-trt-mgpu-cluster 3d16h 2.30.4 healthy
        

Monitoring Data

Metrics

The cluster agent and operator emit Prometheus style metrics. The following metrics and labels are available by default.

Cluster maintainers can scrape the available metrics using the following examples of a PodMonitor for NVCA Operator and ServiceMonitor for NVCA for reference:

Sample NVCA Operator PodMonitor

            

            
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
    labels:
        app.kubernetes.io/component: metrics
        app.kubernetes.io/instance: prometheus-agent
        app.kubernetes.io/name: metrics-nvca-operator
        jobLabel: metrics-nvca-operator
        release: prometheus-agent
        prometheus.agent/podmonitor-discover: "true"
    name: metrics-nvca-operator
    namespace: monitoring
spec:
    podMetricsEndpoints:
    - port: http
        scheme: http
        path: /metrics
    jobLabel: jobLabel
    selector:
        matchLabels:
            app.kubernetes.io/name: nvca-operator
    namespaceSelector:
        matchNames:
        - nvca-operator
        

Sample NVCA ServiceMonitor

            

            
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    labels:
        app.kubernetes.io/component: metrics
        app.kubernetes.io/instance: prometheus-agent
        app.kubernetes.io/name: metrics-nvca
        jobLabel: metrics-nvca
        release: prometheus-agent
        prometheus.agent/servicemonitor-discover: "true"
    name: prometheus-agent-nvca
    namespace: monitoring
spec:
    endpoints:
    - port: nvca
    jobLabel: jobLabel
    selector:
        matchLabels:
            app.kubernetes.io/name: nvca
    namespaceSelector:
        matchNames:
        - nvca-system
        

Logs

Both the Cluster Agent and Cluster Agent Operator emit logs locally by default.

Local logs for the NVIDIA Cluster Agent Operator can be obtained via kubectl:

            

            
kubectl logs -l app.kubernetes.io/instance=nvca-operator -n nvca-operator --tail 20
        

Similarly, NVIDIA Cluster Agent logs can be obtained with the following command via kubectl:

            

            
kubectl logs -l  app.kubernetes.io/instance=nvca -n nvca-system --tail 20
        

Tracing

The NVIDIA Cluster Agent provides OpenTelemetry integration for exporting traces and events to compatible collectors. As of agent version 2.0, the only supported collector is Lightstep. See Advanced: NVCA Operator Configuration Options.

Cluster Key Rotation

To regenerate or rotate a cluster’s key, choose the “Regenerate Key” option from the Clusters table in the Settings page. Please refer to this command snippet for the most up-to-date upgrade instructions.

The below are additional configuration options for reference purposes.

NVCA Operator Parameters

NGC Configuration

Node Selector Configuration

OpenTelemetry Configuration

In order to enable manual instance configuration, remove the “Dynamic GPU Discovery” capability.

All fields in the generated example configuration in the UI are required. Start by choosing “Apply Example” to copy over the example configuration, and then modify it to your cluster’s instance specifications.