Advanced Cluster Configurations

Advanced cluster configurations allow you to customize your NVIDIA Run:ai cluster deployment to support your environment. Some settings may be required for deployment, while others can be fine-tuned to align with organizational policies, security requirements, or other operational preferences.

By adjusting these configurations, you can influence system behavior, including functionality, scheduling policies, and resource management, giving you greater control over how the cluster operates. This article provides guidance on configuring and managing these settings so you can adapt your NVIDIA Run:ai cluster to your organization’s needs.

Configuration Scope

The Helm chart provides the complete set of configuration options for the NVIDIA Run:ai cluster. Some options, such as global.affinity, are available only through Helm. The rest of the configurable settings are grouped under clusterConfig. These clusterConfig settings can be applied through Helm as part of your deployment or upgrade process.

The clusterConfig subset can also be managed at runtime through the runaiconfig Custom Resource (under spec). For details, see Modify cluster configurations at runtime.

At runtime, runaiconfig is the source of truth for the active cluster configuration. If a configuration key is defined in both Helm and runaiconfig and the values differ, a Helm upgrade will overwrite the runaiconfig value to match the chart.

Helm Chart Values

The NVIDIA Run:ai cluster installation can be customized to support your environment via Helm values files or Helm install flags. If you are using Guided Installation, set the same keys under cluster.customHelmValues in your config.yaml instead of a Helm values file - see cluster. For example:

# values.yaml
global:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/runai-system
                operator: Exists

Modify Cluster Configurations at Runtime

The clusterConfig subset of settings can also be managed at runtime via the runaiconfig Kubernetes Custom Resource.

1. To edit the cluster configurations, run:

   kubectl edit runaiconfig runai -n runai

2. To see the full runaiconfig object structure, use:

   kubectl get crds/runaiconfigs.run.ai -n runai -o yaml

When using runaiconfig, the clusterConfig values appear under spec.

Configurations

The following configurations allow you to enable or disable features, control permissions, and customize the behavior of your NVIDIA Run:ai cluster

Helm Only Configurations

All Configurations (Helm and runaiconfig)

Third-Party Service Images

The NVIDIA Run:ai cluster bundles Valkey, oauth2-proxy, and git-sync. For what these services do, see Third-Party Services.

NVIDIA Run:ai Services Resource Management

NVIDIA Run:ai cluster includes many different services. To simplify resource management, the configuration structure allows you to configure the containers CPU / memory resources for each service individually or group of services together.

Apply the following configuration in order to change resources request and limit for a group of services:

clusterConfig:
  global:
   <service-group-name>: # schedulingServices | syncServices | workloadServices
     resources:
       limits:
         cpu: 1000m
         memory: 1Gi
       requests:
         cpu: 100m
         memory: 512Mi

Or, apply the following configuration in order to change resources request and limit for each service individually:

clusterConfig:
  <service-name>: # for example: pod-grouper
    resources:
      limits:
        cpu: 1000m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 512Mi

NVIDIA Run:ai Services Replicas

By default, all NVIDIA Run:ai containers are deployed with a single replica. Some services support multiple replicas for redundancy and performance.

To simplify configuring replicas, a global replicas configuration can be set and is applied to all supported services:

clusterConfig:
  global: 
    replicaCount: 1 # default

This can be overwritten for specific services (if supported). Services without the replicas configuration does not support replicas:

clusterConfig:
  <service-name>: # for example: pod-grouper
    replicas: 1 # default

Prometheus

NVIDIA Run:ai uses two separate Prometheus instances:

• The Prometheus instance used for metrics collection and alerting.
• A dedicated Prometheus instance used the NVIDIA Run:ai Scheduler for tracking historical resource usage and enabling time-based fairshare.

Metrics Collection

The Prometheus instance in NVIDIA Run:ai is used for metrics collection and alerting.

The configuration scheme follows the official PrometheusSpec and supports additional custom configurations. The PrometheusSpec schema is available using the spec.prometheus.spec configuration.

A common use case using the PrometheusSpec is for metrics retention. This prevents metrics loss during potential connectivity issues and can be achieved by configuring local temporary metrics retention. For more information, see Prometheus Storage:

clusterConfig:  
  prometheus:
    spec: # PrometheusSpec
      retention: 2h # default 
      retentionSize: 20GB

In addition to the PrometheusSpec schema, some custom NVIDIA Run:ai configurations are also available:

• Additional labels - Set additional labels for NVIDIA Run:ai’s built-in alerts sent by Prometheus.
• Log level configuration - Configure the logLevel setting for the Prometheus container.
• Image override - Use prometheus.spec.image to manually specify the Prometheus image reference. Due to a known issue, the imageRegistry setting in the Prometheus Helm chart is ignored. To pull the image from a different registry, specify the full image reference. Default: quay.io/prometheus/prometheus.
• Image pull secrets - Use prometheus.spec.imagePullSecrets to list Kubernetes image pull secrets in the runai namespace. This is particularly relevant for air-gapped installations where pulling Prometheus images requires authentication. Default: [].
• Advanced metrics (GPU profiling metrics) - Use prometheus.spec.config.advancedMetricsEnabled to activate GPU profiling metrics from NVIDIA DCGM. When enabled, Prometheus collects and aggregates advanced GPU performance data such as SM activity, memory bandwidth, and tensor core utilization. For setup instructions, see GPU profiling metrics.
• OTLP receiver - Use prometheus.config.enableOTLPReceiver to enable or disable the OpenTelemetry Protocol (OTLP) receiver on the NVIDIA Run:ai Prometheus instance. When enabled, Knative Serving can push inference request metrics to Prometheus over OTLP, surfacing them in the NVIDIA Run:ai API and UI. Default: true.

clusterConfig:  
  prometheus:
    logLevel: info # debug | info | warn | error
    additionalAlertLabels:
      - env: prod # example
    config:
      enableOTLPReceiver: true # default: true

Time-Based Fairshare

Time-based fairshare relies on historical resource usage metrics collected by a dedicated Prometheus instance. By default, persistent storage is not configured for this Prometheus instance. As a result, if the Prometheus pod restarts or fails, all historical usage data is lost. In this scenario, the Scheduler recalculates fairshare from scratch and makes scheduling decisions without taking previously collected data into account until new data is accumulated.

To preserve historical usage data across restarts, you can enable persistent storage for the Prometheus instance used for time-based fairshare. It is recommended to use network-based storage that is accessible from all nodes in the cluster.

To enable persistent storage, configure the following cluster settings:

clusterConfig:
  historical-usage-prometheus:
    enablePersistentStorage: true #default false
    storageSize: 50Gi # default 50Gi
    storageClassName: fast-ssd # defaults to the default storage class configured in the cluster  

NVIDIA Run:ai Managed Nodes

To include or exclude specific nodes from running workloads within a cluster managed by NVIDIA Run:ai, use the nodeSelectorTerms flag. For additional details, see Kubernetes nodeSelector.

Label the nodes using the below:

• key - Label key (e.g., zone, instance-type).
• operator - Operator defining the inclusion/exclusion condition (In, NotIn, Exists, DoesNotExist).
• values - List of values for the key when using In or NotIn.

The below example shows how to include NVIDIA GPUs only and exclude all other GPU types in a cluster with mixed nodes, based on product type GPU label:

clusterConfig:   
  global:
     managedNodes:
       inclusionCriteria:
          nodeSelectorTerms:
          - matchExpressions:
            - key: nvidia.com/gpu.product  
              operator: Exists

Custom Certificate Authority for Git and S3

To override the default global CA used by the system and inject a custom CA certificate for Git or S3 data sources, follow these steps:

1. Create a Kubernetes secret with the custom CA certificate:

   kubectl -n runai create secret generic runai-cluster-git-ca \
       --from-file=runai-ca-git.pem=<ca_bundle_path>
   kubectl label secret runai-cluster-git-ca -n runai run.ai/cluster-wide=true run.ai/name=runai-ca-cert --overwrite

2. When installing the cluster, make sure the following flags are added to the helm command. See Create a Tenant and Cluster.

--set global.customCAGit.enabled=true \
--set global.customCAGit.secret.name=<secret-name>