What's New in Version 3.0

The NVIDIA Run:ai v3.0 what’s new provides a detailed summary of the latest features, enhancements, and updates introduced in this version. It serves as a guide to help you understand the new capabilities and how to leverage them for improved workload management, resource optimization, and more.

Feature Life Cycle

NVIDIA Run:ai uses life cycle labels to indicate the maturity and stability of features across releases:

• Experimental - This feature is in early development. It may not be stable and could be removed or changed significantly in future versions. Use with caution.
• Beta - This feature is still being developed for official release in a future version and may have some limitations. Use with caution.
• Legacy - This feature is scheduled to be removed in future versions. We recommend using alternatives if available. Use only if necessary.

AI Workloads and Applications

Workloads and Scheduling

• Semi-preemptible workloads - Added a third preemptibility mode alongside preemptible and non-preemptible workloads. Semi-preemptible mode allows workloads to guarantee a minimum number of replicas while using preemptible capacity for any replicas above that minimum, balancing workload availability with more efficient use of spare capacity. Supported for native and supported workload types via workload submission or workload policies. Refer to Workload Priority and Preemption for more details. From cluster v3.0 onward
• Extended-resource translation to DRA - The NVIDIA Run:ai Scheduler can now translate extended-resource requests, such as nvidia.com/gpu: 2, into Kubernetes Dynamic Resource Allocation (DRA) ResourceClaim requests, so workloads that request extended resources and workloads that submit DRA claims can run in the same cluster. Translation requires the NVIDIA DRA driver and a DeviceClass for the extended resource name. Refer to Extended-Resource Translation for more details. From cluster v3.0 onward
• Segment-based topology placement - For distributed training workloads that the framework divides into fixed-size segments, you can now request topology constraints per segment so the workers within each segment are co-located within the same topology domain. The framework handles the segmentation itself; NVIDIA Run:ai adds per-segment topology constraints and applies them automatically. This is useful for workloads where groups of workers communicate more frequently within the group than across groups, such as tensor-parallel training jobs that benefit from tighter locality. By default, a preferred topology placement is applied to each segment. Native workload support is available for PyTorch, while supported workload types include PyTorchJob and LeaderWorkerSet (LWS). Refer to Segment-Based Topology Placement for more details. From cluster v3.0 onward
• Per-workload stale gang eviction timeout - You can now set, per workload, how long a gang-scheduled workload, such as distributed training or distributed inference, can remain below its gang-scheduling minimum before the Scheduler evicts it. Set the kai.scheduler/staleness-grace-period annotation to override the cluster-wide default for a specific workload, or set a negative value to disable stale gang eviction for it. Refer to Stale Gang Eviction for more details. From cluster v3.0 onward
• Background workloads - Infrastructure workloads such as health checks, diagnostics, and monitoring agents can now run as background workloads. The Scheduler treats them as invisible when planning resources, places AI workloads as though they were not running, and evicts them on demand when their resources are needed. Background workloads are excluded from quota and fairshare calculations, so they never trigger over-quota usage or resource reclaim. Refer to Background Workloads for more details. From cluster v3.0 onward
• GPU fractions on NVIDIA CUDA - GPU memory fractions and dynamic GPU memory fractions are now powered by the standard NVIDIA CUDA implementation, replacing the previous proprietary GPU sharing engine. This makes the capability part of the standard NVIDIA stack, with broader compatibility and long-term maintenance by NVIDIA. Existing GPU fraction workloads require additional migration steps when upgrading to v3.0. See Upgrade to 3.0 for upgrade considerations. From cluster v3.0 onward
  • GPU compute modes - Workloads using GPU memory fractions can use either time-slicing or the new sm-sharing mode. Time-slicing remains the default, while sm-sharing allows workloads to run concurrently on separate streaming multiprocessors. Both modes are supported with GPU memory fractions and dynamic GPU memory fractions. Refer to GPU Compute Modes for more details.
  • GPU memory fractions - GPU memory fractions and dynamic GPU memory fractions continue to support static and burstable GPU memory allocation with the CUDA-based implementation. Refer to GPU Memory Fractions and Dynamic GPU Memory Fractions for more details.
  • Missing prerequisites indication - The Clusters and Nodes pages show a Missing prerequisites status when the prerequisites for GPU fractions are not met, such as a GPU Operator version below the minimum on a cluster or an unsupported CUDA driver version on a node.

AI Applications

• Upload a local Helm chart - Uploading a packaged Helm chart (.tgz, up to 10 MB) directly from your local machine is now supported when creating an AI application, in addition to using an NGC catalog entry or custom URL. Local chart upload is available through the UI and API. Refer to AI Applications for more details. From cluster v2.25 onward
• AI application structure diagram - The AI application details view now includes a structure diagram that shows how workloads connect to the Kubernetes resources they use, including Secrets, PersistentVolumeClaims, and ConfigMaps. The diagram supports filtering and search to help locate specific workloads or resources in more complex applications. Refer to AI Applications for more details. From cluster v2.23 onward

Inference

• vLLM metrics for LLM inference workloads - NVIDIA Run:ai exposes vLLM metrics for inference workloads running vLLM, including request concurrency, token throughput, latency percentiles, and GPU KV-cache utilization, matching the existing NVIDIA NIM metrics pattern. Refer to Workloads for more details. From cluster v3.0 onward

Agent Skills and MCP Server

• Agent Skills - NVIDIA Run:ai now provides Agent Skills, pre-built and evaluated workflows that teach AI agents how to perform common Run:ai operational tasks using the NVIDIA Run:ai MCP server. Version 3.0 includes two skills: runai-diagnose-failed-workload, which diagnoses failed, crashed, or CrashLoopBackOff workloads, and runai-audit-assigned-quota, which audits GPU quota fairness across a cluster and identifies reclaimable GPUs per node pool. Refer to Agent Skills for more details. From cluster v2.25 onward
• MCP Server - The NVIDIA Run:ai MCP server expands from 16 to 27 tools, including six new write tools that let AI agents perform actions in Run:ai in addition to retrieving and analyzing platform information. New write capabilities include tools such as manage_credentials, submit_workload, and manage_node_pool. The server works with MCP-compatible clients such as Claude Code, Cursor, VS Code/GitHub Copilot, and Codex. Refer to Tools Reference for the full list. From cluster v2.25 onward

Platform Management

Node Pools and Nodes

• NUMA-aware scheduling visibility - NUMA-aware scheduling prerequisites and per-NUMA-node resource information are surfaced directly in the Nodes and Node Pools views in the UI and are also available through the API. This includes the Topology Manager policy and total and allocated GPUs, CPU cores, and CPU memory by NUMA node, helping verify NUMA-aware scheduling configuration and troubleshoot placement issues caused by NUMA fragmentation. Nodes and node pools show a Missing Prerequisites status when the required topology information is unavailable or invalid. Refer to Configuring NUMA-Aware Scheduling for more details. From cluster v3.0 onward
• GPU device placement strategy - The Node pools table includes a GPU device placement strategy column that shows the Bin-pack or Spread strategy applied at the GPU device level, separate from the existing node-level placement strategy. Refer to Node Pools for more details.

Policies

• Multi-scope policies for supported workload types - Policies for supported workload types can be applied at tenant, cluster, department, and project scopes. For rules defined at tenant, cluster, or department scope, you can control whether a policy at a narrower scope is allowed to override the rule. Overrides are disabled by default. When enabled, a narrower-scope policy can define a rule for the same field and that rule takes effect instead. The policy view also shows the source scope for each effective rule. Refer to Supported Workload Types Policies for more details. From cluster v2.26 onward

Authentication and Authorization

• Access rules for all authenticated users and service accounts - Access rules can now use a built-in subject type that applies to all authenticated users and service accounts. This makes it possible to define default access that automatically applies to new users and service accounts at the tenant, department, or project scope. Refer to Access Rules for more details. From cluster v3.0 onward
• Granular asset permission sets - New permission sets provide more granular read and edit access to workload templates, environments, compute resources, and credentials. This makes it possible to create custom roles with access to specific asset types without granting the broader asset-management permissions. Refer to Permissions Reference for more details.

General Enhancements

• Bulk report deletion - You can now select and delete multiple reports from the Reports table in a single action. A confirmation dialog shows how many reports will be deleted, and if part of the operation fails, NVIDIA Run:ai reports the first failed report and the total number of failures. CSV download remains a single-report action. Refer to Reports for more details.
• Email notification recipient verification - The Email Notifications verification flow in General Settings now includes an optional recipient email address. This lets you verify that the configured SMTP server accepts mail for a specific recipient without sending an actual email. Refer to Notifications for more details.

Installation and Configuration

Installation

• Guided installation - The new runai-installer CLI provides a streamlined, end-to-end installation experience for new deployments. A single declarative config.yaml file defines what to install and how, while the installer handles validation, preflight checks, prerequisites, features, networking, and deployment of the control plane and cluster components. This reduces the number of separate installation steps and configuration points customers need to manage. Manual Helm installation remains supported. Refer to Guided Installation for more details. From cluster v3.0 onward
  • Single declarative configuration - The installation is built from the runai-installer binary and a single config.yaml file that serves as the input for the entire installation. The installer persists installation state so interrupted installations can be resumed, and it can be rerun with an updated configuration to add prerequisites or features that were not installed initially.
  • Built-in validation and deployment - Before applying changes, the installer validates the configuration and runs preflight checks. It then configures networking and registry access, installs the required prerequisites and selected features, and deploys the control plane and cluster components.
  • Configuration templates - Ready-to-use configuration templates for common deployment scenarios help customers get started without building the configuration from scratch. Templates are available for full installations and common training, inference, Gateway API, and OpenShift scenarios.
  • Deployment options - Guided installation supports connected and air-gapped environments, same-cluster and multi-cluster deployments, and all supported Kubernetes distributions. It is the recommended installation method for both Open Source and Support-Entitled Release installations.
  • Upgrade support - runai-installer is currently intended for new deployments; upgrades are not yet supported.
• Support-Entitled Release installation - Version 3.0 introduces separate Open Source and Support-Entitled release artifacts. Customers with NVIDIA support entitlements install the Support-Entitled Release, which identifies the installation as entitled and enables access to the corresponding release artifacts and support-related capabilities. With Guided Installation, use the --support-entitled flag. With Manual Helm, add supportEntitled=true and global.imagePullSecrets[0].name, which references the image pull secret that holds your NGC credentials, when installing the control plane. Refer to Install the Control Plane for the full commands. From cluster v3.0 onward
• NVIDIA Run:ai telemetry - NVIDIA Run:ai v3.0 introduces telemetry for Support-Entitled deployments, reporting initial metrics including platform usage and adoption signals, GPU utilization and allocation, and cluster health signals to support proactive troubleshooting and continued product improvement. By default, telemetry is not associated with the customer organization. Customers can opt in to identified telemetry by providing an NGC Service API Key during installation, which associates the telemetry with the customer’s NCA ID. Refer to Installation and Configuration for your installation path. From cluster v3.0 onward
• Third-party service images from upstream registries - Third-party service images are sourced from their official upstream registries for both Open Source and Support-Entitled installations. Select third-party services can also be replaced with externally managed instances, allowing customers to use services they already operate outside the NVIDIA Run:ai installation. Refer to Third-Party Services for supported services, image sources, and configuration options. From cluster v3.0 onward
• Gov-ready installation - NVIDIA Run:ai, including its dependencies, now supports Gov-ready deployments, the NVIDIA standard that makes sure NVIDIA software complies with the federal and governmental security and compliance requirements that regulated customers must meet. A Gov-ready installation follows the standard installation procedures, with FIPS-validated hardened images for the third-party services, an external PostgreSQL instance, the Gov-ready versions of the GPU Operator and Network Operator, and FIPS mode enabled for the control plane and the cluster. Gov-ready requires a new installation. The vulnerability remediation SLA and support for Gov-ready issues apply to NVIDIA Run:ai Support-Entitled Release customers. Refer to Gov-Ready Installation for the steps and to Gov-Ready Third-Party Services for the images and Helm values. From cluster v3.0 onward
• New air-gapped image workflow - Air-gapped installations now use a single bundle containing all required NVIDIA Run:ai platform and third-party service images. You create the bundle on a connected machine with runai-installer airgap pack and push it to your internal mirror registry with airgap load. This workflow applies to both Guided Installation and Manual Helm. Refer to Installation and Configuration for your installation path. From cluster v3.0 onward

Cluster Configuration

• Unified controller for all workload types - The workload-controller and inference-workload-controller are consolidated into a single anyworkload-controller in v3.0. Settings that were previously configured separately by workload type are now configured under anyworkload-controller. Refer to Cluster Configuration for more details. Running and pending native workloads submitted in v2.26 and below using these controllers are affected by the upgrade and lose some capabilities until resubmitted. See Upgrade to 3.0 for more details. From cluster v3.0 onward
• OpenShift Route TLS passthrough - On OpenShift, NVIDIA Run:ai exposes workload URLs through Route objects. By default, Routes use edge termination, where TLS terminates at the OpenShift Router and traffic to the workload pod is plain HTTP. Version 3.0 adds a passthrough termination option, which terminates TLS inside the pod and preserves end-to-end encryption from the client to the workload. The setting applies cluster-wide to all workload Routes created by NVIDIA Run:ai. Refer to External Access to Containers for configuration requirements and impact on existing workloads. From cluster v3.0 onward
• Bootstrap token for cluster installation - The cluster installation command generated by the NVIDIA Run:ai UI now authenticates the cluster with the control plane using --set controlPlane.bootstrapToken=... instead of --set controlPlane.clientSecret=..., which earlier versions used. The bootstrap token is short-lived and expires 2 hours after generation, so generate a new command from the cluster wizard if the installation is not completed in that window. Control planes at v3.0 and later no longer return the clientSecret. If both values are set, the bootstrap token takes precedence. Refer to Install the Cluster for the updated command. From cluster v3.0 onward

Upgrade and Compatibility

Upgrade to 3.0

Upgrading to v3.0 requires administrators to perform additional steps beyond the base upgrade procedure. See Plan the Upgrade for the specific changes to plan for, and Upgrade NVIDIA Run:ai for the base upgrade procedure.

• Upstream third-party images - NVIDIA Run:ai v3.0 changes how third-party service images are sourced. For connected environments, third-party service images are now pulled directly from their official upstream registries, including Docker Hub, Quay, and registry.k8s.io. Before upgrading, ensure that the required registries are accessible from the cluster and back up your NVIDIA Run:ai configurations and third-party services. See Back Up Before Upgrading and Third-Party Services for more details.
• NVIDIA CUDA-based GPU fractions - GPU fractions remain supported in v3.0, now powered by NVIDIA CUDA instead of the previous proprietary engine. This makes the capability part of the standard NVIDIA stack, with broader compatibility and long-term maintenance by NVIDIA. Native workload types use the same UI, API, and CLI fields as before; supported workload types require a manifest change only if their manifest sets a fraction limit. See Migrate GPU Fractions for prerequisites and the full procedure.
• Unified controller for all workload types - The controllers and CRDs behind native workload types (workspace, training, distributed training, standard inference, distributed inference) are removed in v3.0 and replaced by a single, unified controller and CRD. Running and pending native workloads submitted before upgrading through a legacy workload type - workspace, training, distributed training, standard inference, or distributed inference - are affected by the upgrade and lose some capabilities until resubmitted. See Migrate Legacy Workloads for the affected capabilities and the resubmission procedure.
• New flags for Support-Entitled Manual Helm upgrades - When upgrading the control plane with Manual Helm, add two new flags to the helm upgrade command: supportEntitled=true and global.imagePullSecrets[0].name. See Upgrade NVIDIA Run:ai for the full commands.
• DRA driver - Upgrading the GPU Operator does not install or change the NVIDIA DRA driver, so keep a separately installed DRA driver. GPU Operator 26.7 can deploy the DRA driver itself in the new GPUCluster mode, for new installations only. See Plan the Upgrade for details.
• GPU memory swap disabled - Upgrading to v3.0 forces GPU memory swap and the node-level scheduler off, even if they were previously enabled, and neither can be re-enabled afterward. The node-level scheduler was deprecated in v2.25 and GPU memory swap in v2.26. Snapshot, the planned replacement capability, is targeted for v3.1 and is not available in v3.0. See Plan the Upgrade for what to check before upgrading.
• New air-gapped upgrade workflow - Air-gapped upgrades in v3.0 use the new image-bundle workflow. runai-installer airgap pack creates a single bundle on a connected machine containing the NVIDIA Run:ai platform and third-party service images required for the target release. After the bundle is transferred into the air-gapped environment, airgap load pushes the images to the internal mirror registry and generates the Helm values required for the upgrade. See Upgrade NVIDIA Run:ai for the full upgrade procedure.

Support Matrix

NVIDIA Run:ai v3.0 updates supported versions across Kubernetes distributions, NVIDIA operators, and framework operators. Refer to Support Matrix for the complete list of supported versions.

• NVIDIA GPU Operator 26.7.1 - The maximum supported version is now 26.7.1. With GPU Operator 26.7, the GPU Operator can deploy the DRA driver in the new GPUCluster mode, for new installations only.
• NVIDIA NIM Operator 3.1.x - Version 3.1.x is now supported.
• NVIDIA Dynamo Operator 1.5.x - Version 1.5.x is now supported.
• Prometheus / Kube-Prometheus Stack 3.13.1 / 88.6.5 - The minimum supported versions are now 3.13.1 and 88.6.5.
• Knative Serving 1.23 - The maximum supported version is now 1.23.
• Kubeflow Training Operator 1.9.4 - Version 1.9.4 is now supported.
• MPI Operator 0.8.2 - The maximum supported version is now 0.8.2.
• LeaderWorkerSet (LWS) 0.10.0 - The maximum supported version is now 0.10.0.
• Upstream Kubernetes 1.35–1.37 - The supported range is now Kubernetes 1.35 through 1.37. The same range applies to supported managed Kubernetes distributions that track upstream Kubernetes. Kubernetes 1.34 has reached end of life (EOL).
• Red Hat OpenShift 4.20–4.22 - The supported range is now OpenShift 4.20 through 4.22. OpenShift 4.18 has reached end of life (EOL).

Documentation Changes

The Installation and Configuration documentation has been restructured to consolidate setup content that was previously spread across Getting Started, Advanced Setup, and Infrastructure Procedures.

• Unified Installation and Configuration section - Installation content that previously lived under Getting Started, configuration content that was under Advanced Setup, and ongoing operational procedures that were under Infrastructure Procedures are now consolidated under a single Installation and Configuration section.
• System Requirements is now installation-path independent - With v3.0 supporting both Guided Installation and Manual Helm, preparation of the Kubernetes cluster and surrounding infrastructure is no longer documented inside the Helm installation flow. The planning content is now split into focused pages:
  • Deployment Model - Same-cluster and multi-cluster deployment planning.
  • Infrastructure Requirements - Hardware, operating system, Kubernetes distribution, storage, permissions, and other infrastructure requirements that were previously spread across control plane and cluster system requirements.
  • Networking Setup - Load balancer, ingress, FQDNs, DNS, TLS certificates, and related networking preparation that previously appeared in cluster system requirements.
  • Third-Party Services - Services deployed with NVIDIA Run:ai and options for using externally managed instances.
  • External Access to Critical Services - The former Network Requirements content has been reorganized under this page. It covers the external connectivity NVIDIA Run:ai requires, including access to NGC, upstream registries, control plane and cluster communication, and other external services.
• Manual Helm installation flow reorganized - The Manual Helm installation now starts with a single Preparations page that brings together setup tasks that were previously spread across preparation, control plane system requirements, and cluster system requirements. This includes NVIDIA Run:ai registry credentials, TLS certificates, namespaces, and other installation prerequisites. The flow then continues to Prerequisites and Features, followed by the control plane and cluster installation procedures. Upgrade content has been moved out of the Helm installation flow into a separate Upgrade section.
• Prerequisites and Features are now separate from System Requirements - Workload- and environment-specific components that previously appeared under cluster system requirements are now documented under Prerequisites and Features, including the GPU Operator, Prometheus, Network Operator, and components required for supported workload types.
• Aligned to the Day 0, Day 1, and Day 2 lifecycle - The new structure follows the installation lifecycle:
  • Day 0: Planning - Plan the deployment and prepare the Kubernetes cluster and surrounding infrastructure required before installing NVIDIA Run:ai.
  • Day 1: Setup - Install NVIDIA Run:ai using Guided Installation or Manual Helm, then configure the platform for the environment.
  • Day 2: Operations - Perform ongoing operational tasks such as cluster management, scaling, high availability, node maintenance, backup and restore, monitoring, and logging.