NVIDIA Mission Control 2.5.0 Release Notes#

Overview#

NVIDIA Mission Control 2.5.0 adds early access support for NVIDIA Vera Rubin NVL72, updates software components, and introduces scoped beta components. Early access support for Vera Rubin NVL72 is limited to essential bring-up support with NVIDIA Base Command Manager. NVIDIA Run:ai and Autonomous Resiliency Engine are not supported on Vera Rubin NVL72 in this release. Previously supported systems continue to support their existing NVIDIA Mission Control capabilities.

Refer to NVIDIA Mission Control 2.5.0 Software Components Enabled for component availability and platform support.

Other notable features of NVIDIA Mission Control 2.5.0 are Autonomous Resiliency Engine support for Kubernetes workloads with NVIDIA Run:ai, update of NVIDIA Base Command Manager to 11.34.0 with support for Vera Rubin NVL72 bring up, and bug fixes for software components across the Mission Control software stack. Refer to the revised Software Bill of Materials (SBOM) for specific component versions.

Beta Software Components:

NVIDIA Minerva and NVIDIA Dynamic Power Software are available with NVIDIA Mission Control 2.5.0 as pre-release beta software components. They are not included in this release, and are available through a scoped engagement only. Contact your NVIDIA Account Manager to participate.

Feature Updates#

Workload Management#

Update to NVIDIA Run:ai 2.26#

Highlights of the updates for Run:ai 2.26 (self-hosted). Refer to the NVIDIA Run:ai documentation for full details, including screenshots and configuration steps.

NVIDIA Run:ai MCP Server#

The NVIDIA Run:ai MCP Server exposes cluster and workload data to AI coding assistants (Claude Code, Cursor, VS Code with GitHub Copilot) via the Model Context Protocol. AI practitioners can ask why a workload is stuck or failing; platform admins can check GPU availability, inspect utilization across projects, or diagnose node issues. Read-only in this release; supports local (per-user Docker container) and remote (admin-deployed shared server) modes. (cluster v2.25+)

AI Practitioners — Workloads#

  • Enhanced workload structure view now displays workload hierarchies in a full-screen expanded view, with per-element inspection of metrics, pods, logs, terminal access, and live Kubernetes YAML (spec, status, and metadata). Connection endpoints are surfaced directly within the structure view, and the workload Details experience extends to supported workload types. (cluster v2.26+)

  • The Workloads API now returns the actual topology placement of each workload element for multi-component workloads (for example, Dynamo over Grove), and returns workload-level placement for any distributed workload whose node pool is attached to a topology. (cluster v2.25+)

  • The Workloads v2 API now supports PersistentVolumeClaims and user credentials as part of the workload specification, enabling workloads to provision storage and reference credentials without pre-existing resources or exposing secrets in the manifest. (cluster v2.26+)

  • AI applications support private NGC Helm repositories; users are prompted for an NGC API key credential when a private repository is selected as the chart source. (cluster v2.26+)

  • Suspend and resume operations are now supported for any workload type that defines suspend behavior in its Karta description, including CronJob, Job, PyTorchJob, RayJob, and TFJob, via both the UI and API. (cluster v2.26+)

Platform Administrators#

  • NUMA-aware scheduling can now be enabled per node pool, placing GPU, CPU, and CPU memory within the same NUMA node where possible to minimize cross-NUMA fragmentation on multi-NUMA servers such as DGX H100. Requires NFD with topology updater enabled and a non-none Topology Manager policy. Disabled by default. (cluster v2.26+)

  • Node pools now support a separate GPU device-level placement strategy (Bin-pack or Spread) in addition to the node-level strategy, allowing fractional distributed workloads to bin-pack at the node level while spreading across GPU devices within a node. (cluster v2.26+)

  • NVIDIA Run:ai network topologies can now be linked to a corresponding Grove ClusterTopology via the Network Topologies API, surfacing the Grove topology name and levels to help users identify the correct topology for their target node pool.

  • Workload policies now support any supported workload type with built-in policy mappings, enabling governance (requirements, constraints, defaults, enforcement) without additional configuration. Administrators can extend policy support to additional workload types via the Policy Mappings API. (Experimental, cluster v2.26+)

  • The Access rules page now surfaces a banner when rules still use deprecated legacy roles, with a dialog to map each legacy role to its replacement; selected rules are recreated under the new role and legacy rules are deleted. (cluster v2.25+)

  • Users now appear by full name next to their email in the format Full Name (email) throughout the platform. Names for local users can be set on creation or edit; SSO user names are sourced from the identity provider. A new users-minimal permission provides the minimal access needed to display user names.

  • The General settings page is now permission-aware: visible only when the user has at least one settings permission with CRUD access; read-only permissions do not surface the page.

Infrastructure Administrators#

System Requirements:

Transition Notice — GPU Fractions: Move to CUDA-Based GPU Sharing:

NVIDIA Run:ai is transitioning GPU fractions from its proprietary sharing engine to NVIDIA CUDA-native technology for broader industry compatibility. The Run:ai experience, APIs, YAML definitions, and workload management workflows are not changing.

  • v2.26 (current): No changes; existing GPU fractions engine remains in use. No action required.

  • Future releases: CUDA-based fractions will replace the legacy engine, requiring updated CUDA driver and GPU Operator versions on all GPU nodes. GPU fractions will be disabled until both requirements are met; all other workloads continue normally. Once the stack requirements are in place, fractions re-enable automatically.

What to do now: No action required for v2.26. Review current CUDA driver and GPU Operator versions and plan a maintenance window for the future stack upgrade. Contact your NVIDIA Run:ai account team with questions.

Deprecation Notifications#

(Deprecated features/APIs remain available for two versions from notice.)

  • GPU Memory SWAP — will be deprecated and removed in a future release, replaced by Snapshot (CUDA Checkpoint/Restore), which captures full pod state and reduces cold-start time on resume. GPU Memory SWAP remains fully supported in v2.26.

  • TensorFlow, XGBoost, and JAX distributed frameworks — deprecated and will be removed in a future release. These are based on Kubeflow Training Operator v1 CRDs, which are deprecated upstream.

  • Resource Interfaces — renamed to Karta across Run:ai. The resourceInterfaces API field is deprecated; use kartas instead.

  • Node Type feature — will be removed in a future release. Use workload policies with node affinity (nodeAffinityRequired.nodeSelectorTerms) instead of nodeType.

  • Legacy workload submission form — will be removed in a future release. Transition to flexible workload submission (enabled by default; re-enable under General settings → Workloads → Flexible workload submission if disabled).

  • Several API endpoints and parameters (refer to the full list in the source documentation), including /api/v1/logo, /api/v1/org-unit/node-type, resourceInterfaces, schedulingConfiguration.placementStrategy, gpuResourceOptimization, DistributedFramework values TF/XGBoost/JAX, and nodeType.

Autonomous Resiliency Engine#

Autonomous Resiliency Engine (ARE) comprises:

  • NVIDIA Mission Control- autonomous job recovery

  • NVIDIA Mission Control- autonomous hardware recovery

  • NVIDIA Resiliency Extensions (NVRx)

Note 1: NVIDIA Resiliency Extensions (NVRx) is not installed with NVIDIA Mission Control. It must be installed separately. Follow the installation instructions at: https://nvidia.github.io/nvidia-resiliency-ext/

Note 2: Integrate the Autonomous Job Recovery Training Telemetry Client Library into your training application to automatically produce log messages monitored by the job recovery service — covering throughput, checkpoints, and hang detection without manual log formatting. Install from NVIDIA PyPI:

pip install --extra-index-url https://pypi.nvidia.com --upgrade aidot_training_telemetry

NVIDIA Mission Control - autonomous job recovery#

Autonomous job recovery v1.7.0 delivers the following updates:

What’s New:

  • Autonomous Resiliency Engine (ARE) is now integrated into NVIDIA Run:ai KAI scheduler when using the kubeflow pytorch training operators. When submitting a job, NVIDIA Mission Control monitors the jobs output and correlates any system log or telemetry data with job failures to determine the root cause - hardware, or software fault. ARE requeues the job and updates its job specification to exclude nodes identified as contributors to the most recent anomaly.

  • NVRx 0.6.0 asynchronous checkpointing is now supported.

Bug Fixes:

  • If ARE installation fails in the NVIDIA Mission Control wizard, the wizard automatically removes all partially deployed resources, allowing installation to be retried without running the uninstall wizard.

For air-gapped installation instructions, refer to nmc-software-installation-guide.

NVIDIA Mission Control - autonomous hardware recovery#

Autonomous hardware recovery 29.2.31 delivers the following updates:

Features:

Authentication & SSO:

  • Keycloak added as a new identity provider, supporting both OIDC and SAML 2 protocols so users can sign in via SSO.

  • Auto-login when only a single OIDC or SAML identity provider is configured.

Resources & queries:

  • Resource queries support an order by clause.

  • Resource types are computed dynamically at runtime.

  • New event type connection is now available.

  • Filter queries now correctly target actions in Operation History.

  • Improved tag conflict resolution.

  • Search operators are now case-insensitive.

Runs & Runbooks:

  • Updated the built-in runbook category list.

  • Exporting a tag value as a runbook parameter is more intuitive.

  • Runbook Remediator V2 keeps its Alertmanager integration durable across restarts.

Runbook cell, output, & docs:

  • Runbook cells support exporting Terraform / OpenTofu output.

  • Markdown cells are no longer skipped when a run is aborted.

Actions & statements:

  • New fire_query action can export variables to a linked runbook.

  • Statement timeout is propagated through all subsequent execution flows.

  • Statements with multiple resource queries are interpreted consistently across paths.

  • export command handles resource queries with mixed resource types correctly.

  • Exit status of timed-out commands can now be exported.

  • Exporting a variable that is not in run params no longer aborts the run.

  • Action sequences now include resource information when running.

  • Linux commands that exit with status 1 are no longer mistakenly marked as succeeded.

  • Fixed incorrect tag names when using the gpu | controller | export() syntax.

  • Fixed proxy_merge for the new export structure.

  • General action-sequence reliability improvements.

  • Fixed output format for action sequences and dynamic filters.

Alarms:

  • Alarm definitions now have syntax validation.

  • Fixed multiple alarms-dashboard issues caused by time-partitioned reads across databases.

  • Removed the Resource Type column from the alarm configurations dashboard.

Errors & validation:

  • Improved error messages across the platform.

  • SQL statements are no longer exposed in backend error messages.

  • More detail in the Stmt execution aborted unexpectedly error message.

Integrations (removed):

  • Datadog integration removed

  • FluentBit integration removed

  • Elastic integration removed

TUI install wizard:

  • TUI prompts for SSO type during install (Keycloak SAML, or Keycloak OIDC).

  • Grafana resources can be deployed to a local kube-prometheus-stack via the TUI.

  • Runbook dependencies are now recommended (not required) in the agent’s Enroot package.

  • Run:ai nodes detection and custom agent install

Terraform / OpenTofu:

  • export_tf() (OpenTofu) function is now executable from CLI, API, and runbooks.

Monitoring and Observability#

Grafana Visualizations#

The following known issues exist in nmc-grafana-dashboards v27.3.8:

NVBUG ID

Description

NVBUG-6057876

Redfish metric collection on B200/B300 platforms can be slow in polling.

Advanced Cluster Management#

Update to NVIDIA Base Command Manager v11.34.0#

Key Features and Enhancements:

Expanded NVIDIA Platform & OS Support:

  • Added support for Vera Rubin NVL72 bring-up, including new Grafana dashboards, alerts, and enhanced Redfish telemetry.

  • Added support for DGX OS 8.0.0 / BaseOS 9 software images.

  • Added support for Ubuntu 26.04

Mission Control, Monitoring & NVLink Enhancements:

  • Added Grafana dashboards and alerts for Vera Rubin NVL72, along with expanded alerting configuration guidance.

  • Enhanced Redfish monitoring with additional current, power, temperature, and voltage telemetry for NVIDIA GB200 NVL72, NVIDIA GB300 NVL72, and NVL72 platforms.

  • Improved NVLink switch power-state validation and recovery behavior.

  • Prometheus exporter configuration can now automatically enable monitoring for NVIDIA Mission Control environments.

Security & Cluster Access Improvements:

  • Added cluster-wide SSH multi-factor authentication (MFA) support, including staged deployment and enforcement capabilities.

  • Addressed multiple Slurm security vulnerabilities through updated Slurm maintenance releases.

  • Fixed a command-injection vulnerability in the BCM SSH restrictor.

  • Strengthened container-registry certificate permissions.

  • Added architecture validation for cloud instances and software images to prevent incompatible images from being deployed.

Kubernetes & Container Platform Updates:

  • Updated key container and Kubernetes ecosystem components, including containerd, Helm, Harbor, MetalLB, Tigera Operator, and Knative.

  • Added the AgentGateway operator and associated CRDs.

  • Improved Kubernetes version consistency by pinning minor versions for both connected and air-gapped deployments.

  • Improved NVIDIA GPU Operator integration and container runtime configuration.

  • Resolved several Run, Knative, Kubelet, Gateway TLS, and operator installation issues.

High Availability & Cluster Lifecycle:

  • BCM HA setup can now optionally clone all databases when configuring a highly available cluster.

  • Improved etcd backup and recovery behavior when individual etcd members are unavailable.

  • Improved node installation, software-image cloning, disk-layout handling, and ramdisk creation across supported architectures and operating systems.

Workload Power Profile Settings (WPPS)#

WPPS is not yet supported in Vera Rubin NVL72.

Energy-Optimized Workload Power Profiles Support for NVIDIA GB300 NVL72 Systems:

Customer Impact: Energy-optimized workload power profiles are not supported on NVIDIA GB300 NVL72 systems in this release. These power profiles depend on Max-Q power profile functionality that is not yet available on these systems.

Workaround: None available at this time.

Resolution: Max-Q support is planned for system software version 2.0.0. Energy-optimized workload power profiles will be enabled in a future release following successful validation.

Update to NetQ 5.2.1#

NetQ 5.2.1 Release Notes

What’s New in NetQ 5.2.1#

  • Re-designed the network topology dashboard so that you can visualize your network’s topology according to the system labels assigned to a device (beta)

  • You can now back up your NetQ data using an NFS server for disaster recovery scenarios (beta)

  • NetQ now supports fault correlation system events which group events linked to the same underlying issue and displays the association between host-based errors and devices within a network’s fabric (beta)

  • Introduced adaptive routing validations that verify configuration consistencies across switches in your network’s fabric (beta)

  • You can now add nodes to your existing NetQ NVLink + Ethernet combined mode deployments (beta)

  • Introduced RoCE validations that verify configuration consistencies across the entire network fabric, including switches, host NICs, and DPUs (beta)

  • Added support for Arm-based systems

  • You can now create NetQ simulations with NVIDIA DSX Air

  • Updated the Cable Validation Tool to version 1.9

  • Added IPv6 support for switch discovery operations

Pre-Release Beta Software Components#

NVIDIA Dynamic Power Software (DPS) v0.9.0-Beta#

Platform Availability:

  • DPS v0.9.0 supports NVIDIA GB200 NVL72, NVIDIA GB300 NVL72, and early access for Vera Rubin NVL72 systems only.

NOTE: DGX B200 and DGX B300 are not supported: topology activation fails on these platforms during power policy setup and will be addressed in a future release.

Known Issues preventing DPS use on DGX B200 and DGX B300

  • DPS cannot activate DGX_B200 topologies (DCPOW-2510)

  • MaxLPS activation cannot create dps-managed-domain on B300 nodes (6797024)

Added:

  • Added the DGX_B200 Redfish device plugin. DGX B200 systems remain unsupported because topology activation fails.

  • Added the DGX_B300 Redfish device plugin. DGX B300 systems remain unsupported because topology activation fails.

  • Added Redfish GPU index detection for platforms whose GPU endpoint numbering differs from the default.

  • Added configurable GPU endpoint index base support through the indexBase device specification field.

  • Added Helm configuration for Redfish per-operation timeouts.

Improved:

  • Improved Redfish GPU power-limit actuation performance by parallelizing GPU write operations.

  • Improved Redfish connection tuning with separate controls for metrics and actuation paths.

  • Improved telemetry resilience by returning partial topology telemetry when one or more BMC telemetry requests fail.

  • Improved BMC simulator timing handling by migrating workload timing fields to time.Duration.

Fixed:

  • Fixed async resource group activation task status lookup.

  • Fixed Redfish operation behavior by bounding per-operation latency.

  • Fixed Redfish telemetry handling to reject invalid 0xFFFFFFFF sentinel values.

  • Fixed stale platform environment metric handling in Redfish telemetry.

  • Fixed high-cardinality Prometheus metric labels to improve observability scalability.

  • Backported disable-node behavior to prevent node power controls on the 0.8.x release branch.

Security:

No security-specific changes are included in this release.

Deprecated:

No features are deprecated in this release.

Removed:

No features are removed in this release.

Compatibility and Support Notes:

  • Operators should review Redfish timeout and concurrency settings before rollout, especially in environments with BMCs sensitive to concurrent actuation requests.

NVIDIA Minerva 1.0.0 (Beta)#

Minerva is an AIOps layer for AI factory operations. It ships as a limited beta alongside this release.

Minerva collects network fabric telemetry that NVIDIA Mission Control does not collect today, including transport, congestion and buffer measures, plus an overall fabric health score. It then correlates that data with GPU, interconnect, scheduler and workload signals so you can trace an issue back to the job that caused it.

Minerva deploys into the NVIDIA Mission Control control plane and is additive to the existing observability stack. It is not a replacement for the NVIDIA Mission Control observability stack.

Availability: Minerva 1.0.0 is in beta and is offered through a limited early-access engagement. Features may change or be withdrawn, and no release or general availability date is committed.

Getting started: Customers interested in becoming early adopters of Minerva should contact their NVIDIA Account Manager or Developer Relations Manager directly to begin a Lighthouse Agreement conversation.

NVIDIA Mission Control 2.5.0 Software Components Enabled#

Capability

DGX B200 *

DGX B300 *

DGX GB200

OEM GB200 NVL72

DGX GB300

OEM GB300 NVL72

DGX VR NVL72

OEM VR NVL72

Rack Provisioning and Management Software

Base Command Manager

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Workload Managers

Slurm

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Run:ai

Yes

Yes

Yes

Yes

Yes

Yes

No

No

Autonomous Resiliency Engine (ARE)

Mission Control- autonomous hardware recovery

Yes

Yes

Yes

Yes

Yes

Yes

No

No

Mission Control-autonomous job recovery

Yes

Yes

Yes

Yes

Yes

Yes

No

No

Telemetry/Observability

Observability Stack (Prometheus Server, Loki, Alloy, and Grafana)

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Opinionated Grafana Visualizations (JSONs)

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Network Management and Monitoring

NetQ (Formerly NMX-M)

N/A

N/A

Yes

Yes

Yes

Yes

No

No

Minerva- Beta (Early Preview)

N/A

Yes

N/A

Yes

Yes

Mission Control LaunchPad

Mission Control Management and Control Plane Hardware Requirements

Control Plane

10 Nodes (x86 Only)

10 Nodes (x86 Only)

10 nodes (ARM required for BCM Head Nodes and Slurm nodes, exceptions permitted for x86 head nodes)

10 nodes (ARM recommended for Slurm nodes, rest can be x86)

10 nodes (ARM required for BCM Head Nodes and Slurm nodes, exceptions permitted for x86 head nodes)

10 nodes (ARM recommended for Slurm nodes, rest can be x86)

10 nodes (ARM required for BCM Head Nodes and Slurm nodes, exceptions permitted for x86 head nodes)

10 nodes (ARM recommended for Slurm nodes, rest can be x86)

* = HGX B200 and HGX B300 from OEMs are not supported.

Refer to the Software Bill of Materials for version numbers for each component

Software Bill of Materials#

Software Downloads, Installation, and Activation#

Product Documentation#