NVIDIA Mission Control 2.3.1 Release Notes#

Overview#

NVIDIA Mission Control 2.3.1 adds air-gapped deployment support for NVIDIA Mission Control - autonomous job recovery, updates NVIDIA Base Command Manager to 11.33.1 with Kubernetes networking and containerd improvements, and updates NVIDIA Run:ai to 2.25 with AI application deployment, topology-aware scheduling, Dynamic Resource Allocation (DRA), and Kubernetes Gateway API support.

This release also includes fixes across the Mission Control software stack. Refer to the revised Software Bill of Materials (SBOM) for component details.

Key Highlights:

  • NVIDIA Base Command Manager v11.33.1: Adds Kubernetes networking and containerd improvements, and fixes issues in CMDaemon and cm-kubernetes-setup.

  • NVIDIA Run:ai v2.25: Adds AI application deployment, topology-aware scheduling, DRA support, workload endpoint discovery, Kubernetes Gateway API support, and updated platform-administration workflows.

  • NVIDIA Mission Control - autonomous job recovery v1.5.0: Adds air-gapped installation support for observability components.

  • Updated software stack: Updates Mission Control LaunchPad to v0.2.18, Mission Control Air-Gap Tool to v1.10.0, and autonomous hardware recovery to v29.1.97.

Platform Support#

Capability

DGX B200*

DGX B300*

DGX GB200

OEM GB200 NVL72

DGX GB300

OEM GB300 NVL72

NVIDIA Base Command Manager (11.33.1)

Yes

Yes

Yes

Yes

Yes

Yes

NVIDIA Run:ai (2.25)

Yes

Yes

Yes

Yes

Yes

Yes

NVIDIA Mission Control-Autonomous Resiliency Engine

NVIDIA Mission Control-autonomous job recovery (1.5.0)

Yes

Yes

Yes

Yes

Yes

Yes

NVIDIA Mission Control-autonomous hardware recovery (29.1.97) and NVIDIA Mission Control-autonomous hardware recovery-Config files and Runbooks (2.3.26)

Yes

Yes

Yes

Yes

Yes

Yes

NVIDIA Resiliency Extensions (0.4.1)

Yes

Yes

Yes

Yes

Yes

Yes

Observability Stack

Yes

Yes

Yes

Yes

Yes

Yes

NVIDIA Mission Control-Grafana Dashboards (27.3.2)

Yes

Beta

Yes

Yes

Yes

Yes

NVIDIA NetQ (5.1.0)

N/A

N/A

Yes

Yes

Yes

Yes

NVIDIA Mission Control-LaunchPad (0.2.18)

Yes

Yes

Yes

Yes

Yes

Yes

NVIDIA Domain Power Service (0.7.10) (Early Preview)

Yes

Yes

Yes

Yes

Yes

Yes

Kubernetes Security Policies (2.0.15)

Yes

Yes

Yes

Yes

Yes

Yes

Air-Gapped Deployment

Yes

Yes

Yes

Yes

Yes

Yes

Feature Updates#

NVIDIA Base Command Manager#

NVIDIA Base Command Manager v11.33.1 includes the following updates:

CMDaemon#

Fixed Issues#

  • Resolved a CMDaemon database upgrade failure tied to NVLink SDN partition migrations that could occur during upgrades that span a larger version gap.

cm-kubernetes-setup#

New Features#

  • Added routing support from bcm-gateway to Ingress NGINX for NMC installation wizards on clusters that do not support the Gateway API.

  • Standardized containerd configuration to use RUNTIME_DROP_IN_CONFIG at version 3 and removed the NVIDIA_RUNTIME_SET_AS_DEFAULT=true setting from the configuration.

  • Updated the containerd drop-in configuration for sandbox image and certs.d path specification to version 3.

Fixed Issues#

  • Added a port-in-use check for the Gateway proxy port in the Kubernetes wizard, matching the check already in place for the Kubernetes API server proxy port.

  • Fixed an air-gapped NVIDIA Network Operator deployment issue that pointed to a non-existent k8s-launch-kit image. The wizard now correctly references v26.1.0, the compatible version for Network Operator v26.1.1.

  • Fixed a failure of cm-runai-setup --skip-packages in air-gapped environments caused by the wizard incorrectly attempting an apt update.

NVIDIA Run:ai 2.25#

The following highlights summarize NVIDIA Run:ai 2.25 self-hosted updates. Refer to the NVIDIA Run:ai documentation for full details, including configuration steps.

AI Practitioners - Workloads#

  • Blueprints and other Helm charts from the NGC catalog, or by direct URL, can now be deployed as AI applications through the UI and API without direct cluster access. Users can override Helm values, such as GPU fractions, before deployment. This feature is experimental and requires cluster v2.25 or later.

  • Topology placement visibility now shows the actual topology placement of a workload, including name, level, value, requested constraints, and whether constraints were met. This information is available in the Workloads API and in the UI details view for cluster v2.25 or later.

  • YAML-based workload submissions can now specify whether Multi-Node NVLink (MNNVL) acceleration is required on a per-workload basis for cluster v2.24 or later.

  • Enhanced workload structure visibility exposes the full internal hierarchy of supported workload types, from the top-level workload to individual elements and pods. This information is available through the API and a new Structure tab in the UI for cluster v2.25 or later.

  • Added support for scheduling supported workload types with Kubernetes Dynamic Resource Allocation (DRA) ResourceClaims for GPUs, in addition to the existing extended-resources method, for cluster v2.25 or later.

  • Externally reachable endpoints for workloads and AI applications are now automatically discovered and surfaced in the UI Connections column and through the Workloads and AI Applications APIs for cluster v2.25 or later.

  • Updated the distributed training workflow with a new mutual workload setup step and removed the Allow different setup for the master toggle for Flexible submissions.

  • Improved pod log and terminal access with one-click access from Pods and Logs views, a new Terminal tab, synchronized pod selection across views, and the ability to view logs from previous container instances for cluster v2.23 or later.

AI Practitioners - Inference#

  • Added hierarchical, multi-component topology-aware scheduling for Dynamo workloads using Grove topology definitions, enforced at both the workload and component level, for cluster v2.25 or later.

  • Distributed multi-node inference workloads can now be submitted and managed directly from the UI, supporting leader and worker configurations for custom inference, for cluster v2.22 or later.

  • Hugging Face models can now be deployed through the NIM serving path using Multi-LLM NIM, removing the previous one-image-per-model restriction.

  • Added endpoint access control for NIM services deployed through the NIM Operator, configurable for users, groups, and service accounts through the API, for cluster v2.25 or later.

  • Added full CLI lifecycle support for NIM services, including submit, update, list, delete, exec, logs, port-forward, and describe operations. The CLI supports autoscaling, fractional GPU, multi-node, and Multi-LLM workloads for cluster v2.25 or later.

  • Increased the maximum initialization timeout for inference workloads and templates to 720 minutes to accommodate models that need more time to start.

  • Updated the NIM model selection dropdown to show short descriptions sourced from the NGC model card. The Hugging Face model catalog now highlights the top 20 trending models.

  • Predefined inference environment assets, including chatbot-ui, gpt2, and llm-server, can now be deleted.

AI Practitioners - CLI#

  • User credentials, including Generic credentials, Docker registry credentials, and NGC API keys, can now be created, listed, and deleted with the CLI and used during workload submission as image pull secrets or environment variables for cluster v2.22 or later.

  • Added runai workload-type list and runai workload-type describe for supported YAML-based workload types for cluster v2.23 or later.

Platform Administrators#

  • Added a guided in-product tour for the projects and departments experience.

  • Time-based fairshare parameters, including historical usage weight and window, are now configurable directly in the UI, not only through the node pool API, for cluster v2.24 or later.

  • Access rules can now be assigned during local user or service account creation for administrators with the required permissions.

  • Redesigned the Notifications API. Dedicated Email and Slack APIs replace the previous unified notification channel endpoints, and Slack integration is now fully supported in the UI across all deployment types for cluster v2.25 or later. Legacy endpoints are deprecated.

  • Added a blocked rule for workload policies, enabling administrators to prevent practitioners from setting a value for a given field.

  • Overview dashboard access is now permission-based, showing only widgets and data relevant to each user’s read permissions for clusters, node pools, nodes, projects, departments, and workloads.

  • Over-time dashboard widgets now support scope-aware filtering by department and project based on the user’s permissions.

  • Removed legacy Grafana dashboards, including Overview, Analytics, Consumption, and Multi-cluster, in favor of the new dashboards. These dashboards were deprecated in v2.23.

  • Renamed the Session timeout setting to UI inactivity timeout. The default is 24 hours when the field is left blank. CLI and API access are not affected.

  • The Pods modal in the Nodes and Workloads views now supports sorting by pod name.

  • The Workload Metrics view adds a Since first run time range option.

Infrastructure Administrators#

  • NVIDIA Run:ai artifacts, including Helm charts, container images, CLI packages, air-gapped packages, and documentation, are now published to NVIDIA NGC. JFrog support is deprecated for cluster v2.25 or later.

  • Host-based routing is now enabled by default, exposing workload URLs as subdomains. Kubernetes clusters need additional setup, OpenShift clusters do not, and upgrades from path-based routing are unaffected for cluster v2.25 or later.

  • Added opt-in support for the Kubernetes Gateway API as an alternative to Ingress. Gateway API can run alongside Ingress for zero-downtime migration for cluster v2.25 or later.

  • Vertical Pod Autoscaling (VPA) can now be configured for cluster-level services globally, per scaling group, or per component with runaiconfig. Supported update modes include Off, Initial, Auto, and InPlaceOrRecreate for cluster v2.25 or later.

System Requirements#

  • Added support for OpenShift 4.21, Knative 1.19-1.21, GPU Operator 26.3, Network Operator 26.1, and DRA driver 25.15.

  • Kubernetes 1.32 and OpenShift 4.17 are no longer supported.

Deprecation Notifications#

The following deprecated features and APIs remain available for two releases from notice:

  • JFrog as an artifact registry. Migrate to NGC.

  • TGI server for the Hugging Face flow. The TGI server is in maintenance mode and moving toward end of life.

  • Administrator CLI, runai-adm. Use runai diagnostics collect-logs for log collection and kubectl for node role management.

  • Node-level Scheduler feature.

  • Department administrator role. This role has been renamed to Department administrator legacy. A new Department administrator role replaces it without Read Clusters access.

  • Several v1 API endpoints and parameters, including notification-channel and Slack endpoints, /api/v1/token, /api/v1/clusters/{clusterUuid}/nodes, overProvisioningRatio, placementStrategy, externalConnections, and urls.

Autonomous Resiliency Engine#

NVIDIA Mission Control - autonomous job recovery#

Autonomous job recovery v1.5.0 includes the following updates:

  • Adds air-gapped installation support for observability components. Alloy Open Source Software (OSS) packages are downloaded with the nmc-airgap tool and deployed directly through the installation wizard, removing the internet access requirement introduced in v2.3.0.

What changed#

  • Air-gapped installations no longer require internet access for observability components. Alloy .deb packages are bundled in the file/ subdirectory of the nmc-airgap download and specified during the autonomous job recovery installation wizard.

  • The cm-mission-control-setup Base Command Manager wizard now prompts for Grafana repository configuration. Select yes for standard environments and no for air-gapped environments.

  • The cm-mission-control-setup wizard validates the required Alloy packages before proceeding and deploys them when Save config & deploy is selected.

  • Removed manual steps for fetching the cm-setup-ajr-bcm11 plugin and the wizard API token buffer patch, simplifying the installation flow.

For more information about installing NVIDIA Mission Control - autonomous job recovery in an air-gapped setup, refer to the NVIDIA Mission Control 2.3.1 Software Installation Guide.

Known Issues and Limitations#

Known Issues: Domain Power Service#

The following known issues exist in Domain Power Service v0.7.10:

NVBUG ID

Description

NVBUG-5969875

Power distribution data shows via dpsctl but is not displayed by PDN Graphview in the UI.

NVBUG-5965403

UI indicates power policy percent limit accepts 0-100% but rejects any value other than 100%.

NVBUG-5965415

DPS UI allows proceeding to save a Power Policy without limits added, then reports limits as mandatory at the next step.

NVBUG-5951458

DPS UI incorrectly displays “no topology file uploaded” even after a successful upload.

NVBUG-5951531

DPS UI is missing an option to display DPS version info; the only available option is to log out.

NVBUG-5973217

DPS Slurm integration PrologSlurmctld/EpilogSlurmctld scripts block job scheduling, causing jobs to requeue indefinitely. Workaround: Disable DPS prolog/epilog in slurm.conf.

Energy-Optimized Workload Power Profiles Support#

Customer Impact: Energy-optimized workload power profiles are not supported for NVIDIA GB300 NVL72 systems in this release.

Workaround: None available at this time.

Resolution: Support for energy-optimized workload power profiles depends on MaxQ Power profile functionality, planned for system software version 2.0.0 in a future release. The feature will be enabled in a future release following successful validation.

Known Issues: Grafana Visualizations#

The following known issues exist in nmc-grafana-dashboards v27.3.2:

NVBUG ID

Description

NVBUG-6057876

Redfish metric collection on B200/B300 platforms can be slow in polling.

NVIDIA Mission Control 2.3.1 Software Components#

  • NVIDIA Base Command Manager - 11.33.1 (GB200/GB300 NVL72, DGX B200/B300)

  • NVIDIA |runai| - 2.25 (GB200/GB300 NVL72, DGX B200/B300)

  • NetQ - 5.1.0 (GB200 NVL72, GB300 NVL72)

  • Grafana Visualizations - 27.3.2 (GB200/GB300 NVL72, DGX B200/B300)

  • Kubernetes Security Policies - 2.0.15 (GB200/GB300 NVL72, DGX B200/B300)

  • Domain Power Service (DPS) - Early Preview - 0.7.10

  • NVIDIA Mission Control-LaunchPad - 0.2.18

  • NVIDIA Mission Control - Air-Gap Tool for DGX B200/B300 and GB200/GB300 NVL72 Systems - 1.10.0

  • Autonomous Recovery Engine, including:

    • autonomous job recovery - 1.5.0 (GB200/GB300 NVL72, DGX B200/B300)

    • autonomous hardware recovery - 29.1.97 (GB200/GB300 NVL72, DGX B200/B300)

    • autonomous hardware recovery-Config files and Runbooks - 2.3.26 (GB200/GB300 NVL72, DGX B200/B300)

    • NVIDIA Resiliency Extension (NVRx) - 0.4.1

Note

NVIDIA Resiliency Extensions (NVRx) is not bundled with Mission Control. It must be installed separately. Refer to the NVIDIA Resiliency Extension documentation for installation instructions.

Software Bill of Materials (SBOM)#

Software Downloads, Installation, and Activation#

Product Documentation#