> This page is for version 1.0.1. > For other versions, use one of these documentation indexes: > - 1.0.3 (default): https://docs.nvidia.com/nvcf/1.0.3/llms.txt > - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt > - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt > - Development: https://docs.nvidia.com/nvcf/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server. # Release Notes Release history, stack versions, and upgrade guidance for NVIDIA Cloud Functions. Each stack keeps its own artifact version. The [compatibility matrix](/nvcf/1.0.1/release-notes/compatibility-matrix) records the stack releases that are qualified to run together. ## Release 1.0.1 ## Release 1.0.1 Starting with NVCF 1.0.1, the three Helm stacks are versioned and released independently. Release 1.0.1 documents this qualified stack combination: | Stack | Version | | ------------------- | ------- | | Self-Managed Stack | 1.0.1 | | Compute Plane Stack | 1.0.0 | | Observability Stack | 1.0.0 | These notes describe what changed since NVCF self-hosted 0.6.1, the last legacy release. In 0.6.1, all components shipped as a single bundle. See the [compatibility matrix](/nvcf/1.0.1/release-notes/compatibility-matrix) for the stack releases that are qualified to run together, and the [upgrade guide](/nvcf/1.0.1/release-notes/0-6-1-to-1-0-1-upgrade) for the ordered upgrade procedure from 0.6.1. > **Warning** > > Upgrading from 0.6.1 is not a single `helmfile sync`. The control plane has > mandatory upgrade stops (Cassandra, OpenBao, ICMS, NVCT, then the API), and > every GPU cluster must be re-registered against the upgraded control plane > without deleting its registration. Read the upgrade guide before starting. ### Self-Managed Stack 1.0.1 * Release: `nvcf-self-managed-stack` 1.0.1 * Git tag: `deploy/stacks/self-managed/v1.0.1` 1.0.1 is a patch release on top of 1.0.0. It includes the fixes listed in [Fixes in 1.0.1](#fixes-in-101). #### What is new LLM functions: The control plane now ships the LLM API Gateway and the LLM Request Router (Stargate and Pylon) as stack components. Customers can expose OpenAI-compatible chat completion endpoints for LLM functions, route requests across workers with configurable load-balancing algorithms and affinity, and serve LLM routes on custom hosts through the vanity gateway. See [LLM Gateway](/nvcf/1.0.1/overview/llm-gateway), [LLM Function Enablement](/nvcf/1.0.1/self-managed/llm-function-enablement), and [LLM Request Router Load Balancing](/nvcf/1.0.1/self-managed/llm-request-router-load-balancing). Function Autoscaler: A dedicated autoscaler scales function instances from invocation metrics and queue depth, configured per function through the NVCF API. It replaces static instance counts for most workloads. See [Configure Autoscaling](/nvcf/1.0.1/overview/configure-autoscaling) and [Function Autoscaling Overview](/nvcf/1.0.1/self-managed/function-autoscaling-overview). Managed PKI and encrypted transport: A managed ClusterIssuer backed by OpenBao issues certificates for LLM traffic, LLM gRPC ingress uses a dedicated HTTPS listener, and the router and worker components reload certificates without a restart. Plaintext transport is now an explicit development opt-in. NATS can be configured with server-side TLS for split-plane deployments where workers reach NATS over a load balancer. See [Transport TLS Rotation](/nvcf/1.0.1/self-managed/transport-tls-rotation). NVCF UI: An optional administrative UI ships with the stack, disabled by default. See [NVCF UI](/nvcf/1.0.1/self-managed/nvcf-ui). Operability: PodDisruptionBudgets can be enabled for every control-plane chart, API issuer and JWKS endpoints are overridable for custom identity setups, gateway routes can be enabled per service, and every control-plane service exposes a `GET /info` endpoint reporting its version. ICMS gains select-and-terminate instance controls and per-reservation backup opt-out. See [Pod Disruption Budgets](/nvcf/1.0.1/self-managed/pod-disruption-budgets). Documentation Update: This release adds two new guides: * A multi-tenancy guide describes the two supported invocation models, dedicated endpoint and shared endpoint, and the isolation controls available at each layer, from namespace and network to node and container runtime. See [Multi-Tenancy](/nvcf/1.0.1/overview/multi-tenancy). * A quota management runbook explains how operators raise account limits after installation, such as the maximum number of functions or tasks. See [Quota Management](/nvcf/1.0.1/self-managed/quota-management). #### Changes that can affect an existing install * Observability is no longer bundled in the control-plane stack. It is a separately installed and versioned stack; see the Observability section. * Service lineages changed. SIS is replaced by ICMS (`icms-service-oss`), and the ESS and Notary images are renamed (`nvcf-ess`, `nvcf-notary`). Version numbers that appear to go down in the manifest reflect these new lineages, not downgrades. Image mirrors must include the new names. * Cassandra moves from 5.0.8 to 5.0.9 and OpenBao from 2.5.5 to 2.6.2, each with schema or storage migrations that must run in order. The default Cassandra migration image runs every keyspace; the staged upgrade path requires the bounded per-keyspace Jobs described in the upgrade guide. The Cassandra service-role password in the secrets file must match the value supplied to the OpenBao migration. * Zero-downtime upgrades require three Cassandra members and two replicas per service with `maxUnavailable: 0`. Single-member Cassandra installs incur downtime during the upgrade. * OpenBao pods use an `OnDelete` update strategy and must be rotated manually after the chart upgrade, as in 0.6.1. * The environment file gained sections for every service (`api`, `nvctApi`, `icms`, `apikeys`, `ess`, `invocation`, `adminIssuerProxy`, `functionautoscaler`, `reval`, `llmApiGateway`, `llmRequestRouter`) and the secrets file gained a `cassandra` block. Carry your values forward into the new templates; do not replace the files wholesale. See the environment file section of [Helmfile Installation](/nvcf/1.0.1/self-managed/helmfile-installation). * Third-party charts (observability components and the cert-manager ACME solver image) resolve from upstream repositories by default. Disconnected sites must mirror them and set the per-chart `chartRepository` overrides. * `alpine-k8s` is no longer a customer-mirrored artifact in the manifest. #### Fixes in 1.0.1 * Worker discovery targets the backend router endpoint when backend routing is enabled, so LLM workers register against the correct router. * A final Helmfile stage writes an `nvcf-upgrade-receipt` ConfigMap recording the installed stack version. Future upgrades use it to gate unsupported paths; a cluster without the receipt is treated as a 0.x install. Notable fixes since 0.6.1 include correct LLM gateway to API authentication on default installs, Stargate readiness gated on Pylon registration, aligned Cassandra application credentials, and the Function Autoscaler honoring its configured runtime settings. ### Compute Plane Stack 1.0.0 * Release: `nvcf-compute-plane-stack` 1.0.0 * Git tag: `deploy/stacks/nvcf-compute-plane/v1.0.0` This release ships NVCA and the NVCA operator 3.10.0, up from 3.0.3 in 0.6.1. It also includes the `nvca-operator` chart 1.28.0 and a new required chart, `nvcf-cluster-topology`. #### What is new Advanced scheduling: The stack can install the KAI Scheduler, and NVCA supports gang scheduling and topology-aware placement for multi-node NVLink (MNNVL) workloads through KAI and Grove. Functions that need all replicas co-scheduled, or that need NVLink cliques, can express that requirement directly. See [KAI Scheduler](/nvcf/1.0.1/compute-plane/kai-scheduler), [Gang Scheduling](/nvcf/1.0.1/compute-plane/gang-scheduling), and [Topology-Aware Scheduling](/nvcf/1.0.1/compute-plane/topology-aware-scheduling). Helm model caching: NVCA can serve model artifacts from a shared read-write-many volume and mount them read-only into workloads, selecting the cache backend from a storage capability catalog that covers Weka, OCI FSS, and NVMesh. The feature is off by default and enabled per cluster. See [Helm Model Caching](/nvcf/1.0.1/compute-plane/helm-model-caching). Storage and worker settings as first-class values: Storage classes, worker settings, and bring-your-own observability settings are configured through chart values instead of patches, and missing task-data storage classes fail fast at deployment time rather than at first use. Host isolation: A new `HostIsolation` cluster attribute schedules one function instance per node, in addition to the existing `AccountIsolation` mode. The two are mutually exclusive. See [Cluster Configuration](/nvcf/1.0.1/compute-plane/cluster-configuration). Observability from the compute plane: The NVCA OpenTelemetry collector adds Kubernetes object and event collection for the event ledger, per-function collector resource overrides, log and trace sampling, and correlation IDs on instance events. NVCA exposes a `GET /info` endpoint. Secure transport: Worker init containers and Helm-based LLM workers trust the managed transport CA, TLS is validated before workloads render, and insecure trust bundles are rejected. GPU checkpoint and restore. The NVCA agent integrates with NvSnap to checkpoint and restore GPU workloads. It is off by default and enabled per cluster. See the [NvSnap README](https://github.com/NVIDIA/nvcf/tree/main/src/compute-plane-services/nvsnap). > **Note** > > This feature is in beta. Behavior and configuration may change in future releases. Documentation Update: This release adds the following guides: * An Nsight profiling guide explains how to profile function pods with NVIDIA Nsight. See [Nsight Profiling](/nvcf/1.0.1/compute-plane/nsight-profiling). * Consolidated cache documentation covers the model, container, GX, and simulation caches. See [Caches Overview](/nvcf/1.0.1/compute-plane/caches-overview). #### Changes that can affect an existing cluster * Every GPU cluster must be re-registered against the upgraded control plane without deleting its registration. Preserve `clusterID` and `clusterGroupID`; deleting them orphans deployments and tasks. The source cluster must be on NVCA 3.0.3 with a healthy agent. * Registration inputs changed. In 0.6.1 the environment file had to carry the ICMS, ReVal, and NATS endpoints. `make register-cluster` now generates them. Any non-empty `selfManaged.*` endpoint values left in an old environment file act as overrides and will point the cluster at stale addresses. * The operator installs a new `ModelCacheBinding` CRD, and the `nvcf-cluster-topology` chart is a required component. Cluster-admin rights are needed for the CRD install. * KAI Scheduler is optional, but gang scheduling and topology-aware placement do not function without it. * Resource enforcement is enabled by default; workloads without resource requests may be rejected or rescheduled. Review function resource settings before upgrading. * Clusters on NVCA 2.51.0 or earlier must follow the 0.6.0 version-specific guidance before moving to 1.0.0. #### Notable fixes Operator upgrades from 3.0.x no longer hit a nil pointer or lose self-managed defaults; uninstall completes even while ICMS finalizers are being stripped; GPU capacity and usage are reported correctly in heartbeats; the webhook TLS certificate is no longer regenerated on every rollout; workload configuration persists across Helm updates; and queue credentials refresh after registration. ### Observability Stack 1.0.0 * Release: `nvcf-observability-stack` 1.0.0 * Git tag: `deploy/stacks/observability/v1.0.0` Observability is now a separate stack that is installed and versioned on its own. In 0.6.1, it shipped inside the control plane stack as the `nvcf-observability-reference-stack` chart. #### What is new Profiles: A single `observability.profile` value (`disabled`, `control`, `compute`, or `all`) selects the shared infrastructure and monitors to install. The control-plane stack defaults to `control` and the compute-plane stack to `compute`, so a cluster hosting both selects `all`. See [Observability](/nvcf/1.0.1/observability/observability). Bundled metrics pipeline: The stack installs the Prometheus Operator CRDs, the OpenTelemetry Operator, an NVCF OpenTelemetry collector with target allocation and discovery RBAC, VictoriaMetrics as the default metrics store, and the NVCF default monitors. Bring your own backend: Set `metricsBackend.mode: existing` to use an existing Prometheus-compatible metrics backend. A PromQL endpoint is required for the `control` and `all` profiles because the Function Autoscaler queries it; the autoscaler supports no auth, token, and mTLS. See [Observability](/nvcf/1.0.1/observability/observability#self-managed-metrics-stack). Compute-plane collection: The `compute` and `all` profiles enable the NVCA collector and the worker PodMonitor, which covers function and task pods on a stable `worker-metrics` port. Metrics reference: New reference pages document the metrics that each NVCF service exposes, covering control plane services, LLM services, per-function containers, and datastores. Each page lists metric names, types, descriptions, and labels for use in queries and dashboards. See [Metrics Overview](/nvcf/1.0.1/observability/metrics-overview). #### Changes that can affect an existing install * Install the observability stack once per cluster and pick a profile. Do not also keep the legacy reference chart from the control-plane bundle. * Third-party charts (OpenTelemetry Operator, Prometheus Operator CRDs, VictoriaMetrics) are pulled from upstream Helm repositories by default. Restricted-egress sites must mirror them and set per-chart `chartRepository` overrides. * The bundled VictoriaMetrics instance needs a persistent volume; size it for your retention. * The stack is Early Access. Customers own long-term storage, log collection, dashboards, and alerting. Kubernetes support covers the latest minor version and the two before it. #### Notable fixes Worker PodMonitor targets resolve through the named `worker-metrics` port, Function Autoscaler metrics are scraped, VictoriaMetrics installs after the Prometheus CRDs so first installs succeed, and custom observability environments are supported. ## Release 0.6.1 ## Release 0.6.1 NVCF Self-Hosted 0.6.1 is a maintenance release. It updates OpenBao to address security issues in the runtime and migration images. It also replaces the archived Bitnami Cassandra runtime with the official Apache Cassandra 5.0.8 image and an in-house Cassandra Helm chart. ### Highlights | Change | Status | Summary | | ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | OpenBao security update | GA | OpenBao uses the remediated runtime and migration images. Existing installations require a controlled server pod rotation after the Helm upgrade. | | Cassandra off Bitnami | GA | Cassandra runs on the official Apache Cassandra 5.0.8 image, deployed by an in-house Helm chart, replacing the Bitnami image and subchart. | ### Cassandra runtime and chart The Cassandra runtime image is now built on the official Apache Cassandra 5.0.8 image instead of the archived Bitnami image, and the Bitnami Cassandra Helm subchart is replaced by an in-house StatefulSet chart. This removes the CVE-affected Bitnami base and its bundled components while keeping the same operational surface: PasswordAuthenticator and CassandraAuthorizer, stable-DNS seed discovery, the Prometheus metrics exporter, keyspace initialization, and schema migrations. The chart runs the nodes with `storage_compatibility_mode: NONE` to match the existing fleet's on-disk and streaming format. ### Upgrade Notes Fresh installs need no special handling; Cassandra comes up on the official image with the in-house chart. Existing clusters that already hold data need a migration, because the runtime image, the Helm chart shape, and the on-disk data layout all change. A normal in-place Helm upgrade does not carry an existing Bitnami-based cluster onto the new stack. Follow the [0.6.0 to 0.6.1 Upgrade](/nvcf/1.0.1/release-notes/legacy-0-6-0-to-0-6-1-upgrade) procedure, which describes the supported migration methods (datacenter expansion, in-place volume adoption, backup and restore) and their trade-offs. ### OpenBao security update The OpenBao runtime and migration images include security fixes. OpenBao stores control-plane secrets and uses persistent Raft storage, so upgrade it before applying the rest of the patch stack. The OpenBao StatefulSet uses `OnDelete`. The Helm upgrade updates the StatefulSet template and runs the post-upgrade migration Job, but Kubernetes does not restart existing OpenBao server pods. After the OpenBao release is synced from the 0.6.1 stack, rotate the OpenBao server pods manually. The pod rotation steps recreate pods from the existing StatefulSet. They do not delete or recreate the OpenBao persistent volume claims (PVCs). Troubleshooting OpenBao itself is outside the scope of this upgrade guide. If OpenBao is sealed, uninitialized, unhealthy, or failing migration before or during the procedure, stop and resolve OpenBao before continuing. > **Info** > > These steps assume 0.6.0 is already deployed and healthy, that you have > downloaded the 0.6.1 stack, and that you run `make` from the extracted > `nvcf-self-managed-stack/` directory with your environment set. ```bash export HELMFILE_ENV= ``` #### Prepare the 0.6.1 configuration Return to the extracted 0.6.1 stack before running any 0.6.1 sync command. Reconcile the site-specific settings from the 0.6.0 deployment into the 0.6.1 configuration files: * `environments/$HELMFILE_ENV.yaml` * `secrets/$HELMFILE_ENV-secrets.yaml` Start from the files or templates included with 0.6.1 and carry forward the required values from 0.6.0. Do not replace the 0.6.1 files wholesale because available settings can change between releases. Preserve the existing registry, storage, endpoint, credential, and secret values unless this procedure explicitly instructs you to change them. Confirm that both files exist before continuing: ```bash for file in \ "environments/${HELMFILE_ENV}.yaml" \ "secrets/${HELMFILE_ENV}-secrets.yaml"; do [ -f "$file" ] || { echo "missing required file: $file"; exit 1; } done ``` #### Check OpenBao before the patch Confirm that the current OpenBao release is healthy before starting the patch upgrade: ```bash kubectl -n vault-system get pods,jobs kubectl -n vault-system get statefulset openbao-server \ -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}' ``` The OpenBao server pods should be `Running`, the OpenBao Jobs should be `Complete`, and the update strategy should be `OnDelete`. Record the full image references printed from the StatefulSet template. These are OpenBao image references, not NVCF stack version numbers. After step 1, the StatefulSet template should show the updated OpenBao image reference. Existing pods can still show the previous OpenBao image reference until step 2 is complete. Inspect each OpenBao server pod and record which pod is active: ```bash for pod in $(kubectl -n vault-system get pods \ -l app.kubernetes.io/name=openbao,component=server \ -o jsonpath='{.items[*].metadata.name}'); do echo "== $pod ==" kubectl -n vault-system exec "$pod" -c openbao -- sh -c \ 'BAO_ADDR=http://127.0.0.1:8200 bao status' done ``` In the command output, the active pod reports `HA Mode active`. Standby pods report `HA Mode standby` and include the active node address. Do not continue if any pod is sealed, uninitialized, or not ready. #### Step 1: sync OpenBao from the 0.6.1 stack Sync only the OpenBao release first. This updates the OpenBao release and StatefulSet template, rolls the injector deployment, and runs the OpenBao post-upgrade migration Job. ```bash make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=openbao-server ``` The command blocks until Helm finishes. If the command fails, stop and resolve the OpenBao release before continuing. Do not rotate OpenBao server pods until the post-upgrade migration Job has completed. Confirm the OpenBao migration Job succeeded: ```bash kubectl -n vault-system get jobs # the openbao-server-migrations Job should report COMPLETIONS 1/1 ``` If the migration Job fails, stop and resolve the OpenBao release before continuing. Do not rotate OpenBao server pods while the migration Job is failed or incomplete. Confirm that the StatefulSet template now points at the updated OpenBao image reference and still uses `OnDelete`: ```bash kubectl -n vault-system get statefulset openbao-server \ -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{": "}{.image}{"\n"}{end}{"updateStrategy: "}{.spec.updateStrategy.type}{"\n"}' ``` Confirm that the injector now injects the updated OpenBao image reference: ```bash kubectl -n vault-system get deployment openbao-server-agent-injector \ -o jsonpath='{range .spec.template.spec.containers[*].env[?(@.name=="AGENT_INJECT_VAULT_IMAGE")]}{.value}{"\n"}{end}' ``` > **Info** > > At this point the OpenBao Helm release and StatefulSet template are updated, but > existing OpenBao server pods can still be running the previous OpenBao image. > This is expected because the StatefulSet uses `OnDelete`. #### Step 2: rotate OpenBao server pods Rotate OpenBao server pods manually so each pod is recreated from the updated StatefulSet template. Rotate standby pods first. Rotate the active pod last. > **Warning** > > Do not delete all OpenBao server pods at once. Delete one pod, wait for it to > become ready, and verify that it is initialized and unsealed before deleting the > next pod. Do not delete the OpenBao StatefulSet or its PVCs. > **Info** > > If your installation has only one OpenBao server pod, there are no standby pods. > Rotate the single pod after the migration Job completes, then wait for it to > become ready and verify that it is initialized and unsealed. List the pods and identify the standby pods: ```bash for pod in $(kubectl -n vault-system get pods \ -l app.kubernetes.io/name=openbao,component=server \ -o jsonpath='{.items[*].metadata.name}'); do echo "== $pod ==" kubectl -n vault-system exec "$pod" -c openbao -- sh -c \ 'BAO_ADDR=http://127.0.0.1:8200 bao status' done ``` In the command output, rotate pods that report `HA Mode standby` first. Do not rotate the pod that reports `HA Mode active` until all standby pods have been recreated and verified. For each standby pod, delete the pod and wait for it to return. Repeat this block for every standby pod: ```bash export POD= # Waits for the old pod to terminate, not for the replacement pod to be ready. kubectl -n vault-system delete pod "$POD" --wait=true kubectl -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m kubectl -n vault-system exec "$POD" -c openbao -- sh -c \ 'BAO_ADDR=http://127.0.0.1:8200 bao status' ``` The recreated pod must report `Initialized true`, `Sealed false`, and a healthy HA mode before you rotate the next pod. After all standby pods are recreated and healthy, identify the active pod again: ```bash for pod in $(kubectl -n vault-system get pods \ -l app.kubernetes.io/name=openbao,component=server \ -o jsonpath='{.items[*].metadata.name}'); do echo "== $pod ==" kubectl -n vault-system exec "$pod" -c openbao -- sh -c \ 'BAO_ADDR=http://127.0.0.1:8200 bao status' done ``` The active pod is the pod that reports `HA Mode active`. Rotate the active pod last: ```bash export POD= # Waits for the old pod to terminate, not for the replacement pod to be ready. kubectl -n vault-system delete pod "$POD" --wait=true kubectl -n vault-system wait "pod/$POD" --for=condition=Ready --timeout=10m kubectl -n vault-system exec "$POD" -c openbao -- sh -c \ 'BAO_ADDR=http://127.0.0.1:8200 bao status' ``` If a recreated pod does not become ready before the timeout, or if it reports `Sealed true` or `Initialized false`, stop the procedure and resolve that pod before rotating any other OpenBao server pod. #### Verify OpenBao after rotation Confirm that all OpenBao server pods are ready and use the updated OpenBao image reference from the StatefulSet template: ```bash kubectl -n vault-system get pods \ -l app.kubernetes.io/name=openbao,component=server kubectl -n vault-system get pods \ -l app.kubernetes.io/name=openbao,component=server \ -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{" "}{.name}{": "}{.image}{"\n"}{end}{end}' ``` Confirm that the StatefulSet reports all replicas updated and ready: ```bash kubectl -n vault-system get statefulset openbao-server \ -o jsonpath='replicas: {.status.replicas}{"\n"}readyReplicas: {.status.readyReplicas}{"\n"}updatedReplicas: {.status.updatedReplicas}{"\n"}' ``` The `replicas`, `readyReplicas`, and `updatedReplicas` values should match. If `updatedReplicas` briefly lags after the final pod is recreated, wait a few seconds and rerun the command. The per-pod image reference check above and `bao status` check below are the primary verification points. Confirm that each OpenBao server pod is initialized, unsealed, and participating in HA: ```bash for pod in $(kubectl -n vault-system get pods \ -l app.kubernetes.io/name=openbao,component=server \ -o jsonpath='{.items[*].metadata.name}'); do echo "== $pod ==" kubectl -n vault-system exec "$pod" -c openbao -- sh -c \ 'BAO_ADDR=http://127.0.0.1:8200 bao status' done ``` Do not continue if any pod is sealed, uninitialized, not ready, or still running the previous OpenBao image. #### Sync the remaining 0.6.1 stack After OpenBao is healthy on the updated OpenBao image, sync the rest of the 0.6.1 stack: ```bash make install HELMFILE_ENV="$HELMFILE_ENV" ``` Confirm that the control plane is healthy: ```bash kubectl get pods -A ``` This command should print nothing. Any output is a pod that is not yet `Running` or `Completed`. ## Release 0.6.0 ## Release 0.6.0 NVCF Self-Hosted 0.6.0 is a stable release. It adds asynchronous task workloads, multi-cluster control-plane management, OpenAI-compatible LLM routing, single-mode Multi-Instance GPU (MIG) support, and control-plane observability. It also adds Early Access support for optional NVIDIA Dynamo and Grove operator integration. ### Highlights | Feature | Status | Summary | | ---------------------------- | ---------------------- | ----------------------------------------------------------------------------- | | NVIDIA Cloud Tasks (NVCT) | GA | Run asynchronous GPU jobs to completion without keeping an online endpoint. | | Multi-cluster GPU management | GA | Register and manage multiple GPU clusters from one NVCF control plane. | | Dynamo operator integration | Early Access, optional | Install optional NVIDIA Dynamo and Grove operators on GPU clusters. | | LLM API Gateway | GA | Route OpenAI-compatible chat completions, Responses, and embeddings by model. | | Single-mode MIG | GA | Expose single-strategy MIG profiles as NVCF instance types. | | Control-plane observability | GA | Export metrics, logs, and optional traces from control-plane services. | ### NVIDIA Cloud Tasks (NVCT) NVIDIA Cloud Tasks (NVCT) adds an asynchronous workload type for run-to-completion GPU jobs. Tasks are the batch counterpart to Cloud Functions: functions stay online to serve invocation requests, while tasks run to completion and release GPU capacity when they finish. Use NVCT for batch inference, fine-tuning, evaluation, solvers, data preparation, and other GPU workloads that do not require an always-on endpoint. ### Multi-Cluster GPU Management A single NVCF control plane can now register and manage multiple GPU clusters. Each cluster keeps its own identity and configuration, so operators can expand across regions, cloud providers, or on-premises environments without running a separate NVCF control plane for each cluster. After registration, workloads can deploy to the available clusters through the same control-plane API. ### Dynamo Operator Integration NVCF can integrate with optional NVIDIA Dynamo and Grove operators installed on each GPU cluster. KAI Scheduler is a separate prerequisite for using these components. This integration is Early Access and optional. ### LLM API Gateway The LLM API Gateway provides an OpenAI-compatible entry point for LLM workloads on NVCF. It reads the target function and model from the request's `model` field, routes chat completions, Responses, and embeddings to the matching model backend, and applies per-model token rate limits. When cache-aware routing is enabled, the gateway can keep multi-turn requests on the same backend for better cache locality. ### Single-Mode MIG NVCF can detect GPUs configured with the single MIG strategy and expose each MIG profile as an NVCF instance type. Workloads can target an isolated GPU slice instead of a full GPU, improving capacity use on supported nodes. ### Control-Plane Observability Control-plane observability is configured through one Helmfile configuration block. NVCF can export Prometheus-compatible metrics, control-plane logs, and optional OpenTelemetry Protocol (OTLP) traces to an existing observability backend, including in air-gapped environments. ### Upgrade Notes For an existing 0.5.x deployment, follow the [0.5.0 to 0.6.0 Upgrade](/nvcf/1.0.1/release-notes/legacy-0-5-0-to-0-6-0-upgrade) procedure. It covers the required migration stops, full stack sync, and GPU cluster re-registration steps. Customers with GPU clusters running NVIDIA Cluster Agent (NVCA) versions older than 2.51.0 must upgrade those clusters to NVCA 2.51.0 before moving to NVCA 3.x. NVCA 2.51.0 includes fixes for workload lifecycle management and GPU capacity accounting. Skipping this version can cause incorrect GPU resource accounting, untracked workload pods, or capacity mismatches between the GPU cluster and the control plane. | Current NVCA version | Action | | -------------------- | ----------------------------------------------------------------------------- | | 2.50.5 or older | Upgrade to 2.51.0 first, verify cluster health, then upgrade to 3.0 or later. | | 2.51.0 | Upgrade directly to 3.0 or later. | Check the current NVCA version: ```bash kubectl get nvcfbackend -n nvca-operator ``` The `VERSION` column shows the deployed NVCA version. Upgrade the NVCA operator to 2.51.0 before continuing to 3.x: ```bash helm upgrade nvca-operator -n nvca-operator \ --reuse-values \ --wait \ "${CHART_URL}" \ --username='$oauthtoken' \ --password="$(helm get values -n nvca-operator nvca-operator -o json | jq -r '.ngcConfig.serviceKey')" \ --set helmManaged.nvcaVersion="2.51.0" ``` After the upgrade, verify that the cluster is healthy before proceeding: ```bash kubectl get nvcfbackend -n nvca-operator # Verify VERSION shows 2.51.0 and HEALTH shows healthy. ```