> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server.

# NVCF 1.0.1 Release Notes

Starting with NVCF 1.0.1, the three Helm stacks are versioned and released independently.

The current releases are:

| Stack                | Version |
|----------------------|---------|
| Self-Managed Stack   | 1.0.1   |
| Compute Plane Stack  | 1.0.0   |
| Observability Stack  | 1.0.0   |

These notes describe what changed since NVCF self-hosted 0.6.1, the last legacy release. In 0.6.1, all components shipped as a single bundle.

See the [compatibility matrix](/nvcf/overview/compatibility-matrix) for the stack
releases that are qualified to run together, and the
[upgrade guide](/nvcf/overview/0-6-1-to-1-0-1-upgrade) for the ordered upgrade
procedure from 0.6.1.

<Warning>
Upgrading from 0.6.1 is not a single `helmfile sync`. The control plane has
mandatory upgrade stops (Cassandra, OpenBao, ICMS, NVCT, then the API), and
every GPU cluster must be re-registered against the upgraded control plane
without deleting its registration. Read the upgrade guide before starting.
</Warning>

## Self-Managed Stack 1.0.1

- **Release:** `nvcf-self-managed-stack` 1.0.1
- **Git tag:** `deploy/stacks/self-managed/v1.0.1`

1.0.1 is a patch release on top of 1.0.0. It includes the fixes listed in [Fixes in 1.0.1](#fixes-in-101).

### What is new

**LLM functions:** The control plane now ships the LLM API Gateway and the LLM
Request Router (Stargate and Pylon) as stack components. Customers can expose
OpenAI-compatible chat completion endpoints for LLM functions, route requests
across workers with configurable load-balancing algorithms and affinity, and
serve LLM routes on custom hosts through the vanity gateway. See
[LLM Gateway](/nvcf/overview/llm-gateway),
[LLM Function Enablement](/nvcf/self-managed/llm-function-enablement), and
[LLM Request Router Load Balancing](/nvcf/self-managed/llm-request-router-load-balancing).

**Function Autoscaler:** A dedicated autoscaler scales function instances from
invocation metrics and queue depth, configured per function through the NVCF
API. It replaces static instance counts for most workloads. See
[Configure Autoscaling](/nvcf/overview/configure-autoscaling) and
[Function Autoscaling Overview](/nvcf/self-managed/function-autoscaling-overview).

**Managed PKI and encrypted transport:** A managed ClusterIssuer backed by OpenBao
issues certificates for LLM traffic, LLM gRPC ingress uses a dedicated HTTPS
listener, and the router and worker components reload certificates without a
restart. Plaintext transport is now an explicit development opt-in. NATS can be
configured with server-side TLS for split-plane deployments where workers reach
NATS over a load balancer. See
[Transport TLS Rotation](/nvcf/self-managed/transport-tls-rotation).

**NVCF UI:** An optional administrative UI ships with the stack, disabled by
default. See [NVCF UI](/nvcf/self-managed/nvcf-ui).

**Operability:** PodDisruptionBudgets can be enabled for every control-plane chart,
API issuer and JWKS endpoints are overridable for custom identity setups,
gateway routes can be enabled per service, and every control-plane service
exposes a `GET /info` endpoint reporting its version. ICMS gains
select-and-terminate instance controls and per-reservation backup opt-out.
See [Pod Disruption Budgets](/nvcf/self-managed/pod-disruption-budgets).

**Documentation Update:** This release adds two new guides:

- A multi-tenancy guide describes the two supported invocation models,
  dedicated endpoint and shared endpoint, and the isolation controls available
  at each layer, from namespace and network to node and container runtime. See
  [Multi-Tenancy](/nvcf/overview/multi-tenancy).
- A quota management runbook explains how operators raise account limits after
  installation, such as the maximum number of functions or tasks. See
  [Quota Management](/nvcf/self-managed/quota-management).

### Changes that can affect an existing install

- Observability is no longer bundled in the control-plane stack. It is a
  separately installed and versioned stack; see the Observability section.
- Service lineages changed. SIS is replaced by ICMS (`icms-service-oss`), and
  the ESS and Notary images are renamed (`nvcf-ess`, `nvcf-notary`). Version
  numbers that appear to go down in the manifest reflect these new lineages,
  not downgrades. Image mirrors must include the new names.
- Cassandra moves from 5.0.8 to 5.0.9 and OpenBao from 2.5.5 to 2.6.2, each
  with schema or storage migrations that must run in order. The default
  Cassandra migration image runs every keyspace; the staged upgrade path
  requires the bounded per-keyspace Jobs described in the upgrade guide. The
  Cassandra service-role password in the secrets file must match the value
  supplied to the OpenBao migration.
- Zero-downtime upgrades require three Cassandra members and two replicas per
  service with `maxUnavailable: 0`. Single-member Cassandra installs incur
  downtime during the upgrade.
- OpenBao pods use an `OnDelete` update strategy and must be rotated manually
  after the chart upgrade, as in 0.6.1.
- The environment file gained sections for every service (`api`, `nvctApi`,
  `icms`, `apikeys`, `ess`, `invocation`, `adminIssuerProxy`,
  `functionautoscaler`, `reval`, `llmApiGateway`, `llmRequestRouter`) and the
  secrets file gained a `cassandra` block. Carry your values forward into the
  new templates; do not replace the files wholesale. See the environment file
  section of [Helmfile Installation](/nvcf/self-managed/helmfile-installation).
- Third-party charts (observability components and the cert-manager ACME
  solver image) resolve from upstream repositories by default. Disconnected
  sites must mirror them and set the per-chart `chartRepository` overrides.
- `alpine-k8s` is no longer a customer-mirrored artifact in the manifest.

### Fixes in 1.0.1

- Worker discovery targets the backend router endpoint when backend routing is
  enabled, so LLM workers register against the correct router.
- A final Helmfile stage writes an `nvcf-upgrade-receipt` ConfigMap recording
  the installed stack version. Future upgrades use it to gate unsupported
  paths; a cluster without the receipt is treated as a 0.x install.

Notable fixes since 0.6.1 include correct LLM gateway to API authentication on
default installs, Stargate readiness gated on Pylon registration, aligned
Cassandra application credentials, and the Function Autoscaler honoring its
configured runtime settings.

## Compute Plane Stack 1.0.0

- **Release:** `nvcf-compute-plane-stack` 1.0.0
- **Git tag:** `deploy/stacks/nvcf-compute-plane/v1.0.0`

This release ships NVCA and the NVCA operator 3.10.0, up from 3.0.3 in 0.6.1.
It also includes the `nvca-operator` chart 1.28.0 and a new required chart,
`nvcf-cluster-topology`.

### What is new

**Advanced scheduling:** The stack can install the KAI Scheduler, and NVCA
supports gang scheduling and topology-aware placement for multi-node NVLink
(MNNVL) workloads through KAI and Grove. Functions that need all replicas
co-scheduled, or that need NVLink cliques, can express that requirement
directly. See [KAI Scheduler](/nvcf/compute-plane/kai-scheduler),
[Gang Scheduling](/nvcf/compute-plane/gang-scheduling), and
[Topology-Aware Scheduling](/nvcf/compute-plane/topology-aware-scheduling).

**Helm model caching:** NVCA can serve model artifacts from a shared read-write-many
volume and mount them read-only into workloads, selecting the cache backend
from a storage capability catalog that covers Weka, OCI FSS, and NVMesh. The
feature is off by default and enabled per cluster. See
[Helm Model Caching](/nvcf/compute-plane/helm-model-caching).

**Storage and worker settings as first-class values:** Storage classes, worker
settings, and bring-your-own observability settings are configured through
chart values instead of patches, and missing task-data storage classes fail
fast at deployment time rather than at first use.

**Host isolation:** A new `HostIsolation` cluster attribute schedules one function
instance per node, in addition to the existing `AccountIsolation` mode. The two
are mutually exclusive. See
[Cluster Configuration](/nvcf/compute-plane/cluster-configuration).

**Observability from the compute plane:** The NVCA OpenTelemetry collector adds
Kubernetes object and event collection for the event ledger, per-function
collector resource overrides, log and trace sampling, and correlation IDs on
instance events. NVCA exposes a `GET /info` endpoint.

**Secure transport:** Worker init containers and Helm-based LLM workers trust the
managed transport CA, TLS is validated before workloads render, and insecure
trust bundles are rejected.

**GPU checkpoint and restore.** The NVCA agent integrates with NvSnap to
checkpoint and restore GPU workloads. It is off by default and enabled per
cluster. See the
[NvSnap README](https://github.com/NVIDIA/nvcf/tree/main/src/compute-plane-services/nvsnap).

<Note>
This feature is in beta. Behavior and configuration may change in future releases.
</Note>

**Documentation Update:** This release adds the following guides:

- An Nsight profiling guide explains how to profile function pods with NVIDIA
  Nsight. See [Nsight Profiling](/nvcf/compute-plane/nsight-profiling).
- Consolidated cache documentation covers the model, container, GX, and
  simulation caches. See [Caches Overview](/nvcf/compute-plane/caches-overview).

### Changes that can affect an existing cluster

- Every GPU cluster must be re-registered against the upgraded control plane
  without deleting its registration. Preserve `clusterID` and
  `clusterGroupID`; deleting them orphans deployments and tasks. The source
  cluster must be on NVCA 3.0.3 with a healthy agent.
- Registration inputs changed. In 0.6.1 the environment file had to carry the
  ICMS, ReVal, and NATS endpoints. `make register-cluster` now generates them.
  Any non-empty `selfManaged.*` endpoint values left in an old environment
  file act as overrides and will point the cluster at stale addresses.
- The operator installs a new `ModelCacheBinding` CRD, and the
  `nvcf-cluster-topology` chart is a required component. Cluster-admin rights
  are needed for the CRD install.
- KAI Scheduler is optional, but gang scheduling and topology-aware placement
  do not function without it.
- Resource enforcement is enabled by default; workloads without resource
  requests may be rejected or rescheduled. Review function resource settings
  before upgrading.
- Clusters on NVCA 2.51.0 or earlier must follow the 0.6.0 version-specific
  guidance before moving to 1.0.0.

### Notable fixes

Operator upgrades from 3.0.x no longer hit a nil pointer or lose self-managed
defaults; uninstall completes even while ICMS finalizers are being stripped;
GPU capacity and usage are reported correctly in heartbeats; the webhook TLS
certificate is no longer regenerated on every rollout; workload configuration
persists across Helm updates; and queue credentials refresh after
registration.

## Observability Stack 1.0.0

- **Release:** `nvcf-observability-stack` 1.0.0
- **Git tag:** `deploy/stacks/observability/v1.0.0`

Observability is now a separate stack that is installed and versioned on its
own. In 0.6.1, it shipped inside the control plane stack as the
`nvcf-observability-reference-stack` chart.

### What is new

**Profiles:** A single `observability.profile` value (`disabled`, `control`,
`compute`, or `all`) selects the shared infrastructure and monitors to install.
The control-plane stack defaults to `control` and the compute-plane stack to
`compute`, so a cluster hosting both selects `all`. See
[Observability](/nvcf/observability/observability).

**Bundled metrics pipeline:** The stack installs the Prometheus Operator CRDs, the
OpenTelemetry Operator, an NVCF OpenTelemetry collector with target allocation
and discovery RBAC, VictoriaMetrics as the default metrics store, and the NVCF
default monitors.

**Bring your own backend:** Set `metricsBackend.mode: existing` to use an existing
Prometheus-compatible metrics backend. A PromQL endpoint is required for the `control`
and `all` profiles because the Function Autoscaler queries it; the autoscaler
supports no auth, token, and mTLS. See
[Observability](/nvcf/observability/observability#self-managed-metrics-stack).

**Compute-plane collection:** The `compute` and `all` profiles enable the NVCA
collector and the worker PodMonitor, which covers function and task pods on a
stable `worker-metrics` port.

**Metrics reference:** New reference pages document the metrics that each NVCF
service exposes, covering control plane services, LLM services, per-function
containers, and datastores. Each page lists metric names, types, descriptions,
and labels for use in queries and dashboards. See
[Metrics Overview](/nvcf/observability/metrics-overview).

### Changes that can affect an existing install

- Install the observability stack once per cluster and pick a profile. Do not
  also keep the legacy reference chart from the control-plane bundle.
- Third-party charts (OpenTelemetry Operator, Prometheus Operator CRDs,
  VictoriaMetrics) are pulled from upstream Helm repositories by default.
  Restricted-egress sites must mirror them and set per-chart `chartRepository`
  overrides.
- The bundled VictoriaMetrics instance needs a persistent volume; size it for
  your retention.
- The stack is Early Access. Customers own long-term storage, log collection,
  dashboards, and alerting. Kubernetes support covers the latest minor version
  and the two before it.

### Notable fixes

Worker PodMonitor targets resolve through the named `worker-metrics` port,
Function Autoscaler metrics are scraped, VictoriaMetrics installs after the
Prometheus CRDs so first installs succeed, and custom observability
environments are supported.