NVCF 1.0.1 Release Notes

Starting with NVCF 1.0.1, the three Helm stacks are versioned and released independently.

The current releases are:

These notes describe what changed since NVCF self-hosted 0.6.1, the last legacy release. In 0.6.1, all components shipped as a single bundle.

See the compatibility matrix for the stack releases that are qualified to run together, and the upgrade guide for the ordered upgrade procedure from 0.6.1.

Self-Managed Stack 1.0.1

• Release: nvcf-self-managed-stack 1.0.1
• Git tag: deploy/stacks/self-managed/v1.0.1

1.0.1 is a patch release on top of 1.0.0. It includes the fixes listed in Fixes in 1.0.1.

What is new

LLM functions: The control plane now ships the LLM API Gateway and the LLM Request Router (Stargate and Pylon) as stack components. Customers can expose OpenAI-compatible chat completion endpoints for LLM functions, route requests across workers with configurable load-balancing algorithms and affinity, and serve LLM routes on custom hosts through the vanity gateway. See LLM Gateway, LLM Function Enablement, and LLM Request Router Load Balancing.

Function Autoscaler: A dedicated autoscaler scales function instances from invocation metrics and queue depth, configured per function through the NVCF API. It replaces static instance counts for most workloads. See Configure Autoscaling and Function Autoscaling Overview.

Managed PKI and encrypted transport: A managed ClusterIssuer backed by OpenBao issues certificates for LLM traffic, LLM gRPC ingress uses a dedicated HTTPS listener, and the router and worker components reload certificates without a restart. Plaintext transport is now an explicit development opt-in. NATS can be configured with server-side TLS for split-plane deployments where workers reach NATS over a load balancer. See Transport TLS Rotation.

NVCF UI: An optional administrative UI ships with the stack, disabled by default. See NVCF UI.

Operability: PodDisruptionBudgets can be enabled for every control-plane chart, API issuer and JWKS endpoints are overridable for custom identity setups, gateway routes can be enabled per service, and every control-plane service exposes a GET /info endpoint reporting its version. ICMS gains select-and-terminate instance controls and per-reservation backup opt-out. See Pod Disruption Budgets.

Documentation Update: This release adds two new guides:

• A multi-tenancy guide describes the two supported invocation models, dedicated endpoint and shared endpoint, and the isolation controls available at each layer, from namespace and network to node and container runtime. See Multi-Tenancy.
• A quota management runbook explains how operators raise account limits after installation, such as the maximum number of functions or tasks. See Quota Management.

Changes that can affect an existing install

• Observability is no longer bundled in the control-plane stack. It is a separately installed and versioned stack; see the Observability section.
• Service lineages changed. SIS is replaced by ICMS (icms-service-oss), and the ESS and Notary images are renamed (nvcf-ess, nvcf-notary). Version numbers that appear to go down in the manifest reflect these new lineages, not downgrades. Image mirrors must include the new names.
• Cassandra moves from 5.0.8 to 5.0.9 and OpenBao from 2.5.5 to 2.6.2, each with schema or storage migrations that must run in order. The default Cassandra migration image runs every keyspace; the staged upgrade path requires the bounded per-keyspace Jobs described in the upgrade guide. The Cassandra service-role password in the secrets file must match the value supplied to the OpenBao migration.
• Zero-downtime upgrades require three Cassandra members and two replicas per service with maxUnavailable: 0. Single-member Cassandra installs incur downtime during the upgrade.
• OpenBao pods use an OnDelete update strategy and must be rotated manually after the chart upgrade, as in 0.6.1.
• The environment file gained sections for every service (api, nvctApi, icms, apikeys, ess, invocation, adminIssuerProxy, functionautoscaler, reval, llmApiGateway, llmRequestRouter) and the secrets file gained a cassandra block. Carry your values forward into the new templates; do not replace the files wholesale. See the environment file section of Helmfile Installation.
• Third-party charts (observability components and the cert-manager ACME solver image) resolve from upstream repositories by default. Disconnected sites must mirror them and set the per-chart chartRepository overrides.
• alpine-k8s is no longer a customer-mirrored artifact in the manifest.

Fixes in 1.0.1

• Worker discovery targets the backend router endpoint when backend routing is enabled, so LLM workers register against the correct router.
• A final Helmfile stage writes an nvcf-upgrade-receipt ConfigMap recording the installed stack version. Future upgrades use it to gate unsupported paths; a cluster without the receipt is treated as a 0.x install.

Notable fixes since 0.6.1 include correct LLM gateway to API authentication on default installs, Stargate readiness gated on Pylon registration, aligned Cassandra application credentials, and the Function Autoscaler honoring its configured runtime settings.

Compute Plane Stack 1.0.0

• Release: nvcf-compute-plane-stack 1.0.0
• Git tag: deploy/stacks/nvcf-compute-plane/v1.0.0

This release ships NVCA and the NVCA operator 3.10.0, up from 3.0.3 in 0.6.1. It also includes the nvca-operator chart 1.28.0 and a new required chart, nvcf-cluster-topology.

What is new

Advanced scheduling: The stack can install the KAI Scheduler, and NVCA supports gang scheduling and topology-aware placement for multi-node NVLink (MNNVL) workloads through KAI and Grove. Functions that need all replicas co-scheduled, or that need NVLink cliques, can express that requirement directly. See KAI Scheduler, Gang Scheduling, and Topology-Aware Scheduling.

Helm model caching: NVCA can serve model artifacts from a shared read-write-many volume and mount them read-only into workloads, selecting the cache backend from a storage capability catalog that covers Weka, OCI FSS, and NVMesh. The feature is off by default and enabled per cluster. See Helm Model Caching.

Storage and worker settings as first-class values: Storage classes, worker settings, and bring-your-own observability settings are configured through chart values instead of patches, and missing task-data storage classes fail fast at deployment time rather than at first use.

Host isolation: A new HostIsolation cluster attribute schedules one function instance per node, in addition to the existing AccountIsolation mode. The two are mutually exclusive. See Cluster Configuration.

Observability from the compute plane: The NVCA OpenTelemetry collector adds Kubernetes object and event collection for the event ledger, per-function collector resource overrides, log and trace sampling, and correlation IDs on instance events. NVCA exposes a GET /info endpoint.

Secure transport: Worker init containers and Helm-based LLM workers trust the managed transport CA, TLS is validated before workloads render, and insecure trust bundles are rejected.

GPU checkpoint and restore. The NVCA agent integrates with NvSnap to checkpoint and restore GPU workloads. It is off by default and enabled per cluster. See the NvSnap README.

Documentation Update: This release adds the following guides:

• An Nsight profiling guide explains how to profile function pods with NVIDIA Nsight. See Nsight Profiling.
• Consolidated cache documentation covers the model, container, GX, and simulation caches. See Caches Overview.

Changes that can affect an existing cluster

• Every GPU cluster must be re-registered against the upgraded control plane without deleting its registration. Preserve clusterID and clusterGroupID; deleting them orphans deployments and tasks. The source cluster must be on NVCA 3.0.3 with a healthy agent.
• Registration inputs changed. In 0.6.1 the environment file had to carry the ICMS, ReVal, and NATS endpoints. make register-cluster now generates them. Any non-empty selfManaged.* endpoint values left in an old environment file act as overrides and will point the cluster at stale addresses.
• The operator installs a new ModelCacheBinding CRD, and the nvcf-cluster-topology chart is a required component. Cluster-admin rights are needed for the CRD install.
• KAI Scheduler is optional, but gang scheduling and topology-aware placement do not function without it.
• Resource enforcement is enabled by default; workloads without resource requests may be rejected or rescheduled. Review function resource settings before upgrading.
• Clusters on NVCA 2.51.0 or earlier must follow the 0.6.0 version-specific guidance before moving to 1.0.0.

Notable fixes

Operator upgrades from 3.0.x no longer hit a nil pointer or lose self-managed defaults; uninstall completes even while ICMS finalizers are being stripped; GPU capacity and usage are reported correctly in heartbeats; the webhook TLS certificate is no longer regenerated on every rollout; workload configuration persists across Helm updates; and queue credentials refresh after registration.

Observability Stack 1.0.0

• Release: nvcf-observability-stack 1.0.0
• Git tag: deploy/stacks/observability/v1.0.0

Observability is now a separate stack that is installed and versioned on its own. In 0.6.1, it shipped inside the control plane stack as the nvcf-observability-reference-stack chart.

What is new

Profiles: A single observability.profile value (disabled, control, compute, or all) selects the shared infrastructure and monitors to install. The control-plane stack defaults to control and the compute-plane stack to compute, so a cluster hosting both selects all. See Observability.

Bundled metrics pipeline: The stack installs the Prometheus Operator CRDs, the OpenTelemetry Operator, an NVCF OpenTelemetry collector with target allocation and discovery RBAC, VictoriaMetrics as the default metrics store, and the NVCF default monitors.

Bring your own backend: Set metricsBackend.mode: existing to use an existing Prometheus-compatible metrics backend. A PromQL endpoint is required for the control and all profiles because the Function Autoscaler queries it; the autoscaler supports no auth, token, and mTLS. See Observability.

Compute-plane collection: The compute and all profiles enable the NVCA collector and the worker PodMonitor, which covers function and task pods on a stable worker-metrics port.

Metrics reference: New reference pages document the metrics that each NVCF service exposes, covering control plane services, LLM services, per-function containers, and datastores. Each page lists metric names, types, descriptions, and labels for use in queries and dashboards. See Metrics Overview.

Changes that can affect an existing install

• Install the observability stack once per cluster and pick a profile. Do not also keep the legacy reference chart from the control-plane bundle.
• Third-party charts (OpenTelemetry Operator, Prometheus Operator CRDs, VictoriaMetrics) are pulled from upstream Helm repositories by default. Restricted-egress sites must mirror them and set per-chart chartRepository overrides.
• The bundled VictoriaMetrics instance needs a persistent volume; size it for your retention.
• The stack is Early Access. Customers own long-term storage, log collection, dashboards, and alerting. Kubernetes support covers the latest minor version and the two before it.

Notable fixes

Worker PodMonitor targets resolve through the named worker-metrics port, Function Autoscaler metrics are scraped, VictoriaMetrics installs after the Prometheus CRDs so first installs succeed, and custom observability environments are supported.