CSP End-to-End Example Installation (Helmfile)

This page provides a complete end-to-end example for installing NVCF on pre-provisioned managed Kubernetes clusters using the split Helmfile bundles. It covers both topologies:

• Single-cluster: the control plane and the NVCA operator run on the same cluster.
• Multi-cluster: the control plane runs on one cluster, and the NVCA operator is registered and installed on a separate GPU (compute) cluster.

The commands are written to work on any cloud provider (CSP). Amazon EKS is used as the worked example. The only provider-specific pieces are the load balancer annotations on the Gateway, the storageClass name, and the kubectl context names. Substitute the equivalents for GKE, AKS, or on-prem.

For a deeper reference on each release and on values, see Helmfile Installation. For pulling and mirroring the bundles and images, see Image Mirroring.

$
git clone https://github.com/nvidia/nvcf.git

Installation order

The order matters. Each step produces an input that the next step needs. The load balancer address, in particular, must exist before you configure the environment file, because it becomes global.domain and the NVCA Host headers.

1. Install the Gateway          -> external load balancer address
2. Configure the environment    -> environments/<env>.yaml + secrets/<env>-secrets.yaml
3. Install the control plane    -> control-plane services + HTTPRoutes
4. Author the nvcf-cli config   -> points at the load balancer address
5. Register the GPU cluster     -> registration values file
6. Install the NVCA operator    -> agent connects back to the control plane
7. Verify the agent is healthy

Single-cluster and multi-cluster share steps 1 through 4. Step 5 requires a compute-plane environment file for both topologies. Only the cluster target and control-plane endpoint values differ. See Single-cluster vs multi-cluster.

Prerequisites

Tools

Install on the machine you run these commands from:

• kubectl
• helm (3.x)
• helmfile
• nvcf-cli

Clusters

The clusters must be provisioned before you start. This guide does not create them. Each cluster needs:

• A default-capable StorageClass with dynamic provisioning. On EKS this is gp3, backed by the EBS CSI driver. Substitute your provider’s class name.
• The compute (GPU) cluster needs a GPU operator (real or the fake GPU operator for non-GPU validation). See Fake GPU Operator.
• The compute cluster needs the SMB CSI driver (smb.csi.k8s.io). NVCA uses it for shared model cache storage that function worker pods mount. Install and verify the driver before registering the GPU cluster. See the Self-Managed Clusters prerequisites for the installation command.

Both clusters must be reachable through kubectl contexts:

$
kubectl --context "${CONTROL_PLANE_CONTEXT}" get nodes -o name
$
# Multi-cluster only:
$
kubectl --context "${COMPUTE_CONTEXT}" get nodes -o name

Environment variables

Set these once. In single-cluster, COMPUTE_CONTEXT equals CONTROL_PLANE_CONTEXT.

$
# Cluster targeting
$
export CONTROL_PLANE_CONTEXT="<kubectl-context-of-control-plane-cluster>"
$
export COMPUTE_CONTEXT="<kubectl-context-of-gpu-cluster>"   # = control plane in single-cluster
$
export CLUSTER_NAME="<name-to-register-the-gpu-cluster-as>"
$
export CLUSTER_REGION="<region-label>"                      # e.g. us-east-1
$
$
# Bundle environment file name (you create environments/<env>.yaml below)
$
export HELMFILE_ENV="eks"                                   # single-cluster example
$
# export HELMFILE_ENV="eks-multi"                           # multi-cluster example
$
$
# Repository path the bundles use for charts and images
$
export REPOSITORY="<your-ngc-org>/<your-ngc-team>"          # or your mirror path
$
$
# NGC credential used for chart/image pulls and the dockerconfig secret
$
export NGC_API_KEY="<your-ngc-api-key>"
$
$
# Path to the built nvcf-cli binary
$
export NVCF_CLI="<path-to>/nvcf-cli"
$
$
# Storage class for the control-plane bundle (provider specific)
$
export STORAGE_CLASS="gp3"

Log in to the chart and image registry

The bundles pull OCI charts through Helm, so host-side registry auth must exist before any helmfile sync:

$
printf '%s' "${NGC_API_KEY}" | helm registry login nvcr.io --username '$oauthtoken' --password-stdin

Step 1: Install the Gateway and capture the load balancer address

Install the Gateway on the control-plane cluster by following the Gateway quickstart. It installs the Gateway API CRDs, the Envoy Gateway controller, the GatewayClass, and the nvcf-gateway Gateway, and exports GATEWAY_ADDR. Run it against ${CONTROL_PLANE_CONTEXT}.

After the quickstart, confirm the address is set:

$
test -n "${GATEWAY_ADDR}"
$
echo "GATEWAY_ADDR=${GATEWAY_ADDR}"

The gRPC listener is on this same Gateway at port 10081, so the gRPC address is ${GATEWAY_ADDR}:10081 (used in the nvcf-cli config below).

Step 2: Configure the control-plane environment and secrets files

This step produces two files in the nvcf-self-managed-stack bundle: the environment values file environments/<env>.yaml (copied from base.yaml) and the secrets file secrets/<env>-secrets.yaml (copied from secrets.yaml.template). Both are required before the install.

From the nvcf-self-managed-stack directory, copy the base template. The file name (<env>) must match HELMFILE_ENV.

$
cd <path to nvcf-self-managed-stack>/
$
cp environments/base.yaml "environments/${HELMFILE_ENV}.yaml"

Edit environments/${HELMFILE_ENV}.yaml. Every field is explained inline below. Lines marked CHANGE must be updated for your cluster; replace the ${...} values with the literals you exported earlier.

1
global:
2
  domain: "${GATEWAY_ADDR}"        # CHANGE: from "localhost". Builds HTTPRoute hostnames (api.<domain>, etc.)
3
4
  helm:
5
    sources:
6
      registry: "nvcr.io"          # OCI registry NVCF charts are pulled from. Change only if you mirror.
7
      repository: "${REPOSITORY}"  # CHANGE: from "YOUR_ORG/YOUR_TEAM". NGC org/team or mirror path.
8
9
  imagePullSecrets:
10
    - name: nvcr-pull-secret       # CHANGE: from []. Pull secret applied to all workloads (created below).
11
12
  image:
13
    registry: nvcr.io              # Container image registry. Change only if you mirror.
14
    repository: "${REPOSITORY}"    # CHANGE: from "YOUR_ORG/YOUR_TEAM". NGC org/team or mirror path.
15
16
  workerEndpoints:
17
    essServiceURL: ""              # Control-plane endpoints advertised into worker pods. Empty = in-cluster defaults.
18
    invocationServiceURL: ""       # Empty = in-cluster default.
19
20
  nodeSelectors:
21
    enabled: false                 # Pin system workloads to labeled node pools. Leave false unless nodes are labeled.
22
    vault:
23
      key: nvcf.nvidia.com/workload
24
      value: vault                 # Node label for the vault pool (applied only when enabled: true).
25
    cassandra:
26
      key: nvcf.nvidia.com/workload
27
      value: cassandra             # Node label for the cassandra pool.
28
    controlplane:
29
      key: nvcf.nvidia.com/workload
30
      value: control-plane         # Node label for the control-plane pool.
31
32
  tolerations:
33
    enabled: false                 # Tolerations for tainted node pools. Leave false unless nodes are tainted.
34
    all: []                        # Tolerations applied to all system workloads when enabled.
35
36
  storageClass: "${STORAGE_CLASS}" # CHANGE: from "". Dynamic-provisioning StorageClass (gp3 on EKS).
37
  storageSize: "10Gi"              # Per-PVC size. Default works; raise for larger control-plane data (Cassandra).
38
39
  observability:
40
    tracing:
41
      enabled: false               # OpenTelemetry trace export. Set true plus a collector endpoint to enable.
42
      collectorEndpoint: ""        # OTLP collector endpoint when tracing is enabled.
43
      collectorPort: 4317          # OTLP collector port.
44
      collectorProtocol: http      # OTLP protocol (http or grpc).
45
    metrics:
46
      enabled: false               # Prometheus metrics. Set true if you run the Prometheus Operator.
47
48
accounts:
49
  limits:
50
    maxFunctions: 10               # Max functions per account.
51
    maxTasks: 10                   # Max tasks per account.
52
    maxTelemetries: 10             # Max telemetry endpoints per account.
53
    maxRegistryCreds: 10           # Max registry credentials per account.
54
55
nats:
56
  enabled: true                    # Deploy the NATS messaging layer. Keep true; the control plane depends on it.
57
58
cassandra:
59
  enabled: true                    # Deploy Cassandra. Keep true.
60
  resourcesPreset: "xlarge"        # CPU/memory preset. Do not use small for cloud installs (OOM on first boot).
61
62
certManager:
63
  enabled: true                    # Install cert-manager for the self-managed PKI. Keep true.
64
65
openbao:
66
  enabled: true                    # Deploy OpenBao (Vault) for secrets. Keep true.
67
  migrations:
68
    issuerDiscovery:
69
      enabled: true                # CHANGE: from false. Discover the cluster OIDC issuer. Required on managed Kubernetes.
70
  injector:
71
    replicas: 2                    # OpenBao injector replicas (HA). Set 1 on single-node / minimal pools.
72
73
addons:
74
  lls:
75
    enabled: false                 # Low Latency Streaming (TURN) addon. Optional.
76
  llm:
77
    enabled: false                 # LLM gateway + request router (Stargate). Optional.
78
    pki:
79
      enabled: false               # OpenBao-issued QUIC TLS for the request router. Optional.
80
      allowedDomains: ""           # Required only when llm.pki.enabled: comma-separated DNS suffixes.
81
      dnsNames: []                 # Required only when llm.pki.enabled: SANs on the issued certificate.
82
  vanityGateway:
83
    enabled: false                 # Vanity and OpenAI-compatible invocation routes. Optional.
84
    replicaCount: 2                # Vanity gateway replicas (applied only when enabled).
85
86
stateMetrics:
87
  enabled: false                   # State-metrics exporter. Optional.
88
  serviceMonitor:
89
    enabled: false                 # ServiceMonitor for the exporter. Requires the Prometheus Operator.
90
91
rateLimiter:
92
  enabled: false                   # Invocation rate limiter. Optional.
93
  replicaCount: 1                  # Rate limiter replicas (applied only when enabled).
94
95
ingress:
96
  gatewayApi:
97
    enabled: true                  # Enable Gateway API ingress. Keep true.
98
    controllerNamespace: envoy-gateway-system  # CHANGE: from "". Namespace of the gateway controller.
99
    routes:
100
      nvcfApi:
101
        routeAnnotations: {}       # Optional per-route annotations for the NVCF API route.
102
      nvctApi:
103
        routeAnnotations: {}       # Optional annotations for the NVCT API route.
104
      apiKeys:
105
        routeAnnotations: {}       # Optional annotations for the api-keys route.
106
      invocation:
107
        routeAnnotations: {}       # Optional annotations for the invocation route.
108
      llmInvocation:
109
        routeAnnotations: {}       # Optional annotations for the LLM invocation route.
110
      vanityGateway:
111
        hostnames: []              # Override vanity hostnames. Default is vanity.<domain>.
112
        routeAnnotations: {}       # Optional annotations for the vanity route.
113
      grpc:
114
        routeAnnotations: {}       # Optional annotations for the gRPC route.
115
      nats:
116
        enabled: true              # CHANGE: from false. Create the NATS route (the NVCA agent needs it).
117
        routeAnnotations: {}       # Optional annotations for the NATS route.
118
      ess:
119
        enabled: true              # CHANGE: from false. Create the ESS route (the nvcf worker container needs it).
120
        routeAnnotations: {}       # Optional annotations for the ESS route.
121
    gateways:
122
      shared:
123
        name: nvcf-gateway         # CHANGE: from "". Gateway the HTTP routes attach to.
124
        namespace: envoy-gateway   # CHANGE: from "". Namespace of that Gateway.
125
      grpc:
126
        name: nvcf-gateway         # CHANGE: from "". Gateway the gRPC route attaches to.
127
        namespace: envoy-gateway   # CHANGE: from "".
128
      nats:
129
        name: nvcf-gateway         # CHANGE: from "". Gateway the NATS route attaches to (when routes.nats.enabled).
130
        namespace: envoy-gateway   # CHANGE: from "".
131
        listenerName: nats         # Gateway listener name for NATS. Keep as nats.

Create the registry pull secret

The control-plane charts reference an image pull secret named nvcr-pull-secret in each namespace. Create it in every control-plane namespace:

$
for ns in cassandra-system nats-system nvcf api-keys ess sis vault-system cert-manager; do
$
  kubectl --context "${CONTROL_PLANE_CONTEXT}" create namespace "${ns}" \
>
    --dry-run=client -o yaml | kubectl --context "${CONTROL_PLANE_CONTEXT}" apply -f -
$
  kubectl --context "${CONTROL_PLANE_CONTEXT}" create secret docker-registry nvcr-pull-secret \
>
    --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password="${NGC_API_KEY}" \
>
    -n "${ns}" --dry-run=client -o yaml | kubectl --context "${CONTROL_PLANE_CONTEXT}" apply -f -
$
done

Create the bundle secrets file

The control-plane bundle reads a secrets file for the OpenBao migration and API account bootstrap. Create it from the template and set the base64 dockerconfig credential:

$
cp secrets/secrets.yaml.template "secrets/${HELMFILE_ENV}-secrets.yaml"
$
$
DOCKER_CRED_B64=$(printf '%s' '$oauthtoken:'"${NGC_API_KEY}" | base64 | tr -d '\n')
$
# Replace every REPLACE_WITH_BASE64_DOCKER_CREDENTIAL in the secrets file with ${DOCKER_CRED_B64}.
$
sed -i.bak "s|REPLACE_WITH_BASE64_DOCKER_CREDENTIAL|${DOCKER_CRED_B64}|g" \
>
  "secrets/${HELMFILE_ENV}-secrets.yaml"
$
rm "secrets/${HELMFILE_ENV}-secrets.yaml.bak"

Step 3: Install the control plane

Run from the nvcf-self-managed-stack bundle directory:

$
cd <path to nvcf-self-managed-stack>/
$
kubectl config use-context "${CONTROL_PLANE_CONTEXT}"
$
make install HELMFILE_ENV="${HELMFILE_ENV}"

Verify the releases are deployed:

$
helm list --all-namespaces --kube-context "${CONTROL_PLANE_CONTEXT}"

Expected releases include nats, cert-manager, openbao-server, cassandra, api-keys, sis, api, nvct-api, invocation-service, grpc-proxy, ess-api, notary-service, admin-issuer-proxy, reval, nats-auth-callout-service, and ingress.

Confirm global.domain propagated into the API HTTPRoute hostname:

$
kubectl --context "${CONTROL_PLANE_CONTEXT}" get httproute nvcf-api -n envoy-gateway \
>
  -o jsonpath='{.spec.hostnames[0]}'
$
# Expected: api.${GATEWAY_ADDR}

Step 4: Author the nvcf-cli config

Create nvcf-cli.yaml pointing at the load balancer address. The static fields are the same across self-hosted installs; only the URL and Host fields are derived from GATEWAY_ADDR.

$
cat > nvcf-cli.yaml <<EOF
$
# Admin token issuer config (chart-level defaults; identical across installs)
$
api_keys_service_id: "nvidia-cloud-functions-ncp-service-id-aketm"
$
api_keys_issuer_service: "nvcf-api"
$
api_keys_owner_id: "svc@nvcf-api.local"
$
client_id: "nvcf-default"
$
$
# Endpoints, derived from the gateway load balancer address
$
base_http_url: "http://${GATEWAY_ADDR}"
$
invoke_url: "http://${GATEWAY_ADDR}"
$
base_grpc_url: "${GATEWAY_ADDR}:10081"
$
api_keys_service_url: "http://${GATEWAY_ADDR}"
$
icms_url: "http://${GATEWAY_ADDR}"
$
api_host: "api.${GATEWAY_ADDR}"
$
api_keys_host: "api-keys.${GATEWAY_ADDR}"
$
invoke_host: "invocation.${GATEWAY_ADDR}"
$
icms_host: "sis.${GATEWAY_ADDR}"
$
EOF
$
$
export NVCF_CLI_CONFIG="$(pwd)/nvcf-cli.yaml"

Step 5: Register the GPU cluster and install the NVCA operator

This is where single-cluster and multi-cluster diverge. Pick the matching section. Run both from the source repository root.

$
cd <path-to-nvcf-repository>

Create the compute-plane environment file for either topology. The compute-plane Makefile reads deploy/stacks/nvcf-compute-plane/environments/${HELMFILE_ENV}.yaml, and the selfManaged values tell the NVCA agent how to reach the control plane. The sections below use different cluster targets and endpoint values.

$
cp deploy/stacks/nvcf-compute-plane/environments/base.yaml \
>
  "deploy/stacks/nvcf-compute-plane/environments/${HELMFILE_ENV}.yaml"

Set these keys in deploy/stacks/nvcf-compute-plane/environments/${HELMFILE_ENV}.yaml:

1
global:
2
  helm:
3
    sources:
4
      repository: "${REPOSITORY}"
5
  image:
6
    repository: "${REPOSITORY}"
7
  imagePullSecrets:
8
    - name: nvcr-pull-secret
9
  nvcaOperator:
10
    selfManaged:
11
      icmsServiceURL: "http://${GATEWAY_ADDR}"
12
      icmsServiceHostHeaderOverride: "sis.${GATEWAY_ADDR}"
13
      revalServiceURL: "http://${GATEWAY_ADDR}"
14
      revalServiceHostHeaderOverride: "reval.${GATEWAY_ADDR}"
15
      natsURL: "nats://${GATEWAY_ADDR}:4222"
16
      natsHostOverride: "nats.${GATEWAY_ADDR}"

Then follow the matching subsection below.

Single-cluster

The GPU cluster is the same cluster as the control plane, so registration uses the current context. Create the pull secret in nvca-operator; the operator propagates it to the managed namespaces after installation:

$
kubectl --context "${CONTROL_PLANE_CONTEXT}" create namespace nvca-operator \
>
  --dry-run=client -o yaml | kubectl --context "${CONTROL_PLANE_CONTEXT}" apply -f -
$
kubectl --context "${CONTROL_PLANE_CONTEXT}" create secret docker-registry nvcr-pull-secret \
>
  --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password="${NGC_API_KEY}" \
>
  -n nvca-operator --dry-run=client -o yaml | kubectl --context "${CONTROL_PLANE_CONTEXT}" apply -f -

Register, then install:

$
kubectl config use-context "${CONTROL_PLANE_CONTEXT}"
$
$
make -C deploy/stacks/nvcf-compute-plane register-cluster \
>
  CLUSTER_NAME="${CLUSTER_NAME}" \
>
  NCA_ID=nvcf-default \
>
  CLUSTER_REGION="${CLUSTER_REGION}" \
>
  ICMS_URL="http://${GATEWAY_ADDR}" \
>
  NVCF_CLI="${NVCF_CLI}" \
>
  NVCF_CLI_CONFIG="${NVCF_CLI_CONFIG}"
$
$
make -C deploy/stacks/nvcf-compute-plane install \
>
  CLUSTER_NAME="${CLUSTER_NAME}" \
>
  HELMFILE_ENV="${HELMFILE_ENV}" \
>
  NVCF_CLI="${NVCF_CLI}" \
>
  NVCF_CLI_CONFIG="${NVCF_CLI_CONFIG}"

Multi-cluster

The NVCA operator installs on a separate compute cluster. Two extra concerns:

1. Registration must discover the OIDC issuer and JWKS from the compute cluster, not the control-plane cluster. Switch the context to the compute cluster and pass a compute-scoped kubeconfig to register-cluster.
2. The compute-plane environment file must carry the control-plane service URLs and Host headers so the agent on the compute cluster can reach the control plane through the Gateway.

You already created the compute-plane environment file with the selfManaged values above. Create the pull secret in nvca-operator on the compute cluster; the operator propagates it to the managed namespaces after installation. Also create a compute-scoped kubeconfig:

$
kubectl --context "${COMPUTE_CONTEXT}" create namespace nvca-operator \
>
  --dry-run=client -o yaml | kubectl --context "${COMPUTE_CONTEXT}" apply -f -
$
kubectl --context "${COMPUTE_CONTEXT}" create secret docker-registry nvcr-pull-secret \
>
  --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password="${NGC_API_KEY}" \
>
  -n nvca-operator --dry-run=client -o yaml | kubectl --context "${COMPUTE_CONTEXT}" apply -f -
$
$
kubectl --context "${COMPUTE_CONTEXT}" config view --raw --minify --flatten > compute-kubeconfig.yaml
$
export COMPUTE_KUBECONFIG="$(pwd)/compute-kubeconfig.yaml"

Register with the compute context active, then install onto the compute cluster:

$
kubectl config use-context "${COMPUTE_CONTEXT}"
$
$
make -C deploy/stacks/nvcf-compute-plane register-cluster \
>
  CLUSTER_NAME="${CLUSTER_NAME}" \
>
  NCA_ID=nvcf-default \
>
  CLUSTER_REGION="${CLUSTER_REGION}" \
>
  ICMS_URL="http://${GATEWAY_ADDR}" \
>
  KUBECONFIG_FILE="${COMPUTE_KUBECONFIG}" \
>
  NVCF_CLI="${NVCF_CLI}" \
>
  NVCF_CLI_CONFIG="${NVCF_CLI_CONFIG}"
$
$
make -C deploy/stacks/nvcf-compute-plane install \
>
  CLUSTER_NAME="${CLUSTER_NAME}" \
>
  HELMFILE_ENV="${HELMFILE_ENV}" \
>
  KUBECONFIG_FILE="${COMPUTE_KUBECONFIG}" \
>
  NVCF_CLI="${NVCF_CLI}" \
>
  NVCF_CLI_CONFIG="${NVCF_CLI_CONFIG}"

Step 6: Verify the agent is healthy

Confirm the NVCA operator is deployed and the backend reports the agent healthy. Use the compute context (equal to the control-plane context in single-cluster):

$
helm list -n nvca-operator --kube-context "${COMPUTE_CONTEXT}"
$
# Expected: nvca-operator deployed
$
$
kubectl rollout status deployment/nvca-operator -n nvca-operator \
>
  --context "${COMPUTE_CONTEXT}" --timeout=10m
$
$
kubectl wait nvcfbackend "${CLUSTER_NAME}" -n nvca-operator \
>
  --context "${COMPUTE_CONTEXT}" \
>
  --for=jsonpath='{.status.agentStatus}'=healthy --timeout=10m

Confirm that the operator propagated the pull secret to nvca-system:

$
kubectl --context "${COMPUTE_CONTEXT}" get secret nvcr-pull-secret -n nvca-system

The agent reaching healthy confirms registration and the Host-header wiring are correct.

Single-cluster vs multi-cluster

Known limitations

Cross-cluster function execution is not supported by the current NVCA operator chart. Registration and agent health work across clusters and regions, but the agent injects the control plane’s in-cluster service names into worker pods (for example api.nvcf.svc.cluster.local), which do not resolve from a separate compute cluster. Functions deployed to a remote compute cluster cannot fetch artifacts and will error. The selfManaged.*Override values configure the agent’s own connections, not the worker pod environment. Use single-cluster for end-to-end function execution until worker FQDNs can be externalized.

Troubleshooting

• Gateway never becomes Programmed: check the load balancer annotations match your provider and that the controller pod in envoy-gateway-system is running.
• make install reports “Registration values not found”: run make register-cluster first, in the same directory, with the same CLUSTER_NAME.
• Compute agent fails with no matching key(s) found: you registered with the wrong context active. Switch to the compute context and re-run make register-cluster.

See Troubleshooting for more.