NVCA Configuration

This page documents NVCA configuration options that apply to NGC-managed and Helm-managed deployment modes. Where the configuration steps differ per mode, instructions are provided for each.

For mode-specific lifecycle operations (registration, upgrades, deregistration), see:

• NGC-Managed Clusters for clusters managed through the NGC UI
• Helm-Managed Clusters for clusters managed through Helm values

Advanced Settings

See below for descriptions of all capability options in the “Advanced Settings” section of the cluster configuration.

Cluster Features

Cluster Features allow enabling specific features on the cluster. Note that for customer-managed clusters (registered via the Cluster Agent) Dynamic GPU Discovery is enabled by default. For NVIDIA managed clusters, Collect Function Logs is also enabled by default.

See below for descriptions of all cluster features in the cluster configuration.

Caching Support

Enabling caching for models, resources and containers is recommended for optimal performance. You must create StorageClass configurations for caching within your cluster to fully enable “Caching Support” with the Cluster Agent. See examples below.

StorageClass Configurations in GCP

1
  kind: StorageClass
2
  apiVersion: storage.k8s.io/v1
3
  metadata:
4
      name: nvcf-sc
5
  provisioner: pd.csi.storage.gke.io
6
  allowVolumeExpansion: true
7
  volumeBindingMode: Immediate
8
  reclaimPolicy: Retain
9
  parameters:
10
      type: pd-ssd
11
      csi.storage.k8s.io/fstype: xfs

1
  kind: StorageClass
2
  apiVersion: storage.k8s.io/v1
3
  metadata:
4
      name: nvcf-cc-sc
5
  provisioner: pd.csi.storage.gke.io
6
  allowVolumeExpansion: true
7
  volumeBindingMode: Immediate
8
  reclaimPolicy: Retain
9
  parameters:
10
      type: pd-ssd
11
      csi.storage.k8s.io/fstype: xfs

StorageClass Configurations in Azure

1
  kind: StorageClass
2
  apiVersion: storage.k8s.io/v1
3
  metadata:
4
      name: nvcf-sc
5
  provisioner: file.csi.azure.com
6
  allowVolumeExpansion: true
7
  volumeBindingMode: Immediate
8
  reclaimPolicy: Retain
9
  parameters:
10
      skuName: Standard_LRS
11
      csi.storage.k8s.io/fstype: xfs

1
  kind: StorageClass
2
  apiVersion: storage.k8s.io/v1
3
  metadata:
4
      name: nvcf-cc-sc
5
  provisioner: file.csi.azure.com
6
  allowVolumeExpansion: true
7
  volumeBindingMode: Immediate
8
  reclaimPolicy: Retain
9
  parameters:
10
      skuName: Standard_LRS
11
      csi.storage.k8s.io/fstype: xfs

StorageClass Configurations in Oracle Cloud

1
  kind: StorageClass
2
  apiVersion: storage.k8s.io/v1
3
  metadata:
4
    name: nvcf-sc
5
  provisioner: blockvolume.csi.oraclecloud.com
6
  allowVolumeExpansion: true
7
  volumeBindingMode: Immediate
8
  reclaimPolicy: Retain
9
  parameters:
10
    csi.storage.k8s.io/fstype: xfs

1
  kind: StorageClass
2
  apiVersion: storage.k8s.io/v1
3
  metadata:
4
    name: nvcf-cc-sc
5
  provisioner: blockvolume.csi.oraclecloud.com
6
  allowVolumeExpansion: true
7
  volumeBindingMode: Immediate
8
  reclaimPolicy: Retain
9
  parameters:
10
    csi.storage.k8s.io/fstype: xfs

Apply the StorageClass Configurations

Save the StorageClass template to files nvcf-sc.yaml and nvcf-cc-sc.yaml and apply them as:

$
  kubectl create -f nvcf-sc.yaml
$
  kubectl create -f nvcf-cc-sc.yaml

Override the Default Mount Options for Cache Volumes

Cluster Agent with caching support by default will enable linux mount-options with ro,norecovery,nouuid.

If the CSI Driver in the cluster doesn’t support mount options then you may apply the following command on the cluster to disable the mount options

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"
$
  kubectl patch nvcfbackend -n nvca-operator "$nvcf_cluster_name" --type=merge -p '{"spec":{"overrides":{"featureGate":{"cacheCSIVolumeMountOptionsConfig":{"disabled":true}}}}}'

If you want to update the mount-options to a different value for example: ro,norecovery. You may use the following command. Replace these options with desired value as dictated by CSI Driver Volume Mount Options.

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"
$
  kubectl patch nvcfbackend -n nvca-operator "$nvcf_cluster_name" --type=merge -p '{"spec":{"overrides":{"featureGate":{"cacheCSIVolumeMountOptionsConfig":{"disabled":false}}}}}'
$
  kubectl patch nvcfbackend -n nvca-operator "$nvcf_cluster_name" --type=merge -p '{"spec":{"overrides":{"featureGate":{"cacheCSIVolumeMountOptionsConfig":{"mountOptions":"ro,norecovery"}}}}}'

Account-Isolated Clusters

Clusters with the AccountIsolation attribute have enhanced isolation between workloads, ensuring that function and task instances run on nodes isolated by NCAId. This is particularly important for customers with strict security requirements or those who want to ensure complete separation of workloads at the account level.

NVLink-optimized Clusters

Clusters with MNNVL GPUs like GB200 can run multi-node workloads that require inter-GPU data transfer with large performance improvements when properly configured. The Cluster Agent can be directed to configure multi-node workloads with their own ComputeDomains automatically to optimize inter-GPU connections.

Additional prerequisites:

• The NVIDIA GPU DRA driver must be installed.
• The NVLinkOptimized cluster attribute must be added during cluster registration.

Kata Container-Isolated Workloads

Clusters that have this attribute run all function/task Pods in Kata Containers without exception.

Additional cluster restrictions to be aware of:

• Pod containers must at least have resource limits defined for cpu and memory. If unset, runtime behavior is undefined.

• Object count limits are configured for resource fairness in these clusters:

  • ConfigMaps: 20
  • Secrets: 20
  • Services: 20
  • Pods: 100
  • Jobs: 10
  • CronJobs: 10
  • Deployments: 10
  • ReplicaSets: 10
  • StatefulSets: 10

Network Configuration

The NVCA operator requires outbound network connectivity to pull images, charts, and report logs and metrics. During installation, the operator pre-configures the nvca-namespace-networkpolicies configmap with the following network policies:

Key Network Requirements

1. Kubernetes API Access

• NVCA requires access to the Kubernetes API
• Consult your cloud provider’s documentation (e.g., Azure, AWS, GCP) for the Kubernetes API endpoint

2. Container Registry and NVCF Control Plane Access

• Access to nvcr.io and helm.ngc.nvidia.com is required to pull container images, resources, and helm charts.

• NVCA requires access to NVIDIA control plane services for coordination of functions and task deployments and invocation, this includes:

  • connect.pnats.nvcf.nvidia.com
  • grpc.api.nvcf.nvidia.com
  • *.api.nvcf.nvidia.com
  • sqs.*.amazonaws.com
  • spot.gdn.nvidia.com
  • ess.ngc.nvidia.com
  • api.ngc.nvidia.com

3. Monitoring and Logging

• If your environment requires advanced monitoring or logging (e.g., sending logs to external endpoints), ensure your cluster’s NetworkPolicy or firewall rules allow egress to the required monitoring/logging domains

Network Policy Customization via ConfigMap

The NVCA operator pre-configures the nvca-namespace-networkpolicies configmap during installation. If you need to customize these policies for your cluster, you can use a configmap to override the default policies.

To customize a network policy:

1. Create a configmap with your custom network policy, for example:

1
 apiVersion: v1
2
 kind: ConfigMap
3
 metadata:
4
   name: demopatch-configmap
5
   namespace: nvca-operator
6
   labels:
7
     nvca.nvcf.nvidia.io/operator-kustomization: enabled
8
 data:
9
   patches: |
10
     - target:
11
         group: ""
12
         version: v1
13
         kind: ConfigMap
14
         name: nvca-namespace-networkpolicies
15
       patch: |-
16
         - op: replace
17
           path: /data/allow-egress-internet-no-internal-no-api
18
           value: |
19
             apiVersion: networking.k8s.io/v1
20
             kind: NetworkPolicy
21
             metadata:
22
               name: allow-egress-internet-no-internal-no-api
23
               labels:
24
                 app.kubernetes.io/name: nvca
25
                 app.kubernetes.io/instance: nvca
26
                 app.kubernetes.io/version: "1.0"
27
                 app.kubernetes.io/managed-by: nvca-operator
28
             spec:
29
               podSelector: {}
30
               policyTypes:
31
                 - Egress
32
               egress:
33
                 - to:
34
                   - namespaceSelector: {}
35
                     podSelector:
36
                       matchLabels:
37
                         k8s-app: kube-dns
38
                 - to:
39
                   - namespaceSelector:
40
                       matchLabels:
41
                         kubernetes.io/metadata.name: gxcache
42
                   ports:
43
                     - port: 8888
44
                       protocol: TCP
45
                     - port: 8889
46
                       protocol: TCP

2. Apply the configmap:

$
 kubectl apply -f patchcm.yaml

3. Verify the changes:

$
 kubectl logs -n nvca-operator -l app.kubernetes.io/name=nvca-operator

You should see a message indicating successful patching: configmap patched successfully

The changes will be applied to the nvcf-backend namespace and will be used for all new namespaces’ network policies. The network policies will also be updated across all helm chart namespaces.

Network Policy Customization via clusterNetworkCIDRs Flag

You can customize the allow-egress-internet-no-internal-no-api policy with helm, by adding on the networkPolicy.clusterNetworkCIDRs flag. For example:

$
 helm upgrade nvca-operator -n nvca-operator --create-namespace -i --reuse-values --wait "https://helm.ngc.nvidia.com/nvidia/nvcf-byoc/charts/nvca-operator-1.14.0.tgz" --username='$oauthtoken' --password=$(helm get values -n nvca-operator nvca-operator -o json | jq -r '.ngcConfig.serviceKey') --set networkPolicy.clusterNetworkCIDRs="{10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/12}"

This command will override the default k8s networking CIDRs specified in the allow-egress-internet-no-internal-no-api with your input.

Advanced: Additional Configuration Options

CSI Volume Mount Options

The NVIDIA Cluster Agent supports customizing CSI volume mount options for caching. This allows you to configure specific mount options for the CSI volumes used in your cluster.

To configure CSI volume mount options:

1. Get the NVCF cluster name:

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"

2. View current mount options configuration:

$
  kubectl get nvcfbackend -n nvca-operator "$nvcf_cluster_name" -o yaml | grep -A 5 "cacheCSIVolumeMountOptionsConfig"

1. Set mount options (example):

$
  kubectl patch nvcfbackends.nvcf.nvidia.io -n nvca-operator "$nvcf_cluster_name" --type='json' -p='[{"op": "replace", "path": "/spec/overrides/featureGate/cacheCSIVolumeMountOptionsConfig", "value": {"disabled": false, "mountOptions": "ro,norecovery,nouuid"}}]'

4. Verify the changes:

$
  kubectl get nvcfbackend -n nvca-operator "$nvcf_cluster_name" -o yaml | grep -A 5 "cacheCSIVolumeMountOptionsConfig"

The default mount options are:

• ro: Read-only mount
• norecovery: Skip journal recovery
• nouuid: Ignore filesystem UUID

You can modify these options based on your specific requirements. The configuration will be applied to all CSI volumes created by the NVIDIA Cluster Agent for caching purposes.

Node Selection for Cloud Functions

By default, the cluster agent uses all nodes discovered with GPU resources to schedule Cloud Functions and there are no additional configuration required.

In order to limit the nodes that can run Cloud Functions, you may use nvca.nvcf.nvidia.io/schedule=true label on the specific nodes.

If there are no nodes in the cluster with the nvca.nvcf.nvidia.io/schedule=true label set, the cluster agent will switch to the default behavior of using all nodes with GPUs.

For example, to mark specific nodes as schedulable in a cluster:

$
  kubectl label node <node-name> nvca.nvcf.nvidia.io/schedule=true

To mark a single node from the above set as unschedulable for nvcf workloads, you can unlabel using:

$
  kubectl label node <node-name> nvca.nvcf.nvidia.io/schedule-

GPU Product Name Override

The NVIDIA Cluster Agent supports GPU product name override via node label. This is useful for customers who want to use a custom product name or override the default GPU product name.

For example, to set the GPU product name for a node, use the following command:

$
  kubectl label node <node-name> nvca.nvcf.nvidia.io/gpu.product=<product-name>

Note: The GPU Product Name Override via node labeling only takes effect when there are no pre-existing active instances in the cluster. If active instances already exist with the original GPU instance types, the override will not be applied.

Managing Feature Flags

The NVIDIA Cluster Agent supports various feature flags that can be enabled or disabled to customize its behavior. The following are some commonly used feature flags:

Setting Feature Flags at Install Time

Feature flags can be set during the initial NVCA Operator installation through Helm values. The mechanism differs by deployment mode.

1
selfManaged:
2
featureGateValues: ["DynamicGPUDiscovery", "SelfHosted", "LogPosting"]

$
helm upgrade --install nvca-operator \
>
oci://${REGISTRY}/${REPOSITORY}/nvca-operator \
>
--version 1.2.7 \
>
--namespace nvca-operator --create-namespace \
>
-f nvca-operator-values.yaml \
>
--set 'selfManaged.featureGateValues={DynamicGPUDiscovery,SelfHosted,LogPosting}'

$
helm upgrade nvca-operator \
>
oci://${REGISTRY}/${REPOSITORY}/nvca-operator \
>
--version 1.2.7 \
>
--namespace nvca-operator \
>
-f nvca-operator-values.yaml \
>
--set 'selfManaged.featureGateValues={DynamicGPUDiscovery,SelfHosted,LogPosting,MultiNodeWorkloads}'

1
- selfManaged:
2
featureGateValues: ["DynamicGPUDiscovery", "SelfHosted", "LogPosting"]
3
imageCredHelper:
4
imageRepository: {{ .Values.global.image.registry }}/{{ .Values.global.image.repository }}/nvcf-image-credential-helper
5
sharedStorage:
6
imageRepository: {{ .Values.global.image.registry }}/{{ .Values.global.image.repository }}/samba

$
helm upgrade nvca-operator \
>
oci://helm.ngc.nvidia.com/nvidia/nvcf-byoc/charts/nvca-operator \
>
--version `<version>` \
>
--namespace nvca-operator \
>
--set helmManaged.featureGateValues='["LogPosting","CachingSupport"]'

Verifying Feature Flags

After installing or upgrading, verify the active feature flags:

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"
$
  kubectl get nvcfbackends -n nvca-operator "$nvcf_cluster_name" -o jsonpath='{.spec.featureGate.values}' && echo ""

The NVCA agent pod command-line args also reflect the active flags:

$
  kubectl get pods -n nvca-system -o yaml | grep -i feature

Modifying Feature Flags at Runtime

For NGC-managed and Helm-managed clusters, feature flags can also be modified at runtime by patching the NVCFBackend resource directly. This is useful for quick changes without running a helm upgrade.

1. Get the NVCF cluster name:

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"

2. View current feature flags:

$
  kubectl get nvcfbackends -n nvca-operator -o yaml | grep -A 5 "featureGate:"

3. Patch the feature flags. Note that this will override all feature flags.

$
  kubectl patch nvcfbackends.nvcf.nvidia.io -n nvca-operator "$nvcf_cluster_name" --type=merge -p '{"spec":{"overrides":{"featureGate":{"values":["LogPosting","CachingSupport"]}}}}'

As an alternative to the patch command, you can also modify the feature flags using the edit command:

$
  kubectl edit nvcfbackend -n nvca-operator
$
  ...
$
  spec:
$
    featureGate:
$
      values:
$
      - LogPosting                # Existing feature flag
$
    overrides:
$
      featureGate:
$
        values:
$
        - LogPosting              # Existing feature flag copied over
$
        - -CachingSupport         # Caching support disabled
$
        ...

4. Verify the changes:

$
  kubectl get pods -n nvca-system -o yaml | grep -i feature

Enable Helm Shared Storage

The NVIDIA Cluster Agent supports shared storage for Helm charts through the SMB CSI driver. This feature is required for enabling Helm-based tasks in your cluster.

1. First, install the SMB CSI driver using Helm:

$
  helm repo add csi-driver-smb https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
$
  helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.16.0

2. Get the NVCF cluster name:

$
  nvcf_cluster_name="$(kubectl get nvcfbackends -n nvca-operator -o name | cut -d'/' -f2)"

3. Enable the Helm shared storage feature flag:

$
  kubectl patch nvcfbackends.nvcf.nvidia.io -n nvca-operator "$nvcf_cluster_name" --type=merge -p '{"spec":{"overrides":{"featureGate":{"values":["LogPosting","HelmSharedStorage", "CachingSupport"]}}}}'

4. Verify that the feature flag is enabled:

$
  kubectl get pods -n nvca-system -o yaml | grep HelmSharedStorage

Agent Config Merging

The NVIDIA Cluster Agent supports merging custom configuration into the generated NVCA config via the agentConfig.mergeConfig Helm value. This allows you to override or extend NVCA runtime settings without modifying the operator’s config generation logic.

When agentConfig.mergeConfig is set, the Helm chart creates a ConfigMap called agent-config-merge containing the provided YAML. This ConfigMap is mounted into the NVCA pod and merged with the generated config at runtime.

Example values.yaml:

1
 agentConfig:
2
   mergeConfig: |
3
     agent:
4
       logLevel: debug

Apply via Helm:

$
 helm upgrade nvca-operator -n nvca-operator --create-namespace -i \
>
   <chart-reference> \
>
   --set-file agentConfig.mergeConfig=my-nvca-config.yaml

Or include it in a values file passed to helm upgrade -f values.yaml.

Manual Instance Configuration

Manual instance configuration allows you to disable Dynamic GPU Discovery and instead provide a static list of instance types that NVCA will register with the NVCF control plane. This is useful when:

• You have a known, fixed set of GPU configurations
• Dynamic GPU discovery isn’t working correctly for your environment
• You want to control exactly which instance types are available

By default, NVCA uses Dynamic GPU Discovery to automatically detect GPUs on cluster nodes and register appropriate instance types. When this is disabled, NVCA instead reads a static GPU configuration from a ConfigMap.

Prerequisites

• A working NVCF cluster with nvca-operator installed
• Access to modify Helm values for nvca-operator
• Since you are not using the GPU Operator, you must ensure each GPU node has the instance-type label that matches the “value” field in your manual configuration:

$
 kubectl label nodes <node-name> nvca.nvcf.nvidia.io/instance-type=<instance-type-value>

For example, if your configuration specifies "value": "OCI.GPU.A10", you would label the node with:

$
 kubectl label nodes gpu-node-1 nvca.nvcf.nvidia.io/instance-type=OCI.GPU.A10

Step 1: Create the GPU Configuration JSON

Create a JSON file defining your GPU types and instance configurations. The configuration is an array of GPU types, each containing an array of instance types.

Example Configuration (gpu-config.json):

1
 [
2
     {
3
         "name": "H100",
4
         "capacity": 8,
5
         "instanceTypes": [
6
             {
7
                 "name": "ON-PREM.GPU.H100_1x",
8
                 "value": "ON-PREM.GPU.H100",
9
                 "description": "One Nvidia Hopper GPU",
10
                 "default": true,
11
                 "cpuCores": 16,
12
                 "systemMemory": "128G",
13
                 "gpuMemory": "80G",
14
                 "gpuCount": 1,
15
                 "os": "linux",
16
                 "driverVersion": "535.135.05",
17
                 "cpuArch": "amd64",
18
                 "storage": "1Ti"
19
             },
20
             {
21
                 "name": "ON-PREM.GPU.H100_8x",
22
                 "value": "ON-PREM.GPU.H100",
23
                 "description": "Eight Nvidia Hopper GPUs (Full Node)",
24
                 "default": false,
25
                 "cpuCores": 128,
26
                 "systemMemory": "1Ti",
27
                 "gpuMemory": "640G",
28
                 "gpuCount": 8,
29
                 "os": "linux",
30
                 "driverVersion": "535.135.05",
31
                 "cpuArch": "amd64",
32
                 "storage": "8Ti"
33
             }
34
         ]
35
     }
36
 ]

Step 2: Base64 Encode the Configuration

The GPU configuration must be Base64-encoded for the Helm values. Use the following command:

$
 # Base64 encode the configuration (without line wrapping)
$
 GPU_CONFIG_B64=$(cat gpu-config.json | base64 -w 0)
$
 echo $GPU_CONFIG_B64

$
GPU_CONFIG_B64=$(cat gpu-config.json | base64)

Step 3: Configure Helm Values

Update your nvca-operator Helm values to disable Dynamic GPU Discovery and provide the manual configuration. The values path depends on your cluster source mode.

1
 selfManaged:
2
   # Disable dynamic GPU discovery
3
   featureGateValues: []
4
5
   # Your Base64-encoded GPU configuration
6
   gpuManualInstanceConfigB64: "<your-base64-encoded-config>"

1
 ngcConfig:
2
   clusterSource: "helm-managed"
3
4
 helmManaged:
5
   cloudProvider: "on-prem"
6
   clusterRegion: "us-west-1"
7
   clusterGroupID: "your-group-id"
8
   clusterGroupName: "your-group-name"
9
   nvcaVersion: "2.97.0"
10
11
   # Disable dynamic GPU discovery
12
   featureGateValues:
13
     - "-DynamicGPUDiscovery"
14
15
   # Your Base64-encoded GPU configuration
16
   gpuManualInstanceConfigB64: "<your-base64-encoded-config>"

Step 4: Install or Upgrade the Operator

Apply the configuration using Helm:

$
 helm upgrade nvca-operator -n nvca-operator --create-namespace -i \
>
   <chart-reference> \
>
   -f values.yaml

If you are using the NVCF self-hosted Helmfile, add your values file as an entry in the nvca-operator release values: list and run helmfile sync or helmfile apply instead.

Configuration Fields Reference

GPU Type Fields:

Instance Type Fields:

Verification

After applying the configuration, verify that NVCA is using the static configuration:

1. Check the NVCFBackend resource:

$
 kubectl get nvcfbackend -n nvca-operator -o yaml

Look for -DynamicGPUDiscovery in the feature gates and verify the GPU configuration is present.

2. Check the nvca-config ConfigMap:

$
 kubectl get configmap nvca-config -n nvca-system -o yaml

The gpus key should contain your JSON configuration.

3. Check NVCA logs for registration:

$
 kubectl logs -n nvca-system -l app=nvca | grep -i "registration\|instance"

Troubleshooting

Configuration Not Applied:

1. Verify Dynamic GPU Discovery is disabled in the feature gate values
2. Ensure the Base64 encoding is correct and doesn’t contain line breaks
3. Check that the JSON is valid before encoding

Invalid JSON Format:

1. Validate your JSON using a JSON validator before encoding
2. Ensure all required fields are present
3. Check that numeric values (cpuCores, gpuCount) are not quoted as strings

Memory/Storage Format Errors:

Memory and storage values must use valid Kubernetes quantity format:

• Valid: "28G", "128Gi", "1Ti", "512Mi"
• Invalid: "28GB", "128 Gi", "1TB"

Cloud Provider-Specific Notes

Oracle Cloud Infrastructure (OCI)

When using Oracle Container Engine for Kubernetes (OKE), ensure that:

• Your compute nodes and GPU nodes are in the same availability domain
• This is required for proper network connectivity between the NVIDIA Cluster Agent and GPU nodes
• Flannel CNI is the current recommended and validated CNI vs OCI native CNI for OKE cluster networking.

AWS

When using AWS EKS, note that the following limitations exist:

• Caching is currently not supported