Concepts

Platform Detection & Overrides

View as Markdown

Auto-detection

The controller detects two dimensions at runtime:

DimensionSource
Cloud platformspec.providerID on the Node object (e.g. aws://..., gce://...)
GPU architecturenvidia.com/gpu.product node label (e.g. NVIDIA-GB200, NVIDIA-H100-80GB-HBM3)

The live controller writes detection results to status.orchestration.detectedPlatform and status.orchestration.detectedGPUArchitecture on the Workflow. When using nvcrectl workflow render (client-side), these values are also written as annotations (nvcrectl.nvidia.com/detected-platform, nvcrectl.nvidia.com/detected-gpu-architecture) on the rendered manifest for offline inspection.

Override matching

Catalog entries define a base WorkflowSpec. Overrides are matched by platform + GPU architecture and applied in order. Two patch mechanisms are supported:

jobTemplate (strategic merge patch)

Replaces entire arrays. Use when you want to set all values for a field (e.g. the full env list):

1jobTemplate:
2  spec:
3    workload:
4      trainJob:
5        trainer:
6          env:
7            - name: MY_VAR
8              value: "value"

Arrays in jobTemplate overrides are replaced entirely, not merged by name. If the base spec has env vars, they will be wiped unless you include them in the override.

jobTemplatePatch (RFC 6902 JSON Patch)

Appends or modifies individual fields without replacing arrays. Use op: add with path ending in /- to append to an array:

1jobTemplatePatch:
2  - op: add
3    path: /spec/workload/trainJob/trainer/env/-
4    value:
5      name: EXTRA_VAR
6      value: "extra"

A jobTemplatePatch that appends to an array must come after any jobTemplate override that sets that array, or the appended values will be wiped.

Architecture-specific resources

Different GPU architectures and cloud platforms require different Kubernetes resources:

ArchitecturePlatformInterconnectKey resources
GB200AWSEFAhugepages-2Mi, vpc.amazonaws.com/efa, EFA hostPath volume, ComputeDomain
GB200AzureInfiniBandmlnxnics dep, topo ConfigMap, ComputeDomain
GB300AWSRoCEroce-channel resource claim (DRA), no hugepages, no EFA
GB300AzureInfiniBandmlnxnics dep, topo ConfigMap, ComputeDomain
H100AWSEFAvpc.amazonaws.com/efa: 32, no hugepages
H100AzureInfiniBandmlnxnics dep, topo ConfigMap

The live controller tracks which overrides matched in status.orchestration.appliedOverrides. When using nvcrectl workflow render, the same information is also written to the nvcrectl.nvidia.com/applied-overrides annotation on the rendered manifest.