Catalog

The catalog is the registry of all supported certification categories. It maps {domain, variant} pairs to workload definitions — the concrete specs that run during certification.

Structure

Each catalog entry is a YAML file at pkg/catalog/entries/<domain>/<variant>.yaml. The file defines the base workload spec (dependencies, job template, orchestration) plus platform- and GPU-specific overrides. Entries are embedded into the NVCRE binaries at compile time via //go:embed (pkg/catalog/loader.go); the loader walks the embedded tree and registers every entry automatically. Adding or changing an entry therefore requires rebuilding nvcrectl and building and deploying a new controller image; entries cannot be dropped in at runtime.

Domains and variants

Selecting categories

In a Certification spec, list any combination of domain/variant pairs under categories:

spec:
  categories:
    - domain: communication
      variant: nccl-all-reduce
    - domain: training
      variant: nemotron5-8b

To see all available categories from the CLI:

nvcrectl certification list-categories

Per-node category options

nccl-loopback, nccl-loopback-nvswitch, and dcgm-level4 run one Job per node. They support two additional options:

categories:
  - domain: diagnostics
    variant: dcgm-level4
    options:
      maxConcurrent: 4
      timeoutPerJob: 30m

DCGM endpoint

dcgm-level4 runs dcgmi diag against an existing DCGM hostengine; installing DCGM stays the cluster operator’s job. By default it connects to the GPU Operator’s nvidia-dcgm.gpu-operator.svc:5555 (on Nscale NKS, nvidia-dcgm-dra.nvidia-platform.svc:5555). To use a DCGM Service in another namespace, or on another port, set dcgmHost to host[:port]:

categories:
  - domain: diagnostics
    variant: dcgm-level4
    options:
      dcgmHost: nvidia-dcgm.gpu-system.svc:5555

Each per-node Job must reach the hostengine on its own node; otherwise the diagnostics exercise another node’s GPUs, and the result is reported against the wrong node. The dcgm-level4 pod runs on the host network, so the address must route node-locally. Either use a Service with internalTrafficPolicy: Local, as the GPU Operator’s and NKS’s Services are, or localhost:5555 when the hostengine itself listens on the node’s network. Pointing dcgmHost at a Service that load-balances across nodes does not give that routing.

nvcrectl setup status checks only the GPU Operator’s gpu-operator/nvidia-dcgm Service, so it reports DCGM as missing when only your own endpoint exists.

Training category options

The training entries (nemotron5-8b, nemotron5-56b) size their containers for DGX-class nodes by default: limits of cpu: "128" / memory: 800Gi and requests of cpu: "64" / memory: 500Gi. On smaller GPU nodes those defaults make training pods unschedulable (Insufficient cpu / Insufficient memory), so the CPU and memory values can be overridden with the resources option:

categories:
  - domain: training
    variant: nemotron5-8b
    options:
      resources:
        limits:
          cpu: "6"
          memory: 48Gi
        requests:
          cpu: "4"
          memory: 32Gi

Each of the four values independently falls back to its default when unset. Values are standard Kubernetes resource quantities. The GPU count is not part of resources — it is controlled by gpusPerNode. Like all category options, resources can also be set at the Certification spec level as a global default for every category; a per-category resources block replaces the global one entirely (the two are not merged field by field). Non-training entries ignore it.

When a request and its matching limit are both set in the same resources block, the request must not exceed the limit — the Certification CRD rejects an inverted pair at admission (requests.cpu must not exceed limits.cpu). A request set without its matching limit is only checked against the resolved default when the training pod is admitted, so raise the limit alongside the request when overriding upward.

Note: platform-specific catalog overrides may still supersede these values. On AWS with H100 GPUs, the training entries apply an instance-sized override (limits cpu: "192" / memory: 1800Gi, requests cpu: "128" / memory: 1200Gi for p5-class instances) that takes precedence over resources.

Overrides

Catalog entries define a base workload spec. Platform-specific and GPU-specific overrides within the same YAML are applied at render time based on the detected environment. Supported GPU architectures include GB200, GB300, H100, H200, B200, and B300. See Platform Detection & Overrides for override semantics.

Adding a custom entry

See How-to: Custom Catalog Entries. For running a custom test without rebuilding the binaries, see that guide’s runtime alternatives.