NVIDIA Health Validation: Architecture

Package roles

Data flow

Sensor.Measure()
    --> []health.Measurement
        --> Catalog.EvaluateAll()
            --> per-category check routing
                --> CEL evaluation per check
                    --> []CheckVerdict
                        --> AggregateNodeVerdict()
                            --> NodeVerdict

Each sensor produces one or more measurements. Device sensors normally emit one per device; the CPU sensor emits one host-wide measurement. Each measurement is routed to the checks whose Category matches the measurement’s category. The evaluation engine resolves threshold overrides, checks architecture and execution context prerequisites, runs CEL rules, and returns a CheckVerdict. Aggregation computes the node-level worst-case severity. The process exit code is a coarse mapping of that severity (0 for PASS/INFO/WARNING, 1 for CRITICAL/UNKNOWN, 2 when the tool could not run to a verdict); the precise level is conveyed in the JSON.

Sensors and backends

Two distinct concepts:

• Sensor implements toolkit.Sensor. It holds a backend and produces []health.Measurement for Collect. For example, GPUSensor wraps a GPUBackend, and CPUSensor wraps a CPUBackend.
• Backend is a data source defined as an interface in toolkit. The NVML-backed implementation lives in toolkit/nvml/ (Linux only, requires cgo). SysfsCPU (deprecated; the CLI uses the collect/sensors/cpu collect sensor through the toolkit bridge) and its replacement read /sys/devices/system/cpu/possible, /sys/devices/system/cpu/online, and /sys/devices/system/node/online without sudo or an external executable. Mock backends are available for testing without hardware.

SysfsCPU observes the kernel CPU topology exposed in its execution environment. The cpu_count check compares the possible and online logical CPU counts, requires no operator value, and reports CRITICAL when they differ. It assumes the target platform does not advertise intentionally unused CPU hotplug capacity and does not validate an exact VM/SKU count, process affinity, or cgroup CPU limits; inside a container, it evaluates the sysfs view exposed by the runtime.

The numa_node_count check compares the online NUMA node count against expected_numa_node_count and reports CRITICAL when they differ. The expected value resolves from the detected architecture, and an operator override replaces whatever the architecture resolved to. Each read is attributed to its own signal, so an unreadable NUMA node list fails numa_node_count while cpu_count still evaluates, and vice versa. The check applies only to the architectures it lists and skips on any other, including a node whose architecture cannot be identified.

health-validation collect runs the same sensors without evaluating any checks and prints the measurements as one ObservationBatch (protojson). An optional config’s check IDs select which sensor domains run; without one, every built-in domain runs. Use it to retain node-local evidence and evaluate it separately from acquisition. See the README for the output shape.

To add a sensor for a new hardware domain: define a measurement struct in catalog/<domain>/, declare a backend interface in toolkit/, implement a sensor wrapping it, and add a mock constructor.

Catalog construction options

Scheduling cadence is not a construction option. Set Cadence in per-run catalog.EvaluationOptions or toolkit.CollectionOptions; either zero value evaluates every registered check, and zero-value collection options also use the default timeout. Checks outside the requested window are reported with CADENCE_MISMATCH. One catalog can serve Continuous, Prolog, and Maintenance windows.

health-validation run defaults to TENANT_GUEST; --execution-context selects another context. The CLI applies the selected context through catalog.WithExecutionContext when constructing the catalog.

Execution contexts

Output schema

The NodeVerdict JSON is a versioned contract. Its schema_version field uses MAJOR.MINOR versioning (health.NodeVerdictSchemaVersion in catalog/health/verdict.go, mirrored by catalog/health/node-verdict.schema.json); MAJOR increments on breaking changes and MINOR on additive ones. Consumers must check this field before acting on a verdict and treat an unrecognized or missing value as untrusted.

NodeVerdict

NodeVerdict carries no node-level RecommendedAction. Each CheckVerdict carries its own remediation sequence; with several checks potentially tying at overall_severity, there is not a single, clearly-ranked action to surface at the node level. Consumers that need node-level remediation should walk CheckResults.

CheckVerdict

ResolvedThreshold

Skip reason codes

Adding a check

1. Add any new signal fields to the measurement struct in catalog/<domain>/measurement.go. Add matching Signal* string constants.
2. Write a CheckDefinition factory in catalog/<domain>/. Declare RequiredSignals, Thresholds, and Rules (CEL expressions referencing signal names and threshold names). Declare Dependencies when another check explains this one’s complete failure to read an entity, as the GPU and NVLink checks do for gpu_count. A partial read, where only some required signals failed, is an independent fault and is not attributed.
3. Register it in catalog/checks.go via builtinChecks() so catalog.New() and the stock CLI run it. Declare SchedulingCadence on the check. Omit --cadence (or use a zero-value evaluation/collection context) to evaluate every registered check; set a cadence to run checks whose cadence covers that window (Continuous ⊂ Prolog ⊂ Maintenance) and report the others as skipped.
4. Add a test covering the PASS path, at least one non-PASS path, and the skipped path (errored measurement).

Build

# Canonical build (Bazel)
bazel build //toolkit/cmd/health-validation
bazel test //...
# Local iteration
go build -o health-validation ./toolkit/cmd/health-validation
go test ./catalog/...

The repo uses Bazel exclusively in CI. go build and go test work for local iteration but do not run all CI checks (nogo, link-mode validation). On macOS, run bazel test --config=ci --platforms=//build/platforms:linux_amd64 //... from a Linux VM or container to reproduce the Linux CI build. See Developer Guide on GitHub for the container workflow.