FAQ
What does the process exit code mean?
The exit code is a coarse health signal:
The exit code does not distinguish CRITICAL from UNKNOWN or INFO from WARNING. For the precise level, read overall_severity from the JSON — Slurm prolog scripts that need more than the coarse signal should do so.
The catalog library’s health.NodeVerdict.ExitCode() method uses a separate 0/1/2/3 scheme that the CLI does not use.
What happens if a sensor cannot read from a GPU?
It depends on how the sensor reports the failure. If the sensor can identify the GPU but cannot read one or more fields, it calls AddError() on the measurement for those failing fields. The catalog evaluates each check against the partial measurement: checks that require only the available signals proceed normally; checks that require a failed signal produce a CRITICAL verdict rather than silently passing.
If the sensor cannot identify the GPU at all (e.g. a device handle failure), it returns an ErroredGPU measurement. All checks for that GPU produce CRITICAL verdicts.
What does skipped: true mean?
The check did not execute. The skip_reason field says why:
Skipped checks are excluded from overall_severity, so a node can look healthy while some checks never ran. Inspect skipped_checks as well as overall_severity, and investigate any node where every check was skipped.
How do I run only a subset of checks?
List the check IDs under disabled_checks: in the config file passed to --config:
Each ID is validated against the registered checks, so a typo is an error rather than a check that silently keeps running. Disabled checks appear in skipped_checks with skip_reason: CHECK_DISABLED and are excluded from overall_severity — a disable lowers the bar the node is measured against, so it is worth reviewing what is in that list.
The equivalent for a consumer building their own binary is catalog.WithDisabledChecks() at catalog construction time. There is still no positive filter (no “run only these”); disabling is subtractive.
To select checks by scheduling cadence, use the optional --cadence flag. Each check declares a SchedulingCadence (CONTINUOUS, PROLOG, or MAINTENANCE), and the windows nest (CONTINUOUS ⊂ PROLOG ⊂ MAINTENANCE): PROLOG runs the CONTINUOUS and PROLOG checks, and MAINTENANCE runs every check. Checks outside the requested window appear in skipped_checks with skip_reason: CADENCE_MISMATCH. Omit the flag and every registered check still runs, including Prolog checks such as gpu_power_limit. Job-time integrators that must not run Prolog checks pass --cadence=CONTINUOUS. Cadence is not a YAML config field: one cluster config can serve every window.
Can I add custom checks?
Yes. The catalog is open for extension. Implement a health.CheckDefinition, define a measurement struct that implements health.Measurement, and register the check via catalog.Register() or by adding it to builtinChecks() in a fork. The evaluation engine treats built-in and custom checks identically.
Custom checks you register carry provenance: private. Check results set nvidia_endorsed: false for any check whose provenance is not nvidia, so your tooling can filter to NVIDIA-authored checks without custom logic.
The community contribution program (allowing checks to ship with provenance: community in the NVIDIA-maintained catalog) is not yet open.
Are thresholds adjustable?
It depends on the ThresholdCategory:
- DEFAULT: Operator can override via config file. NVIDIA’s value is used if no override is supplied.
- ADJUSTABLE: Operator can override within documented bounds.
- FIXED: Non-overridable. Attempting to override returns an error before the check runs.
All active overrides appear in thresholds[name].overridden: true in the output, alongside the original NVIDIA value in thresholds[name].original_value.
What Go version is required?
Go 1.27.2, as declared in each module’s go.mod. The repo has four Go modules: catalog, toolkit, collect, and agentkit.
What hardware is covered?
The catalog currently centers on GB200 and GB300 GPU nodes, with some checks scoped to further NVIDIA architectures, plus host CPU availability and topology. Registered checks cover host CPU (possible-versus-online logical CPU count and online NUMA node count), GPU power (power limit, power state, average draw, hardware throttle), leftover GPU compute processes, GPU count, NVLink (active links, fabric health, fabric status, bandwidth/count/speed, capabilities, and peer-to-peer), fatal mlx5 RDMA link flaps, and Xid-driven GPU events (a CRITICAL check per critical Xid code, a consolidated Xid warning check, and an INFO check for Xid 154). See the README’s GB300 support note — GB200 coverage has been validated, GB300 coverage has not yet completed the same validation pass.
The sensor interface is extensible. Partners can implement toolkit.GPUBackend for other GPU types and supply measurements directly to cat.EvaluateAll(m) without modifying catalog source. NVIDIA does not author or validate checks for non-NVIDIA hardware.
Can I test without a GPU?
Yes. The toolkit ships mock constructors: toolkit.MockGPU(measurements...) returns a GPUBackend, and toolkit.MockCPU(measurement) returns a CPUBackend. Pass them to their matching sensor constructors and call toolkit.Collect(cat, sensor) normally.
The catalog itself has no hardware dependency. Populate gpu.GPUMeasurement directly and call cat.EvaluateAll(m) for fully in-process testing.