> This page is for Health Validation.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/health-validation/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/health-validation/_mcp/server.

# Extending the catalog with private checks

This guide is for **downstream consumers** who want to run their own hardware
health checks — proprietary thresholds, vendor-specific conditions, signals the
public catalog does not carry — *on top of* health-validation, without
forking it.

The pattern is: keep your consumer code in its own module that imports this one,
register your check definitions with the catalog, and supply the signals they
read. The snippets below are self-contained; nothing here requires changes to
health-validation itself.

## What is stable, and what is not

> [!IMPORTANT]
> **Build against the catalog. Treat the toolkit's sensor/backend layer as a
> demonstration.**
>
> - The **catalog** — the `catalog/health` vocabulary (`CheckDefinition`,
>   `Rule`, `ThresholdSpec`, the `Measurement` interface, the enums) and the
>   `catalog` registry/evaluation engine — is the extension surface. Your check
>   definitions are data written against it.
> - The **toolkit** sensor and backend implementations (the NVML/sysfs/journal
>   sensors, the `toolkit.Collect` pipeline, the `app.App` builder, and the
>   probe seams shown below) are **reference implementations for demonstration
>   and are subject to change.** They are functional today, but do not treat
>   their API as stable; expect the collection wiring to churn.

The practical consequence: put your *judgment* (what condition is unhealthy, at
what threshold, with what remediation) in check definitions, which are durable.
Keep your dependence on the collection wiring thin and localized so it is cheap
to follow when it changes.

## A check is data

A check is a `health.CheckDefinition` — no code, no hardware dependency. The
engine evaluates its CEL `Rules` against signal values a measurement supplies.

```go
package mychecks

import "github.com/dsx-ai-factory/health-validation/catalog/health"

func GPUClockCapped() health.CheckDefinition {
	return health.CheckDefinition{
		ID:          "gpu_clock_capped",
		Name:        "GPU Clock Capped",
		Description: "Flags a GPU whose sustained clock is pinned below the expected floor.",
		Category:    health.GPU,

		// The signals this check reads. A measurement must carry all of them
		// (see "Routing") for the check to run.
		RequiredSignals: []string{"gpu_id", "sm_clock_mhz"},

		ExecutionContexts: []health.ExecutionContext{health.OperatorInfra},
		Architectures:     []string{"GB200", "H100"},

		// Your checks stay yours: the engine honors this classification as
		// written. Use health.Private, the closest provenance for a definition
		// authored outside the public catalog; it excludes your checks from
		// canonical-only filters.
		Provenance: health.Private,

		// Operator-tunable threshold. FixedThreshold values cannot be overridden.
		Thresholds: map[string]health.ThresholdSpec{
			"min_sm_clock_mhz": {
				Value:       int64(1200),
				Category:    health.DefaultThreshold,
				Description: "Minimum acceptable sustained SM clock.",
				Unit:        "MHz",
			},
		},

		Rules: []health.Rule{
			{
				Expression:        "sm_clock_mhz < min_sm_clock_mhz",
				Severity:          health.Critical,
				Message:           "GPU ${gpu_id}: SM clock ${sm_clock_mhz} MHz below floor ${min_sm_clock_mhz} MHz.",
				RecommendedAction: health.GenericDrain(),
			},
		},
	}
}
```

Fields worth knowing:

- **`Category`** and **`RequiredSignals`** are the routing key — see below.
- **`Rules`** are CEL expressions over the union of the measurement's signal
  names and the check's threshold names. The worst-severity matching rule sets
  the verdict; `${name}` placeholders interpolate signal and threshold values.
- **`Thresholds`** are operator-tunable by default. `FixedThreshold` values can
  never be overridden; `RequiredOverride` means the check skips until a fleet
  supplies a value.
- **`RecommendedAction`**'s `RemediationActions` is an open `[]RemediationAction`
  (a string type, not a closed enum): use the `health.RemediationAction*`
  constants for steps NVIDIA's own catalog uses, or supply your own string for
  a remediation step specific to your extension. A downstream remediation
  engine should treat an unrecognized value as an opaque step, not an error.
- **`Provenance`**, `MaturityLevel`, and the other classification fields are
  passed through untouched. Use `health.Private`.
- **`ConditionScope`** records how far the detected condition reaches: `NodeScope`,
  `DomainScope`, or `ClusterScope`. It is `NodeScope` when omitted, so an existing
  check keeps its current meaning without change. Unlike the fields above it is
  not passed through untouched: a collection-failure verdict reports
  `NodeScope` whatever the check declares.

## Routing: how a measurement reaches your check

The engine feeds a measurement to a check when **both** hold:

1. they share a `Category`, and
2. the measurement's schema carries **every** one of the check's
   `RequiredSignals`.

Your check is then evaluated once per matching measurement (so a per-GPU check
yields one verdict per GPU). If no supplied measurement carries its required
signals, the check is skipped `NO_APPLICABLE_MEASUREMENT`; if its whole category
supplied no measurement at all, it is skipped `SENSOR_MISSING`. Registering the
check is what declares that its sensor is expected, so every registered check
produces a verdict — a result or a skip — and none disappears from the output.

So a private check only does something if **some measurement in its category
carries its signals.** That is the real work of extending: getting your signals
onto a measurement.

Two invariants keep routing unambiguous. Enforce both in your tests:

- **Signal names are unique within a category.** Two measurement types in the
  same category must not share a signal name, or a check could match the wrong
  one. Assert with `catalog.CheckSignalUniqueness(measurements...)`.
- **Every check has a provider.** A check whose signals no measurement carries
  can never run. Assert with `cat.CheckCoverage(measurements...)`.

```go
if err := catalog.CheckSignalUniqueness(mygpu.Measurement{}, /* … */); err != nil {
	t.Fatal(err)
}
if err := cat.CheckCoverage(mygpu.Measurement{}, /* … */); err != nil {
	t.Fatal(err)
}
```

## Getting your signals collected (demonstration wiring — subject to change)

> The mechanics in this section use the toolkit's sensor/backend layer, which is
> demonstration-grade and **subject to change.** Keep this wiring thin.

There are two situations.

**1. Your signals come from a domain the toolkit already enumerates** (e.g. GPU
via NVML). Attach a *prober* to the shared backend so the existing
category-measurement carries your extra signals. A prober implements
`nvml.DeviceProber` (`Measure(ProbeContext) health.Measurement` plus an
`Errored` fallback); it reads per-device state into your own measurement type
whose `Category()` is `health.GPU` and whose struct fields (via `json` tags)
are exactly your check's `RequiredSignals`. Install it on the app's shared NVML
backend:

```go
// Layer your prober onto the one NVML enumeration the node and NVLink
// sensors already share — no second session.
a = a.WithGPUBackend(a.NVMLBackend().WithGPUProbe(mygpu.Probe()))
```

**2. Your signals come from a new source.** You would implement collection here,
but this is exactly the layer flagged as unstable — approach it knowing the
interfaces may move.

## Registering and running your checks

The app builder is the demonstration entry point. `WithChecks` registers your
definitions alongside the built-ins (declared classification honored; a
duplicate ID fails):

```go
func main() {
	a := app.New().WithChecks(mychecks.All()...)
	a = configureGPU(a) // installs your prober; keep this platform-split (see below)
	os.Exit(a.Run(os.Args[1:], os.Stdout, os.Stderr))
}
```

If you do not want the node runner at all, use the catalog directly — the most
stable path:

```go
cat := catalog.New(
	catalog.WithArchitecture("GB200"),
	catalog.WithExecutionContext(health.OperatorInfra),
	catalog.WithOverrides(catalog.NewOverrideSet(catalog.Override{
		CheckID: "gpu_clock_capped", ThresholdName: "min_sm_clock_mhz", Value: int64(1400),
	})),
)
_ = cat.Register(mychecks.GPUClockCapped())
verdict, err := cat.EvaluateAll(measurements...) // measurements you produced yourself
```

## A recommended consumer layout

Keep your module thin and mirror the public per-domain structure. For a private
GPU vertical, that is roughly:

| File | Role |
|---|---|
| <nobr>`gpu/measurement.go`</nobr> | Your measurement type (`Category() == health.GPU`) whose `json`-tagged fields are exactly your checks' `RequiredSignals`. |
| <nobr>`gpu/*.go`</nobr> | One `health.CheckDefinition` per condition. |
| <nobr>`gpu/checks.go`</nobr> | An `All()` that returns every definition; passed to `app.WithChecks`. |
| <nobr>`gpu/probe_linux.go`</nobr> | Your `nvml.DeviceProber` that reads the signals into the measurement. |
| `main.go` + `backend_{linux,other}.go` | Wires `WithChecks` and `WithGPUProbe`, **platform-split** — the NVML prober is linux+cgo, so gate it behind `//go:build linux` and provide a no-op `configureGPU` for other platforms, so the binary still builds and `version` runs off-linux. |

health-validation owns the NVML lifecycle, device enumeration, and the
built-in sensors; your module owns only its measurement, its checks, and its
per-device read.

## Testing your extension

- Unit-test each check with `catalog.NewEmpty()`, `Register`, then `Evaluate`
  (single check) or `EvaluateAll` (batch) against measurements you construct
  directly — no hardware needed.
- Assert `CheckSignalUniqueness` and `CheckCoverage` over your full
  measurement-plus-check set, so a renamed signal or a check with no provider
  fails a test rather than silently never running.

See [Architecture](/health-validation/getting-started/architecture) for package roles and the evaluation data
flow.