Extending the catalog with private checks

This guide is for downstream consumers who want to run their own hardware health checks — proprietary thresholds, vendor-specific conditions, signals the public catalog does not carry — on top of health-validation, without forking it.

The pattern is: keep your consumer code in its own module that imports this one, register your check definitions with the catalog, and supply the signals they read. The snippets below are self-contained; nothing here requires changes to health-validation itself.

What is stable, and what is not

[!IMPORTANT] Build against the catalog. Treat the toolkit’s sensor/backend layer as a demonstration.

• The catalog — the catalog/health vocabulary (CheckDefinition, Rule, ThresholdSpec, the Measurement interface, the enums) and the catalog registry/evaluation engine — is the extension surface. Your check definitions are data written against it.
• The toolkit sensor and backend implementations (the NVML/sysfs/journal sensors, the toolkit.Collect pipeline, the app.App builder, and the probe seams shown below) are reference implementations for demonstration and are subject to change. They are functional today, but do not treat their API as stable; expect the collection wiring to churn.

The practical consequence: put your judgment (what condition is unhealthy, at what threshold, with what remediation) in check definitions, which are durable. Keep your dependence on the collection wiring thin and localized so it is cheap to follow when it changes.

A check is data

A check is a health.CheckDefinition — no code, no hardware dependency. The engine evaluates its CEL Rules against signal values a measurement supplies.

package mychecks
import "github.com/dsx-ai-factory/health-validation/catalog/health"
func GPUClockCapped() health.CheckDefinition {
	return health.CheckDefinition{
		ID:          "gpu_clock_capped",
		Name:        "GPU Clock Capped",
		Description: "Flags a GPU whose sustained clock is pinned below the expected floor.",
		Category:    health.GPU,
		// The signals this check reads. A measurement must carry all of them
		// (see "Routing") for the check to run.
		RequiredSignals: []string{"gpu_id", "sm_clock_mhz"},
		ExecutionContexts: []health.ExecutionContext{health.OperatorInfra},
		Architectures:     []string{"GB200", "H100"},
		// Your checks stay yours: the engine honors this classification as
		// written. Use health.Private, the closest provenance for a definition
		// authored outside the public catalog; it excludes your checks from
		// canonical-only filters.
		Provenance: health.Private,
		// Operator-tunable threshold. FixedThreshold values cannot be overridden.
		Thresholds: map[string]health.ThresholdSpec{
			"min_sm_clock_mhz": {
				Value:       int64(1200),
				Category:    health.DefaultThreshold,
				Description: "Minimum acceptable sustained SM clock.",
				Unit:        "MHz",
			},
		},
		Rules: []health.Rule{
			{
				Expression:        "sm_clock_mhz < min_sm_clock_mhz",
				Severity:          health.Critical,
				Message:           "GPU ${gpu_id}: SM clock ${sm_clock_mhz} MHz below floor ${min_sm_clock_mhz} MHz.",
				RecommendedAction: health.GenericDrain(),
			},
		},
	}
}

Fields worth knowing:

• Category and RequiredSignals are the routing key — see below.
• Rules are CEL expressions over the union of the measurement’s signal names and the check’s threshold names. The worst-severity matching rule sets the verdict; ${name} placeholders interpolate signal and threshold values.
• Thresholds are operator-tunable by default. FixedThreshold values can never be overridden; RequiredOverride means the check skips until a fleet supplies a value.
• RecommendedAction’s RemediationActions is an open []RemediationAction (a string type, not a closed enum): use the health.RemediationAction* constants for steps NVIDIA’s own catalog uses, or supply your own string for a remediation step specific to your extension. A downstream remediation engine should treat an unrecognized value as an opaque step, not an error.
• Provenance, MaturityLevel, and the other classification fields are passed through untouched. Use health.Private.
• ConditionScope records how far the detected condition reaches: NodeScope, DomainScope, or ClusterScope. It is NodeScope when omitted, so an existing check keeps its current meaning without change. Unlike the fields above it is not passed through untouched: a collection-failure verdict reports NodeScope whatever the check declares.

Routing: how a measurement reaches your check

The engine feeds a measurement to a check when both hold:

1. they share a Category, and
2. the measurement’s schema carries every one of the check’s RequiredSignals.

Your check is then evaluated once per matching measurement (so a per-GPU check yields one verdict per GPU). If no supplied measurement carries its required signals, the check is skipped NO_APPLICABLE_MEASUREMENT; if its whole category supplied no measurement at all, it is skipped SENSOR_MISSING. Registering the check is what declares that its sensor is expected, so every registered check produces a verdict — a result or a skip — and none disappears from the output.

So a private check only does something if some measurement in its category carries its signals. That is the real work of extending: getting your signals onto a measurement.

Two invariants keep routing unambiguous. Enforce both in your tests:

• Signal names are unique within a category. Two measurement types in the same category must not share a signal name, or a check could match the wrong one. Assert with catalog.CheckSignalUniqueness(measurements...).
• Every check has a provider. A check whose signals no measurement carries can never run. Assert with cat.CheckCoverage(measurements...).

if err := catalog.CheckSignalUniqueness(mygpu.Measurement{}, /* … */); err != nil {
	t.Fatal(err)
}
if err := cat.CheckCoverage(mygpu.Measurement{}, /* … */); err != nil {
	t.Fatal(err)
}

Getting your signals collected (demonstration wiring — subject to change)

The mechanics in this section use the toolkit’s sensor/backend layer, which is demonstration-grade and subject to change. Keep this wiring thin.

There are two situations.

1. Your signals come from a domain the toolkit already enumerates (e.g. GPU via NVML). Attach a prober to the shared backend so the existing category-measurement carries your extra signals. A prober implements nvml.DeviceProber (Measure(ProbeContext) health.Measurement plus an Errored fallback); it reads per-device state into your own measurement type whose Category() is health.GPU and whose struct fields (via json tags) are exactly your check’s RequiredSignals. Install it on the app’s shared NVML backend:

// Layer your prober onto the one NVML enumeration the node and NVLink
// sensors already share — no second session.
a = a.WithGPUBackend(a.NVMLBackend().WithGPUProbe(mygpu.Probe()))

2. Your signals come from a new source. You would implement collection here, but this is exactly the layer flagged as unstable — approach it knowing the interfaces may move.

Registering and running your checks

The app builder is the demonstration entry point. WithChecks registers your definitions alongside the built-ins (declared classification honored; a duplicate ID fails):

func main() {
	a := app.New().WithChecks(mychecks.All()...)
	a = configureGPU(a) // installs your prober; keep this platform-split (see below)
	os.Exit(a.Run(os.Args[1:], os.Stdout, os.Stderr))
}

If you do not want the node runner at all, use the catalog directly — the most stable path:

cat := catalog.New(
	catalog.WithArchitecture("GB200"),
	catalog.WithExecutionContext(health.OperatorInfra),
	catalog.WithOverrides(catalog.NewOverrideSet(catalog.Override{
		CheckID: "gpu_clock_capped", ThresholdName: "min_sm_clock_mhz", Value: int64(1400),
	})),
)
_ = cat.Register(mychecks.GPUClockCapped())
verdict, err := cat.EvaluateAll(measurements...) // measurements you produced yourself

A recommended consumer layout

Keep your module thin and mirror the public per-domain structure. For a private GPU vertical, that is roughly:

health-validation owns the NVML lifecycle, device enumeration, and the built-in sensors; your module owns only its measurement, its checks, and its per-device read.

Testing your extension

• Unit-test each check with catalog.NewEmpty(), Register, then Evaluate (single check) or EvaluateAll (batch) against measurements you construct directly — no hardware needed.
• Assert CheckSignalUniqueness and CheckCoverage over your full measurement-plus-check set, so a renamed signal or a check with no provider fails a test rather than silently never running.

See Architecture for package roles and the evaluation data flow.