Skip to navigation

Verdict Schema

View as Markdown

Type definitions behind the verdict JSON. Enums serialize as strings, and struct fields are listed with their JSON key names. The machine-readable contract is node-verdict.schema.json.

Structs

AffectedComponent

AffectedComponent summarizes a category with non-Pass results.

FieldJSON nameTypeDescription
CheckIDscheck_ids[]string
Entitiesentities[]string
CategorycategoryCategory
MaxSeveritymax_severitySeverity

CheckDefinition

CheckDefinition is the canonical shape of a health check in the catalog. Each definition encodes NVIDIA’s operational judgment about a specific hardware or driver condition: which signals to examine, what thresholds apply, what action to recommend, and how far the condition reaches. Check definitions are data. They carry no sensor or hardware dependencies. Evaluation happens when a caller provides signal values to [Catalog.Evaluate].

FieldJSON nameTypeDescription
Thresholdsthresholdsmap[string]ThresholdSpec
FeasibilityClassfeasibility_classstring
Descriptiondescriptionstring
Namenamestring
IDidstring
Dependenciesdependencies[]stringDependencies names check IDs whose Critical verdict explains this check’s inability to collect. Checks never read each other’s results, so a dependency has no effect on whether this check runs, what it reports, or its severity. Its only effect is that a verdict for an entity this check could not read at all gains a message naming the dependency as the cause. An ID that is not registered is an inactive edge by design, so a consumer may register any subset of the catalog. Validate rejects only a check that names itself.
Architecturesarchitectures[]string
Rulesrules[]Rule
ExecutionContextsexecution_contexts[]ExecutionContext
Prerequisitesprerequisites[]Prerequisite
RequiredSignalsrequired_signals[]string
RecommendedActionrecommended_actionRecommendedAction
Timeouttimeouttime.Duration
MaturityLevelmaturity_levelMaturityLevel
SchedulingCadencescheduling_cadenceSchedulingCadence
CategorycategoryCategory
ConditionScopecondition_scopeConditionScope
ProvenanceprovenanceProvenance
ExecutionClassexecution_classExecutionClass

CheckVerdict

CheckVerdict is the outcome of evaluating a single check against signal values. It carries all context needed for downstream systems to make remediation decisions without referencing catalog source. Every Rule whose expression matched contributes one entry to Messages. Severity is the most severe across the matching rules; RecommendedAction comes from the highest-priority matching rule’s override (or the check default when no override applies).

FieldJSON nameTypeDescription
Timestamptimestamptime.Time
ObservedValuesobserved_valuesmap[string]any
Thresholdsthresholdsmap[string]ResolvedThreshold
CheckIDcheck_idstring
SpanIDspan_idstringUnique per check invocation.
TraceIDtrace_idstringShared by all checks in the same run.
SkipDetailskip_detailstring
Messagesmessages[]string
AffectedEntitiesaffected_entities[]string
RecommendedActionrecommended_actionRecommendedAction
SeverityseveritySeverity
CategorycategoryCategory
ConditionScopecondition_scopeConditionScopeBlast radius; NODE when collection failed.
SkipReasonskip_reasonSkipReason
MaturityLevelmaturity_levelMaturityLevel
Skippedskippedbool
NVIDIAEndorsednvidia_endorsedbool

ErroredMeasurement

ErroredMeasurement records one per-signal collection failure on a measurement. Sensors append entries via the typed measurement’s AddError method (e.g. [GPUMeasurement.AddError]) as they encounter failures while measuring different fields, so a single measurement may carry several entries — one per failing call. The catalog fails a check Critical when any entry’s FailedSignals overlaps with the check’s RequiredSignals; the verdict’s message joins the matching entries’ Err.Error() strings. Checks whose required signals were all collected still run normally on the partial data. The zero value is not valid; construct entries with [NewErroredMeasurement] (or via the typed measurement’s AddError), which guarantees Err is non-nil so consumers always have a surfaceable failure cause.

FixedThresholdError

FixedThresholdError is returned when an operator attempts to override a threshold classified as Fixed.

FieldJSON nameTypeDescription
ThresholdNamestring
CheckIDstring

NodeVerdict

NodeVerdict aggregates per-check verdicts into a node-level health assessment.

FieldJSON nameTypeDescription
SchemaVersionschema_versionstring
CatalogVersioncatalog_versionstringCatalog module version compiled into the producer.
Timestamptimestamptime.Time
NodeIDnode_idstringOpaque identifier for the node under test.
TraceIDtrace_idstringShared by all checks in this run; echoed on each CheckVerdict.
SeverityByDomainseverity_by_domainmap[Category]Severity
AffectedComponentsaffected_components[]AffectedComponent
CheckResultscheck_results[]CheckVerdict
SkippedChecksskipped_checks[]CheckVerdict
OverallSeverityoverall_severitySeverity

Prerequisite

Prerequisite declares a minimum software requirement for a check.

FieldJSON nameTypeDescription
Namenamestring
MinVersionmin_versionstring

RecommendedAction

RecommendedAction is the structured remediation a check carries. OperationalState is the high-level state change to apply; RemediationActions is the ordered sequence of remediation steps a cluster operator’s remediation engine should work through, and is only meaningful (and only emitted in JSON) when OperationalState is [OperationalDrain]. Use [Drain] / [Monitor] / [NoAction] for readable call sites instead of constructing the struct directly.

FieldJSON nameTypeDescription
RemediationActionsremediation_actions[]RemediationAction
OperationalStateoperational_stateOperationalState

ResolvedThreshold

ResolvedThreshold is a threshold value after operator overrides have been applied. It tracks provenance so downstream systems can audit whether NVIDIA defaults or operator-supplied values were used.

FieldJSON nameTypeDescription
Valuevalueany
OriginalValueoriginal_valueany
CategorycategoryThresholdCategory
Overriddenoverriddenbool

Rule

Rule defines a single evaluation condition within a check. Every rule is evaluated and the matches are aggregated: the verdict’s severity is the worst across matching rules, Messages holds one entry per matching rule in declaration order, and RecommendedAction comes from the highest-severity matching rule (ties go to declaration order); its override applies when present, otherwise the check-level action is used.

FieldJSON nameTypeDescription
ExpressionexpressionstringExpression is a CEL expression that evaluates to a boolean. Variables available are the union of signal field names and threshold names from the parent CheckDefinition.
MessagemessagestringMessage describes the detected condition. May contain ${variable} placeholders interpolated from signal and threshold values.
RecommendedActionrecommended_actionRecommendedActionRecommendedAction, when non-zero (see [RecommendedAction.IsZero]), overrides the parent [CheckDefinition.RecommendedAction] in the resulting [CheckVerdict]. Use it when a specific failure mode warrants a different remediation sequence than the check’s default. A zero value leaves the check-level action in place.
SeverityseveritySeveritySeverity produced when this rule matches.

ThresholdSpec

ThresholdSpec defines a threshold in a check definition. It carries the catalog-authored value and its governance classification.

FieldJSON nameTypeDescription
Valuevalueany
PerArchitectureper_architecturemap[string]anyPerArchitecture overrides Value for a specific detected architecture (keyed by the catalog architecture identifier, e.g. “H100”). It lets one check carry per-architecture defaults — e.g. NVLink speed differs by GPU generation. Architectures absent from the map fall back to Value. When the check declares Architectures, registration requires this map to include every listed architecture; Value may then be omitted.
Descriptiondescriptionstring
Unitunitstring
CategorycategoryThresholdCategory
RequiredOverriderequired_overrideboolRequiredOverride marks a threshold with no meaningful catalog value that the operator must supply per fleet. It is independent of Category, which governs whether an override is permitted at all; pairing it with [FixedThreshold] is rejected at registration. A check whose RequiredOverride threshold is unset is skipped with [ThresholdUnconfigured] rather than evaluated.

Enums

Category

Category groups related checks by the hardware or interconnect domain they monitor. It determines how failure domains are reported in node-level verdicts.

NameJSON valueDescription
GPUgpuGPU covers GPU device health: driver-reported events (XID), thermal state, driver status, power, and clock behavior.
NVLinknvlinkNVLink covers NVLink interconnect health: replay counters, link errors, and fabric state.
PCIepciePCIe covers PCI Express bus health: link width, speed degradation, and error counters.
InfiniBandinfinibandInfiniBand covers InfiniBand network health: port state, error rates, and link integrity.
StoragestorageStorage covers storage health in NVIDIA-validated configurations: NVMe state, SMART counters.
NodenodeNode covers host-wide GPU-system signals that are not bound to any single device — GPU count, driver version, and similar inventory checks.
XidxidXid covers NVRM Xid fault events logged by the NVIDIA driver to the kernel ring buffer. Xid measurements are collected separately from GPU telemetry and route only to Xid checks.
CPUcpuCPU covers host processor inventory and availability signals.

ConditionScope

ConditionScope records how far the condition a check detects reaches: the smallest boundary that must be inspected to explain it. It is independent of [RecommendedAction], which stays node-local: a ClusterScope verdict reports what the fault implicates, not that the cluster should be acted on. NodeScope is the zero value, so a check that omits it is treated as node-local, which is how consumers already treat every check. The NODE value is distinct from the [Node] Category: one is a blast radius, the other a hardware domain. They serialize with different casing.

NameJSON valueDescription
NodeScopeNODEThis node’s own hardware explains the condition.
DomainScopeDOMAINExplaining it requires the NVLink domain shared with peer nodes.
ClusterScopeCLUSTERExplaining it requires the cluster network fabric beyond the node.

ExecutionClass

ExecutionClass distinguishes passive observation from active probing.

NameJSON valueDescription
PassivePASSIVEReads existing state without perturbing the system.
ActiveACTIVEMay exercise hardware or run diagnostic commands.

ExecutionContext

ExecutionContext identifies where a health check can run within a GPU cluster.

NameJSON valueDescription
OperatorInfraOPERATOR_INFRAHost OS or CSP privileged control-plane VM.
TenantGuestTENANT_GUESTTenant VM, container, or K8s worker node.
OutOfBandOUT_OF_BANDBMC via Redfish API.

MaturityLevel

MaturityLevel describes how production-ready a check is.

NameJSON valueDescription
StablestableVetted for production use.
ExperimentalexperimentalUnder evaluation; may change or be withdrawn.

OperationalState

OperationalState names the node-level state change a verdict recommends. It is the high-level “what should happen to the node?” answer; the [RemediationAction] companion sequence narrows the why when the state is [OperationalDrain].

NameJSON valueDescription
OperationalReadyreadyOperationalReady is the zero value: the node stays ready for work and no node-level action is recommended.
OperationalMonitormonitorOperationalMonitor keeps the node in service but flags the finding for trending; no action required right now.
OperationalDraindrainOperationalDrain removes the node from scheduling so the finding can be remediated. The accompanying [RemediationAction] sequence names the specific remediation steps.

Provenance

Provenance identifies who authored a check definition and determines the trust level of its verdicts.

NameJSON valueDescription
NVIDIAnvidiaAuthored and maintained by NVIDIA.
CommunitycommunityCommunity-contributed, NVIDIA-reviewed.
PrivateprivatePrivately authored for a specific deployment.

SchedulingCadence

SchedulingCadence describes how often a check is meant to run in a node’s operational lifecycle. The cadences nest by frequency — Continuous > Prolog > MaintenanceWindow — so a check declared at a more frequent cadence is also eligible to run in every less frequent window: a Continuous check runs at prolog and in maintenance windows, a Prolog check runs in maintenance windows. See [SchedulingCadence.Covers]. Continuous is the zero value: a check that omits SchedulingCadence defaults to the most frequent cadence, so it runs in every window rather than being silently excluded from more frequent ones.

NameJSON valueDescription
ContinuousCONTINUOUSDuring normal operation (most frequent); the zero-value default.
PrologPROLOGBefore job dispatch.
MaintenanceWindowMAINTENANCEDuring scheduled maintenance (least frequent).

Severity

Severity represents the graduated health severity of a check result. The levels form a total order: Pass < Info < Warning < Critical. Unknown is a special case indicating the check could not complete.

NameJSON valueDescription
PassPASSPass indicates no condition detected. The monitored state is within normal operating bounds.
InfoINFOInfo indicates an anomaly detected within safe operating bounds. The node may remain in service; monitor for trend.
WarningWARNINGWarning indicates a condition that degrades performance or increases failure risk. The node may continue in service for a bounded window pending operator action.
CriticalCRITICALCritical indicates imminent risk of failure, data corruption, or unrecoverable state. The node should be removed from service without delay.
UnknownUNKNOWNUnknown indicates the check could not complete. Service eligibility for the affected component cannot be assessed. This typically results from tool failures or insufficient data.

SkipReason

SkipReason provides a machine-readable code for why a check did not run.

NameJSON valueDescription
NotSkippedNOT_SKIPPEDCheck ran normally.
ArchitectureUnsupportedSIGNAL_UNAVAILABLE_ARCHITECTUREWrong GPU architecture.
DeploymentUnsupportedSIGNAL_UNAVAILABLE_DEPLOYMENT_MODELRequires different execution context.
CheckDisabledCHECK_DISABLEDOperator disabled this check.
InsufficientHistoryINSUFFICIENT_HISTORYTime-window data not yet available.
ThresholdUnconfiguredTHRESHOLD_UNCONFIGUREDRequired threshold not supplied for this fleet.
NoApplicableMeasurementNO_APPLICABLE_MEASUREMENTNo supplied measurement carries the check’s required signals.
SensorMissingSENSOR_MISSINGSensorMissing marks a check whose whole category was expected but never arrived: the sensor is absent, unconfigured, or its backend was never constructed. It is distinct from NoApplicableMeasurement, which means the category was collected but no measurement carried this check’s required signals, and from a Critical errored verdict, which means the sensor ran and failed.
CadenceMismatchCADENCE_MISMATCHCheck is not scheduled for the requested cadence.

ThresholdCategory

ThresholdCategory classifies how a threshold value may be modified by operators. The catalog enforces governance rules: Fixed thresholds reject override attempts with an error; Default and Adjustable thresholds accept operator-supplied values.

NameJSON valueDescription
DefaultThresholdDEFAULTDefaultThreshold is NVIDIA’s recommended starting value. Operators may override it via a parameter file.
FixedThresholdFIXEDFixedThreshold is a hard boundary that cannot be overridden. Attempts to override produce an error and the check will not run with an invalid override applied.
AdjustableThresholdADJUSTABLEAdjustableThreshold may be tuned by operators within documented bounds.