HealthEvent Data Model Reference

This document is the authoritative reference for the HealthEvent data model — the fundamental unit of data that flows through NVSentinel. Every fault detected by any health monitor is represented as a HealthEvent, and every downstream module (quarantine, drain, remediation, export) consumes and enriches this structure.

Audience: Developers building custom health monitors, writing CEL quarantine rules, integrating external systems via the event exporter, or debugging the fault-remediation pipeline.

Document type: Reference. Expected lifetime: same as the proto schema it describes. This document must be updated in the same PR that modifies data-models/protobufs/health_event.proto or data-models/protobufs/external_remediation.proto. If you change a field, enum value, or status transition in those files, update the corresponding section here before merging.

Table of Contents

• Where the Model Lives
• Core Message: HealthEvent
• Field Reference
• RecommendedAction Enum
• ProcessingStrategy Enum
• BehaviourOverrides
• Entity (Impacted Resources)
• Storage Wrapper: HealthEventWithStatus
• HealthEventStatus Lifecycle
• CRD Projection: HealthEventResource
• External Remediation Request
• Tracing Correlation
• Examples

Where the Model Lives

Core Message: HealthEvent

1
message HealthEvent \{
2
  uint32 version = 1;
3
  string agent = 2;
4
  string componentClass = 3;
5
  string checkName = 4;
6
  bool isFatal = 5;
7
  bool isHealthy = 6;
8
  string message = 7;
9
  RecommendedAction recommendedAction = 8;
10
  repeated string errorCode = 9;
11
  repeated Entity entitiesImpacted = 10;
12
  map<string, string> metadata = 11;
13
  google.protobuf.Timestamp generatedTimestamp = 12;
14
  string nodeName = 13;
15
  BehaviourOverrides quarantineOverrides = 14;
16
  BehaviourOverrides drainOverrides = 15;
17
  ProcessingStrategy processingStrategy = 16;
18
  string id = 17;
19
  string customRecommendedAction = 18;
20
\}

Field Reference

If you are building a custom health monitor, the table below shows what happens when a field is omitted or left at its zero value. See Writing a Health Monitor for the end-to-end walkthrough.

RecommendedAction Enum

Directs the fault-remediation module on what repair action to trigger.

Routing logic (in fault-remediation):

1
// data-models/pkg/model/health_event_extentions.go
2
func GetEffectiveActionName(he *protos.HealthEvent) string \{
3
    if he.RecommendedAction == protos.RecommendedAction_CUSTOM \{
4
        return he.CustomRecommendedAction
5
    \}
6
    return he.RecommendedAction.String()
7
\}

The effective action name is looked up in the Helm-configured maintenance.actions map to find the CRD template, API group, kind, and completion condition to create.

ProcessingStrategy Enum

Controls whether downstream modules (fault-quarantine, node-drainer, fault-remediation) may modify cluster state in response to this event.

Where strategy is enforced:

• Platform connector normalizes UNSPECIFIED → EXECUTE_REMEDIATION at ingestion.
• Deduplication transformer downgrades repeated unhealthy events to STORE_AND_ANALYSE within a suppression window (see ADR-039).
• Fault-quarantine skips events where processingStrategy != EXECUTE_REMEDIATION.
• Health-events-analyzer queries for STORE_AND_ANALYSE events specifically.

BehaviourOverrides

1
message BehaviourOverrides \{
2
  bool force = 1;
3
  bool skip = 2;
4
\}

Per-event overrides that let a health monitor bypass default pipeline behavior.

quarantineOverrides

drainOverrides

Use case: Debug/test events (e.g. fault-injection demos) set drainOverrides.skip = true to exercise the remediation pipeline without waiting for pod eviction.

Entity (Impacted Resources)

1
message Entity \{
2
  string entityType = 1;
3
  string entityValue = 2;
4
\}

Identifies specific hardware entities affected by the fault.

The entitiesImpacted array is propagated to:

• Tracing: each entity becomes a span attribute health_event.entities_impacted.<type>.
• Event exporter: serialized into the CloudEvents payload.
• GPU Reset: the janitor uses GPU UUIDs from entities to target the reset CRD.

Storage Wrapper: HealthEventWithStatus

Events are stored in MongoDB (or PostgreSQL) wrapped in a status envelope:

1
// data-models/pkg/model/health_event_extentions.go
2
type HealthEventWithStatus struct \{
3
    CreatedAt         time.Time                 `bson:"createdAt"`
4
    HealthEvent       *protos.HealthEvent       `bson:"healthevent,omitempty"`
5
    HealthEventStatus *protos.HealthEventStatus `bson:"healtheventstatus"`
6
\}

The HealthEvent portion is immutable after ingestion (except for dedup strategy downgrade). The HealthEventStatus portion is progressively enriched by each downstream module as it processes the event.

HealthEventStatus Lifecycle

1
message HealthEventStatus \{
2
  string nodeQuarantined = 1;
3
  google.protobuf.Timestamp quarantineFinishTimestamp = 2;
4
  OperationStatus userPodsEvictionStatus = 3;
5
  google.protobuf.Timestamp drainFinishTimestamp = 4;
6
  google.protobuf.BoolValue faultRemediated = 5;
7
  google.protobuf.Timestamp lastRemediationTimestamp = 6;
8
  map<string, string> spanIds = 7;
9
\}

Each field is written by a specific module at a specific pipeline stage:

OperationStatus values (userPodsEvictionStatus.status)

Status progression diagram

CRD Projection: HealthEventResource

For Kubernetes-native storage (experimental), the HealthEvent maps to a CRD:

1
message HealthEventResource \{
2
  option (protoc_gen_crd.k8s_crd) = \{
3
    api_group: "healthevents.dgxc.nvidia.com",
4
    kind: "HealthEventResource",
5
    plural: "healtheventresources",
6
    singular: "healtheventresource",
7
    categories: ["nvidia", "gpu"]
8
  \};
9
  HealthEvent spec = 1;
10
  HealthEventStatus status = 2;
11
\}

The spec field holds the immutable HealthEvent; the status field holds the progressively-enriched HealthEventStatus. This mirrors the standard Kubernetes spec/status convention.

External Remediation Request

When the configured remediation action produces an ExternalRemediationRequest (ExtRR) — for any recommendedAction, not only CUSTOM — the full HealthEvent is embedded in the CRD spec:

1
message ExternalRemediationRequestSpec \{
2
  HealthEvent healthEvent = 1;
3
\}

The ExtRR is cluster-scoped (like RebootNode, TerminateNode, GPUReset CRDs) and carries status conditions for coordination with external repair systems:

See ADR-040 for the full design.

Tracing Correlation

Each module that processes an event writes its span ID into HealthEventStatus.spanIds[serviceName]. This enables end-to-end trace correlation:

platform-connector → fault-quarantine → node-drainer → fault-remediation

The span ID written to the datastore becomes the parent span for the next module’s processing, creating a causal chain across independent services that communicate only through MongoDB change streams.

See Distributed Tracing for setup and querying.

Examples

GPU XID fault (fatal, triggers reboot)

1
\{
2
  "version": 1,
3
  "agent": "syslog-health-monitor",
4
  "componentClass": "GPU",
5
  "checkName": "xid-check",
6
  "isFatal": true,
7
  "isHealthy": false,
8
  "message": "XID 79: GPU has fallen off the bus",
9
  "recommendedAction": "RESTART_BM",
10
  "errorCode": ["XID_79"],
11
  "entitiesImpacted": [
12
    \{"entityType": "GPU", "entityValue": "GPU-a1b2c3d4-e5f6-a7b8-c9d0-e1f2a3b4c5d6"\},
13
    \{"entityType": "PCI", "entityValue": "0000:17:00.0"\}
14
  ],
15
  "metadata": \{"driverVersion": "570.86.15"\},
16
  "nodeName": "gpu-node-42",
17
  "processingStrategy": "EXECUTE_REMEDIATION"
18
\}

Recovery event (un-quarantine)

1
\{
2
  "version": 1,
3
  "agent": "gpu-health-monitor",
4
  "componentClass": "GPU",
5
  "checkName": "dcgm-health-check",
6
  "isFatal": false,
7
  "isHealthy": true,
8
  "message": "All GPU health checks passing",
9
  "recommendedAction": "NONE",
10
  "nodeName": "gpu-node-42",
11
  "processingStrategy": "EXECUTE_REMEDIATION"
12
\}

Custom remediation with drain skip

1
\{
2
  "version": 1,
3
  "agent": "custom-monitor",
4
  "componentClass": "Software",
5
  "checkName": "memory-pressure",
6
  "isFatal": true,
7
  "isHealthy": false,
8
  "message": "Uncorrectable memory pressure detected",
9
  "recommendedAction": "CUSTOM",
10
  "customRecommendedAction": "slack-notify-ops",
11
  "drainOverrides": \{"skip": true\},
12
  "nodeName": "worker-7",
13
  "processingStrategy": "EXECUTE_REMEDIATION"
14
\}

Related Documentation

• Data Flow — end-to-end sequence diagrams
• Platform Connectors — ingestion and validation
• Fault Quarantine — CEL rule evaluation
• Node Drainer — eviction strategies
• Fault Remediation — CRD creation and action routing
• Event Exporter — CloudEvents transformation
• ADR-021: Health Event Property Overrides
• ADR-025: Processing Strategy
• ADR-039: Health Event Deduplication