Tutorial: Writing a New Health Monitor

This tutorial walks you through building a brand-new health monitor from an empty directory to a deployed, fault-reporting component — without needing any GPU hardware.

By the end you will have:

• A working Go health monitor that detects a fault and reports it to NVSentinel.
• A container image for it, deployed to a live cluster as a DaemonSet.
• Verification that NVSentinel sees the fault (it surfaces as a Kubernetes node condition).

Who is this for? Teams already running NVSentinel who want to extend it with a new fault detector.

Just want the AI to do it? Jump to Appendix: One-shot AI prompt.

Prerequisites

• Tools: docker and kubectl.
• A cluster with NVSentinel running. Verification needs NVSentinel already running — at minimum platform-connectors (it sets the node condition this tutorial verifies). See DEVELOPMENT.md if you need to bring a cluster up locally, and confirm the core pods are green with kubectl get pods -n nvsentinel before verifying — otherwise a failed verification just means an unhealthy cluster, not a broken monitor.

1. What is a health monitor

NVSentinel detects faults on cluster nodes and remediates them (cordon → drain → break-fix). The thing that detects a fault is a health monitor.

A health monitor is just a standalone process that:

1. Observes something (GPU via DCGM, kernel logs, a cloud API, a file in /proc, …).
2. Decides whether a node/component is healthy or not.
3. Sends a HealthEvent to the platform-connector over a local gRPC Unix domain socket.

The platform-connector turns fatal events into Kubernetes node conditions — and that’s where this tutorial ends. Your monitor only has to detect and report.

Key consequences of this design:

• Your monitor never talks to the Kubernetes API to cordon/drain. It only emits events.
• The transport is a Unix socket on the node, shared via a hostPath volume.
• The contract is a single gRPC method and a single message type. That’s all you implement.

2. The contract (the only API you must speak)

The contract lives in data-models/protobufs/health_event.proto. Go types are generated into the package github.com/nvidia/nvsentinel/data-models/pkg/protos.

There is exactly one RPC:

1
service PlatformConnector \{
2
  rpc HealthEventOccurredV1(HealthEvents) returns (google.protobuf.Empty) \{\}
3
\}
4
5
message HealthEvents \{
6
  uint32 version = 1;             // set to 1
7
  repeated HealthEvent events = 2;
8
\}

You send a batch (HealthEvents) containing one or more HealthEvents.

The HealthEvent message

The two enums you care about:

1
enum ProcessingStrategy \{
2
  UNSPECIFIED = 0;          // normalized to EXECUTE_REMEDIATION by platform-connector
3
  EXECUTE_REMEDIATION = 1;  // normal: downstream modules may cordon/drain/remediate
4
  STORE_ONLY = 2;           // observability-only: persisted/exported, never mutates the cluster
5
\}

RecommendedAction values (from data-models/protobufs/health_event.proto):

The golden rules

1. agent is your identity. Downstream consumers match on it, so pick a stable, unique name.
2. Prefer emitting on state transitions, not every poll. Send an unhealthy event when a fault appears and a healthy event when it clears. This is a best-effort optimization to cut noise, not a correctness requirement — repeated events are deduped, so re-sending the same state is safe.
3. A healthy event clears the node condition. Send isHealthy=true, recommendedAction=NONE when things recover.
4. isFatal=true is what triggers action. Fatal events surface as a Kubernetes node condition; non-fatal events become Kubernetes Events instead.

3. Choose your workload model

A monitor can observe almost anything — node-local state (/proc, /sys, GPUs, logs), the Kubernetes API, or an external service. That choice only affects the Kubernetes workload type:

• DaemonSet (one pod per node) — for node-local state; each pod reports its own node.
• Deployment (single replica) — for a single cluster-wide source (the K8s API or an external API); it sets nodeName from the observed entity instead of from NODE_NAME.

The publishing path is identical either way. This tutorial builds a DaemonSet, the most common shape.

Our worked example is a demo-health-monitor that periodically opens a TCP connection to a configurable endpoint (CHECK_TARGET) the node depends on. It reports the node unhealthy when the endpoint is unreachable and healthy again once it recovers.

4. Scaffold the module

Your monitor is an ordinary, standalone Go module. It depends on two NVSentinel Go modules:

• github.com/nvidia/nvsentinel/data-models — the event contract (the HealthEvent types).
• github.com/nvidia/nvsentinel/commons — the shared publisher (commons/pkg/healthpub), which adds socket-presence gating, retries, and Prometheus metrics over the raw gRPC client.

Create the module (any module path works):

$
mkdir demo-health-monitor && cd demo-health-monitor
$
go mod init github.com/<your-org>/demo-health-monitor

Add the two dependencies. They live in subdirectories of the NVSentinel repo and aren’t published as tagged releases, so fetch them by commit (@main, or pin a specific commit for reproducibility). Because commons refers to data-models by an internal placeholder version, add one replace so your build resolves the real commit:

$
export GOTOOLCHAIN=auto   # the modules need Go 1.26+; this fetches it automatically
$
$
# 1. Fetch the contract module, then capture the exact version go resolved.
$
go get github.com/nvidia/nvsentinel/data-models@main
$
DM=$(go list -m github.com/nvidia/nvsentinel/data-models | awk '\{print $2\}')
$
$
# 2. Point commons' internal data-models placeholder at that same version.
>
go mod edit -replace github.com/nvidia/nvsentinel/data-models=github.com/nvidia/nvsentinel/data-models@"$DM"
>
>
# 3. Fetch the publisher module.
>
go get github.com/nvidia/nvsentinel/commons@main

5. Write the monitor

Create main.go. The structure is: parse config from env → dial the socket → poll → build an event on each state change → publish it through the shared healthpub publisher.

1
package main
2
3
import (
4
	"context"
5
	"errors"
6
	"fmt"
7
	"log/slog"
8
	"net"
9
	"os"
10
	"os/signal"
11
	"strconv"
12
	"syscall"
13
	"time"
14
15
	"google.golang.org/grpc"
16
	"google.golang.org/grpc/credentials/insecure"
17
	"google.golang.org/protobuf/types/known/timestamppb"
18
19
	"github.com/nvidia/nvsentinel/commons/pkg/healthpub"
20
	pb "github.com/nvidia/nvsentinel/data-models/pkg/protos"
21
)
22
23
const (
24
	agentName      = "demo-health-monitor"
25
	componentClass = "Node"
26
	checkName      = "NetworkReachability"
27
)
28
29
func main() \{
30
	if err := run(); err != nil \{
31
		slog.Error("fatal error", "error", err)
32
		os.Exit(1)
33
	\}
34
\}
35
36
func run() error \{
37
	// --- Configuration from environment ---
38
	nodeName := os.Getenv("NODE_NAME")
39
	if nodeName == "" \{
40
		return fmt.Errorf("NODE_NAME env var must be set")
41
	\}
42
	// checkTarget is a host:port the node should always be able to reach. When it
43
	// becomes unreachable we report the node unhealthy. The default is the in-cluster
44
	// Kubernetes API, which is reachable from any pod; override it for your dependency.
45
	checkTarget := envOrDefault("CHECK_TARGET", "kubernetes.default.svc:443")
46
	socketPath := envOrDefault("SOCKET_PATH", "/var/run/nvsentinel.sock")
47
48
	pollSeconds, err := strconv.Atoi(envOrDefault("POLL_INTERVAL_SECONDS", "10"))
49
	if err != nil \{
50
		return fmt.Errorf("invalid POLL_INTERVAL_SECONDS: %w", err)
51
	\}
52
	// time.NewTicker panics on a zero or negative duration, so reject bad values
53
	// before we build the ticker.
54
	if pollSeconds <= 0 \{
55
		return fmt.Errorf("POLL_INTERVAL_SECONDS must be > 0, got %d", pollSeconds)
56
	\}
57
58
	dialTimeout, err := strconv.Atoi(envOrDefault("CHECK_TIMEOUT_SECONDS", "3"))
59
	if err != nil \{
60
		return fmt.Errorf("invalid CHECK_TIMEOUT_SECONDS: %w", err)
61
	\}
62
	if dialTimeout <= 0 \{
63
		return fmt.Errorf("CHECK_TIMEOUT_SECONDS must be > 0, got %d", dialTimeout)
64
	\}
65
66
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
67
	defer stop()
68
69
	// --- Connect to platform-connector over the Unix socket ---
70
	target := "unix://" + socketPath
71
	conn, err := grpc.NewClient(target, grpc.WithTransportCredentials(insecure.NewCredentials()))
72
	if err != nil \{
73
		return fmt.Errorf("dial %s: %w", target, err)
74
	\}
75
	defer conn.Close()
76
	client := pb.NewPlatformConnectorClient(conn)
77
78
	// Publish through the shared publisher instead of the raw client: it gates on
79
	// socket presence, retries transient errors, and exports Prometheus metrics.
80
	pub := healthpub.New(client, target, agentName)
81
82
	slog.Info("starting", "agent", agentName, "node", nodeName,
83
		"checkTarget", checkTarget, "socket", socketPath)
84
85
	// --- Poll loop (edge-triggered: only send on health changes) ---
86
	ticker := time.NewTicker(time.Duration(pollSeconds) * time.Second)
87
	defer ticker.Stop()
88
89
	var lastHealthy *bool
90
	for \{
91
		select \{
92
		case <-ctx.Done():
93
			slog.Info("shutting down")
94
			return nil
95
		case <-ticker.C:
96
			// Healthy when we can open a TCP connection to checkTarget within the timeout.
97
			checkErr := checkReachable(checkTarget, time.Duration(dialTimeout)*time.Second)
98
			healthy := checkErr == nil
99
			slog.Info("connectivity check", "target", checkTarget, "healthy", healthy)
100
101
			// Only emit when health state flips.
102
			if lastHealthy != nil && *lastHealthy == healthy \{
103
				continue
104
			\}
105
106
			event := buildEvent(nodeName, checkTarget, checkErr)
107
			err := pub.Publish(ctx, &pb.HealthEvents\{
108
				Version: 1,
109
				Events:  []*pb.HealthEvent\{event\},
110
			\})
111
			if errors.Is(err, healthpub.ErrPlatformConnectorUnavailable) \{
112
				// PC is down: do NOT advance lastHealthy so the next poll
113
				// re-emits this state with a fresh timestamp.
114
				slog.Warn("platform-connector unavailable; will retry next tick")
115
				continue
116
			\}
117
			if err != nil \{
118
				slog.Error("publish failed after retries", "error", err)
119
				continue // also do not advance lastHealthy
120
			\}
121
122
			slog.Info("sent health event", "healthy", healthy)
123
			lastHealthy = &healthy
124
		\}
125
	\}
126
\}
127
128
// checkReachable returns nil if a TCP connection to addr (host:port) succeeds within
129
// timeout, otherwise the underlying dial error (surfaced verbatim in the event message).
130
func checkReachable(addr string, timeout time.Duration) error \{
131
	conn, err := net.DialTimeout("tcp", addr, timeout)
132
	if err != nil \{
133
		return err
134
	\}
135
	return conn.Close()
136
\}
137
138
func buildEvent(nodeName, target string, checkErr error) *pb.HealthEvent \{
139
	healthy := checkErr == nil
140
	event := &pb.HealthEvent\{
141
		Version:            1,
142
		Agent:              agentName,
143
		ComponentClass:     componentClass,
144
		CheckName:          checkName,
145
		NodeName:           nodeName,
146
		GeneratedTimestamp: timestamppb.Now(),
147
		ProcessingStrategy: pb.ProcessingStrategy_EXECUTE_REMEDIATION,
148
		IsHealthy:          healthy,
149
		IsFatal:            !healthy,
150
	\}
151
	if healthy \{
152
		event.Message = fmt.Sprintf("%s reachable", target)
153
		event.RecommendedAction = pb.RecommendedAction_NONE
154
	\} else \{
155
		// Surface the raw dial error verbatim.
156
		event.Message = checkErr.Error()
157
		event.RecommendedAction = pb.RecommendedAction_CONTACT_SUPPORT
158
	\}
159
	return event
160
\}
161
162
func envOrDefault(key, fallback string) string \{
163
	if v := os.Getenv(key); v != "" \{
164
		return v
165
	\}
166
	return fallback
167
\}

Tidy and build:

$
go mod tidy
$
go build ./...

That’s a complete, contract-correct health monitor. Everything else is packaging, deployment, and hardening.

6. Containerize

Package the monitor as a container image with a small multi-stage Dockerfile in the monitor directory. Because dependencies are fetched from the module proxy (not local paths), the build context is just the monitor directory itself:

1
FROM public.ecr.aws/docker/library/golang:1.26.3-trixie AS builder
2
WORKDIR /src
3
4
# Manifests first for better layer caching. go.mod/go.sum already pin the
5
# data-models/commons versions and the replace directive.
6
COPY go.mod go.sum ./
7
RUN go mod download
8
9
COPY . .
10
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o demo-health-monitor .
11
12
FROM gcr.io/distroless/static-debian13
13
COPY --from=builder /src/demo-health-monitor /demo-health-monitor
14
ENTRYPOINT ["/demo-health-monitor"]

Build the image and push it to a registry your cluster can pull from. We use Docker Hub here — replace <dockerhub-user> with your Docker Hub username. Run from the monitor directory:

$
cd demo-health-monitor
$
$
docker login     # once, to authenticate to Docker Hub
$
$
docker build -t docker.io/<dockerhub-user>/demo-health-monitor:dev .
$
$
# Push so the cluster can pull it.
$
docker push docker.io/<dockerhub-user>/demo-health-monitor:dev

If you use a private repository, make sure the cluster has the required image pull credentials (e.g. an imagePullSecret in the nvsentinel namespace, referenced from the DaemonSet’s spec.template.spec.imagePullSecrets). A public repository needs none.

7. Run and verify

Deploy the image as a DaemonSet, then trigger a fault and watch it flow through the pipeline. (Needs the Prerequisites.)

Apply the DaemonSet. We start CHECK_TARGET at a guaranteed-unreachable, reserved TEST-NET address so the monitor reports a fault right away — then we’ll point it at a reachable endpoint to clear it. It mounts the platform-connector socket (hostPath /var/run/nvsentinel → /var/run) and sets NODE_NAME via the downward API:

$
kubectl apply -f - <<'YAML'
$
apiVersion: apps/v1
$
kind: DaemonSet
$
metadata:
$
  name: demo-health-monitor
$
  namespace: nvsentinel
$
  labels:
$
    app.kubernetes.io/name: demo-health-monitor
$
spec:
$
  selector:
$
    matchLabels:
$
      app.kubernetes.io/name: demo-health-monitor
$
  template:
$
    metadata:
$
      labels:
$
        app.kubernetes.io/name: demo-health-monitor
$
    spec:
$
      containers:
$
        - name: demo-health-monitor
$
          image: docker.io/<dockerhub-user>/demo-health-monitor:dev
$
          imagePullPolicy: IfNotPresent
$
          env:
$
            - name: NODE_NAME
$
              valueFrom:
$
                fieldRef:
$
                  fieldPath: spec.nodeName
$
            - name: CHECK_TARGET
$
              value: "192.0.2.1:9"
$
            - name: POLL_INTERVAL_SECONDS
$
              value: "10"
$
            - name: SOCKET_PATH
$
              value: "/var/run/nvsentinel.sock"
$
          volumeMounts:
$
            - name: socket
$
              mountPath: /var/run
$
      volumes:
$
        - name: socket
$
          hostPath:
$
            path: /var/run/nvsentinel
$
            type: DirectoryOrCreate
$
YAML

Confirm the pods are up:

$
kubectl rollout status ds/demo-health-monitor -n nvsentinel
$
kubectl get pods -l app.kubernetes.io/name=demo-health-monitor -n nvsentinel

Trigger and verify

Assumes the Prerequisites are met, these steps are cluster-agnostic.

Because CHECK_TARGET is unreachable, within one poll interval the monitor emits an unhealthy, fatal HealthEvent. Platform-connector turns fatal events into a Kubernetes node condition automatically, using your checkName (NetworkReachability) as the condition Type. Pick any node a monitor pod runs on and watch the condition appear:

$
# Any node running the monitor (each pod reports its own node).
$
NODE=$(kubectl get pods -n nvsentinel -l app.kubernetes.io/name=demo-health-monitor \
>
  -o jsonpath='\{.items[0].spec.nodeName\}')
$
$
# 1. Monitor detected the unreachable endpoint and sent an unhealthy (fatal) event.
$
kubectl logs -l app.kubernetes.io/name=demo-health-monitor -n nvsentinel
$
$
# 2. Platform-connector recorded it on the node as a condition (Type == your checkName).
$
kubectl get node "$NODE" \
>
  -o jsonpath='\{range .status.conditions[?(@.type=="NetworkReachability")]\}\{.type\}\{"  "\}\{.status\}\{"  "\}\{.reason\}\{"  "\}\{.message\}\{"\n"\}\{end\}'

You should see the condition reported as a fault:

NetworkReachability  True  NetworkReachabilityIsNotHealthy  dial tcp 192.0.2.1:9: ... Recommended Action=CONTACT_SUPPORT;

Note the inverted semantics: NVSentinel sets Status=True to mean a fault is present (and Status=False for healthy). The condition Type is exactly your checkName, and Reason is <checkName>IsNotHealthy / <checkName>IsHealthy. The same condition is visible under Conditions in kubectl describe node "$NODE".

Clear the fault — point CHECK_TARGET at a reachable endpoint (kubernetes.default.svc:443 is reachable from any in-cluster pod). On the next poll the monitor emits a healthy event and the condition flips back (Status=False, Reason=NetworkReachabilityIsHealthy, Message="No Health Failures"):

$
kubectl set env ds/demo-health-monitor -n nvsentinel CHECK_TARGET=kubernetes.default.svc:443
$
kubectl rollout status ds/demo-health-monitor -n nvsentinel
$
$
kubectl get node "$NODE" \
>
  -o jsonpath='\{range .status.conditions[?(@.type=="NetworkReachability")]\}\{.type\}\{"  "\}\{.status\}\{"  "\}\{.reason\}\{"  "\}\{.message\}\{"\n"\}\{end\}'

The condition now reports healthy:

NetworkReachability  False  NetworkReachabilityIsHealthy  No Health Failures

That’s the end of the path this tutorial covers: your monitor detects a fault and NVSentinel surfaces it as a node condition.

If nothing arrives at the platform-connector, the usual cause is the socket mount. See docs/runbooks/health-monitor-uds-failures.md.

Appendix: One-shot AI prompt

Paste this to an AI coding agent. It is self-contained: the monitor is a standalone module that can be created in any repository (it does not need to be inside the NVSentinel repo). Replace the bracketed parts.

Create a new NVSentinel health monitor named "[my-monitor]" that detects [the condition]
by [the method, e.g. "checking TCP reachability of an endpoint", "reading /proc/loadavg",
or "watching <K8s resource>"]. It is a standalone Go program that can live in any repo;
follow this spec exactly.
- Create a new Go module in its own directory with any module path (e.g.
  github.com/<your-org>/[my-monitor]). It depends on two NVSentinel modules that are NOT
  published as tagged releases, so fetch them by commit and add one replace so commons can
  resolve data-models:
    export GOTOOLCHAIN=auto
    go get github.com/nvidia/nvsentinel/data-models@main
    DM=$(go list -m github.com/nvidia/nvsentinel/data-models | awk '\{print $2\}')
    go mod edit -replace github.com/nvidia/nvsentinel/data-models=github.com/nvidia/nvsentinel/data-models@"$DM"
    go get github.com/nvidia/nvsentinel/commons@main
- Emit contract-correct HealthEvents (agent="[my-monitor]", a stable componentClass and
  checkName), edge-triggered (only on health-state changes), with
  ProcessingStrategy_EXECUTE_REMEDIATION; healthy events use RecommendedAction_NONE.
- Publish via commons/pkg/healthpub (github.com/nvidia/nvsentinel/commons/pkg/healthpub) to
  the platform-connector Unix socket (unix:///var/run/nvsentinel.sock) and treat
  ErrPlatformConnectorUnavailable as "retry next cycle" without advancing dedup state.
- Validate env config: NODE_NAME (required) plus the poll interval and any check timeout;
  reject non-positive durations before constructing a time.Ticker.
- Add a multi-stage Dockerfile whose build context is the monitor directory itself (COPY
  go.mod/go.sum, go mod download, COPY ., build a static binary). No local module copying.
- Provide a DaemonSet manifest (namespace nvsentinel) that mounts the platform-connector
  socket (hostPath /var/run/nvsentinel -> /var/run) and sets NODE_NAME via the downward API.
- Show the deploy commands against a running NVSentinel cluster: docker build, docker push
  to a registry the cluster can pull from (e.g. Docker Hub), then kubectl apply the DaemonSet
  (image referencing the pushed tag).
- Show how to verify: a fatal event surfaces as a Kubernetes node condition whose Type equals
  the checkName (kubectl get node <node> -o jsonpath over .status.conditions; Status=True
  means a fault is present). Stop at the node condition.
Ensure `go mod tidy` and `go build ./...` pass.