Tutorial: Plugin Custom Remediation

This tutorial walks you through extending NVSentinel’s fault-remediation stage with your own repair automation — for example, a custom controller.

By the end you will understand:

• Where custom remediation fits in the NVSentinel pipeline.
• How to register a custom remediation action with no NVSentinel code changes.
• The status-condition completion contract your controller must implement.

Who is this for? Teams already running NVSentinel — who need a repair step NVSentinel does not ship out of the box (CSP RMA tickets, multi-step orchestration, …) instead of the built-in reboot / terminate / GPU-reset actions.

Just want the AI to do it? Jump to Appendix: One-shot AI prompt.

1. Where custom remediation fits

NVSentinel remediates a faulty node in stages:

health-monitor → fault-quarantine → node-drainer → fault-remediation → your system

fault-remediation is the extension point. After a node is cordoned and drained, fault-remediation:

1. Resolves the health event’s recommended action (a built-in enum value or a custom string).
2. Looks up a matching entry in the maintenance.actions Helm config.
3. Renders a Go template into a Kubernetes Custom Resource.
4. Creates that CR in the cluster.
5. Polls the CR’s status conditions (completeConditionType) to avoid duplicate repair requests.

Your controller watches the CR and performs the actual repair.

  fault-remediation ──create CR──► Kubernetes API ◄──watch── Your controller
        ▲                              │
        └── poll completeConditionType ┘
                    (repair + set status condition)

Built-in vs custom actions

2. Integration approach

fault-remediation creates a Kubernetes CR for your action; your controller watches it, runs the repair, and reports completion via a status condition.

Why your CR must expose status conditions

fault-remediation’s status checker only understands Kubernetes-style conditions — it looks for a status.conditions[] entry whose type equals completeConditionType and reads its status (True → done, False → failed/retry).

Many external orchestrators report completion differently (e.g. a phase field, a job exit code, or a ticket status). Use a thin wrapper CRD whose controller translates external completion into status.conditions[Complete]=True.

3. The completion contract

Whatever CR NVSentinel creates, fault-remediation decides “is this repair done?” by reading a status condition:

1
status:
2
  conditions:
3
    - type: <completeConditionType>   # e.g. "Complete"
4
      status: "True"                  # True = success, False = failed (retry allowed)
5
      reason: RepairSucceeded
6
      message: Node hardware repaired and validated

Behavior of the status checker:

• Condition True → repair succeeded; equivalent events are marked covered.
• Condition False → repair failed; NVSentinel drops the group and creates a new CR to retry. There is no “terminal failure” state — if a repair is unrecoverable, keep the condition off True (leave it Unknown/absent) rather than setting False, or stop requeueing after your own retry budget.
• Condition missing / Unknown → repair in progress; NVSentinel skips creating a duplicate CR.

fault-remediation derives an effective equivalence group from the matched action — and, when impactedEntityScope is set, the impacted entity (e.g. reset-GPU-123). It skips creating a new CR only while a CR in that effective group (or a superseding group) is in progress or already succeeded; it does not deduplicate every repair on the node.

supersedingEquivalenceGroups lets a broader action cover a narrower one. The subordinate action lists the group that supersedes it. For example, a node reboot has the same effect as an RMA, so the hardware-rma action lists restart:

1
hardware-rma:
2
  equivalenceGroup: external-rma
3
  supersedingEquivalenceGroups: [restart]   # a restart supersedes external-rma
4
RESTART_VM:
5
  equivalenceGroup: restart                 # must be a defined action with no impactedEntityScope

4. Walkthrough — custom CR plugin (step by step)

Step 1 — Emit a custom action from a health event

A health monitor must set both fields — the action type and the custom action name:

1
recommendedAction: CUSTOM
2
customRecommendedAction: hardware-rma

Platform-connector overrides cannot configure custom actions. Overrides remap recommendedAction between built-in enum values only (RESTART_VM, REPLACE_VM, …) and do not set customRecommendedAction. If an override sets recommendedAction: CUSTOM, the custom string stays empty and fault-remediation skips remediation as an unsupported action.

Step 2 — Register the action in fault-remediation Helm values

Add your action and its template to distros/kubernetes/nvsentinel/charts/fault-remediation/values.yaml:

1
maintenance:
2
  actions:
3
    hardware-rma:
4
      apiGroup: remediation.example.com
5
      version: v1alpha1
6
      kind: RemediationRequest
7
      scope: Namespaced
8
      namespace: remediation
9
      completeConditionType: Complete
10
      templateFileName: hardware-rma.yaml
11
      equivalenceGroup: external-rma
12
  templates:
13
    hardware-rma.yaml: |
14
      apiVersion: \{\{ .ApiGroup \}\}/\{\{ .Version \}\}
15
      kind: RemediationRequest
16
      metadata:
17
        name: maintenance-\{\{ .HealthEvent.NodeName \}\}-\{\{ .HealthEventID \}\}
18
        namespace: \{\{ .Namespace \}\}
19
        labels:
20
          app.kubernetes.io/managed-by: nvsentinel
21
      spec:
22
        nodeName: \{\{ .HealthEvent.NodeName \}\}
23
        action: hardware-rma
24
        healthEventId: \{\{ .HealthEventID \}\}

Key fields:

RBAC: fault-remediation auto-generates RBAC from maintenance.actions (resource name = lowercase kind + s). Use CRD kinds with regular plurals (RemediationRequest → remediationrequests); irregular plurals (Policy → policys) break RBAC.

Step 3 — Template variables

Available in every maintenance template:

Step 4 — Build your controller

Scaffold a controller that:

1. Watches your CR kind (RemediationRequest).
2. Executes the repair (call a CSP API, open a support ticket, run diagnostics, …).
3. Patches the status condition:

1
status:
2
  conditions:
3
    - type: Complete          # must match completeConditionType
4
      status: "True"          # "True" = done; "False" makes NVSentinel retry with a new CR
5
      reason: RepairSucceeded
6
      message: Node hardware repaired and validated

Golden rules (same as any NVSentinel plugin):

1. Reconcile is idempotent — a no-op once Complete=True.
2. Make non-idempotent repairs durably idempotent. A controller restart (or a retry after a False condition, which spawns a fresh CR) can re-run your repair before Complete=True is persisted. Guard one-shot side effects (opening a ticket, calling a CSP API) with a durable key such as spec.healthEventId or a status phase, so the effect happens at most once.
3. Only set Complete=True when the repair is actually done; requeue until then. Set Complete=False only when you want NVSentinel to retry with a new CR.
4. One CR per health event — NVSentinel manages its lifecycle via the equivalence group.

Build and push the controller image with Kubebuilder’s Makefile, pointing IMG at a registry your cluster can pull from:

$
docker login   # once, to authenticate
$
$
# Replace YOUR_USER and my-remediation with your Docker Hub user and image name.
$
export IMG=docker.io/YOUR_USER/my-remediation:dev   # or your cluster's private registry
>
make docker-build docker-push IMG=$IMG
>
make install && make deploy IMG=$IMG

Use a public repo so the cluster can pull without credentials. For a private registry (e.g. NVCR), add the cluster’s imagePullSecret to the controller Deployment via config/.

For a full Kubebuilder scaffold, reconciler walkthrough, and one-shot AI prompt, see Writing a Drain Plugin — Appendix: One-shot AI prompt — the pattern is the same (NVSentinel creates the CR and polls a status condition; your controller reconciles it and sets completion), even though the upstream stage is node-drainer instead of fault-remediation.

For a full working example (custom health monitor, CRD, controller, and Helm values), see the local custom remediation demo — a memory-pressure monitor and reclaim controller on a local KIND cluster (no GPU required).

Appendix: One-shot AI prompt

Paste this to an AI coding agent. It is self-contained: the controller is a standalone Kubebuilder project in any repository; produce Helm values snippets for fault-remediation registration (separate from the controller repo). Replace the bracketed parts.

Create a new NVSentinel custom remediation plugin named "[my-action]" (e.g. "hardware-rma") that
runs [the repair, e.g. "opens a CSP support ticket and waits for hardware replacement"] after
fault-remediation has cordoned and drained the node. Follow this spec exactly.
Architecture (direct CR plugin):
- A health monitor emits recommendedAction=CUSTOM with customRecommendedAction="[my-action]".
- fault-remediation looks up maintenance.actions["[my-action]"], renders a Go template into a
  Kubernetes CR, creates it, and polls status.conditions[] until completeConditionType is True.
- Your controller watches that CR, performs the repair, and sets the completion condition.
Health event contract (emit from a health monitor):
- Set BOTH recommendedAction=CUSTOM and customRecommendedAction="[my-action]" on the HealthEvent
  (protobuf field customRecommendedAction on datamodels.HealthEvent).
- Platform-connector overrides cannot configure custom actions — they remap built-in
  recommendedAction enums only and do not set customRecommendedAction.
Controller (standalone Kubebuilder project, any repo):
- Scaffold: kubebuilder init --domain example.com --repo github.com/<your-org>/[my-remediation]
  then kubebuilder create api --group remediation --version v1alpha1 --kind RemediationRequest
  --resource --controller.
- Spec: nodeName (string), action (string), healthEventId (string) — minimal fields from the
  fault-remediation template below.
- Status: conditions []metav1.Condition with a subresource.
- Reconciler: idempotent; no-op once Complete=True; run the repair at most once (guard non-idempotent
  side effects with a durable key such as spec.healthEventId so a restart or retry cannot repeat them);
  patch status with Type="Complete", Status="True" when repaired (or "False" to make NVSentinel retry
  with a new CR) using append-or-update-by-type (meta.SetStatusCondition + Status().Update).
  RBAC: remediationrequests + status; add any API permissions your repair needs.
- Build and push with the generated Dockerfile/Makefile, pointing IMG at a registry your cluster can
  pull from (e.g. docker.io/<your-user>/[my-remediation]:dev or a private registry such as NVCR).
  Run docker login once, then make docker-build docker-push IMG=$IMG. Use a public repo so the
  cluster can pull without credentials; for a private registry, add the cluster's imagePullSecret
  to the controller Deployment via config/. Deploy with make install && make deploy IMG=$IMG.
fault-remediation registration (Helm values snippet — merge into your NVSentinel release's
fault-remediation chart values; exact file/chart path depends on your deployment):
- Add under maintenance.actions:
    [my-action]:
      apiGroup: remediation.example.com
      version: v1alpha1
      kind: RemediationRequest
      scope: Namespaced
      namespace: remediation
      completeConditionType: Complete
      templateFileName: [my-action].yaml
      equivalenceGroup: [my-equivalence-group]
- Add under maintenance.templates.[my-action].yaml a Go text/template body:
    apiVersion: \{\{ .ApiGroup \}\}/\{\{ .Version \}\}
    kind: RemediationRequest
    metadata:
      name: maintenance-\{\{ .HealthEvent.NodeName \}\}-\{\{ .HealthEventID \}\}
      namespace: \{\{ .Namespace \}\}
      labels:
        app.kubernetes.io/managed-by: nvsentinel
    spec:
      nodeName: \{\{ .HealthEvent.NodeName \}\}
      action: [my-action]
      healthEventId: \{\{ .HealthEventID \}\}
- Template rules: plain Go text/template ONLY (no Sprig — no quote, trunc, or hash helpers).
  Use full \{\{ .HealthEvent.NodeName \}\}-\{\{ .HealthEventID \}\} for metadata.name. Do not template
  free-form message text into YAML.
- Use a CRD kind with a regular plural (RemediationRequest -> remediationrequests); irregular
  plurals break fault-remediation's auto-generated RBAC.
- Deploy the values via your usual NVSentinel Helm upgrade (e.g. helm upgrade <release>
  <chart> -n nvsentinel --reuse-values -f <overrides>.yaml), then wait for the fault-remediation
  Deployment rollout.
Verification (against a running NVSentinel cluster):
- Trigger a health event with CUSTOM + customRecommendedAction="[my-action]" on a test node.
- Confirm fault-remediation created the RemediationRequest (kubectl get remediationrequests -n
  remediation).
- Confirm your controller reconciles it and sets status.conditions Type=Complete Status=True.
- Confirm fault-remediation logs show the repair as complete and does not create a duplicate CR.
Ensure controller `go mod tidy` and `go build ./...` pass. Do not run docker push, helm upgrade,
or kubectl apply unless credentials/cluster are available — but do create the Dockerfile, CRD,
controller, and Helm values snippets.