Kubernetes Object Monitor Configuration

Overview

The Kubernetes Object Monitor module watches Kubernetes resources and generates health events when resources enter unhealthy states. This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the kubernetes-object-monitor module is deployed in the cluster.

1
global:
2
  kubernetesObjectMonitor:
3
    enabled: true

Resources

Defines CPU and memory resource requests and limits for the kubernetes-object-monitor pod.

1
kubernetes-object-monitor:
2
  resources:
3
    limits:
4
      cpu: 500m
5
      memory: 256Mi
6
    requests:
7
      cpu: 100m
8
      memory: 128Mi

Logging

Sets the verbosity level for kubernetes-object-monitor logs.

1
kubernetes-object-monitor:
2
  logLevel: info  # Options: debug, info, warn, error

Controller Configuration

Controls behavior of the Kubernetes controller watching resources.

1
kubernetes-object-monitor:
2
  maxConcurrentReconciles: 1
3
  resyncPeriod: 5m

maxConcurrentReconciles

Maximum number of concurrent reconciliation workers. Higher values allow parallel processing of multiple resources.

resyncPeriod

How often the controller re-evaluates all watched resources even without changes.

Policies Configuration

Policies define which Kubernetes resources to monitor and when to generate health events.

Policy Structure

1
kubernetes-object-monitor:
2
  policies:
3
    - name: "policy-name"
4
      enabled: true
5
      resource:
6
        group: ""
7
        version: v1
8
        kind: Node
9
        # namespace: gpu-operator  # Optional, only for namespaced resources
10
      predicate:
11
        expression: |
12
          resource.status.conditions.filter(c, c.type == "Ready" && c.status == "False").size() > 0
13
      nodeAssociation:
14
        expression: resource.spec.nodeName
15
      healthEvent:
16
        componentClass: Node
17
        isFatal: true
18
        message: "Error message"
19
        recommendedAction: CONTACT_SUPPORT
20
        errorCode:
21
          - ERROR_CODE
22
        quarantineOverrides:
23
          force: true  # Or use skip: true; do not set both
24
        drainOverrides:
25
          skip: true   # Or use force: true; do not set both

Parameters

name

Unique identifier for the policy used in logs and metrics.

enabled

Enables or disables the policy. Disabled policies are not compiled or evaluated.

resource

Specifies the Kubernetes resource type to monitor.

group

API group of the resource. Use empty string "" for core resources (Pod, Node, Service, etc.).

version

API version of the resource (e.g., v1, v1beta1).

kind

Kubernetes Kind of the resource (e.g., Node, Pod, Deployment).

namespace

Optional namespace restriction for namespaced resources. When set, the monitor creates the informer cache for this resource kind only in that namespace. Leave unset to watch all namespaces. Do not set this for cluster-scoped resources such as Node.

predicate

CEL expression that evaluates to true when the resource is in an unhealthy state. Evaluated with resource variable containing the full resource object.

expression

CEL expression accessing the resource via resource variable.

nodeAssociation

Optional CEL expression that maps the resource to a specific Kubernetes node name.

expression

CEL expression that returns a string node name.

healthEvent

Defines the health event to generate when the predicate matches.

componentClass

Component type for the health event (e.g., Node, GPU, Pod).

isFatal

Boolean indicating if this is a fatal error that should trigger quarantine.

message

Human-readable error message included in the health event.

recommendedAction

Action code from health event proto (see health_event.proto).

errorCode

Array of error code strings for categorization and filtering.

quarantineOverrides

Optional behavior override for fault-quarantine. force forces node cordoning regardless of normal rules; skip skips node cordoning for the generated health event. Set at most one of force or skip.

drainOverrides

Optional behavior override for node-drainer. force forces immediate pod eviction regardless of configured namespace drain modes; skip skips pod eviction and marks the event as already drained. Set at most one of force or skip.

CEL Expressions

Predicate Expressions

Access the resource object via the resource variable.

Common Patterns

Check if condition exists and is True:

1
expression: |
2
  resource.status.conditions.filter(c, c.type == "Ready" && c.status == "True").size() > 0

Check field value:

1
expression: |
2
  has(resource.status.phase) && resource.status.phase == "Failed"

Check label exists:

1
expression: |
2
  'failure' in resource.metadata.labels && resource.metadata.labels['failure'] == 'true'

Node Association Expressions

Map resources to nodes using CEL expressions.

Direct Field Reference

1
nodeAssociation:
2
  expression: resource.spec.nodeName

Using lookup() Function

The lookup() function retrieves other Kubernetes resources during evaluation.

Signature:

lookup(version, kind, namespace, name) -> resource object

Parameters:

• version (string) - API version (e.g., “v1”, “apps/v1”)
• kind (string) - Resource Kind (e.g., “Pod”, “Node”)
• namespace (string) - Namespace (use empty string "" for cluster-scoped resources)
• name (string) - Resource name

Examples:

Get node from pod reference:

1
nodeAssociation:
2
  expression: |
3
    lookup('v1', 'Pod', resource.metadata.namespace, resource.spec.podName).spec.nodeName

Monitoring by resource kind

Each policy watches one Kubernetes resource type. Set resource.group, resource.version, and resource.kind accordingly. Write predicates with the common CEL patterns on the resource object.

Node

1
resource:
2
  group: ""
3
  version: v1
4
  kind: Node

Nodes are cluster-scoped — do not set resource.namespace. Do not set nodeAssociation; the monitor uses the Node name automatically (ADR-011).

Pod

1
resource:
2
  group: ""
3
  version: v1
4
  kind: Pod
5
  namespace: default   # optional; omit to watch all namespaces

Pods must include nodeAssociation (usually resource.spec.nodeName; see direct reference). Gate predicates so every matched Pod has a non-empty node — e.g. has(resource.spec.nodeName) && resource.spec.nodeName != "" — so unscheduled Pods never match; otherwise the reconciler falls back to the Pod name when association is empty. For production DaemonSet operator monitoring, see Monitoring Critical Operators.

Event

1
resource:
2
  group: events.k8s.io
3
  version: v1
4
  kind: Event

Events must include nodeAssociation that resolves the node from the object the event is about — typically a lookup() on resource.regarding (ADR-011).

Policy name and output

The policy name is sent as the health event checkName. Whether that becomes a node condition or a Kubernetes Event on the node depends on isFatal — see INTEGRATIONS.

Policy Examples

Example 1: Node Not Ready

Monitor nodes that are not in Ready state.

1
policies:
2
  - name: node-not-ready
3
    enabled: true
4
    resource:
5
      group: ""
6
      version: v1
7
      kind: Node
8
    predicate:
9
      expression: |
10
        resource.status.conditions.filter(c, c.type == "Ready" && c.status == "False").size() > 0
11
    healthEvent:
12
      componentClass: Node
13
      isFatal: true
14
      message: "Node is not ready"
15
      recommendedAction: CONTACT_SUPPORT
16
      errorCode:
17
        - NODE_NOT_READY

Example 2: Node Needs Repair

Monitor custom node conditions.

1
policies:
2
  - name: NodeNeedsRepair
3
    enabled: true
4
    resource:
5
      group: ""
6
      version: v1
7
      kind: Node
8
    predicate:
9
      expression: |
10
        resource.status.conditions.filter(c, c.type == "kubernetes.acme.com/NeedsRepair" && c.status == "True").size() > 0
11
    healthEvent:
12
      componentClass: Node
13
      isFatal: true
14
      message: "Node needs repair"
15
      recommendedAction: REPLACE_VM
16
      errorCode:
17
        - NODE_NEEDS_REPAIR

Example 3: Warning Event on Pod

Event objects do not store the node name directly. Resolve the node by looking up the Pod referenced by resource.regarding.

1
policies:
2
  - name: pod-warning-event
3
    enabled: true
4
    resource:
5
      group: events.k8s.io
6
      version: v1
7
      kind: Event
8
    predicate:
9
      # Guard the Pod lookup so a deleted/absent Pod or empty spec.nodeName cannot
10
      # fall back to the Event name for node association.
11
      expression: |
12
        resource.type == 'Warning' &&
13
        resource.regarding.kind == 'Pod' &&
14
        has(resource.regarding.namespace) &&
15
        has(resource.regarding.name) &&
16
        (now - timestamp(resource.eventTime)) < duration('10m') &&
17
        lookup('v1', 'Pod', resource.regarding.namespace, resource.regarding.name) != null &&
18
        has(lookup('v1', 'Pod', resource.regarding.namespace, resource.regarding.name).spec.nodeName) &&
19
        lookup('v1', 'Pod', resource.regarding.namespace, resource.regarding.name).spec.nodeName != ""
20
    nodeAssociation:
21
      expression: |
22
        lookup('v1', 'Pod', resource.regarding.namespace, resource.regarding.name).spec.nodeName
23
    healthEvent:
24
      componentClass: Software
25
      isFatal: false
26
      message: "Recent Warning event on pod"
27
      recommendedAction: CONTACT_SUPPORT
28
      errorCode:
29
        - POD_WARNING_EVENT

Example 4: Pod Failed phase

Phase predicate from Check field value plus Pod nodeAssociation.

1
policies:
2
  - name: pod-failed-on-node
3
    enabled: true
4
    resource:
5
      group: ""
6
      version: v1
7
      kind: Pod
8
      namespace: default
9
    predicate:
10
      expression: |
11
        has(resource.spec.nodeName) && resource.spec.nodeName != "" &&
12
        resource.status.phase == 'Failed'
13
    nodeAssociation:
14
      expression: resource.spec.nodeName
15
    healthEvent:
16
      componentClass: Pod
17
      isFatal: false
18
      message: "Pod entered Failed phase"
19
      recommendedAction: CONTACT_SUPPORT
20
      errorCode:
21
        - POD_FAILED

For a fatal GPU-runtime Event policy, see the NVMLError example in ADR-011.

Observe a policy

$
NODE=\{test-node\}
$
$
# Example 4 — Pod exits immediately; the generated Warning events also exercise Example 3
$
kubectl run kom-pod-fail --restart=Never -n default --image=busybox \
>
  --overrides='\{"spec":\{"nodeName":"'"$NODE"'","containers":[\{"name":"kom-pod-fail","image":"busybox","command":["sh","-c","exit 1"]\}]\}\}'
$
$
kubectl logs -n nvsentinel -l app.kubernetes.io/name=kubernetes-object-monitor --tail=30
$
$
# Examples 3 & 4 use isFatal: false — verify Node events ([INTEGRATIONS](../INTEGRATIONS.md#using-events-for-non-fatal-errors))
$
kubectl get events --field-selector involvedObject.kind=Node,involvedObject.name=$NODE \
>
  --sort-by='.lastTimestamp' | tail -10
$
$
# Cleanup
$
kubectl delete pod kom-pod-fail -n default --ignore-not-found

One-shot AI prompt

Create a Helm values overlay for NVSentinel kubernetes-object-monitor policies on Node, Pod, or
events.k8s.io/v1/Event. Include nodeAssociation for Pod/Event (omit for Node). Output values
YAML plus kubectl trigger/verify commands.

RBAC Permissions

RBAC permissions are automatically generated based on configured policies:

• Node resources: Get write permissions (patch/update) for annotations
• All other resources: Get read-only permissions (get/list/watch)

When adding a new policy for a Custom Resource, ensure the CRD is installed before deploying the kubernetes-object-monitor.

Policy Design Guidelines

1. Predicate Specificity: Write predicates that clearly identify unhealthy states
2. Node Association: Provide nodeAssociation for non-Node resources to enable quarantine
3. Error Codes: Use descriptive error codes for filtering and categorization
4. Fatal vs NonFatal: Set isFatal: true only for errors requiring node quarantine
5. Testing: Use dry-run mode to test policy expressions before production deployment
6. Performance: Avoid expensive operations in predicates (e.g., multiple nested lookups)
7. Namespace Scope: For namespaced resources with many objects, set resource.namespace to reduce informer cache memory usage