Event Exporter Configuration

View as Markdown

Overview

The Event Exporter module exports health events from NVSentinel to external systems using CloudEvents format over HTTP. This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the event-exporter module is deployed in the cluster.

1global:
2 eventExporter:
3 enabled: true

Note: This module depends on the datastore being enabled. Therefore, ensure the datastore is also enabled.

Resources

Defines CPU and memory resource requests and limits for the event-exporter pod.

1event-exporter:
2 resources:
3 limits:
4 cpu: "1"
5 memory: "1Gi"
6 requests:
7 cpu: "500m"
8 memory: "512Mi"

OIDC Secret

Name of the Kubernetes secret containing OIDC client secret for authentication.

1event-exporter:
2 oidcSecretName: "event-exporter-oidc-secret"

The secret must contain a key named oidc-client-secret with the client secret value. Create the secret before deploying:

$kubectl create secret generic event-exporter-oidc-secret \
> --from-literal=oidc-client-secret='your-client-secret-here' \
> -n nvsentinel

Metadata Configuration

Custom metadata fields included in all exported CloudEvents.

1event-exporter:
2 exporter:
3 metadata:
4 cluster: "my-cluster"
5 environment: "production"

Metadata fields are included in the CloudEvent data.metadata object. The cluster field is required and used to generate the CloudEvent source field.

Custom Metadata Fields

Add any additional metadata fields:

1event-exporter:
2 metadata:
3 cluster: "prod-us-west-2"
4 environment: "production"
5 region: "us-west-2"
6 datacenter: "dc01"
7 tenant: "acme-corp"

All fields are included in exported events and can be used for filtering, routing, or categorization in downstream systems.

Sink Configuration

Defines the destination endpoint for exported events.

1event-exporter:
2 exporter:
3 sink:
4 endpoint: "https://events.example.com/api/v1/events"
5 timeout: "30s"
6 insecureSkipVerify: false

Parameters

endpoint

HTTP/HTTPS URL where CloudEvents will be POSTed.

timeout

Request timeout for HTTP calls to the sink endpoint.

insecureSkipVerify

Skip TLS certificate verification. Set to true only for testing with self-signed certificates.

OIDC Authentication

Configuration for OAuth 2.0 Client Credentials flow authentication.

1event-exporter:
2 exporter:
3 oidc:
4 tokenUrl: "https://auth.example.com/oauth2/token"
5 clientId: "nvsentinel-exporter"
6 scope: "events:write"
7 insecureSkipVerify: false

Parameters

tokenUrl

OAuth 2.0 token endpoint URL for obtaining access tokens.

clientId

OAuth 2.0 client identifier.

scope

OAuth 2.0 scope requested for access token.

insecureSkipVerify

Skip TLS certificate verification for token endpoint. Set to true only for testing.

Authentication Flow

The event exporter uses OAuth 2.0 Client Credentials grant:

  1. Requests access token from tokenUrl using clientId and client secret
  2. Caches the token until expiration
  3. Includes token in Authorization: Bearer <token> header for event POSTs
  4. Automatically refreshes expired tokens

Backfill Configuration

Controls whether historical events are exported when the exporter starts.

1event-exporter:
2 exporter:
3 backfill:
4 enabled: true
5 maxAge: "720h"
6 maxEvents: 1000000
7 batchSize: 500
8 rateLimit: 1000

Parameters

enabled

Enable backfilling of historical events from the datastore.

maxAge

Maximum age of events to backfill (e.g., “720h” = 30 days).

maxEvents

Maximum number of historical events to process during backfill.

batchSize

Number of events to process in each batch during backfill.

rateLimit

Maximum events per second to export during backfill to avoid overwhelming the sink.

Backfill Examples

Conservative Backfill

1backfill:
2 enabled: true
3 maxAge: "168h" # 7 days
4 maxEvents: 10000
5 batchSize: 100
6 rateLimit: 100

Aggressive Backfill

1backfill:
2 enabled: true
3 maxAge: "2160h" # 90 days
4 maxEvents: 5000000
5 batchSize: 1000
6 rateLimit: 5000

Disabled Backfill

1backfill:
2 enabled: false

Workers

Number of concurrent goroutines that process and publish events to the sink in parallel.

1event-exporter:
2 exporter:
3 workers: 10

Each worker independently picks events from the dispatch queue, processes them (unmarshal, transform, publish), and reports the result. A sequence tracker ensures resume tokens advance in strict order regardless of which worker finishes first, so increasing workers scales throughput while preserving at-least-once delivery guarantees. Note that concurrent publishing means events may arrive at the sink out of order.

The default of 10 handles clusters up to ~3,300 nodes at typical event rates.

Scale-Up Guide

Event production rate: ~10 events/sec per 1,000 nodes (~36,000 events/hour) Per-worker throughput: ~3.3 events/sec (at 300ms publish latency)

WorkersThroughput (events/sec)Max Nodes Supported
13.3~330
26.6~660
39.9~990
516.5~1,650
1033~3,300
1549.5~5,000
2066~6,600

If your publish latency is lower (e.g., 100ms for a co-located endpoint), each worker handles proportionally more events — divide the latency ratio to estimate your actual throughput.

Failure Handling

Configures retry behavior for failed export attempts.

1event-exporter:
2 exporter:
3 failureHandling:
4 maxRetries: 17
5 initialBackoff: "1s"
6 maxBackoff: "5m"
7 backoffMultiplier: 2.0

Parameters

maxRetries

Maximum number of retry attempts for failed exports before giving up.

initialBackoff

Initial delay before first retry attempt.

maxBackoff

Maximum delay between retry attempts (caps exponential backoff).

backoffMultiplier

Multiplier for exponential backoff calculation.

Retry Examples

Fast Retries

1failureHandling:
2 maxRetries: 10
3 initialBackoff: "100ms"
4 maxBackoff: "10s"
5 backoffMultiplier: 1.5

Conservative Retries

1failureHandling:
2 maxRetries: 30
3 initialBackoff: "5s"
4 maxBackoff: "15m"
5 backoffMultiplier: 2.5

Sink Endpoint Requirements

The external event sink must:

  1. Accept POST requests at the configured endpoint
  2. Accept Content-Type: application/cloudevents+json header
  3. Validate Authorization: Bearer <token> header
  4. Return HTTP 2xx status codes for successful ingestion
  5. Return HTTP 4xx/5xx status codes for failures
  6. Handle CloudEvents 1.0 JSON format

Example Sink Implementation

A minimal sink endpoint should:

1@app.route('/api/v1/events', methods=['POST'])
2def receive_event():
3 # Verify Bearer token
4 auth_header = request.headers.get('Authorization')
5 if not auth_header or not auth_header.startswith('Bearer '):
6 return {'error': 'Unauthorized'}, 401
7
8 # Verify Content-Type
9 if request.content_type != 'application/cloudevents+json':
10 return {'error': 'Unsupported Media Type'}, 415
11
12 # Parse CloudEvent
13 event = request.json
14 if event.get('specversion') != '1.0':
15 return {'error': 'Unsupported CloudEvents version'}, 400
16
17 # Process event
18 process_health_event(event['data']['healthEvent'])
19
20 return {'status': 'accepted'}, 202