Operations

Day-2 operational reference for the DSX Event Bus: monitoring, configuration tuning, service internals, and the chart structure.

Monitoring

Surveyor Configuration

NATS Surveyor exports Prometheus metrics from the NATS cluster. Configure via the surveyor section in Helm values:

1
surveyor:
2
  config:
3
    servers: "nats://nats:4222"
4
    expectedServers: 3
5
    accounts: true
6
    jsz: all
7
    nkey:
8
      secret:
9
        name: nats-surveyor
10
        key: seed
11
  serviceMonitor:
12
    enabled: true

Prometheus Operator must be installed for the ServiceMonitor CRD.

Monitoring Architecture

The mTLS cluster’s SYS account is federated to the main cluster via leaf node, enabling centralized monitoring of both NATS instances from a single Surveyor.

NATS Metrics

Surveyor exposes metrics on port 7777 at /metrics:

• nats_core_* — core server metrics (connections, messages, bytes)
• nats_account_* — per-account metrics
• nats_jetstream_* — JetStream stream and consumer metrics

Auth-Callout Metrics

Auth-callout exposes Prometheus metrics at :9090/metrics:

Health Endpoints

Internal Services

Kubernetes service names and ports within the dsx namespace:

Configuration Reference

mTLS Endpoint

The mTLS NATS cluster is enabled by default. It deploys a separate NATS instance that accepts MQTT connections authenticated with client certificates. This instance has no local JetStream; it connects to the main NATS cluster via leaf nodes.

1
global:
2
  eventBus:
3
    mtls:
4
      enabled: true   # default

When disabled (global.eventBus.mtls.enabled: false):

• No nats-mtls pods, services, or config deployed
• No mqttMtls gateway route created
• mTLS-specific environment variables omitted
• mTLS leaf NKey entries omitted from auth-callout permissions
• mTLS secrets not required

MQTT Stream Configuration

JetStream streams for MQTT persistence are managed declaratively by the NACK controller:

1
global:
2
  eventBus:
3
    mqttStreams:
4
      maxBytes: 67108864   # 64MB per stream
5
      replicas: 3          # match NATS cluster size
6
      storage: memory      # memory or file

Extra Accounts

Add cluster-wide NATS accounts beyond the defaults:

1
global:
2
  eventBus:
3
    extraAccounts:
4
      LaunchLayer:
5
        jetstream: true
6
      Kiwi: {}             # minimal account with defaults

Properties are passed through to the NATS account configuration on each cluster. CPC leaf nodes bridge enabled extra accounts to CSC, while each account keeps its own permissions and JetStream API surface.

Subchart Configuration

Configure subcharts by prefixing values with the chart alias:

1
# NATS cluster
2
nats:
3
  config:
4
    cluster:
5
      replicas: 3
6
    jetstream:
7
      enabled: true
8
    mqtt:
9
      enabled: true
10
11
# Auth callout
12
auth-callout:
13
  serviceConfig:
14
    nats:
15
      url: "nats://nats:4222"
16
    jwks:
17
      url: "https://keycloak.example.com/realms/event-bus/protocol/openid-connect/certs"
18
      issuer: "https://keycloak.example.com/realms/event-bus"
19
20
# NACK controller
21
nack:
22
  jetstream:
23
    enabled: true
24
    nats:
25
      url: "nats://nats:4222"

Teardown

helm uninstall removes chart-managed resources but leaves operator-provisioned secrets and service accounts in place. This is intentional — the chart does not own those resources.

Chart uninstall

Run on each cluster (CSC and every CPC):

$
helm uninstall dsx -n dsx

Removing operator-provisioned resources

If you want a full teardown, delete the secrets and service accounts that were created during pre-deployment. These survive helm uninstall because they were created outside the chart:

$
# NKey and TLS secrets
$
kubectl -n dsx delete secret \
>
  auth-callout-keys event-bus-server-tls-certificate \
>
  nats-auth-signing nats-authx-user nats-leaf-csc \
>
  nats-nack-user nats-surveyor nats-xkey \
>
  --ignore-not-found
$
$
# mTLS secrets (if mTLS was enabled)
$
kubectl -n dsx delete secret \
>
  nats-mtls-server-tls nats-mtls-leaf nats-mtls-authx-leaf nats-mtls-sys-leaf \
>
  --ignore-not-found
$
$
# Secrets pipeline service accounts (if applicable)
$
kubectl -n dsx delete sa event-bus-pki nats-event-bus-vso --ignore-not-found

Full namespace reset

To remove everything including the namespace:

$
kubectl delete ns dsx --ignore-not-found

If using a Vault-backed secrets pipeline, also remove the Vault PKI role, KV paths, and per-cluster Kubernetes auth mounts for a true clean slate.

Chart Dependencies

The nats-event-bus umbrella chart bundles these subcharts: