Credential monitoring
NVKE observes actual certificate expiration dates independently of automated
renewal. Enable inspection on managed T2 nodes with the node agent that implements
InspectKubernetesCertificates. The controller polls joined nodes, persists
public observations in PostgreSQL and exports them without contacting nodes
while Prometheus scrapes /metrics. Inspection does not renew certificates,
write files, restart processes or request private key material over gRPC.
The current alpha inventory assumes the kubeadm layout NVKE provisions:
super-admin.conf is optional: kubeadm init creates it, while joined control-plane
nodes need not have it. Its absence is visible without triggering the required
credential alert. A present optional certificate still receives expiry warnings.
The first inspection runs as soon as a node is marked joined, and a kubelet
serving certificate can still be missing then, while TLS bootstrap completes. A
required credential that is missing or invalid is therefore inspected again
after a backoff of 1, 2, 4, 8, 16 and then 32 minutes, capped at
inspectionInterval. With prompt probes, these retries fall approximately at
minutes 1, 3, 7, 15 and 31 after the first inspection.
NVKECertificateMissingOrInvalid needs the condition to hold for fifteen
minutes. The missing observation remains stored until a successful retry sees
the certificate, so slower bootstrap, delayed probes or timeouts can still
cause an alert. Early retries reduce that window; they do not guarantee that a
brief bootstrap gap cannot alert.
Custom certificate locations, external PKI, arbitrary kubeconfig certificate references, worker bootstrap tokens and copied operator kubeconfigs outside this inventory are not inspected. The current RPC reads only the fixed catalog and caps each credential file at 1 MiB. It returns the earliest expiration and public DER SHA-256 for a certificate bundle, never PEM, private keys, serials or subjects. A fixed file that is missing or invalid remains unavailable. Monitoring CAs does not enable CA rotation: approaching CA expiry requires an operator procedure outside the automatic alpha renewal scope.
T1 management-cluster renewal remains an operator-run procedure. This collector covers database-managed nodes; it does not discover unmanaged T1 nodes or scrape their credential files. Establish equivalent operator evidence for those nodes rather than treating absent T1 series as healthy.
Configuration
The existing renewal defaults remain a twelve-hour check interval and a thirty-day
renewal threshold. The following bounds are a proposed alpha support policy
for review, not a statement of product-approved SLOs. Startup checks the bounds
of each enabled feature and names the offending setting in the error. The
certRenewal.* bounds apply only while certRenewal.controllerEnabled is true,
and the credentialMonitoring.* bounds only while credentialMonitoring.enabled
is true. The credential metrics collector is registered either way and reads
credentialMonitoring.observationMaxAge and credentialMonitoring.warningDays,
so those two must be positive even while monitoring is disabled. The chart
refuses the same values when it renders, so helm install or helm upgrade
fails with the offending setting instead of the new pod failing to start.
These settings live in the controller ConfigMap named by controllerConfig.name
(default nvke-controller-config); the chart renders them from
scheduledMaintenance.certRenewal.* and credentialMonitoring.*. The
controller no longer reads the certRenewal.* settings from their former
CERT_RENEWAL_* environment variables, and the chart refuses an env entry
that names one.
The warning margin gives an operator notice before the automated renewal
threshold, with at least two check opportunities. A node-agent connection and
inspection attempt has a thirty-second timeout. Unreachable or older agents
retain last-known expiry dates but set inspection availability to zero. A failed
or incomplete inspection is retried after a short backoff (1 minute, doubling to
32 minutes, never beyond inspectionInterval) rather than after a full interval,
so a transient agent restart or network error normally clears before
NVKECertificateInspectionUnavailable has held for fifteen minutes. A successful
inspection with every required credential present, or an agent that does not
implement the RPC, waits the full interval. No prior
observation yields an age of -1. Expiration alerts use only fresh observations;
unavailable inspection alerts cover unknown or stale data. Re-enabling monitoring
resumes persisted observations and refreshes them on the configured interval.
The tenant manager reads a complete kubelet-serving CSR snapshot every minute. It stores the snapshot when the counts or the oldest pending time change, and otherwise refreshes the stored observation time only every two to three minutes. The write goes around the cluster reconcile notifications, so observing does not queue a reconcile in every cluster-watching controller. Approved requests without an issued certificate still count as pending. Failed and denied requests remain visible until the Kubernetes CSR object is removed; counts are inventory gauges, not lifetime event counters. A missing or stale snapshot has its own alert, so zero cached counts cannot imply health when tenant access has failed.
Metrics and labels
Every metric has the nvke_controller_ prefix. Identity labels are limited to
managed tenant_cluster and node, plus the fixed credential catalog or fixed
lifecycle state values. There are no CSR names, Secret names, workflow/request
IDs, fingerprints, certificate subjects or arbitrary error text in labels.
Fingerprints are stored as public evidence, never labels. Deleting a node or
cluster removes its series on the next scrape; a controller restart reconstructs
metrics from database state.
nvke_controller_credential_audit_writes_total counts credential audit ledger
writes by result; Credential lifecycle audit evidence
gives its alert expression for sink_failed, invalid_event and rate_limited.
nvke_controller_tenant_kubeconfig_rejected_total counts tenant kubeconfigs the
controller refused to load, by source: node_agent for an admin.conf a
control-plane node returned, secret for a published kubeconfig Secret. A node
or Secret is never a label; the controller logs the cluster, node and Secret
with each rejection.
A renewal due timestamp is a scheduling timestamp. It is never substituted for certificate expiry. Unknown rotation state means no recorded rotation and is not itself a failed operation.
Install and evaluate alerts
The controller chart can install the shipped PrometheusRule:
release: prometheus is an example selector label. Use labels matching your
Prometheus instance’s serviceMonitorSelector and ruleSelector. Install
Prometheus Operator CRDs before enabling the optional rule. The umbrella
global.monitoring.enabled=false disables both monitoring resources. The chart
does not install Prometheus, a rule evaluator or an Alertmanager notification
route. Successful scraping alone does not deliver notifications.
Operators must confirm the ServiceMonitor selects the controller service, the
chart’s metrics NetworkPolicy admits the Prometheus namespace (by default only
monitoring and dgxc-alloy; add others to
metrics.networkPolicy.additionalNamespaces, as described in
Read the management plane metrics), the
Prometheus instance loads nvke-credential-lifecycle, and its Alertmanager route
has a receiver for the rule severities. Add a separate scrape/target-down alert:
these rules cannot report from a controller that is unreachable. With
credentialMonitoring.enabled=false, nothing refreshes the stored observations.
Inspection availability keeps its last stored value until that observation is
older than observationMaxAge, then reads 0. The CSR observation age and the
last stored CSR inventory’s pending age keep growing.
The two availability alerts, NVKECertificateInspectionUnavailable and
NVKEKubeletServingCSRObservationUnavailable, and the two CSR alerts,
NVKEKubeletServingCSRStale and NVKEKubeletServingCSRFailed, therefore fire
only while credential_monitoring_enabled is 1, so disabling monitoring does not
page for every node and cluster or for a frozen CSR snapshot. Unknown observations must still not be mistaken for
healthy credentials: with monitoring disabled, no rule reports expiring or missing
certificates once the last observation is older than observationMaxAge. Until
then, the expiry and missing alerts keep evaluating that last observation.
Alert responses
The shipped thresholds below are proposed alpha operational bounds. Expiry of
kubeadm leaves, CAs and kubeconfig certificates uses the configurable warning
window; the remaining rules use fixed evaluation thresholds in
deploy/charts/controller/files/credential-alerts.yaml. Kubelet client and
serving certificates have their own seven-day rule: the kubelet rotates them
itself at 70–90% of their lifetime, so warningDays would fire on healthy
nodes, while seven days catches rotation that has stopped working.
Alert summaries and responses are shipped in each rule. Renewal or rotation failure does not establish that the agent stopped mutating state. Do not release locks or start competing maintenance from an alert alone.
Offline verification and live acceptance
make chart-check, which the merge gate runs, includes the credential rule
fixtures, a schema-validated render with alerts enabled, and renders asserting
that no PrometheusRule is emitted by default or with
global.monitoring.enabled=false. The check installs
Prometheus promtool at the version in make/tool-versions.mk into the ignored
.local/ directory, verifying the release archive against a committed SHA-256.
The first run needs network access to GitHub releases; subsequent runs reuse the
binary. Every run also fetches the PrometheusRule schema from the pinned
datreeio/CRDs-catalog commit on raw.githubusercontent.com and verifies its
committed SHA-256, so the check is never fully offline. To run only the credential checks:
Put the repository-pinned Helm and kubeconform on PATH for these checks. Fixtures exercise warning crossing before expiry, holds, firing and recovery for every shipped alert; optional super-admin absence and unavailable observations are also covered. Fixture format follows the Prometheus rule-testing reference.
Offline tests do not prove a site’s scrape selection, rule evaluation, actual certificate layout or notification delivery. Before acceptance, retain sanitized live evidence of a supported node approaching the warning window, the loaded rule firing before expiry, receipt by the configured receiver, and resolution after renewal. Exercise unavailable inspection and a stale/failed CSR as well. Offline results do not establish live renewal qualification.