Monitoring and Health

This page covers monitoring and health workflows for NICo sites after deployment: hardware health, DPU health, aggregate host health, health overrides, Prometheus scraping, Grafana dashboards, and Loki queries.

Use aggregate host health as the starting point for operational decisions. NICo combines hardware health, DPU health, validation and discovery checks, rack health, and health overrides into a single host-level result. Component health explains which source is responsible for the aggregate result.

Use this page as the entry point for health triage. It gives the primary inspection path, commands, metrics, dashboards, and log queries needed to start an investigation. For subsystem-specific behavior, follow the linked hardware, DPU, health aggregation, and classification references rather than treating this page as a replacement for those manuals.

For reference, see:

• Health Checks and Health Aggregation
• Health Probe IDs
• Health Alert Classifications
• Redfish Workflow

Health Sources

NICo builds health from health reports. A health report contains successes and alerts from a reporting source. Common health sources are:

Each alert has an ID, an optional target, a message, a start time, and one or more classifications. Classifications define operational impact. For example, PreventAllocations blocks new allocations while the alert is active, and ExcludeFromStateMachineSla excludes the host from state-machine SLA evaluation.

Hardware Health Monitoring

NICo monitors hardware through the hardware health service. The Helm chart is nico-hardware-health.

The service discovers BMC endpoints from NICo and queries them through Redfish. It monitors host BMCs, DPU BMCs, and configured switch or power-shelf BMCs. The primary monitoring path is sensor collection. Additional collectors can gather entity metrics (see Hardware Entity Metrics), firmware, log, NMX-T, NMX-C, NVUE REST, and leak-related data when configured.

Helm Configuration

Enable hardware health in Helm values:

1
nico-hardware-health:
2
  enabled: true

Enable metrics scraping with its ServiceMonitor:

1
nico-hardware-health:
2
  enabled: true
3
  replicas: 1
4
5
  serviceMonitor:
6
    enabled: true
7
    interval: 30s
8
    scrapeTimeout: 25s

By default, the chart exposes hardware health metrics on port 9009. Log collection is disabled by default:

1
env:
2
  NICO_HEALTH__COLLECTORS__LOGS__ENABLED: "false"

Enable log collection only through the target site’s deployment values.

Hardware Health Service Configuration

The hardware health service config example, crates/health/example/config.example.toml, documents endpoint discovery, sinks, rate limits, collectors, processors, and metrics.

Production endpoint discovery uses the NICo API source. The checked-in hardware-health example config currently names that source [endpoint_sources.nico_api]:

1
[endpoint_sources.nico_api]
2
root_ca = "/var/run/secrets/spiffe.io/ca.crt"
3
client_cert = "/var/run/secrets/spiffe.io/tls.crt"
4
client_key = "/var/run/secrets/spiffe.io/tls.key"
5
api_url = "https://nico-api.forge-system.svc.cluster.local:1079"

Static BMC endpoints are supported for local, mock, or special deployments:

1
[[endpoint_sources.static_bmc_endpoints]]
2
ip = "10.0.0.1"
3
port = 443
4
mac = "aa:bb:cc:dd:ee:ff"
5
username = "admin"
6
password = "secret"
7
labels = { site = "rno-dev7", cluster = "cluster-01", environment = "development" }

Static endpoints can define up to 32 custom telemetry labels. Label names must match [a-zA-Z_][a-zA-Z0-9_]*. Names owned by the health service, such as machine_id, system_uuid, endpoint_ip, and collector_type, are reserved. Custom labels are exported as Prometheus labels and OTLP resource attributes, as a labels object in JSONL logs, and as a structured map in tracing events.

Collector defaults from the example config:

NMX-C connects directly to eligible primary switch-host gRPC endpoints whose switch config has NMX-C enabled; it does not use BMC or NICo API TLS material. For static switch-host endpoints, switch.nmxc_enabled controls this target eligibility after the endpoint_role = "host" and is_primary = true checks; it defaults to switch.is_primary when omitted. NMX-C notifications emit log events for tracing, log-file, and OTLP log sinks only; Prometheus metrics and switch health reports are separate scope.

NMX-C collection uses plaintext gRPC over HTTP/2. TLS, certificate bypass, custom certificate loading, and mTLS are intentionally separate scope; do not model them with the NICo API SPIFFE certificate fields.

BMC Proxy

The full Helm example enables nico-bmc-proxy, the authenticating proxy for BMC Redfish access. The proxy chart exposes proxy traffic on port 1079 and metrics on port 1080.

Example proxy settings:

1
listen = "[::]:1079"
2
metrics_endpoint = "[::]:1080"
3
allowed_principals = ["spiffe-service-id/nico-api", "spiffe-service-id/nv-dps"]

The BMC proxy ServiceMonitor follows the same serviceMonitor.enabled, interval, and scrapeTimeout pattern as other NICo services.

Sensor Alerts

Hardware sensor alerts are derived from BMC-reported health, sensor readings, and thresholds. Sensor classifications include:

If numeric threshold data indicates a problem but the BMC reports the sensor as healthy, NICo treats the sensor as healthy. In that case the BMC health state is the authority.

Hardware Entity Metrics

Beyond sensors, BMCs expose scalar values on Redfish *Metrics resources — error counters, throttle durations, bandwidth utilization, power figures — that have no sensor backing. The entity metrics collector polls these and exports them as Prometheus series.

The entity metrics collector is disabled by default. Add the [collectors.metrics] section to the hardware health service config to enable it:

1
[collectors.metrics]
2
fetch_interval = "2m"     # default
3
fetch_concurrency = 4     # default; parallel per-entity fetches

What it collects, per entity type discovered on the BMC:

Sensor-backed values (carrying a Redfish DataSourceUri) are skipped: the sensor collector already publishes them as hw_sensor series, so nothing is double-reported.

Exported series are named {prefix}_hw_metric_{metric_type}_{unit} — with the default carbide_hardware_health prefix, for example:

carbide_hardware_health_hw_metric_correctable_core_errors_count
carbide_hardware_health_hw_metric_pcie_fatal_errors_count
carbide_hardware_health_hw_metric_bandwidth_percent
carbide_hardware_health_hw_metric_input_power_watts

On the Prometheus endpoint, series carry entity labels (processor_id, memory_id, drive_id, powersupply_id, system_id, model, …) plus the standard identity labels added by the sink (machine_id, endpoint_ip, serial_number, rack_id, …), with collector_type="metrics_collector".

The OTLP sink ([sinks.otlp]) emits the same metric names, but places the identity context on OTLP resource attributes rather than datapoint labels; whether those appear as query labels depends on the backend (VictoriaMetrics, for example, flattens resource attributes onto every series).

Entity discovery runs as its own periodic task ([collectors.discovery], always on) that walks each BMC’s Redfish Systems and Chassis trees and publishes an inventory snapshot; the metrics collector only reads that snapshot. Until the first discovery pass completes, the metrics collector emits nothing.

Hardware Health Logs

Use Loki or Grafana Explore to inspect hardware health logs for a host:

{k8s_container_name="nico-hardware-health"} |= "<machine-id>"

Health report events include fields such as:

collector=sensor_collector
report_source=bmc-sensors
machine_id=Some(<machine-id>)
alert_count=<n>
success_count=<n>

For leak-related events, look for:

report_source=tray-leak-detection

DPU Health Checks

dpu-agent runs on managed DPUs and reports DPU health to NICo. The BlueField chart is named nico-dpu-agent. In service names and logs, the DPU agent currently appears as forge-dpu-agent.service.

The agent checks DPU service health, networking state, HBN/NVUE configuration, DHCP behavior, BGP status, and heartbeat. DPU health is part of aggregate host health, so an otherwise healthy host can still be unavailable when its DPU is unhealthy.

See also:

• DPU Lifecycle Management
• DPU Configuration
• Stuck in WaitingForNetworkConfig and DPU Health

DPU Agent Configuration

Key nico-dpu-agent chart values:

The DaemonSet renders these core arguments:

nico-dpu-agent run
  --hbn-config-mode=nvue-rest
  --agent-platform-type=containerized
  --dhcp-grpc-server=http://<dhcp-grpc-service>:10079
  --fmds-grpc-server=http://<fmds-grpc-service>:50052

If dhcp_server.interface_prepend is set, the chart also adds:

--dhcp-server-interface-prepend=<prefix>

The pod sets these runtime environment variables:

Common DPU Alerts

Common DPU alert IDs include:

• ContainerExists
• ServiceRunning
• DhcpServer
• BgpStats
• BgpPeeringTor
• BgpPeeringRouteServer
• Ifreload
• BgpDaemonEnabled
• PostConfigCheckWait
• DpuDiskUtilizationCritical
• HeartbeatTimeout

HeartbeatTimeout means NICo has not received a recent health report from the DPU agent. Check whether the DPU is powered, the agent is running, DPU time is correct, and the DPU can reach NICo.

DPU Logs

Use Loki to inspect DPU-agent logs:

{systemd_unit="forge-dpu-agent.service", machine_id="<machine-id>"}

Alternative labels can be used when available:

{systemd_unit="forge-dpu-agent.service", host_name="<host-name>"}

On the DPU, use journalctl for direct service logs:

$
journalctl -u forge-dpu-agent.service -e --no-pager

Restart the agent when required:

$
systemctl restart forge-dpu-agent.service

Health Alert Lifecycle

NICo health alerts are source-based. A health source submits a fresh report, and NICo uses the latest report from each source to calculate aggregate health.

A typical alert flow:

1. A source reports an alert such as PoweredOff with target <bmc-ip>.
2. NICo adds the alert to aggregate host health.
3. Classifications such as PreventAllocations define the operational effect.
4. The health view shows the alert ID, target, message, start time, and classifications.
5. Metrics and logs identify the responsible source.
6. After remediation, the source submits a fresh report that marks the check successful or omits the previous alert.
7. NICo merges the fresh report and aggregate host health returns to healthy.

If a health override created the alert, remove or replace the override after the operational reason ends.

Inspect Current Health

Start with the host health page in the Admin Web UI:

https://<nico-api-hostname>/admin/machine/<machine-id>/health

Inspect the aggregate health table first. For each alert, note:

• ID
• Target
• In Alert Since
• Message
• Tenant Message
• Classifications

Then inspect component health to identify the source: hardware health, DPU health, validation, discovery, rack health, or health override.

Use the health history table to review recent transitions. This helps identify whether an alert is new, recurring, or already cleared by a later health report.

Admin CLI examples:

$
nico-admin-cli machine show <machine-id>
$
nico-admin-cli machine health-override show <machine-id>

Health Overrides

Health overrides add manual or service-created health reports into the same aggregate health model as automated checks. Overrides are shared health mechanisms; they are not specific to hardware health.

Use overrides for controlled states such as maintenance, validation, repair, break-fix, or temporary automation control. Do not use an override as a substitute for resolving the underlying condition.

Merge and Replace

DPU replace overrides are rejected by the API.

Override Templates

The nico-admin-cli machine health-override add command supports templates for common workflows:

Examples:

$
nico-admin-cli machine health-override show <machine-id>
$
$
nico-admin-cli machine health-override add <machine-id> \
>
  --template RequestRepair \
>
  --message "Manual repair trigger for tenant-reported issue"
$
$
nico-admin-cli machine health-override add <machine-id> \
>
  --template OutForRepair \
>
  --message "Automated repair failed, requires manual investigation"
$
$
nico-admin-cli machine health-override remove <machine-id> repair-request
$
nico-admin-cli machine health-override remove <machine-id> tenant-reported-issue

Before creating an override, identify the current aggregate health, choose the smallest effect that matches the workflow, include a clear message, and define the removal condition. After remediation, remove or replace the override and verify aggregate health.

Prometheus Metrics

NICo charts expose Prometheus scraping through ServiceMonitor resources. ServiceMonitors are disabled by default in chart values and enabled in the full example for selected services.

Example:

1
nico-api:
2
  serviceMonitor:
3
    enabled: true
4
    interval: 30s
5
    scrapeTimeout: 25s
6
7
nico-hardware-health:
8
  serviceMonitor:
9
    enabled: true
10
    interval: 30s
11
    scrapeTimeout: 25s

ServiceMonitors from the charts:

Check rendered ServiceMonitors:

$
kubectl get servicemonitor -n nico-system
$
kubectl get servicemonitor -n nico-system nico-api-metrics
$
kubectl get servicemonitor -n nico-system nico-hardware-health-metrics
$
kubectl get servicemonitor -n nico-system nico-dsx-exchange-consumer-metrics

Core health metrics currently use carbide_* metric names. Some dashboards and site configurations also expose host-health rollups with the forge_* prefix. Use the literal metric name that exists in the target site.

Useful core metric families:

Use the Host Health dashboard panels for fleet-level rollups, including host health status, health overrides, probe alerts, and alert classifications. Example dashboard queries:

sum by (healthy, in_use) (
  max by(healthy, in_use) (
    forge_hosts_health_status_count{fresh="true"}
  )
)

sum by(override_type) (
  max by(override_type, in_use) (
    forge_hosts_health_overrides_count{fresh="true"}
  )
)

sum by(probe_id) (
  max by(probe_id, probe_target) (
    forge_hosts_unhealthy_by_probe_id_count{fresh="true"}
  )
)

sum by(classification) (
  max by(classification, in_use) (
    forge_hosts_unhealthy_by_classification_count{fresh="true"}
  )
)

Per-Object Health Metrics

The aggregate metrics above report counts of unhealthy objects. To identify which objects carry a given health-alert classification, NICo can emit one additional time series per affected object:

carbide_object_unhealthy_by_classification_count{object_type="machine",object_id="fm100...",classification="Hardware",in_use="true"} 1

Labels:

Emission is opt-in per classification to contain cardinality: series count still scales with fleet size (one series per matching object per listed classification — an object carrying two enabled classifications emits two series), but only for the classifications you list. It is disabled by default; enable it in the NICo API config by listing the classifications to emit:

1
[observability]
2
per_object_metrics_for_classifications = ["Hardware", "PreventAllocations"]

With an empty list (the default) the metric is not registered at all; the aggregate health metrics are unaffected either way. Series disappear automatically when the object becomes healthy, loses the classification, or is deleted — entries are retained for the registry’s hold period, which is configured slightly longer than the state controllers’ metric_hold_time.

For example, use the following PromQL query to list hosts blocked from allocations by a hardware problem, or alert when hardware-unhealthy machines accumulate fleet-wide:

carbide_object_unhealthy_by_classification_count{object_type="machine",classification="Hardware",in_use="false"}
count(carbide_object_unhealthy_by_classification_count{object_type="machine",classification="Hardware"}) > 10

Per-object state progress metrics

NICo can expose current state progress for individual machines, switches, power shelves, and racks (plus network segments, VPC prefixes, SPDM attestation, and IB partitions) from a dedicated Prometheus listener. This endpoint is disabled by default and uses a separate registry, so enabling it does not add high-cardinality series to the existing /metrics endpoint.

1
[observability.per_object_state_metrics]
2
enabled = true
3
listen_address = "[::]:9091"
4
# Defaults to all supported object types; also valid: "network_segment",
5
# "vpc_prefix", "spdm_attestation", "ib_partition".
6
object_types = ["machine", "switch", "power_shelf", "rack"]

Scrape /metrics on the configured address at a relatively slow interval (60–120 seconds is normally sufficient). Queries joining these metrics with aggregate or health metrics require both the main and per-object endpoints to be scraped into the same Prometheus.

When deploying with the Helm chart, set nico-api.service.perObjectStateMetrics.enabled=true; this configures the application listener, container port, and Service together. Enable its ServiceMonitor with nico-api.perObjectStateMetricsServiceMonitor.enabled=true. If configFiles.nicoApiConfig replaces the chart’s bundled application configuration, that custom TOML must include the section above explicitly.

For example, this query finds objects that have exceeded their own resolved SLA:

(time() - carbide_object_state_entered_timestamp_seconds)
  > on(object_type, object_id, state, substate) group_left()
    carbide_object_state_sla_seconds

DPU metrics:

API Health and Availability

The NICo API is required for health inspection, health report ingestion, administrative workflows, and state-machine visibility. Check API health before debugging a host-specific health issue.

Check Kubernetes status:

$
kubectl get deploy -n nico-system nico-api
$
kubectl get pods -n nico-system -l app.kubernetes.io/name=nico-api
$
kubectl get svc -n nico-system nico-api

Check API metrics scraping:

$
kubectl get servicemonitor -n nico-system nico-api-metrics

Use Loki or Grafana Explore to inspect API logs:

{k8s_container_name="nico-api"} |= "<machine-id>"

{k8s_container_name="nico-api"} |= "<bmc-ip>" != "SPAN"

Grafana, Loki, and Logs

Use Grafana dashboards for fleet-level triage and Loki for source-specific logs. Start from aggregate host health, identify the alert source and inAlertSince, then query logs around that time.

Common Loki patterns:

{systemd_unit="forge-dpu-agent.service", machine_id="<machine-id>"}

{k8s_container_name="nico-hardware-health"} |= "<machine-id>"

Some sites expose machine identity as a log label. When that label is present, prefer a label filter over a free-text match:

{machine_id="<machine-id>"}

Console logs are shipped by the nico-ssh-console-rs OpenTelemetry Collector sidecar when enabled:

1
nico-ssh-console-rs:
2
  lokiLogCollector:
3
    enabled: true
4
    image:
5
      repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib
6
      tag: "0.81.0"

The sidecar tails:

/var/log/consoles/{machineid}_{bmc_ip}.log

It labels console logs with machineid and an SSH console exporter label:

{machineid="<machine-id>", exporter="nico-ssh-console-rs"}

Labels vary by log source. Use the Loki label browser to choose the most specific label available. Common labels include:

• k8s_container_name
• k8s_namespace_name
• k8s_pod_name
• machine_id
• machineid
• host_machine_id
• host_name
• systemd_unit
• exporter
• level

logcli can be used for repeatable terminal-based Loki queries when direct Loki access is configured for the site. Use the same LogQL selectors shown above. For example:

$
logcli query --since=1h '{k8s_container_name="nico-hardware-health"} |= "<machine-id>"'
$
logcli query --since=1h '{systemd_unit="forge-dpu-agent.service"} |= "<machine-id>"'

Dashboard Starting Points

Use the site-level health dashboard for fleet triage before drilling into logs. Start with these panels when they are available:

For host-health triage, the highest-value panels are Healthy Host Percentage, Health Status, Health Overrides, Health Probe Alerts, and Health Alert Classifications.

Troubleshooting

Triage Workflow

1. Open aggregate host health.
2. Record the alert ID, target, message, inAlertSince, and classifications.
3. Identify the source: hardware health, DPU health, validation, discovery, rack health, or override.
4. Use the source-specific metrics and logs for that alert.
5. Remediate the underlying condition.
6. Wait for a fresh health report from the responsible source.
7. Confirm aggregate host health returns to healthy.
8. Remove temporary overrides used during the investigation.