Monitoring and Health

This page covers monitoring and health workflows for NICo sites after deployment: hardware health, DPU health, aggregate host health, health overrides, Prometheus scraping, Grafana dashboards, and Loki queries.

Use aggregate host health as the starting point for operational decisions. NICo combines hardware health, DPU health, validation and discovery checks, rack health, and health overrides into a single host-level result. Component health explains which source is responsible for the aggregate result.

Use this page as the entry point for health triage. It gives the primary inspection path, commands, metrics, dashboards, and log queries needed to start an investigation. For subsystem-specific behavior, follow the linked hardware, DPU, health aggregation, and classification references rather than treating this page as a replacement for those manuals.

For reference, see:

• Health Checks and Health Aggregation
• Health Probe IDs
• Health Alert Classifications
• Redfish Workflow

Health Sources

NICo builds health from health reports. A health report contains successes and alerts from a reporting source. Common health sources are:

Each alert has an ID, an optional target, a message, a start time, and one or more classifications. Classifications define operational impact. For example, PreventAllocations blocks new allocations while the alert is active, and ExcludeFromStateMachineSla excludes the host from state-machine SLA evaluation.

Hardware Health Monitoring

NICo monitors hardware through the hardware health service. The Helm chart is nico-hardware-health.

The service discovers BMC endpoints from NICo and queries them through Redfish. It monitors host BMCs, DPU BMCs, and configured switch or power-shelf BMCs. The primary monitoring path is sensor collection. Additional collectors can gather firmware, log, NMX-T, NVUE REST, and leak-related data when configured.

Helm Configuration

Enable hardware health in Helm values:

1
nico-hardware-health:
2
  enabled: true

Enable metrics scraping with its ServiceMonitor:

1
nico-hardware-health:
2
  enabled: true
3
  replicas: 1
4
5
  serviceMonitor:
6
    enabled: true
7
    interval: 30s
8
    scrapeTimeout: 25s

By default, the chart exposes hardware health metrics on port 9009. Log collection is disabled by default:

1
env:
2
  NICO_HEALTH__COLLECTORS__LOGS__ENABLED: "false"

Enable log collection only through the target site’s deployment values.

Hardware Health Service Configuration

The hardware health service config example, crates/health/example/config.example.toml, documents endpoint discovery, sinks, rate limits, collectors, processors, and metrics.

Production endpoint discovery uses the NICo API source. The checked-in hardware-health example config currently names that source [endpoint_sources.nico_api]:

1
[endpoint_sources.nico_api]
2
root_ca = "/var/run/secrets/spiffe.io/ca.crt"
3
client_cert = "/var/run/secrets/spiffe.io/tls.crt"
4
client_key = "/var/run/secrets/spiffe.io/tls.key"
5
api_url = "https://nico-api.forge-system.svc.cluster.local:1079"

Static BMC endpoints are supported for local, mock, or special deployments:

1
[[endpoint_sources.static_bmc_endpoints]]
2
ip = "10.0.0.1"
3
port = 443
4
mac = "aa:bb:cc:dd:ee:ff"
5
username = "admin"
6
password = "secret"

Collector defaults from the example config:

BMC Proxy

The full Helm example enables nico-bmc-proxy, the authenticating proxy for BMC Redfish access. The proxy chart exposes proxy traffic on port 1079 and metrics on port 1080.

Example proxy settings:

1
listen = "[::]:1079"
2
metrics_endpoint = "[::]:1080"
3
allowed_principals = ["spiffe-service-id/nico-api", "spiffe-service-id/nv-dps"]

The BMC proxy ServiceMonitor follows the same serviceMonitor.enabled, interval, and scrapeTimeout pattern as other NICo services.

Sensor Alerts

Hardware sensor alerts are derived from BMC-reported health, sensor readings, and thresholds. Sensor classifications include:

If numeric threshold data indicates a problem but the BMC reports the sensor as healthy, NICo treats the sensor as healthy. In that case the BMC health state is the authority.

Hardware Health Logs

Use Loki or Grafana Explore to inspect hardware health logs for a host:

1
{k8s_container_name="nico-hardware-health"} |= "<machine-id>"

Health report events include fields such as:

collector=sensor_collector
report_source=bmc-sensors
machine_id=Some(<machine-id>)
alert_count=<n>
success_count=<n>

For leak-related events, look for:

report_source=tray-leak-detection

DPU Health Checks

dpu-agent runs on managed DPUs and reports DPU health to NICo. The BlueField chart is named nico-dpu-agent. In service names and logs, the DPU agent currently appears as forge-dpu-agent.service.

The agent checks DPU service health, networking state, HBN/NVUE configuration, DHCP behavior, BGP status, and heartbeat. DPU health is part of aggregate host health, so an otherwise healthy host can still be unavailable when its DPU is unhealthy.

See also:

• DPU Lifecycle Management
• DPU Configuration
• Stuck in WaitingForNetworkConfig and DPU Health

DPU Agent Configuration

Key nico-dpu-agent chart values:

The DaemonSet renders these core arguments:

nico-dpu-agent run
  --hbn-config-mode=nvue-rest
  --agent-platform-type=containerized
  --dhcp-grpc-server=http://<dhcp-grpc-service>:10079
  --fmds-grpc-server=http://<fmds-grpc-service>:50052

If dhcp_server.interface_prepend is set, the chart also adds:

--dhcp-server-interface-prepend=<prefix>

The pod sets these runtime environment variables:

Common DPU Alerts

Common DPU alert IDs include:

• ContainerExists
• ServiceRunning
• DhcpServer
• BgpStats
• BgpPeeringTor
• BgpPeeringRouteServer
• Ifreload
• BgpDaemonEnabled
• PostConfigCheckWait
• DpuDiskUtilizationCritical
• HeartbeatTimeout

HeartbeatTimeout means NICo has not received a recent health report from the DPU agent. Check whether the DPU is powered, the agent is running, DPU time is correct, and the DPU can reach NICo.

DPU Logs

Use Loki to inspect DPU-agent logs:

1
{systemd_unit="forge-dpu-agent.service", machine_id="<machine-id>"}

Alternative labels can be used when available:

1
{systemd_unit="forge-dpu-agent.service", host_name="<host-name>"}

On the DPU, use journalctl for direct service logs:

$
journalctl -u forge-dpu-agent.service -e --no-pager

Restart the agent when required:

$
systemctl restart forge-dpu-agent.service

Health Alert Lifecycle

NICo health alerts are source-based. A health source submits a fresh report, and NICo uses the latest report from each source to calculate aggregate health.

A typical alert flow:

1. A source reports an alert such as PoweredOff with target <bmc-ip>.
2. NICo adds the alert to aggregate host health.
3. Classifications such as PreventAllocations define the operational effect.
4. The health view shows the alert ID, target, message, start time, and classifications.
5. Metrics and logs identify the responsible source.
6. After remediation, the source submits a fresh report that marks the check successful or omits the previous alert.
7. NICo merges the fresh report and aggregate host health returns to healthy.

If a health override created the alert, remove or replace the override after the operational reason ends.

Inspect Current Health

Start with the host health page in the Admin Web UI:

https://<nico-api-hostname>/admin/machine/<machine-id>/health

Inspect the aggregate health table first. For each alert, note:

• ID
• Target
• In Alert Since
• Message
• Tenant Message
• Classifications

Then inspect component health to identify the source: hardware health, DPU health, validation, discovery, rack health, or health override.

Use the health history table to review recent transitions. This helps identify whether an alert is new, recurring, or already cleared by a later health report.

Admin CLI examples:

$
nico-admin-cli machine show <machine-id>
$
nico-admin-cli machine health-override show <machine-id>

Health Overrides

Health overrides add manual or service-created health reports into the same aggregate health model as automated checks. Overrides are shared health mechanisms; they are not specific to hardware health.

Use overrides for controlled states such as maintenance, validation, repair, break-fix, or temporary automation control. Do not use an override as a substitute for resolving the underlying condition.

Merge and Replace

DPU replace overrides are rejected by the API.

Override Templates

The nico-admin-cli machine health-override add command supports templates for common workflows:

Examples:

$
nico-admin-cli machine health-override show <machine-id>
$
$
nico-admin-cli machine health-override add <machine-id> \
>
  --template RequestRepair \
>
  --message "Manual repair trigger for tenant-reported issue"
$
$
nico-admin-cli machine health-override add <machine-id> \
>
  --template OutForRepair \
>
  --message "Automated repair failed, requires manual investigation"
$
$
nico-admin-cli machine health-override remove <machine-id> repair-request
$
nico-admin-cli machine health-override remove <machine-id> tenant-reported-issue

Before creating an override, identify the current aggregate health, choose the smallest effect that matches the workflow, include a clear message, and define the removal condition. After remediation, remove or replace the override and verify aggregate health.

Prometheus Metrics

NICo charts expose Prometheus scraping through ServiceMonitor resources. ServiceMonitors are disabled by default in chart values and enabled in the full example for selected services.

Example:

1
nico-api:
2
  serviceMonitor:
3
    enabled: true
4
    interval: 30s
5
    scrapeTimeout: 25s
6
7
nico-hardware-health:
8
  serviceMonitor:
9
    enabled: true
10
    interval: 30s
11
    scrapeTimeout: 25s

ServiceMonitors from the charts:

Check rendered ServiceMonitors:

$
kubectl get servicemonitor -n nico-system
$
kubectl get servicemonitor -n nico-system nico-api-metrics
$
kubectl get servicemonitor -n nico-system nico-hardware-health-metrics
$
kubectl get servicemonitor -n nico-system nico-dsx-exchange-consumer-metrics

Core health metrics currently use carbide_* metric names. Some dashboards and site configurations also expose host-health rollups with the forge_* prefix. Use the literal metric name that exists in the target site.

Useful core metric families:

Use the Host Health dashboard panels for fleet-level rollups, including host health status, health overrides, probe alerts, and alert classifications. Example dashboard queries:

1
sum by (healthy, in_use) (
2
  max by(healthy, in_use) (
3
    forge_hosts_health_status_count{fresh="true"}
4
  )
5
)

1
sum by(override_type) (
2
  max by(override_type, in_use) (
3
    forge_hosts_health_overrides_count{fresh="true"}
4
  )
5
)

1
sum by(probe_id) (
2
  max by(probe_id, probe_target) (
3
    forge_hosts_unhealthy_by_probe_id_count{fresh="true"}
4
  )
5
)

1
sum by(classification) (
2
  max by(classification, in_use) (
3
    forge_hosts_unhealthy_by_classification_count{fresh="true"}
4
  )
5
)

DPU metrics:

API Health and Availability

The NICo API is required for health inspection, health report ingestion, administrative workflows, and state-machine visibility. Check API health before debugging a host-specific health issue.

Check Kubernetes status:

$
kubectl get deploy -n nico-system nico-api
$
kubectl get pods -n nico-system -l app.kubernetes.io/name=nico-api
$
kubectl get svc -n nico-system nico-api

Check API metrics scraping:

$
kubectl get servicemonitor -n nico-system nico-api-metrics

Use Loki or Grafana Explore to inspect API logs:

1
{k8s_container_name="nico-api"} |= "<machine-id>"

1
{k8s_container_name="nico-api"} |= "<bmc-ip>" != "SPAN"

Grafana, Loki, and Logs

Use Grafana dashboards for fleet-level triage and Loki for source-specific logs. Start from aggregate host health, identify the alert source and inAlertSince, then query logs around that time.

Common Loki patterns:

1
{systemd_unit="forge-dpu-agent.service", machine_id="<machine-id>"}

1
{k8s_container_name="nico-hardware-health"} |= "<machine-id>"

Some sites expose machine identity as a log label. When that label is present, prefer a label filter over a free-text match:

1
{machine_id="<machine-id>"}

Console logs are shipped by the nico-ssh-console-rs OpenTelemetry Collector sidecar when enabled:

1
nico-ssh-console-rs:
2
  lokiLogCollector:
3
    enabled: true
4
    image:
5
      repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib
6
      tag: "0.81.0"

The sidecar tails:

/var/log/consoles/{machineid}_{bmc_ip}.log

It labels console logs with machineid and an SSH console exporter label:

1
{machineid="<machine-id>", exporter="nico-ssh-console-rs"}

Labels vary by log source. Use the Loki label browser to choose the most specific label available. Common labels include:

• k8s_container_name
• k8s_namespace_name
• k8s_pod_name
• machine_id
• machineid
• host_machine_id
• host_name
• systemd_unit
• exporter
• level

logcli can be used for repeatable terminal-based Loki queries when direct Loki access is configured for the site. Use the same LogQL selectors shown above. For example:

$
logcli query --since=1h '{k8s_container_name="nico-hardware-health"} |= "<machine-id>"'
$
logcli query --since=1h '{systemd_unit="forge-dpu-agent.service"} |= "<machine-id>"'

Dashboard Starting Points

Use the site-level health dashboard for fleet triage before drilling into logs. Start with these panels when they are available:

For host-health triage, the highest-value panels are Healthy Host Percentage, Health Status, Health Overrides, Health Probe Alerts, and Health Alert Classifications.

Troubleshooting

Triage Workflow

1. Open aggregate host health.
2. Record the alert ID, target, message, inAlertSince, and classifications.
3. Identify the source: hardware health, DPU health, validation, discovery, rack health, or override.
4. Use the source-specific metrics and logs for that alert.
5. Remediate the underlying condition.
6. Wait for a fresh health report from the responsible source.
7. Confirm aggregate host health returns to healthy.
8. Remove temporary overrides used during the investigation.