NICo Logging

How NICo components emit logs, where they go, what format they use and how to tune them.

TL;DR

• All NICo components log to stdout. In Kubernetes the kubelet writes container stdout to files under /var/log/pods/, where a collector (typically an OpenTelemetry Collector DaemonSet) picks them up.
• Most components use logfmt (key=value pairs). nico-dns uses JSON. nico-ssh-console uses a compact single-line format. nico-pxe uses plain text.
• Default log level is INFO. Override at startup with RUST_LOG or the --debug flag.
• nico-api supports runtime log-level changes via nico-admin-cli set log-filter. Changes auto-expire (default 1 hour) and revert to the startup level - useful for time-boxed debugging without forgetting to turn verbosity back down.
• Centralized logging uses a pull model: a DaemonSet collector reads pod log files and ships them to your backend (Loki, Elasticsearch, VictoriaLogs, Datadog, etc.).

1. Where logs go

Every NICo binary writes its own service logs to stdout. There is no file-based logging for service logs - the expectation is Kubernetes-native log collection.

In a Kubernetes cluster:

1. The container runtime captures stdout and writes it to a log file under /var/log/pods/<namespace>_<pod>_<uid>/<container>/*.log.
2. A log collector (DaemonSet) tails those files and forwards entries to a backend.

This is the standard Kubernetes pattern. NICo does not require any special log driver or sidecar - just configure your collector to read pod logs.

Note: nico-ssh-console is an exception - in addition to stdout, it writes machine console output (BMC serial console streams) to local files. These are not service logs but captured output from managed machines. See section 2.5 for details.

2. Which components log and in what format

2.1 logfmt format

Most NICo components use logfmt - a line-oriented format of space-separated key=value pairs, easy for both humans and machines to parse.

Event lines — one per log call:

level=INFO component=nico-api span_id=0x4f… msg="Starting reconciliation" location="handlers/machine.rs:142"

Span lines — emitted when a unit of work closes (level=SPAN), carrying timing data:

level=SPAN component=site-explorer span_id=0xf7… span_name=explore_site timing_elapsed_us=1523 timing_busy_ns=1200000 timing_idle_ns=323000

Common fields:

The logfmt layer is implemented in the logfmt crate (crates/logfmt). It emits span lifecycle events (open/close) as level=SPAN lines with timing data - useful for identifying slow operations without enabling full distributed tracing.

The component field

On logfmt lines, NICo sets the component field to identify the emitting service or subsystem:

nico-api                       — API handlers, DB, startup: anything not in a subsystem below
├── site-explorer
├── machine_state_controller
├── switch_controller
├── rack_controller
├── power_shelf_controller
├── network_segments_controller
├── vpc_prefix_controller
├── ib_partition_controller
└── attestation_controller
nico-bmc-proxy
nico-dhcp
nico-dsx-exchange-consumer
nico-fmds
nico-hardware-health
nico-rvs
nico-test-artifact-cache
nico-dpu-agent
nico-scout

State-controller lines also carry a controller=<name> field with the same value.

This enables filtering logs by component in your backend. For example, with Loki:

{namespace="nico-system"} | logfmt | component="site-explorer"

Convention: NICo uses component for the emitting service/subsystem. Don’t reuse this key for domain data - give those their own keys (e.g. machine_id, controller).

Coverage

Components that do not use the logfmt layer and carry no component field:

2.2 JSON format (nico-dns)

nico-dns uses tracing-subscriber’s JSON formatter. Each line is a self-contained JSON object:

1
{"timestamp":"2026-01-15T10:23:45.123Z","level":"INFO","target":"carbide_dns","message":"DNS query","query_type":"A","name":"host1.example.com"}

2.3 Compact format (nico-ssh-console)

nico-ssh-console uses tracing-subscriber’s compact formatter - a human-readable single-line format similar to traditional log output:

2026-01-15T10:23:45.123Z  INFO carbide_ssh_console: Session started session_id=abc-123

2.4 Plain text (nico-pxe)

nico-pxe uses println! for startup messages and request logging middleware. Output is unstructured plain text.

2.5 Machine console logs (nico-ssh-console)

In addition to its own service logs (compact format on stdout), nico-ssh-console captures machine and DPU serial console output to local files. This is separate from the service’s tracing output.

When a BMC console session is established, nico-ssh-console streams the serial output to a per-machine log file:

/var/log/consoles/<machine-id>_<bmc-ip>.log

Key details:

The console logger strips ANSI escape sequences and adds timestamps at session start/stop. Rotation happens automatically - old logs are renamed with numeric suffixes and the oldest is deleted when the limit is reached.

These files contain raw machine boot output, kernel messages and anything else that appears on the serial console. They are useful for debugging boot failures, kernel panics and hardware issues.

Centralizing console logs: The nico-ssh-console Helm chart includes an optional OpenTelemetry Collector sidecar (lokiLogCollector.enabled: true) that ships console logs to Loki. The sidecar reads from /var/log/consoles/*.log, extracts the machine ID and BMC IP from filenames, and exports to your Loki endpoint. To use a different backend, customize the configFiles.otelcolConfig value in the chart. A separate document covers machine log collection workflows in detail.

3. How to tune log levels

3.1 At startup: RUST_LOG and —debug

All components respect the RUST_LOG environment variable, which uses tracing-subscriber’s EnvFilter syntax:

$
# Global level
$
RUST_LOG=debug
$
$
# Per-module levels
$
RUST_LOG=info,carbide_api_core::handlers=debug,sqlx=warn
$
$
# Combined
$
RUST_LOG=info,carbide=debug,hyper=error

Several binaries also accept a --debug flag that sets the default level to DEBUG (equivalent to RUST_LOG=debug if RUST_LOG is unset):

$
carbide-api run --debug -c /etc/carbide/config.toml
$
carbide-bmc-proxy --debug --config-path /etc/carbide/bmc-proxy.toml

Default log level is INFO for all components.

Noisy dependencies

NICo components automatically suppress verbose output from common dependencies. For example, nico-api applies these directives by default:

sqlxmq::runner=warn,sqlx::query=warn,rustify=off,hyper=error,rustls=warn,h2=warn,vaultrs=error

Your RUST_LOG directives are merged with these defaults, so you don’t need to manually silence framework noise.

3.2 At runtime: nico-admin-cli (nico-api only)

nico-api supports live log-level changes without a restart. This is useful for debugging production issues - you can increase verbosity temporarily, then let it automatically revert.

$
# Increase verbosity for 1 hour (default expiry)
$
nico-admin-cli set log-filter -f "debug"
$
$
# Target specific modules for 30 minutes
$
nico-admin-cli set log-filter -f "info,carbide_api_core::handlers::machine=debug" --expiry 30min
$
$
# Very verbose, short window
$
nico-admin-cli set log-filter -f "trace,h2=warn,hyper=warn" --expiry 5min

When the expiry elapses, the log filter automatically reverts to the startup value. This prevents accidentally leaving debug logging on and filling your storage.

How it works

The admin CLI calls a gRPC endpoint on nico-api that updates the EnvFilter in the running process. The new filter applies immediately to all subsequent log events - no restart, no config file change. Under the hood, tracing-subscriber’s reload::Handle swaps the active filter.

Scope: runtime log-level changes affect nico-api only. Other components (nico-dns, nico-dhcp, etc.) require a restart to change log levels.

3.3 Kubernetes deployment

Set RUST_LOG in your pod spec or Helm values:

1
# Pod spec
2
spec:
3
  containers:
4
    - name: nico-api
5
      env:
6
        - name: RUST_LOG
7
          value: "info,carbide=debug"
8
9
# Or via Helm values
10
env:
11
  RUST_LOG: "info,carbide=debug"

For temporary debugging, use nico-admin-cli to avoid redeploying:

$
kubectl exec -it deploy/nico-api -- nico-admin-cli set log-filter -f "debug" --expiry 15min

4. Centralizing logs with OpenTelemetry Collector

The recommended approach for centralized logging is an OpenTelemetry Collector DaemonSet that reads pod log files and exports them to your backend. This is the standard Kubernetes pattern and works with any OTLP-compatible backend: Grafana Loki, Elasticsearch, VictoriaLogs, Datadog, Splunk, etc.

┌─────────────────────────────────────────────────────────────────────────┐
│  Node                                                                   │
│  ┌──────────────┐    stdout    ┌──────────────┐                         │
│  │   NICo Pod   │ ──────────▶  │  /var/log/   │                         │
│  │  (nico-api)  │              │   pods/...   │                         │
│  └──────────────┘              └──────┬───────┘                         │
│                                       │                                 │
│                                       ▼ filelog receiver                │
│                              ┌────────────────────┐                     │
│                              │   otel-collector   │                     │
│                              │    (DaemonSet)     │                     │
│                              └─────────┬──────────┘                     │
└────────────────────────────────────────┼────────────────────────────────┘
                                         │ OTLP / Loki API / etc.
                                         ▼
                              ┌────────────────────┐
                              │   Logs Backend     │
                              │ (Loki, ES, VL...)  │
                              └────────────────────┘

4.1 Collector configuration

Below is a reference Helm values file for deploying the OpenTelemetry Collector Helm chart to collect NICo logs. Adapt the exporter section for your backend.

The logsCollection preset parses the Kubernetes container log format; the explicit filelog receiver below controls which pod log files are collected.

1
# values.yaml for opentelemetry-collector Helm chart
2
# Install: helm install otel-collector open-telemetry/opentelemetry-collector -f values.yaml
3
mode: daemonset
4
image:
5
  repository: otel/opentelemetry-collector-k8s
6
7
presets:
8
  kubernetesAttributes:
9
    enabled: true      # Adds k8s metadata (pod name, namespace, etc.)
10
  logsCollection:
11
    enabled: true      # Enables filelog receiver for pod logs
12
13
config:
14
  receivers:
15
    filelog:
16
      include:
17
        # Adjust this pattern if your NICo namespace uses a different prefix
18
        - /var/log/pods/nico-*/*/*.log
19
      exclude:
20
        # Exclude collector's own logs to avoid feedback loops
21
        - /var/log/pods/*/opentelemetry-collector*/*.log
22
23
  processors:
24
    memory_limiter:
25
      check_interval: 1s
26
      limit_percentage: 75
27
      spike_limit_percentage: 20
28
29
    batch:
30
      send_batch_size: 1024
31
      timeout: 5s
32
33
    # Add labels for collected NICo logs
34
    resource:
35
      attributes:
36
        - action: insert
37
          key: component
38
          value: "nico"
39
        - action: insert
40
          key: service.name
41
          from_attribute: k8s.container.name
42
43
  exporters:
44
    # === Choose your backend ===
45
46
    # Option A: OTLP over HTTP (VictoriaLogs, etc.)
47
    otlphttp:
48
      logs_endpoint: http://victorialogs.monitoring.svc.cluster.local:9428/insert/opentelemetry/v1/logs
49
50
    # Option B: OTLP over gRPC (Datadog, Splunk, etc.)
51
    # otlp:
52
    #   endpoint: <your-otlp-endpoint>:4317
53
    #   tls:
54
    #     insecure: false
55
56
    # Option C: Grafana Loki
57
    # loki:
58
    #   endpoint: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push
59
    #   default_labels_enabled:
60
    #     exporter: false
61
62
    # Option D: Elasticsearch / OpenSearch
63
    # elasticsearch:
64
    #   endpoints: ["https://elasticsearch.elastic.svc.cluster.local:9200"]
65
    #   logs_index: "nico-logs"
66
    #   tls:
67
    #     insecure: false
68
    #     ca_file: /etc/otel/certs/ca.crt
69
70
  service:
71
    pipelines:
72
      logs:
73
        receivers: [filelog]
74
        processors: [memory_limiter, resource, batch]
75
        exporters: [otlphttp]   # Change to your exporter

4.2 Extracting structured fields

Since most NICo components use logfmt, you can parse structured fields at the collector level. This makes fields like level, msg, machine_id, etc. available for filtering and querying in your backend.

Add a transform processor to parse logfmt:

1
processors:
2
  transform/parse-logfmt:
3
    log_statements:
4
      - context: log
5
        statements:
6
          # Parse logfmt body into attributes
7
          # This regex extracts key=value pairs, handling quoted values
8
          - merge_maps(attributes, ParseKeyValue(body, "=", " "), "upsert")
9
            where IsMatch(body, "^level=")

If you cannot use the transform processor, a minimal filelog receiver regex can extract level and msg while keeping the remaining key-value pairs in rest. Use the ParseKeyValue approach above for full field extraction.

1
receivers:
2
  filelog:
3
    include:
4
      - /var/log/pods/nico-*/*/*.log
5
    operators:
6
      - type: container
7
      - type: regex_parser
8
        if: 'body matches "^level="'
9
        regex: 'level=(?P<level>\w+)\s+(?:msg="(?P<msg>[^"]*)")?\s*(?P<rest>.*)'
10
        parse_to: attributes

4.3 Filtering and sampling

For high-volume deployments, filter or sample logs at the collector to reduce noise and storage costs.

Dropping noisy log messages

Some log messages are expected but not useful for debugging. For example, DHCP relay requests that don’t match a defined network segment (because switches relay DHCP device-wide, not per-network), or DPU kernel messages about mlx5_core module state changes during normal operation. These can flood logs without adding signal.

Use the filter processor to drop specific messages by pattern:

1
processors:
2
  # Drop known noisy messages
3
  filter/noise:
4
    error_mode: silent
5
    logs:
6
      log_record:
7
        # Drop messages matching these patterns
8
        - IsMatch(body, "Module ID not recognized")
9
        - IsMatch(body, "No network segment defined for relay address")
10
11
  # Or drop by log level
12
  filter/drop-debug:
13
    error_mode: silent
14
    logs:
15
      log_record:
16
        - IsMatch(body, "^level=DEBUG")
17
        - IsMatch(body, "^level=TRACE")

The error_mode: silent setting prevents the filter processor from logging errors when a record doesn’t match - otherwise you’d get noise about the noise filtering.

Managing filter patterns with Helm

When deploying via Helm, keep filter patterns separate from the main collector config for easier maintenance. There are two approaches:

Option A: Patterns in chart files

Place patterns in a file within your chart package at files/drop-patterns.txt:

# files/drop-patterns.txt
# Noisy DHCP messages from cross-network relay
No network segment defined for relay address
# DPU mlx5_core noise
Module ID not recognized

Then use .Files.Lines in your template to load them:

1
# templates/configmap.yaml (collector config section)
2
config:
3
  processors:
4
    filter/noise:
5
      error_mode: silent
6
      logs:
7
        log_record:
8
          {{- range .Files.Lines "files/drop-patterns.txt" }}
9
          {{- $line := . | trim }}
10
          {{- if and $line (not (hasPrefix "#" .)) }}
11
          - IsMatch(body, {{ $line | quote }})
12
          {{- end }}
13
          {{- end }}

This reads patterns at helm install time from the chart’s files/ directory. To update patterns, you must update the chart and redeploy.

Option B: Patterns in values.yaml

For patterns that operators can change without modifying the chart, use values:

1
# values.yaml
2
filterPatterns:
3
  - "Module ID not recognized"
4
  - "No network segment defined for relay address"

1
# templates/configmap.yaml (collector config section)
2
config:
3
  processors:
4
    filter/noise:
5
      error_mode: silent
6
      logs:
7
        log_record:
8
          {{- if .Values.filterPatterns }}
9
          {{- range .Values.filterPatterns }}
10
          - IsMatch(body, {{ . | quote }})
11
          {{- end }}
12
          {{- end }}

This approach lets operators add or remove filter patterns by updating Helm values (--set-json or a values file override) without modifying the chart itself.

Routing logs to different pipelines

For more control, use a routing connector to send different log types to different pipelines. This lets you apply different filters or export to different backends:

1
connectors:
2
  routing/logs:
3
    default_pipelines:
4
      - logs/default
5
    table:
6
      # Route console logs to a separate pipeline
7
      - statement: route() where attributes["component"] == "nico-ssh-console-rs"
8
        pipelines:
9
          - logs/console-local    # Keep all locally
10
          - logs/console-remote   # Filter before remote export
11
12
service:
13
  pipelines:
14
    logs/input:
15
      receivers: [filelog]
16
      exporters: [routing/logs]
17
18
    logs/console-local:
19
      receivers: [routing/logs]
20
      processors: [batch]
21
      exporters: [loki/local]
22
23
    logs/console-remote:
24
      receivers: [routing/logs]
25
      processors: [filter/noise, batch]
26
      exporters: [loki/remote]
27
28
    logs/default:
29
      receivers: [routing/logs]
30
      processors: [batch]
31
      exporters: [loki/local]

Probabilistic sampling

For extremely high-volume logs where you only need a statistical sample:

Log support in the OpenTelemetry probabilistic_sampler processor is alpha. Verify that your collector distribution supports log pipelines for this processor before relying on it in production.

1
processors:
2
  probabilistic_sampler:
3
    sampling_percentage: 10   # Keep 10% of logs

Use sampling cautiously - you may miss the one error message that matters.

4.4 Backend-specific notes

VictoriaLogs

VictoriaLogs accepts OTLP logs. Use the otlphttp exporter with logs_endpoint:

1
exporters:
2
  otlphttp:
3
    logs_endpoint: http://victorialogs.monitoring.svc.cluster.local:9428/insert/opentelemetry/v1/logs

Grafana Loki

Loki works well with logfmt - it can parse key=value pairs natively in queries using | logfmt. Set loki.format: raw in the resource processor to send logs unparsed, letting Loki handle parsing at query time:

1
processors:
2
  resource:
3
    attributes:
4
      - action: insert
5
        key: loki.format
6
        value: raw
7
      - action: insert
8
        key: loki.resource.labels
9
        value: k8s.namespace.name, k8s.pod.name, k8s.container.name

Query example:

{k8s_container_name="nico-api"} | logfmt | level="ERROR"

Elasticsearch / OpenSearch

Parse logfmt at ingest time (in the collector or an ingest pipeline) to get structured documents. This enables efficient field-based queries and aggregations.

Datadog

Use the Datadog exporter or OTLP endpoint. Datadog automatically parses common log formats.

5. Troubleshooting

Verifying log output

To see raw logs from a NICo component:

$
# Stream logs from nico-api
$
kubectl logs -f deploy/nico-api
$
$
# Last 100 lines from nico-dns
$
kubectl logs --tail=100 deploy/nico-dns
$
$
# All containers in a pod
$
kubectl logs <pod-name> --all-containers

For more advanced log tailing across multiple pods, use stern:

$
# Stream logs from all nico-api pods
$
stern nico-api
$
$
# Filter by log level (logfmt)
$
stern nico-api --include 'level=ERROR'
$
$
# Stream logs from multiple components
$
stern 'nico-(api|dns|dhcp)'
$
$
# Include timestamps and pod name
$
stern nico-api -t

stern is particularly useful when you have multiple replicas or want to watch several components at once.

Checking current log level (nico-api)

The current log filter is visible in nico-api’s startup log and via the admin API. Look for:

level=INFO msg="current log level: info,carbide=debug" location="setup.rs:142"

6. References

• tracing-subscriber EnvFilter syntax
• OpenTelemetry Collector filelog receiver
• logfmt specification
• stern - Multi-pod log tailing for Kubernetes