NICo Tracing

How NICo component tracing works, what it covers, how to turn it on and off and what it costs.

TL;DR

• nico-api (the carbide-api binary) is NICo’s primary tracing source and the subject of this document. nico-dns also emits traces, but with a separate simpler opt-in setup. nico-bmc-proxy emits traces for each proxied BMC request when configured (see nico-bmc-proxy tracing) and nico-pxe for each boot request it serves (see nico-pxe tracing).

• nico-api traces are off by default; two things must both be true before any spans are emitted:

  • An OTLP endpoint is configured at startup, either in the nico-api config TOML:

    [tracing]
    otlp_endpoint = "http://<otel_endpoint_host>:4317" # gRPC (default port 4317)

    or with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, which overrides the TOML value.

  • Tracing is enabled, either in the same config section with enabled = true, or at runtime with nico-admin-cli set tracing-enabled true when tracing.allow_runtime_changes = true.

• Tracing is resource-intensive when on, so turn it on for a debugging session and then off after.

  # Once the endpoint is configured and runtime changes are allowed:
  nico-admin-cli set tracing-enabled true     # start capturing
  # ... reproduce the issue, examine traces in your backend ...
  nico-admin-cli set tracing-enabled false    # stop capturing traces

  Leaving the OTLP endpoint configured while tracing is disabled costs almost nothing.

• Transport is OTLP/gRPC, plaintext; nico-api cannot do OTLP/HTTP or originate TLS

• nico-api propagates W3C trace context at its network boundaries: it reads traceparent/ tracestate from inbound REST and gRPC requests and continues that trace, injecting the same headers into its outbound requests. Propagation links traces across services, but does not by itself enable recording (see W3C trace-context propagation).

• NICo REST API services share one OpenTelemetry bootstrap configured through standard OTEL_* variables. Unlike nico-api, they export over OTLP/HTTP by default and can use TLS. See REST service tracing.

1. How tracing works

Which components emit traces

The following binaries build an OTLP span exporter:

• nico-api (crates/api-core/src/logging/setup.rs) - the rich, control-plane tracing this document is mostly about, off by default behind endpoint plus enabled-flag configuration
• nico-dns (crates/dns/src/main.rs) - a separate, much simpler opt-in setup.
• nico-bmc-proxy (crates/bmc-proxy/src/setup.rs) - one span per proxied BMC request, off by default behind endpoint plus [tracing] enabled (see nico-bmc-proxy tracing).
• nico-pxe (crates/pxe/src/main.rs) - request spans, off by default unless an OTLP endpoint is configured (see nico-pxe tracing).
• nico-dhcp (crates/dhcp/src/tracing_setup.rs) - the Kea hook library the nico-dhcp pods load. One span per discovery or lease expiration, off by default unless an OTLP endpoint is configured (see nico-dhcp tracing).
• nico-dhcp-server (crates/dhcp-server/src/main.rs) - the Rust DHCP server the BlueField DaemonSet runs. One span per packet, configured the same way (see nico-dhcp tracing).
• NICo REST API services (rest-api/common/pkg/otel), off by default until an OTLP endpoint variable is set, plus tracing.enabled on nico-rest-api and the workflow workers (see REST service tracing).

The other binaries (nico-hardware-health, nico-ssh-console-rs, and nico-dsx-exchange-consumer) carry the OpenTelemetry crates in the workspace but do not build a span exporter, so they do not emit traces.

Unless noted otherwise, the rest of this document describes nico-api tracing. nico-dns differs as described in nico-dns tracing. NICo REST API services are described separately in REST service tracing.

What operations are covered

nico-api links many library crates in-process and the #[tracing::instrument] spans live in those crates. When tracing is enabled, the instrumented operations are:

There is also a metric, carbide_api_tracing_spans_open, that reports the number of currently open spans (exported by the spancounter crate) - useful for spotting span leaks or runaway trace volume.

These cover the control-plane paths an operator most often needs to debug: machine provisioning/reconcile loops, power control and firmware updates against the BMC/power/rack backends, plus the database work underneath them - which maps directly to the EPIC’s “time on a given state of the machine, nodes stuck” need.

RMS RPC auditing

NICo configures librms with a caller-owned transport layer that injects W3C trace context before the V1 or V2 tonic client sends each request. The layer also applies to readiness probes and connections rebuilt after certificate rotation. The generated unary wrappers open an rms_rpc span and report decoded request/response bodies, the protobuf method and message type, numeric grpc_status_code, UTC RFC 3339 request_timestamp and response_timestamp, and monotonic elapsed_milliseconds. Request time is the start of the logical call, before lazy connection setup and its readiness retries; response time is when that call returns. elapsed_milliseconds spans that same interval, so it also includes NICo’s own request-side redaction and span setup, but not response redaction. These are client-side call boundaries, not server receive times.

The same fields appear in completion logs under the rms_rpc_audit target at DEBUG level. The rms_rpc span sets logfmt.suppress, so the logfmt output carries that DEBUG record and no level=SPAN close line repeating the bodies. The admin UI log stream, when enabled, is a separate layer and still lists the span with its fields. In nico-api, DEBUG audit logging is on when the current runtime log filter, including changes made through dynamic settings, allows rms_rpc_audit at DEBUG; another layer accepting DEBUG does not count. When that filter and the API runtime tracing flag are both off, payload encoding and redaction are skipped; transport propagation remains installed. The transport runs without the caller’s span, so connections it opens do not keep the rms_rpc span open after the call returns, and the client’s own events are not recorded.

To turn on the audit log in nico-api, add the target to the server’s runtime filter:

nico-admin-cli set log-filter --filter info,rms_rpc_audit=debug --expiry 30min

The required --filter replaces the whole filter, so include the base level (info here). The optional --expiry accepts a duration such as 30min or 1h, defaults to 1h, and cannot exceed 60h. After expiry, the server restores its startup filter on the next 15-minute reset check. A filter with span directives cannot be evaluated by target alone, so nico-api enables payload encoding and redaction even if the filter ultimately suppresses the audit record. For nico-admin-cli rms, set RUST_LOG=rms_rpc_audit=debug; the records go to stderr.

Bodies are redacted before being attached to a span or log: credentials, passwords, tokens, URLs, arbitrary attributes, config values, error text, embedded JSON, and binary fields are withheld. Only reviewed identifier, address, domain, and version string fields are retained; new string fields are withheld until reviewed. Numeric, boolean, and enum values remain. Unknown or malformed messages and bodies exceeding 65,536 bytes before or after JSON serialization produce a [BODY OMITTED: <reason>] marker, for example [BODY OMITTED: exceeds 65536 bytes], [BODY OMITTED: unknown message type], or [BODY OMITTED: invalid protobuf], with no raw-payload fallback. Withheld field values appear as [REDACTED]. Status messages, status details, and request/response metadata are not recorded. Transport failures record response_body as the literal null; cancellation records gRPC CANCELLED (numeric code 1). This is diagnostic auditing in logs and sampled traces, not a durable audit database.

The API’s shared clients cover every caller listed in the RMS RPCs row of What operations are covered. Existing RED (rate, errors, and duration) metrics are unchanged. Readiness probes carry trace headers but do not produce their own body audit record. Raw tonic clients and streaming RPCs are outside the decoded observer; RMS’s V1/V2 service definitions contain only unary RPCs.

nico-admin-cli rms installs the same policy and emits DEBUG audit logs through its own subscriber, but does not export nico-api OTLP spans. Without a configured propagator and active OpenTelemetry context, transport injection is a no-op. API spans follow the existing endpoint and enabled flag. An RMS RPC without a local parent is an eligible trace root; child RPCs inherit their parent’s sampling decision. Rack reconciliation adds a rack span under the state processor’s existing trace root.

How spans are selected (sampler)

nico-api uses a custom CarbideSpanSampler:

• A root span is recorded only if both are true:
  • the in-process tracing_enabled flag is on, from [tracing] enabled = true at startup or from the dynamic tracing-enabled setting
  • the span carries the carbide.trace_root marker attribute, set explicitly on the request span and a few deliberate roots (the state-controller reconcile loops and site-explorer)
• In-process child spans inherit the root’s decision, so once a trace is sampled the whole call tree beneath it is captured - except tokio spans, which are always dropped (they leak and would exhaust memory).
• For a span parented to a remote (ingress-extracted) trace, the decision stays local: an inbound sampled flag does not override tracing_enabled (see W3C trace-context propagation).
• The exporter resource is service.name = carbide-api; the tracer is named carbide.

How traces leave nico-api

nico-api pushes spans over OTLP/gRPC to a collector endpoint you configure. It does not discover or get injected with anything - it simply connects out to the endpoint from [tracing] otlp_endpoint or, if set, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. The environment variable overrides the TOML value. The transport details: gRPC-only, plaintext.

nico-dns tracing (separate, opt-in)

nico-dns has its own tracing setup (crates/dns/src/main.rs), independent of and simpler than nico-api’s:

• Off by default. nico-dns builds the span exporter only when an OTLP endpoint is configured
  • via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, the --otlp-endpoint CLI flag, or the otlp_endpoint config field (crates/dns/src/config.rs). With none of those set, no exporter is built and no traces are sent. The env var takes precedence over the CLI flag/config file.
• No hardcoded default endpoint. Earlier versions defaulted to http://opentelemetry-collector.otel.svc.cluster.local:4317 and exported unconditionally, which meant every deployment silently exported traces to that address whether or not a collector was listening there for OTLP/gRPC (see NVBUG 6717563). That default has been removed; set the endpoint explicitly to enable tracing.
• Default sampler. It uses the OpenTelemetry SDK’s default sampler (no CarbideSpanSampler), so it records broadly, filtered only by the log-level directives in its EnvFilter. It instruments retrieve_records, among others.
• Resource / output: service.name = nico-dns; logs are logfmt on stdout, matching nico-api.
• Same transport constraints: OTLP/gRPC, plaintext (with_tonic, no tls feature)

nico-bmc-proxy tracing

nico-bmc-proxy traces each proxied Redfish request through the BMC credential proxy (crates/bmc-proxy/src/proxy/). It follows the same W3C propagation model as nico-api (issue #2438) so a call from nico-api or DPS stays one trace across the proxy hop (issue #2355).

• Off by default. Spans are exported only when an OTLP endpoint is configured and [tracing] enabled = true (or the process is started with --debug). There is no runtime toggle on this binary.
• Environment variable override. Tracing can be enabled via environment variable using the NICO_BMC_PROXY__TRACING__ENABLED=true. The double underscore (__) maps to nested TOML sections, so NICO_BMC_PROXY__TRACING__ENABLED overrides [tracing] enabled. This prefix takes precedence over TOML configuration.
• Endpoint. Set the standard OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, or OTEL_EXPORTER_OTLP_ENDPOINT to cover every signal at once; the trace-specific variable wins when both are set. [tracing] otlp_endpoint in the proxy TOML is the fallback for when neither variable is set. The remaining standard OTLP transport settings (OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_HEADERS, …) are read by the exporter itself and apply as well. A malformed endpoint is rejected when the exporter is built; the proxy logs a warning and serves BMC traffic without tracing rather than refusing to start.
• Ingress. Each proxied request opens a bmc_proxy_request span and adopts any inbound traceparent/tracestate via trace_propagation::set_span_parent_from_headers.
• Egress to BMC. Upstream Redfish calls use a reqwest-tracing client so the active proxy span’s W3C context is injected on the BMC leg. The inbound headers are dropped before the upstream request is assembled — trace_propagation::is_propagated_header asks the configured propagator which headers are its own — so the BMC parents under the proxy’s span rather than the caller’s.
• Egress to nico-api (gRPC). Credential lookup uses the shared ForgeApiClient, which already wraps the transport with TraceInjectService.
• Resource / tracer: service.name = nico-bmc-proxy, tracer name nico-bmc-proxy.
• Span fields: HTTP method and request path, the status the proxy answered its caller with (not the BMC’s — a request the proxy rejects never reaches one), BMC target IP (span attribute, not a Prometheus label), the request class (bmc_proxy.class) after the request passes its ACL, and the target type (bmc_proxy.target: bmc, or switch_nvos for a switch’s NVOS management interface) after its credentials resolve. Only a 5xx sets the span status to error; a 4xx is the caller’s error. A 429 the proxy answers for want of a slot at the BMC leaves the span ok as well; carbide_bmc_proxy_admission_refused_total counts those.

Example config:

[tracing]
enabled = true
otlp_endpoint = "http://otel-collector.observability.svc.cluster.local:4317"

Point this at the same collector nico-api uses: spans only join into one trace if every hop’s exporter reaches the same backend. The components stay distinguishable by their service.name.

nico-pxe tracing

nico-pxe traces each HTTP request it serves, including the iPXE script, cloud-init and TLS bootstrap routes. It uses the shared setup in carbide_instrument::otlp_tracing, enabled by that crate’s otlp-tracing feature.

• Off by default. nico-pxe exports spans only when an OTLP endpoint is configured. It has no separate enabled flag and no runtime toggle.
• Endpoint. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (otlpEndpoint Helm value) or OTEL_EXPORTER_OTLP_ENDPOINT, which also applies to metrics and logs. The OTEL_EXPORTER_OTLP_TRACES_ENDPOINT var takes precedence when both are set. If no endpoint is set, span export stays off and nico-pxe logs at debug. If the exporter rejects the endpoint, it logs a warning and disables span export.
• Span level, separate from the log level. NICO_TRACES_SPAN_LEVEL sets the most verbose span level exported, defaulting to info. A span’s level is set by the macro that creates it, such as info_span! or #[instrument(level = "debug")], and decides only whether the exporter receives the span. It accepts off, error, warn, info, debug, trace, or 0-5. It is independent of RUST_LOG: set it to debug or trace to export more spans without adding lines to stdout, and changing RUST_LOG does not change which spans are exported. If the value is invalid, nico-pxe logs a warning and keeps the default level.
• Inbound requests. Each request opens a request span (crates/pxe/src/middleware/logging.rs) and uses any inbound traceparent or tracestate header as its parent.
  • A booting node cannot send those headers, so most requests span starts a new trace.
  • A request from another traced service continues that service’s trace.
• Outbound gRPC to nico-api. Calls use the shared ForgeTlsClient, which wraps its transport with TraceInjectService and sends the trace context on every request. No per-call code is needed.
• Resource / tracer: service.name = nico-pxe, tracer name nico-pxe.
• Span fields: the same fields the request log line carries - span_id, client IP and port, method, path, query, the Host, Content-Length and User-Agent headers when present and the response status.
• Sampling. The standard OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG variables are used to configure a sampler. Prefer parentbased_traceidratio over traceidratio. A parent-based sampler applies the ratio only at the service that starts a trace and later services reuse it. The nico-api has a different tracing support approach. It does not read these variables, because it installs CarbideSpanSampler instead.
• Shutdown. On SIGTERM, nico-pxe stops accepting connections, lets in-flight requests finish, then sends the last batch of spans.

nico-dhcp tracing

Two separate programs serve DHCP, and both use the shared setup in carbide_instrument::otlp_tracing with the same variables as nico-pxe.

• The Kea hook (crates/dhcp, built as libdhcp.so) runs inside the kea-dhcp4 and kea-dhcp6 processes that the nico-dhcp chart deploys. It opens a dhcp_discovery, dhcpv6_discovery or dhcp_lease_expiration span around each callout that calls nico-api. service.name = nico-dhcp.

• nico-dhcp-server (crates/dhcp-server, the forge-dhcp-server binary) runs on BlueField DPUs through bluefield/charts/nico-dhcp-server. It opens one span per DHCPv4 and DHCPv6 packet it processes. service.name = nico-dhcp-server.

• Off by default. Both export spans only when an OTLP endpoint is configured.

• Endpoint. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT, with the traces-only variable taking precedence. The otlpEndpoint Helm value sets the first one in the nico-dhcp chart, where it reaches both the DHCPv4 and the DHCPv6 deployment, and in the BlueField chart.

• Span level. NICO_TRACES_SPAN_LEVEL works as it does for nico-pxe, defaulting to info.

• Each packet starts a new trace. A DHCP client sends no traceparent, so there is no inbound context to continue. Spans opened inside that root span, including the outbound gRPC call, join the same trace.

• Outbound gRPC to nico-api. Calls use the shared ForgeTlsClient, which sends the trace context on every request, so a DHCP trace continues into nico-api.

• Hook logging is unchanged. The hook logs through Kea’s logger, and installing span export does not add log lines or route them through RUST_LOG.

• Shutdown. The hook sends its last batch when Kea unloads it. nico-dhcp-server sends its last batch only after its serving loop returns to main. It does not handle SIGTERM, so spans still buffered when the pod is terminated are lost.

W3C trace-context propagation

nico-api accepts and produces W3C Trace Context headers (traceparent and tracestate) at its network boundaries, so a request already traced by another service stays one trace as it passes through nico-api. The standard TraceContextPropagator is installed once at startup (crates/api-core/src/logging/setup.rs); there is no custom header parsing.

• Ingress (REST + gRPC). The shared per-request layer (crates/api-core/src/logging/api_logs.rs) extracts any inbound traceparent or tracestate and makes the upstream span the parent of nico-api’s request span. REST and gRPC flow through this single layer, so both are covered. A missing or malformed traceparent leaves the request span a fresh root.
• Egress. When nico-api makes an outbound call from within a traced request, it injects the current traceparent and tracestate so the downstream service can continue the trace. Covered:
  • gRPC - Forge and NMX-C (crates/rpc), the NSM and power-shelf (PSM) backends (crates/component-manager), and the NMX-C client pool (crates/libnmxc), through a shared tower layer applied to every request.
  • gRPC (RMS) - the RMS clients use the same trace_propagation::TraceInjectService, but librms owns the transport, so rms_observability::configure() (crates/rms-observability) installs it as the caller-supplied transport layer on the RmsClientConfig. nico-api calls it from crates/api-core/src/setup.rs. See RMS RPC auditing.
  • HTTP - the BMC/Redfish handler, machine-identity token exchange, admin-UI OAuth2, NRAS, the MQTT OAuth2 token provider, and firmware downloads.
• Interaction with the enable flag. tracing-enabled is the master switch for what nico-api records: an inbound sampled flag never turns recording on here. When tracing-enabled is on, the inbound trace_id is inherited, so nico-api’s spans join the caller’s trace.
• Forwarding vs. recording. Forwarding the context is separate from recording it, but both currently depend on the exporter being built:
  • Exporter built, tracing off: records nothing, yet still forwards the inbound trace_id and tracestate marked not sampled (sampled=0).
  • No endpoint configured (exporter not built): does not forward at all, so the trace breaks at this hop. This is a known limitation.
• Scope. Trace context only (traceparent or tracestate).

Adding a new network client

Propagation is automatic on ingress but opt-in on egress. Keep the following in mind when adding code:

• New ingress (a REST route or gRPC method): nothing to do. Every inbound request flows through the shared per-request layer (crates/api-core/src/logging/api_logs.rs), which extracts the inbound context for you.

• New outbound gRPC client (tonic/hyper): wrap its channel/service with trace_propagation::TraceInjectService at construction. Better yet, build through an existing shared client that already wraps the transport (see crates/rpc/src/forge_tls_client.rs).

• New outbound HTTP client (reqwest): build it through the reqwest-tracing middleware instead of using a bare reqwest::Client. The wrapped client injects the current traceparent and tracestate into every request automatically, so there is no per-call code:

  let client = reqwest_middleware::ClientBuilder::new(reqwest::Client::new())
      .with(reqwest_tracing::TracingMiddleware::default())
      .build(); // -> reqwest_middleware::ClientWithMiddleware, a drop-in for request-building
  let resp = client.get(url).send().await?;

  See crates/nras/src/client.rs for a real example.

• When another crate owns the HTTP call (manual fallback): if the request is built and sent by code you don’t control (for example, the oauth2 client (crates/api-web/src/auth.rs), which owns its own reqwest request) inject into that request’s headers directly:

  trace_propagation::inject_current_context(request.headers_mut());

Injection is always a no-op when no trace is active, so it is safe to add it unconditionally.

2. How to enable and disable tracing

Enabling tracing has two parts: startup configuration for the exporter endpoint, and an enabled flag that can come from startup config or, when allowed, the runtime switch. An endpoint without the enabled flag emits no traces. The enabled flag without an endpoint also emits no traces because no OTLP exporter is built.

 Startup configuration                             Enable/disable policy
 ┌───────────────────────────────┐                  ┌─────────────────────────────────────┐
 │ a. a traces backend           │                  │ [tracing] enabled = true|false      │
 │ b. a collector to receive     │   ── then ──▶    │ and optionally:                     │
 │    OTLP from nico-api         │                  │ nico-admin-cli set tracing-enabled  │
 │ c. [tracing] otlp_endpoint    │                  │   true|false                        │
 │    or OTEL_EXPORTER... env    │                  │ if allow_runtime_changes = true     │
 └───────────────────────────────┘                  └─────────────────────────────────────┘

Deploy-time configuration

(a) A traces backend. Anything that accepts OTLP traces: e.g. Tempo, Jaeger, Grafana Cloud, Datadog, Elastic APM or another OTEL collector acting as a gateway.

(b) A collector to receive OTLP from nico-api. nico-api should send to a collector, not straight to the backend - the collector is where you do sampling, batching, attribute normalization and (importantly) TLS for anything leaving the cluster. There are two common ways to give nico-api a collector to talk to:

Option A - a shared collector (Deployment or DaemonSet) that many workloads send to. A minimal otel-collector traces pipeline:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }   # nico-api connects here
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  tail_sampling:              # optional but recommended; keeps trace volume sane
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: probabilistic-baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  batch/traces:
    send_batch_size: 1024     # keep batches small if the backend is Tempo (gRPC msg-size limits)
    send_batch_max_size: 2048
exporters:
  otlp/traces:
    endpoint: <backend-host>:4317   # Tempo / Jaeger / Grafana Cloud / Datadog / Elastic OTLP
    tls: { insecure: true }         # in-cluster plaintext; set real TLS/mTLS per backend
    retry_on_failure: { enabled: false }   # best-effort; don't queue traces if backend is down
service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch/traces]
      exporters:  [otlp/traces]

With Option A, nico-api’s endpoint is the collector’s in-cluster Service, e.g. http://otel-collector.observability.svc.cluster.local:4317.

Option B - a per-pod sidecar collector injected by the OpenTelemetry Operator. If your cluster runs the OpenTelemetry Operator, you can have it inject a collector container into the nico-api pod via a pod annotation. nico-api then talks to the collector over localhost (same pod, same network namespace)

The annotation value follows the form <namespace>/<collector-name>:

# nico-api pod template
metadata:
  annotations:
    sidecar.opentelemetry.io/inject: "observability/otel-sidecar"
spec:
  template:
    spec:
      containers:
        - name: nico-api
          env:
            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: http://localhost:4317   # overrides [tracing] otlp_endpoint

(c) Point nico-api at the collector. nico-api builds its OTLP span exporter only if an endpoint is configured at startup. If no endpoint is configured, no tracing layer is constructed at all and nothing is ever emitted - regardless of the enabled flag.

Preferred config-file form:

[tracing]
# Option A (shared collector): the collector's Service
otlp_endpoint = "http://otel-collector.observability.svc.cluster.local:4317"
# Option B (injected sidecar): the in-pod collector on localhost
# otlp_endpoint = "http://localhost:4317"

The deployment environment variable form is still supported and takes precedence over the TOML endpoint:

# nico-api container env (e.g. via the nico-api Helm values)
env:
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4317

Notes:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is the only trace-related setting nico-api reads from the environment. Other standard OTEL env vars are ignored.
• The endpoint must be a plaintext gRPC target (http://…, h2c); 4317 is the default OTLP/gRPC port. Do not point it at a 4318 HTTP receiver and do not use https://.
• Configuring only the endpoint puts the plumbing in place but does not start emission on its own. enabled must also be true.

Enable / Disable Policy

With the endpoint configured, emission is controlled by [tracing] enabled, which defaults off:

[tracing]
otlp_endpoint = "http://otel-collector.observability.svc.cluster.local:4317"
enabled = true
allow_runtime_changes = true  # default; permits nico-admin-cli set tracing-enabled

When allow_runtime_changes = true, toggle tracing live without a restart:

# start capturing traces (e.g. while reproducing an issue)
nico-admin-cli set tracing-enabled true
# stop capturing, turn it back off when done
nico-admin-cli set tracing-enabled false

Under the hood this sets the dynamic config ConfigSetting::TracingEnabled, which flips the in-process tracing_enabled flag that CarbideSpanSampler reads. If allow_runtime_changes = false, the SetDynamicConfig call is rejected with PermissionDenied; the startup value from [tracing] enabled remains authoritative until nico-api restarts with a new config.

Leaving tracing off in steady state is the intended operating mode. If you need startup-only control, set allow_runtime_changes = false and change [tracing] enabled through the config file plus a pod roll.

Do I need to restart nico-api?

It depends on which part you are changing:

Why: [tracing] otlp_endpoint, [tracing] enabled, [tracing] allow_runtime_changes, and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT are read at process startup (crates/api-core/src/logging/setup.rs). If no endpoint was configured when nico-api started, the OTLP exporter and tracing layer were never constructed and there is no way to add them at runtime. The runtime switch, when allowed, only flips an in-process flag and never needs a restart.

Recommendation: set [tracing] otlp_endpoint at deploy time and leave it in place permanently - the plumbing is cheap while tracing is toggled off. Keep enabled = false and allow_runtime_changes = true for debug-on-demand environments, or set allow_runtime_changes = false when the config file should be the only control plane for tracing.

Verifying it works

1. [tracing] otlp_endpoint or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set on nico-api and points at the collector’s gRPC endpoint.
2. The collector has a traces pipeline and its logs show the OTLP receiver listening on 4317.
3. [tracing] enabled = true is configured, or nico-admin-cli set tracing-enabled true has been run while tracing.allow_runtime_changes = true.
4. Exercise a traced operation (e.g. a machine power/firmware action), then look in your backend for spans with service.name = carbide-api.
5. Watch carbide_api_tracing_spans_open to confirm spans are being opened.

3. Downsides and overhead

Tracing has real cost, which is the reason it defaults off. The cost depends on which of three states nico-api is in:

When tracing is ON

This is the expensive mode the dev team warns about:

• Because a span’s in-process children inherit its sampling decision, a sampled root span pulls in its entire child subtree (the component-manager, controller, and DB spans beneath it). A single traced operation can therefore produce many spans.
• Costs land in several places: extra CPU and memory on nico-api, added latency on instrumented hot paths, network egress to the collector and storage in the backend.
• Mitigate with tail_sampling at the collector (keep errors/slow traces, sample the rest) and - most importantly - only enable it during an active investigation, then turn it back off.

When the endpoint is set but tracing is OFF

This is the common steady state if you follow the recommendation to leave the endpoint configured with [tracing] enabled = false, or after disabling tracing dynamically. The overhead here is near-zero but not exactly zero:

• At startup, because the endpoint is set, nico-api builds the OTLP exporter, a tracer provider with a batch span processor and installs the OpenTelemetry tracing layer into its subscriber stack. That layer stays present.
• Per span, the layer is invoked on each (non-tokio) instrumented span and does a little bookkeeping/allocation before the sampler returns “drop”. A background batch task exists but idles.
• What does not happen: no span recording, no attribute serialization, no batches to flush, no network or gRPC export. There is no I/O.
• Net: a small, roughly constant per-span CPU cost - negligible next to the “on” mode, but not the literal zero you get with the endpoint unset.

Practical guidance

• Leave [tracing] otlp_endpoint configured and keep tracing off in steady state - cheap and avoids a pod roll when you need traces.
• Treat “on” as a temporary debugging state. Turn it off when done; watch carbide_api_tracing_spans_open and nico-api CPU/latency while it is on.

4. How traces are sent (transport & security)

• nico-api speaks OTLP/gRPC only (no OTLP/HTTP).
• nico-api cannot originate TLS or mTLS for traces. The endpoint must be plaintext
• Therefore keep the nico-api → collector hop local (in-cluster Service, or the in-pod sidecar) and make the collector the TLS boundary for anything leaving the cluster.
• Traces are push-based: nico-api connects out to the collector. There is no scrape/discovery annotation involved for traces

5. Troubleshooting

6. DPU workload tracing

DPU workloads (such as ovnkube-node) can emit OpenTelemetry tracing spans through the DPU’s otelcol-contrib collector. The collector provides a localhost-only OTLP/gRPC receiver that forwards spans to the site-level OpenTelemetry receiver using the DPU’s mTLS credentials.

Endpoint configuration

Workload configuration

Configure your workload’s OpenTelemetry exporter to send spans to the local collector:

# Environment variables (standard OTLP configuration)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

For Kubernetes workloads running on DPUs managed by DPF:

# Pod spec environment variables
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://127.0.0.1:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"

Security

• The OTLP receiver binds only to loopback (127.0.0.1), preventing access from outside the node
• Loopback does not authenticate callers - any process in the host network namespace can send spans
• Workloads do not need access to DPU mTLS credentials
• The collector authenticates to the site-level receiver using existing mTLS configuration

Resource attributes

Spans exported through this pipeline include:

7. NICo REST API service tracing

NICo REST API services share one OpenTelemetry bootstrap, rest-api/common/pkg/otel, which each service runs once at startup. It reads the standard OTEL_* environment variables, so changing any setting needs a pod restart. There is no runtime toggle like nico-api’s tracing-enabled.

Which REST services export

An OTLP endpoint variable is OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT with a non-empty value. No Helm chart or Kustomize base sets one, so nothing is exported until you add it.

A service that does not export still reads an inbound traceparent and tracestate, and forwards them on its own HTTP, gRPC, and Temporal calls. So a hop that does not export leaves a gap in the trace rather than splitting it.

What each service traces

Spans are recorded only while a service exports. Apart from the database query hook, the instrumentation stays installed either way, which is what carries trace context through a service that does not export.

A database query span records the SQL statement with ? placeholders, so bound parameter values are never exported.

Configuration

nico-rest-api and nico-rest-workflow read two tracing keys from their config file:

Every REST service reads these environment variables:

An unknown OTEL_PROPAGATORS name is a bootstrap error. Once a service exports, so are an unknown protocol and an OTEL_BSP_* value outside its range. nico-rest-api and nico-rest-workflow exit with failed to initialize tracing. The other services log the error and run without tracing or trace propagation.

Trace context propagation

The API’s Echo middleware, the Temporal interceptor, and the gRPC client and server handlers stay installed whether or not a service exports. With OTEL_PROPAGATORS=none, a service that does not export drops them. A service that exports keeps recording its own spans, but neither reads nor sends trace context.

The API reads OpenTracing ot-tracer-* headers only when OTEL_PROPAGATORS includes ottrace, for example tracecontext,baggage,ottrace.

Baggage also crosses Temporal, where the Temporal SDK writes it into workflow headers. Those headers are kept in workflow history, so treat baggage as durable and keep sensitive values out of it. OTEL_PROPAGATORS=tracecontext propagates trace context without baggage. Changing the propagators while workflows are open is safe: a workflow header the new setting cannot read starts a new trace instead of failing the workflow task.

Finding a request’s trace

When nico-rest-api exports, it returns the trace ID in the X-Nico-Trace-Id response header on every request except /healthz and /readyz. When it does not export, the header only repeats the trace ID the caller sent.

curl -sS -D - -o /dev/null -H "Authorization: Bearer $TOKEN" \
  "https://<api-host>/v2/org/<org>/nico/site" | grep -i x-nico-trace-id

Helm and Kustomize

The nico-rest-api, nico-rest-workflow, nico-rest-cert-manager, and nico-rest-site-manager charts take an extraEnv map of names to values and render it into the container environment. The workflow chart also takes cloudWorker.extraEnv and siteWorker.extraEnv, merged over the shared map with the worker’s keys winning. extraEnv cannot override a variable the chart sets itself, which is CONFIG_FILE_PATH and, on the workers, TEMPORAL_NAMESPACE and TEMPORAL_QUEUE. Rendering fails if it tries.

nico-rest-api:
  config:
    tracing:
      enabled: true
      serviceName: nico-rest-api
  extraEnv:
    OTEL_EXPORTER_OTLP_PROTOCOL: grpc
    OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4317
    OTEL_RESOURCE_ATTRIBUTES: service.namespace=nico-rest,deployment.environment=prod
nico-rest-workflow:
  config:
    tracing:
      enabled: true
      serviceName: nico-rest-workflow
  extraEnv:
    OTEL_EXPORTER_OTLP_PROTOCOL: grpc
    OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4317
    OTEL_RESOURCE_ATTRIBUTES: service.namespace=nico-rest,deployment.environment=prod
  cloudWorker:
    extraEnv:
      OTEL_SERVICE_NAME: nico-rest-cloud-worker
  siteWorker:
    extraEnv:
      OTEL_SERVICE_NAME: nico-rest-site-worker
nico-rest-cert-manager:
  extraEnv:
    OTEL_EXPORTER_OTLP_PROTOCOL: grpc
    OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4317
nico-rest-site-manager:
  extraEnv:
    OTEL_EXPORTER_OTLP_PROTOCOL: grpc
    OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4317

The other services take the same variables through their own charts or manifests:

• nico-rest-site-agent takes them in its envConfig map.
• nico-flow takes a list of EnvVar entries in extraEnv.flow.
• nico-ipam, nico-nvswitch-manager, and nico-powershelf-manager read them from their container environment.

The Kustomize bases in rest-api/deploy/kustomize/base/ set tracing.enabled: false in the API and workflow config maps, and add no OTEL_* variables. To trace a Kustomize deployment, set tracing.enabled: true in an overlay and add the variables to each workload’s container env.

Cost

With the default sampler, a service that exports records every request it handles, including a span for each database query. OTEL_TRACES_SAMPLER=parentbased_traceidratio keeps a fraction of new traces instead. A slow or unreachable collector costs spans rather than request latency, since a full queue drops new spans. On shutdown each service flushes the spans still queued.

Verifying REST tracing

1. Point every service at the same collector. Spans only join one trace when every hop exports to the same backend.
2. Check each service’s startup log. tracing enabled, OTLP tracer provider installed means it exports, and the line also shows the resolved serviceName and protocol.
3. Send an API request that starts a workflow. It should appear as one trace with the API server span, the Temporal client and worker spans, and the database spans beneath them.
4. Look a single request up by the X-Nico-Trace-Id value it returned.

Troubleshooting REST tracing

8. References

• NICo core metrics catalogue - includes carbide_api_tracing_spans_open.