Logging Reference | NVIDIA Dynamo Documentation

Dynamo writes readable text or newline-delimited JSON (JSONL) to standard error. With OTLP export enabled, it also sends structured log records to the configured OpenTelemetry Collector. For a local Loki workflow, see Observe a Local Deployment.

Output Formats

Set DYN_LOGGING_JSONL=false for readable text:

2025-09-02T15:50:01.770028Z INFO main.init: VllmWorker for Qwen/Qwen3-0.6B has been initialized

Set DYN_LOGGING_JSONL=true for structured output:

1 {"time":"2025-10-31T21:06:45.397194Z","level":"DEBUG","target":"dynamo_runtime::pipeline::network::tcp::server","message":"Registering new TcpStream","span_name":"http-request","trace_id":"80196f3e3a6fdf06d23bb9ada3788518","span_id":"f7e487a9d2a6bf38","x_request_id":"test-trace-001"}

Fields vary by event and active span. Common fields include:

Field	Meaning
`time`	Event timestamp
`level`	Event severity
`target`	Rust or Python logging target
`message`	Event message
`trace_id`	Distributed trace identifier
`span_id`	Current span identifier
`span_name`	Current span name
`x_request_id`	Caller-provided request identifier, when present
`component` and `endpoint`	Dynamo component and endpoint context, when present

DYN_LOGGING_SPAN_EVENTS=true adds span creation and exit events. This is useful for detailed debugging but increases log volume.

Log Levels

DYN_LOG accepts standard tracing filter syntax, including per-target overrides.

Level	Intended use
`ERROR`	Unrecoverable failures or resource exhaustion
`WARN`	Degraded or unexpected conditions
`INFO`	Startup, shutdown, and major operational events
`DEBUG`	Request flow and diagnostic state
`TRACE`	Low-level implementation details

For example, keep the default at info and enable trace output for the system-status server:

$ export DYN_LOG='info,dynamo_runtime::system_status_server=trace'

OTLP Export

OTEL_EXPORT_ENABLED=true is the master switch for runtime trace and log export. Configure a generic endpoint for both signals:

$ export DYN_LOGGING_JSONL=true
$ export OTEL_EXPORT_ENABLED=true
$ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
$ export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Use OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_LOGS_ENDPOINT for signal-specific destinations. Signal-specific endpoints are used as-is. For a generic http/protobuf endpoint, Dynamo appends /v1/traces or /v1/logs.

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT does not fall back to OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. If only the traces endpoint is set, logs use the generic endpoint or the protocol default and can be silently dropped when no collector is listening there.

Set these variables on every process whose telemetry you want to export. Use OTEL_TRACES_SAMPLE_RATIO to apply head sampling to traces; leaving it unset exports every trace.

Request Payload Export

Request payload logging uses the request trace pipeline rather than the runtime log stream. Enable the request_payload record and include the otel sink:

$ export DYN_REQUEST_TRACE_RECORDS=request_payload
$ export DYN_REQUEST_TRACE_SINKS=otel
$ export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4317
$ export OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=grpc

Each request trace row maps to one OTLP LogRecord in scope dynamo.request_trace. Combining sinks, such as file,otel, writes the same row to every selected destination. stderr alone never exports over OTLP.

Canceled and failed chat requests retain the request payload and omit the response. Set DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST to capture selected request headers. Captured values are unredacted, so do not allowlist credentials.

For limits and record semantics, see Request Trace Reference.

Request Correlation

When a request includes an x-request-id header, Dynamo propagates it through the distributed trace context and records it as x_request_id on participating spans and log events. JSONL logs also carry trace_id and span_id, allowing Loki results to link to the corresponding Tempo trace.

For the span hierarchy and propagation model, see Observability Architecture.

Backend Engine Controls

DYN_LOG controls Dynamo. Inference engines retain separate controls:

Backend	Control	Notes
vLLM	`VLLM_LOGGING_LEVEL`	Standard vLLM level such as `INFO` or `DEBUG`
TensorRT-LLM	`TLLM_LOG_LEVEL`	Set before process startup; read during import
SGLang	`DYN_SKIP_SGLANG_LOG_FORMATTING=true`	Returns formatting and level control to SGLang, including its `--log-level` flag

See Environment Variables for defaults and the complete logging and OTLP variable catalog.