> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Logging Reference

Dynamo writes readable text or newline-delimited JSON (JSONL) to standard error. With OTLP export
enabled, it also sends structured log records to the configured OpenTelemetry Collector. For a local
Loki workflow, see [Observe a Local Deployment](/dynamo/cli/operations/observability#inspect-traces-and-exported-logs).

## Output Formats

Set `DYN_LOGGING_CONSOLE_FORMAT=readable` for readable text (the default):

```text
2025-09-02T15:50:01.770028Z INFO main.init: VllmWorker for Qwen/Qwen3-0.6B has been initialized
```

Set `DYN_LOGGING_CONSOLE_FORMAT=jsonl` for structured output:

```json
{"time":"2025-10-31T21:06:45.397194Z","level":"DEBUG","target":"dynamo_runtime::pipeline::network::tcp::server","message":"Registering new TcpStream","span_name":"http-request","trace_id":"80196f3e3a6fdf06d23bb9ada3788518","span_id":"f7e487a9d2a6bf38","x_request_id":"test-trace-001"}
```

`DYN_LOGGING_CONSOLE_FORMAT` accepts only `readable` or `jsonl`, ignoring surrounding whitespace.
When it is unset or blank, Dynamo falls back to the legacy `DYN_LOGGING_JSONL` switch. The legacy
switch also continues to enable local trace-context propagation for backward compatibility, even
when the new setting explicitly selects readable output.

Fields vary by event and active span. Common fields include:

| Field                      | Meaning                                             |
| -------------------------- | --------------------------------------------------- |
| `time`                     | Event timestamp                                     |
| `level`                    | Event severity                                      |
| `target`                   | Rust or Python logging target                       |
| `message`                  | Event message                                       |
| `trace_id`                 | Distributed trace identifier                        |
| `span_id`                  | Current span identifier                             |
| `span_name`                | Current span name                                   |
| `x_request_id`             | Caller-provided request identifier, when present    |
| `component` and `endpoint` | Dynamo component and endpoint context, when present |

`DYN_LOGGING_SPAN_EVENTS=true` adds span creation and exit events. This is useful for detailed
debugging but increases log volume.

## Log Levels

`DYN_LOG` accepts standard tracing filter syntax, including per-target overrides.

| Level   | Intended use                                    |
| ------- | ----------------------------------------------- |
| `ERROR` | Unrecoverable failures or resource exhaustion   |
| `WARN`  | Degraded or unexpected conditions               |
| `INFO`  | Startup, shutdown, and major operational events |
| `DEBUG` | Request flow and diagnostic state               |
| `TRACE` | Low-level implementation details                |

For example, keep the default at `info` and enable trace output for the system-status server:

```bash
export DYN_LOG='info,dynamo_runtime::system_status_server=trace'
```

## OTLP Export

`OTEL_EXPORT_ENABLED=true` is the master switch for runtime trace and log export. Configure a
generic endpoint for both signals:

```bash
export DYN_LOGGING_CONSOLE_FORMAT=readable
export OTEL_EXPORT_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
```

Console formatting does not affect OTLP export; use `jsonl` instead if the local stderr stream also
needs machine-readable records.

Use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` or `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` for signal-specific
destinations. Signal-specific endpoints are used as-is. For a generic `http/protobuf` endpoint,
Dynamo appends `/v1/traces` or `/v1/logs`.

> **Warning**
>
> `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` does not fall back to
> `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`. If only the traces endpoint is set, logs use the generic
> endpoint or the protocol default and can be silently dropped when no collector is listening there.

Set these variables on every process whose telemetry you want to export. Use
`OTEL_TRACES_SAMPLE_RATIO` to apply head sampling to traces; leaving it unset exports every trace.

## Request Payload Export

Request payload logging uses the request trace pipeline rather than the runtime log stream. Enable
the `request_payload` record and include the `otel` sink:

```bash
export DYN_REQUEST_TRACE_RECORDS=request_payload
export DYN_REQUEST_TRACE_SINKS=otel
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4317
export OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=grpc
```

Each request trace row maps to one OTLP `LogRecord` in scope `dynamo.request_trace`. Combining
sinks, such as `file,otel`, writes the same row to every selected destination. `stderr` alone never
exports over OTLP.

Canceled and failed chat requests retain the request payload and omit the response. Set
`DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST` to capture selected request headers. Captured values
are unredacted, so do not allowlist credentials.

For limits and record semantics, see [Request Trace Reference](/dynamo/reference/observability/request-traces).

## Request Correlation

When a request includes an `x-request-id` header, Dynamo propagates it through the distributed trace
context and records it as `x_request_id` on participating spans and log events. JSONL logs also carry
`trace_id` and `span_id`, allowing Loki results to link to the corresponding Tempo trace.

For the span hierarchy and propagation model, see
[Observability Architecture](/dynamo/knowledge-base/concepts/observability-architecture#trace-and-log-correlation).

## Backend Engine Controls

`DYN_LOG` controls Dynamo. Inference engines retain separate controls:

| Backend      | Control                               | Notes                                                                            |
| ------------ | ------------------------------------- | -------------------------------------------------------------------------------- |
| vLLM         | `VLLM_LOGGING_LEVEL`                  | Standard vLLM level such as `INFO` or `DEBUG`                                    |
| TensorRT-LLM | `TLLM_LOG_LEVEL`                      | Set before process startup; read during import                                   |
| SGLang       | `DYN_SKIP_SGLANG_LOG_FORMATTING=true` | Returns formatting and level control to SGLang, including its `--log-level` flag |

See [Environment Variables](/dynamo/reference/observability/environment-variables#logging) for defaults and the complete logging
and OTLP variable catalog.