Is this page helpful?

Logging and Observability#

NIM LLM and VLM provides structured logging, Prometheus-compatible metrics, and distributed tracing support to integrate with standard observability stacks.

Structured Logging#

NIM outputs logs to stderr. The following environment variables control log behavior:

Variable	Description	Default
`NIM_LOG_LEVEL`	Log verbosity. Accepts standard Python logging levels: `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`.	`WARNING`
`NIM_JSONL_LOGGING`	Emit logs as JSON Lines for machine parsing. Set to `true` to enable.	`false`

Set the Log Level#

To set the log level, complete the following step. Set NIM_LOG_LEVEL when starting the container:

docker run --gpus all -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_LOG_LEVEL=INFO \
  <image>

Default log output uses a human-readable format:

INFO 2026-03-03 12:00:10.079 start_server.py:42] Server starting on port 8000

NIM_LOG_LEVEL controls both NIM and the inference backend. To override the backend log level independently, set the backend-specific variable for your image in addition to NIM_LOG_LEVEL:

Backend	Log-Level Override
vLLM	`VLLM_LOGGING_LEVEL`
SGLang	`SGLANG_LOG_LEVEL`

NIM applies the override when it configures the backend. On SGLang, direct --log-level arguments are ignored because NIM manages logging.

Enable JSON Lines Output#

When NIM_JSONL_LOGGING=true, all log output from both NIM and the inference backend (vLLM or SGLang) is formatted as one JSON object per line using a unified format. This is recommended for production deployments where logs are ingested by a collector such as Fluentd, Logstash, or CloudWatch.

To enable JSON Lines output, complete the following step:

docker run --gpus all -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_JSONL_LOGGING=true \
  <image>

Example JSON Lines output:

{"level":"INFO","time":"2026-03-03 12:00:10","file_name":"start_server.py","line_number":42,"message":"Server starting on port 8000"}

NIM and backend logs share the same JSON schema, so a single log parser configuration handles all container output.

Metrics#

NIM exposes Prometheus-compatible metrics at the /v1/metrics endpoint.

To retrieve the metrics, run the following command:

curl -s http://localhost:8000/v1/metrics

Design Principle#

NIM passes through the inference backend’s native Prometheus metrics without modification. The metrics at /v1/metrics are the bundled engine’s native metrics, covering request latency, throughput, queue depth, token counts, and GPU utilization. NIM does not wrap, rename, or abstract these metrics, so dashboards and alerting rules built for the matching engine work unchanged. They do not transfer between engines. The metric family depends on the inference engine in your image:

Backend	Metric Prefix
vLLM	`vllm:`
SGLang	`sglang:`

NIM enables the metrics exporter for the engine in the image.

For the full list of available metrics and their descriptions, refer to the vLLM production metrics documentation (vLLM images) or the SGLang production metrics documentation (SGLang images).

Prometheus Scrape Configuration#

To scrape NIM metrics with Prometheus, add the following job to your prometheus.yml:

scrape_configs:
  - job_name: "nim-llm"
    scrape_interval: 15s
    metrics_path: "/v1/metrics"
    static_configs:
      - targets: ["<nim-host>:8000"]

Distributed Tracing#

NIM supports request correlation and distributed tracing through standard HTTP headers. The proxy layer forwards tracing headers from clients to the inference backend on the following inference endpoints:

/v1/chat/completions
/v1/completions
/v1/embeddings
/v1/responses

X-Request-Id#

Include an X-Request-Id header in your request to tag it with a correlation identifier. NIM echoes it on the response and forwards it to the backend. If the header is not present, NIM does not add one and the response does not carry an X-Request-Id.

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Request-Id: req-abc-123" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 64
  }'

On vLLM, the engine uses the header value as its internal request ID, so it appears in engine log entries. On SGLang, use the echoed response header for correlation. SGLang’s internal request ID comes from the optional rid field in the request body.

vLLM images only. To stamp a custom request header into logs (for example a caller-supplied call ID), pass an ASGI middleware class as the direct container argument --middleware <import.path>. Use NIM_PASSTHROUGH_ARGS only when direct container arguments are unavailable, as described in Advanced Configuration. NIM keeps redirect and request-id outermost and installs your class innermost, so it can read any caller-supplied X-Request-Id after NIM records it. Prefer X-Request-Id when a custom class is not required. --middleware is not currently supported in NIM with Dynamo deployments (NIM_DYNAMO_SINGLE / NIM_DYNAMO_WORKER / NIM_DYNAMO_FRONTEND, including DynamoGraphDeployment): the flag is stripped before dynamo.vllm starts, and the frontend cannot install operator ASGI classes.

W3C Traceparent#

NIM forwards the W3C Trace Context traceparent header to the inference backend. If your application uses OpenTelemetry or another W3C-compatible tracing system, include the traceparent header to propagate trace context through NIM:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 64
  }'

When OpenTelemetry tracing is enabled on the backend, this allows inference spans to appear in your existing distributed traces alongside upstream and downstream services.

NIM echoes the traceparent header back on the response only when the caller supplied it on the request. Requests without an inbound traceparent receive a response without one.

Note

The proxy layer always forwards the traceparent header to the backend. However, the backend creates OpenTelemetry spans from it only when tracing is enabled on the inference engine. Enable engine tracing as described for your image in the following table.

Backend	Tracing Arguments
vLLM	`--otlp-traces-endpoint <endpoint>`
SGLang	`--enable-trace --otlp-traces-endpoint <ip>:<port>`

For backend setup, refer to the vLLM OpenTelemetry documentation or SGLang server arguments. Forwarding the header alone does not enable backend tracing.

To export inference request spans, pass the tracing arguments for your backend through NIM_PASSTHROUGH_ARGS:

docker run --gpus=all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_PASSTHROUGH_ARGS="--otlp-traces-endpoint http://<collector-host>:4317" \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  <nim-image>

Setting NIM_PASSTHROUGH_ARGS replaces the value that the container sets by default. If the model’s container already sets it, include those arguments as well. For example, the NVIDIA-Nemotron-3.5-Super-VL-120B-A12B container sets --chat-template /opt/nim/etc/chat_templates/nemotron-3.5-super-vl-reasoning-effort.jinja, so pass NIM_PASSTHROUGH_ARGS="--otlp-traces-endpoint http://<collector-host>:4317 --chat-template /opt/nim/etc/chat_templates/nemotron-3.5-super-vl-reasoning-effort.jinja".

Supported Tracing Headers#

The following table describes the tracing headers that NIM forwards on inference endpoints:

Header	Standard	Description
`X-Request-Id`	Custom	Request correlation identifier. Echoed on the response and forwarded to the backend. On vLLM images, the engine adopts it as the internal request ID for logging and diagnostics.
`traceparent`	W3C Trace Context	Propagates distributed trace context to the inference backend. Requires OpenTelemetry to be enabled on the backend to generate spans.

Request correlation does not imply that an endpoint produces inference spans. Backend tracing must be enabled to collect those spans.