> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Forward Pass Metrics Trace Reference Forward Pass Metrics (FPM) tracing persists finalized FPM payloads immediately before the Rust publication path sends them to the event plane. Persistence does not replace or reroute event-plane publication. ## Configuration | Variable | Default when enabled | Description | | ----------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `DYN_FPM_TRACE` | unset | Environment form of `--fpm-trace` and `--no-fpm-trace`. Accepts `1`/`0`, `true`/`false`, `on`/`off`, and `yes`/`no`, case-insensitively. | | `DYN_FPM_OUTPUT_PATH` | `/tmp/dynamo-fpm` | Output prefix. Files use `...jsonl.gz`. | | `DYN_FPM_MODE` | `sampled` | `sampled` keeps the latest changed record per worker and data-parallel rank; `full` captures every valid payload. | | `DYN_FPM_SAMPLE_INTERVAL_MS` | `5000` | Positive sampling interval. Validated in both modes and used only in `sampled` mode. | | `DYN_FPM_JSONL_GZ_ROLL_BYTES` | `268435456` | Positive uncompressed-byte threshold. Dynamo rolls before the next JSONL row would exceed it. | | `DYN_FPM_MAX_SEGMENTS` | `4` | Positive number of segments retained per producer, including the active segment. | The configuration variables do not enable tracing by themselves. An invalid value or unwritable output path disables tracing only for that worker and emits a warning. Inference and normal FPM publication continue. `DYN_FORWARDPASS_METRIC_PORT` remains a separate legacy backend-generation opt-in and takes precedence when set. It cannot add a missing Dynamo relay to an unsupported topology. `DYN_FPM_BENCHMARK_OUTPUT_PATH` is benchmark-only and is not used for live tracing. ## Supported Topologies | Backend | Worker topology | Trace activation support | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | vLLM (`python -m dynamo.vllm`) | Aggregated, prefill, or decode, including native multimodal workers | Supported | | vLLM (`python -m dynamo.vllm`) | Embedding, multimodal encode, or headless | Not supported; Dynamo warns and does not inject the FPM scheduler | | vLLM unified worker | All topologies | Not supported; the unified path does not construct an FPM relay | | SGLang (`python -m dynamo.sglang`) | Aggregated, prefill, decode, or LLM diffusion, including `--enable-multimodal` without a dedicated encoder | Supported | | SGLang (`python -m dynamo.sglang`) | Embedding or dedicated multimodal encoder | Not supported; Dynamo warns and does not auto-enable FPM | | SGLang (`python -m dynamo.sglang`) | Image diffusion, video generation, or snapshot mode | Not supported | | SGLang unified worker | All topologies | Not supported; the unified path does not construct an FPM relay | | TensorRT-LLM and mocker | Direct-publisher paths | Persistence supported; no Python relay activation is required | For vLLM, an explicit `DYN_FORWARDPASS_METRIC_PORT` wins; otherwise trace activation uses port `20380`. SGLang uses its existing per-worker IPC endpoint. ## Capture Modes In `sampled` mode, Dynamo retains the newest pending payload for each `(namespace, component, worker_id, dp_rank)` key. At each monotonic interval it writes keys whose `counter_id` changed. It does not repeat an unchanged counter, and it flushes pending values during graceful shutdown. In `full` mode, Dynamo writes every valid payload, including idle heartbeats. Each writer flushes a non-empty batch every second and can flush sooner when its 1 MiB buffer fills. Producer enqueueing is nonblocking in both modes; a bounded queue drops trace records instead of delaying inference. ## File and Retention Semantics The default files are: ```text /tmp/dynamo-fpm..000000.jsonl.gz /tmp/dynamo-fpm..000001.jsonl.gz ``` The producer ID is the sanitized runtime connection ID. Rotation counts uncompressed JSONL bytes. A single oversized row is written intact to an otherwise empty segment. After rolling, Dynamo removes only the oldest files that exactly match that producer's prefix. On restart, the next index is one greater than the highest matching index. `DYN_FPM_MAX_SEGMENTS` applies independently to every producer, including producer IDs left by old worker instances. ## Capacity Planning The roll threshold counts uncompressed JSONL bytes; disk usage is the compressed gzip size. Estimate uncompressed daily volume with: ```text average JSONL row bytes * records per second * 86400 ``` At the default five-second interval, one continuously changing rank writes about 17,280 periodic rows per day, plus a possible shutdown flush. A 600-byte row is about 10.4 MB per rank per day before compression. In `full` mode, the same row at 10 forward passes per second is about 518 MB per rank per day. ## Record Schema Each line uses the shared gzip JSONL envelope: ```json { "timestamp": 1250, "event": { "schema": "dynamo.fpm.trace.v1", "source": { "namespace": "default", "component": "backend", "producer_id": "4192" }, "capture_mode": "sampled", "observed_at_unix_ms": 1782777601250, "fpm": { "version": 1, "worker_id": "4192", "dp_rank": 0, "counter_id": 42, "wall_time": 0.025, "scheduled_requests": { "num_prefill_requests": 2, "sum_prefill_tokens": 256, "sum_prefill_kv_tokens": 64, "num_decode_requests": 3, "sum_decode_kv_tokens": 1024 }, "queued_requests": { "num_prefill_requests": 1, "sum_prefill_tokens": 128, "num_decode_requests": 0, "sum_decode_kv_tokens": 0 } } } } ``` `observed_at_unix_ms` is the absolute observation time. The outer `timestamp` is milliseconds since the writer started. The nested `fpm` object is the canonical payload. A `counter_id` gap can result from sampling, local queue pressure, an upstream ZMQ drop, a crash, or node loss. ## Related * [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#capture-forward-pass-metrics) * [Observability Architecture](/dynamo/dev/knowledge-base/design-documents/observability-architecture#forward-pass-metrics-persistence) * [SGLang Observability](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/observability#forward-pass-metrics-fpm)