Forward Pass Metrics Trace Reference

Configuration, topology support, file rotation, and the dynamo.fpm.trace.v1 schema
View as Markdown

Forward Pass Metrics (FPM) tracing persists finalized FPM payloads immediately before the Rust publication path sends them to the event plane. Persistence does not replace or reroute event-plane publication.

Configuration

VariableDefault when enabledDescription
DYN_FPM_TRACEunsetEnvironment form of --fpm-trace and --no-fpm-trace. Accepts 1/0, true/false, on/off, and yes/no, case-insensitively.
DYN_FPM_OUTPUT_PATH/tmp/dynamo-fpmOutput prefix. Files use <prefix>.<producer-id>.<index>.jsonl.gz.
DYN_FPM_MODEsampledsampled keeps the latest changed record per worker and data-parallel rank; full captures every valid payload.
DYN_FPM_SAMPLE_INTERVAL_MS5000Positive sampling interval. Validated in both modes and used only in sampled mode.
DYN_FPM_JSONL_GZ_ROLL_BYTES268435456Positive uncompressed-byte threshold. Dynamo rolls before the next JSONL row would exceed it.
DYN_FPM_MAX_SEGMENTS4Positive number of segments retained per producer, including the active segment.

The configuration variables do not enable tracing by themselves. An invalid value or unwritable output path disables tracing only for that worker and emits a warning. Inference and normal FPM publication continue.

DYN_FORWARDPASS_METRIC_PORT remains a separate legacy backend-generation opt-in and takes precedence when set. It cannot add a missing Dynamo relay to an unsupported topology. DYN_FPM_BENCHMARK_OUTPUT_PATH is benchmark-only and is not used for live tracing.

Supported Topologies

BackendWorker topologyTrace activation support
vLLM (python -m dynamo.vllm)Aggregated, prefill, or decode, including native multimodal workersSupported
vLLM (python -m dynamo.vllm)Embedding, multimodal encode, or headlessNot supported; Dynamo warns and does not inject the FPM scheduler
vLLM unified workerAll topologiesNot supported; the unified path does not construct an FPM relay
SGLang (python -m dynamo.sglang)Aggregated, prefill, decode, or LLM diffusion, including --enable-multimodal without a dedicated encoderSupported
SGLang (python -m dynamo.sglang)Embedding or dedicated multimodal encoderNot supported; Dynamo warns and does not auto-enable FPM
SGLang (python -m dynamo.sglang)Image diffusion, video generation, or snapshot modeNot supported
SGLang unified workerAll topologiesNot supported; the unified path does not construct an FPM relay
TensorRT-LLM and mockerDirect-publisher pathsPersistence supported; no Python relay activation is required

For vLLM, an explicit DYN_FORWARDPASS_METRIC_PORT wins; otherwise trace activation uses port 20380. SGLang uses its existing per-worker IPC endpoint.

Capture Modes

In sampled mode, Dynamo retains the newest pending payload for each (namespace, component, worker_id, dp_rank) key. At each monotonic interval it writes keys whose counter_id changed. It does not repeat an unchanged counter, and it flushes pending values during graceful shutdown.

In full mode, Dynamo writes every valid payload, including idle heartbeats. Each writer flushes a non-empty batch every second and can flush sooner when its 1 MiB buffer fills. Producer enqueueing is nonblocking in both modes; a bounded queue drops trace records instead of delaying inference.

File and Retention Semantics

The default files are:

/tmp/dynamo-fpm.<producer-id>.000000.jsonl.gz
/tmp/dynamo-fpm.<producer-id>.000001.jsonl.gz

The producer ID is the sanitized runtime connection ID. Rotation counts uncompressed JSONL bytes. A single oversized row is written intact to an otherwise empty segment. After rolling, Dynamo removes only the oldest files that exactly match that producer’s prefix. On restart, the next index is one greater than the highest matching index.

DYN_FPM_MAX_SEGMENTS applies independently to every producer, including producer IDs left by old worker instances.

Capacity Planning

The roll threshold counts uncompressed JSONL bytes; disk usage is the compressed gzip size. Estimate uncompressed daily volume with:

average JSONL row bytes * records per second * 86400

At the default five-second interval, one continuously changing rank writes about 17,280 periodic rows per day, plus a possible shutdown flush. A 600-byte row is about 10.4 MB per rank per day before compression. In full mode, the same row at 10 forward passes per second is about 518 MB per rank per day.

Record Schema

Each line uses the shared gzip JSONL envelope:

1{
2  "timestamp": 1250,
3  "event": {
4    "schema": "dynamo.fpm.trace.v1",
5    "source": {
6      "namespace": "default",
7      "component": "backend",
8      "producer_id": "4192"
9    },
10    "capture_mode": "sampled",
11    "observed_at_unix_ms": 1782777601250,
12    "fpm": {
13      "version": 1,
14      "worker_id": "4192",
15      "dp_rank": 0,
16      "counter_id": 42,
17      "wall_time": 0.025,
18      "scheduled_requests": {
19        "num_prefill_requests": 2,
20        "sum_prefill_tokens": 256,
21        "sum_prefill_kv_tokens": 64,
22        "num_decode_requests": 3,
23        "sum_decode_kv_tokens": 1024
24      },
25      "queued_requests": {
26        "num_prefill_requests": 1,
27        "sum_prefill_tokens": 128,
28        "num_decode_requests": 0,
29        "sum_decode_kv_tokens": 0
30      }
31    }
32  }
33}

observed_at_unix_ms is the absolute observation time. The outer timestamp is milliseconds since the writer started. The nested fpm object is the canonical payload. A counter_id gap can result from sampling, local queue pressure, an upstream ZMQ drop, a crash, or node loss.