> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Metrics Catalog Dynamo exposes metrics in Prometheus exposition format at the `/metrics` HTTP endpoint. All Dynamo-generated metrics use the `dynamo_*` prefix and carry the hierarchy labels documented in [Metric Labels](/dynamo/dev/reference/observability/metric-labels). This page is the field catalog; for setup and dashboards see the [Metrics guide](/dynamo/dev/cli/operations/observability#view-metrics-and-dashboards). ## Metric types The `type` on each field below is a [Prometheus metric type](https://prometheus.io/docs/concepts/metric_types/): * **counter** — a cumulative value that only increases (or resets to zero on restart), e.g. total requests. * **gauge** — a single value that can go up or down, e.g. in-flight requests. * **histogram** — samples observations into configurable buckets and exposes `_bucket`, `_sum`, and `_count` series, e.g. request duration. Labeled histograms/counters/gauges register a metric *family*, not one series. Labeled metrics (`HistogramVec`, `CounterVec`, `GaugeVec`) register a metric *family*, not individual time series. A series for a given label combination appears at `/metrics` only after the first request that populates it. A freshly-started process therefore shows empty families until the first matching request — this is expected Prometheus client behavior, not a missing metric. ## Where each family is exposed | Family | Emitted by | Scrape endpoint | | -------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------- | | `dynamo_frontend_*` | HTTP frontend (`python -m dynamo.frontend`) | `/metrics` on `DYN_HTTP_PORT` (default 8000) | | `dynamo_component_*` | Backend workers, standalone router | `/metrics` on `DYN_SYSTEM_PORT` (must be set) | | `dynamo_router_overhead_*` | Frontend with KV routing | `/metrics` on `DYN_HTTP_PORT` | | `dynamo_epp_*` | Endpoint picker (Gateway API Inference Extension routing) | `/metrics` on `DYN_EPP_METRICS_PORT` (default 9090) | | `dynamo_operator_*` | Kubernetes operator | See [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics) | Backend workers expose `dynamo_component_*` only when `DYN_SYSTEM_PORT` is set (disabled by default; the Kubernetes operator typically sets `9090`, local examples use `8081`). See [Environment Variables](/dynamo/dev/reference/observability/environment-variables#system-and-metrics). Engine pass-through metrics (`vllm:*`, `sglang:*`, `trtllm_*`) are emitted by the backend engines themselves and are cataloged separately — see [Metrics Comparison](/dynamo/dev/reference/observability/metrics-comparison). NIXL transfer metrics are exposed on their own port and owned by the [upstream NIXL project](https://github.com/ai-dynamo/nixl/blob/main/docs/telemetry.md). ## Frontend metrics Emitted by the HTTP frontend at `/metrics` on port 8000 by default. Most carry a `model` label. **`dynamo_frontend_active_requests`** `gauge` Requests currently being handled by the frontend, from HTTP handler entry until the response stream completes. The top-level in-flight count with no stage breakdown. --- **`dynamo_frontend_stage_requests`** `gauge` Requests currently in a given frontend pipeline stage. Labeled by `stage` and `phase` — see [Stage values](/dynamo/dev/reference/observability/metric-labels#stage-values) and [Phase values](/dynamo/dev/reference/observability/metric-labels#phase-values). --- **`dynamo_frontend_inflight_requests`** `gauge` — deprecated Inflight requests. Kept for backward compatibility; prefer `dynamo_frontend_active_requests`, which has identical semantics with a clearer name. --- **`dynamo_frontend_queued_requests`** `gauge` — deprecated Requests in the HTTP processing queue. Kept for backward compatibility; the "waiting for first token" window is now the sum of `dynamo_frontend_stage_requests` across the `preprocess`, `route`, and `dispatch` stages. --- **`dynamo_frontend_disconnected_clients`** `gauge` Number of disconnected clients. --- **`dynamo_frontend_input_sequence_tokens`** `histogram` Input sequence length in tokens. --- **`dynamo_frontend_cached_tokens`** `histogram` Cached tokens (prefix cache hits) per request. --- **`dynamo_frontend_tokenizer_cache_hits_total`** `counter` L1 tokenizer prefix-cache hits across all models. One outcome is recorded per encode operation or batch item. --- **`dynamo_frontend_tokenizer_cache_misses_total`** `counter` L1 tokenizer prefix-cache misses across all models. --- **`dynamo_frontend_tokenizer_cache_cached_tokens_total`** `counter` Tokens returned from the L1 tokenizer prefix cache. Labeled by `model`. --- **`dynamo_frontend_tokenizer_cache_uncached_tokens_total`** `counter` Tokens freshly encoded after an L1 tokenizer cache lookup. Labeled by `model`. --- Use this PromQL expression to calculate the five-minute token reuse ratio by model: ```promql sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m])) / ( sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m])) + sum by (model) (rate(dynamo_frontend_tokenizer_cache_uncached_tokens_total[5m])) ) ``` The ratio is defined only when the model has an active L1 cache and observed tokens. Partial hits increment both token counters. When the cache is disabled with `DYN_TOKENIZER_CACHE=0`, the per-model series are never created, so they are absent from the scrape rather than reported as zero; detect that state with `absent()`, since a `== 0` comparison never matches a missing series. Only the literal value `0` disables the cache; values such as `false` or `off` leave it enabled. With the cache enabled, the counters remain zero when encoding fails or the tokenizer has no registered special-token boundaries. **`dynamo_frontend_inter_token_latency_seconds`** `histogram` Inter-token latency in seconds. --- **`dynamo_frontend_output_sequence_tokens`** `histogram` Output sequence length in tokens. --- **`dynamo_frontend_output_tokens_total`** `counter` Total output tokens generated. --- **`dynamo_frontend_request_duration_seconds`** `histogram` End-to-end LLM request duration in seconds. --- **`dynamo_frontend_requests_total`** `counter` Total LLM requests. --- **`dynamo_frontend_time_to_first_token_seconds`** `histogram` Time to first token in seconds. --- ### Migration Counters for [request migration](/dynamo/dev/kubernetes/fault-tolerance/request-migration), the mechanism that continues in-flight requests on a healthy worker when the original worker fails. See [Request Migration Architecture](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-migration-architecture) for the internals. **`dynamo_frontend_model_migration_total`** `counter` Total request migrations due to worker unavailability. Labeled by `model` and [`migration_type`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels) (`new_request` for an initial connection failure, `ongoing_request` for a mid-stream disconnection). --- **`dynamo_frontend_model_migration_max_seq_len_exceeded_total`** `counter` Total times migration was disabled for a request because its sequence length exceeded `--migration-max-seq-len`. A rising value may indicate the limit needs adjustment. Labeled by `model`. --- ### Cancellation and rejection Counters for requests that end early: [cancellations](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture) (the client or frontend aborts an in-flight request) and [rejections](/dynamo/dev/kubernetes/fault-tolerance/request-rejection) (the Frontend sheds load with HTTP 529 by default when all workers are busy). The worker-side counterparts are under [Component metrics](#component-metrics). **`dynamo_frontend_model_cancellation_total`** `counter` Total request cancellations detected by the frontend (client disconnect or stream close). Labeled by `model`, `endpoint` (the API route — `chat_completions`, `completions`, `embeddings`, …), and [`request_type`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels) (`unary` or `stream`). --- **`dynamo_frontend_model_rejection_total`** `counter` Total overload responses surfaced to the client as HTTP 529 by default. Incremented for worker-scoped `WorkerOverloaded` and pool-scoped `ResourceExhausted` errors. Labeled by `model` and `endpoint` (the API route). --- ### Per-worker load and timing gauges `dynamo_frontend_worker_*` gauges appear once workers register and begin serving. They live on the frontend's local registry (not component-scoped) and do **not** carry `dynamo_namespace` or `dynamo_component` labels. Frontend-only — not available on the standalone router. Labeled by `worker_id`, `dp_rank`, and `worker_type`. **`dynamo_frontend_worker_active_decode_blocks`** `gauge` Active KV cache decode blocks per worker. --- **`dynamo_frontend_worker_active_prefill_tokens`** `gauge` Active prefill tokens queued per worker. --- **`dynamo_frontend_worker_last_time_to_first_token_seconds`** `gauge` Last observed time to first token per worker, in seconds. --- **`dynamo_frontend_worker_last_input_sequence_tokens`** `gauge` Last observed input sequence length per worker. --- **`dynamo_frontend_worker_last_inter_token_latency_seconds`** `gauge` Last observed inter-token latency per worker, in seconds. --- ### Model configuration metrics `dynamo_frontend_model_*` gauges are populated from worker backend registration and all carry a `model` label. When multiple workers register the same model name, only the first instance's configuration is recorded. **`dynamo_frontend_model_total_kv_blocks`** `gauge` Total KV blocks available for a worker serving the model. --- **`dynamo_frontend_model_max_num_seqs`** `gauge` Maximum number of sequences for a worker serving the model. --- **`dynamo_frontend_model_max_num_batched_tokens`** `gauge` Maximum number of batched tokens for a worker serving the model. --- **`dynamo_frontend_model_context_length`** `gauge` Maximum context length for a worker serving the model. Sourced from the Model Deployment Card. --- **`dynamo_frontend_model_kv_cache_block_size`** `gauge` KV cache block size for a worker serving the model. Sourced from the Model Deployment Card. --- **`dynamo_frontend_model_migration_limit`** `gauge` Request migration limit for a worker serving the model. Sourced from the Model Deployment Card. --- ## Component metrics Emitted by backend workers (`python -m dynamo.vllm`, `python -m dynamo.sglang`, etc.) at `/metrics` on `DYN_SYSTEM_PORT`. All carry the hierarchy labels (`dynamo_namespace`, `dynamo_component`, `dynamo_endpoint`) — see [Runtime-injected labels](/dynamo/dev/reference/observability/metric-labels#runtime-injected-labels). **`dynamo_component_inflight_requests`** `gauge` Requests currently being processed by the component. --- **`dynamo_component_request_bytes_total`** `counter` Total bytes received in requests. --- **`dynamo_component_request_duration_seconds`** `histogram` Request processing time in seconds. --- **`dynamo_component_requests_total`** `counter` Total requests processed. --- **`dynamo_component_errors_total`** `counter` Total errors encountered while handling a request. Labeled by `error_type` — see [Component error types](/dynamo/dev/reference/observability/metric-labels#component-error-types). --- **`dynamo_component_response_bytes_total`** `counter` Total bytes sent in responses. --- **`dynamo_component_uptime_seconds`** `gauge` DistributedRuntime uptime. Updated before each Prometheus scrape on both the frontend and the system-status server. --- **`dynamo_component_cancellation_total`** `counter` Total requests cancelled by the work handler. Carries the hierarchy labels. Records cancellation *signals* received by the worker, not whether the engine actually aborted — deduplicated so a control message plus a socket close for the same request counts once. See [Request Cancellation Architecture](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture). --- The following worker-side gauges and counter appear only when a hard concurrency cap is set with `--engine-request-limit`; see [Worker-Side Request Admission](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-rejection-architecture#worker-side-request-admission). **`dynamo_rejection_request_total`** `counter` Cumulative requests rejected because the worker was at capacity (engine in-flight limit and Dynamo queue both full). --- **`dynamo_engine_request`** `gauge` Current requests being handled by the engine. --- **`dynamo_request_queue`** `gauge` Current requests queued in Dynamo, not yet in the engine. --- Specialized components expose additional families under their own prefix, e.g. `dynamo_preprocessor_*` for preprocessor components. ## Router metrics The router exposes metrics for routing decisions and overhead. Not every metric appears in every deployment — see [Availability by configuration](#availability-by-configuration). ### Router request metrics `dynamo_component_router_*` histograms and counters for aggregate request-level statistics. On the frontend, exposed at `/metrics` on the HTTP port; on the standalone router, exposed on `DYN_SYSTEM_PORT`. Populated per-request when `--router-mode kv` is active; registered with zero values in non-KV modes. **`dynamo_component_router_requests_started_total`** `counter` Requests admitted by the router scheduler. --- **`dynamo_component_router_requests_total`** `counter` Total requests processed by the router. --- **`dynamo_component_router_time_to_first_token_seconds`** `histogram` Time to first token in seconds. --- **`dynamo_component_router_inter_token_latency_seconds`** `histogram` Average inter-token latency in seconds. --- **`dynamo_component_router_input_sequence_tokens`** `histogram` Input sequence length in tokens. --- **`dynamo_component_router_output_sequence_tokens`** `histogram` Output sequence length in tokens. --- **`dynamo_component_router_kv_hit_rate`** `histogram` Predicted KV cache hit rate at routing time (0.0–1.0). --- **`dynamo_component_router_non_max_overlap_selections_total`** `counter` Admitted prefill scheduler selections routed to a worker with less KV cache overlap than another eligible worker. Pinned requests and equal-overlap ties are excluded. Labeled by `worker_type` (`prefill`). --- **`dynamo_component_router_overlap_blocks_lost`** `histogram` Difference in effective KV cache overlap between the highest-overlap eligible prefill worker and the selected worker for each non-max-overlap selection. Labeled by `worker_type` (`prefill`). --- ### Per-request routing overhead `dynamo_router_overhead_*` histograms (milliseconds) track time spent in each phase of the routing decision. Registered on the frontend port with a `router_id` label (the frontend's discovery instance ID). Created only when the frontend has DRT discovery enabled (`--router-mode kv`); absent in non-KV modes and on the standalone router. **`dynamo_router_overhead_block_hashing_ms`** `histogram` Time computing block hashes. --- **`dynamo_router_overhead_indexer_find_matches_ms`** `histogram` Time in indexer `find_matches`. --- **`dynamo_router_overhead_seq_hashing_ms`** `histogram` Time computing sequence hashes. --- **`dynamo_router_overhead_scheduling_ms`** `histogram` Time in scheduler worker selection. --- **`dynamo_router_overhead_total_ms`** `histogram` Total routing overhead per request. --- ### Router queue metrics The frontend registers these metrics when queueing is enabled by `--router-queue-threshold` or a threshold in `--router-policy-config`. They carry `model`, `worker_type`, and `policy_class`. With policy-family or cache-bucket configuration, `policy_class` is the resolved physical queue. **`dynamo_frontend_router_queue_pending_requests`** `gauge` Requests pending in the router scheduler queue. --- **`dynamo_frontend_router_queue_pending_isl_tokens`** `gauge` Raw input-sequence tokens pending in the queue. --- **`dynamo_frontend_router_queue_pending_cached_tokens`** `gauge` Cached-token estimate captured when each request enters the queue. --- **`dynamo_frontend_router_queue_backpressure_total`** `counter` Queue rejections by configured limit reason. Also labeled by `reason`. --- ### KV indexer metrics **`dynamo_component_kv_cache_events_applied`** `counter` KV cache events applied to the router's radix-tree index. Includes events applied to the device tier and lower tiers such as host-pinned memory and disk. Appears only when `--router-kv-overlap-score-credit` is greater than 0 and workers publish KV events. Labeled by `status` and `event_type` — see [Metric-specific labels](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels). --- The standalone indexer exposes the device-tier-only counter `dynamo_kvrouter_kv_cache_events_applied`; see [Standalone KV Indexer](/dynamo/dev/knowledge-base/modular-components/router/standalone-indexer). ### Availability by configuration Not all router metrics appear in every deployment. This matrix shows which groups are **registered** and **populated** in each configuration. | Metric group | Frontend + KV (agg) | Frontend + KV (disagg) | Frontend + non-KV | Standalone router | | ------------------------------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------- | --------------------------------- | | `dynamo_component_router_*` | Registered and populated | Registered and populated | Registered, **always zero** | Populated (on `DYN_SYSTEM_PORT`) | | `dynamo_router_overhead_*` | Registered and populated | Registered and populated | **Not registered** | **Not created** | | `dynamo_frontend_router_queue_*` | Registered; populated when a CLI or policy-class queue threshold is set | Registered; populated when a CLI or policy-class queue threshold is set | **Not registered** | **Not created** | | `dynamo_component_kv_cache_events_applied` | Populated when KV events received | Populated when KV events received | **Not registered** | Populated when KV events received | | `dynamo_frontend_worker_*` | Registered and populated | Registered and populated (`worker_type` = `prefill`/`decode`) | Registered and populated (`worker_type` = `decode`) | **Not created** | **Key:** * **Registered and populated** — the metric appears at `/metrics` with real values. * **Registered, always zero** — the metric appears at `/metrics` but is never incremented (useful for dashboards that expect it to exist). * **Not registered / Not created** — the metric does not appear at `/metrics` at all. ## Endpoint picker metrics Emitted by the endpoint picker (EPP) when routing through the Gateway API Inference Extension, at `/metrics` on `DYN_EPP_METRICS_PORT` (default 9090). Set the port to `0` to disable the endpoint. **`dynamo_epp_cached_tokens`** `histogram` Prompt tokens the model server served from its KV cache per request, read from `usage.prompt_tokens_details.cached_tokens` in the response body. This is the *observed* cache hit, as opposed to the overlap the KV router estimates at selection time. Buckets match `dynamo_frontend_cached_tokens` so the two can share a dashboard. The `model` label is the model the pool serves, resolved once at startup rather than from the request body. --- Requires the client to request usage accounting on streaming calls (`"stream_options": {"include_usage": true}`); without it the response carries no usage block and nothing is recorded. ## Related * [Metric Labels](/dynamo/dev/reference/observability/metric-labels) — the dimensions attached to these metrics. * [Environment Variables](/dynamo/dev/reference/observability/environment-variables) — variables that enable and configure metric emission. * [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics) — Kubernetes operator metrics catalog. * [Metrics Comparison](/dynamo/dev/reference/observability/metrics-comparison) — engine pass-through metrics (`vllm:*`, `sglang:*`, `trtllm_*`).