> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Metrics Catalog Dynamo exposes metrics in Prometheus exposition format at the `/metrics` HTTP endpoint. All Dynamo-generated metrics use the `dynamo_*` prefix and carry the hierarchy labels documented in [Metric Labels](/dynamo/reference/observability/metric-labels). This page is the field catalog; for setup and dashboards see the [Metrics guide](/dynamo/cli/operations/observability#view-metrics-and-dashboards). ## Metric types The `type` on each field below is a [Prometheus metric type](https://prometheus.io/docs/concepts/metric_types/): * **counter** — a cumulative value that only increases (or resets to zero on restart), e.g. total requests. * **gauge** — a single value that can go up or down, e.g. in-flight requests. * **histogram** — samples observations into configurable buckets and exposes `_bucket`, `_sum`, and `_count` series, e.g. request duration. Labeled histograms/counters/gauges register a metric *family*, not one series. > **Info** > > Labeled metrics (`HistogramVec`, `CounterVec`, `GaugeVec`) register a metric *family*, not individual time series. For request-populated metrics, a series for a given label combination appears at `/metrics` only after the first matching request. Other families create or remove series from runtime state; their entries below document those semantics. ## Where each family is exposed | Family | Emitted by | Scrape endpoint | | -------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------ | | `dynamo_frontend_*` | HTTP frontend (`python -m dynamo.frontend`) | `/metrics` on `DYN_HTTP_PORT` (default 8000) | | `dynamo_component_*` | Backend workers, standalone router | `/metrics` on `DYN_SYSTEM_PORT` (must be set) | | `dynamo_router_overhead_*` | Frontend with KV routing | `/metrics` on `DYN_HTTP_PORT` | | `dynamo_epp_*` | Endpoint picker (Gateway API Inference Extension routing) | `/metrics` on `DYN_EPP_METRICS_PORT` (default 9090) | | `dynamo_operator_*` | Kubernetes operator | See [Operator Metrics](/dynamo/reference/observability/operator-metrics) | Backend workers expose `dynamo_component_*` only when `DYN_SYSTEM_PORT` is set (disabled by default; the Kubernetes operator typically sets `9090`, local examples use `8081`). See [Environment Variables](/dynamo/reference/observability/environment-variables#system-and-metrics). Engine pass-through metrics (`vllm:*`, `sglang:*`, `trtllm_*`) are emitted by the backend engines themselves and are cataloged separately — see [Metrics Comparison](/dynamo/reference/observability/metrics-comparison). NIXL transfer metrics are exposed on their own port and owned by the [upstream NIXL project](https://github.com/ai-dynamo/nixl/blob/main/docs/telemetry.md). ## Frontend metrics Emitted by the HTTP frontend at `/metrics` on port 8000 by default. Most carry a `model` label. For endpoints that canonicalize aliases, that label holds the primary served model name — the first `--served-model-name` value — so a request sent to an alias is counted under the primary. A name the frontend has not registered is counted under `unknown_model`. **`dynamo_frontend_active_requests`** `gauge` Requests currently being handled by the frontend, from HTTP handler entry until the response stream completes. The top-level in-flight count with no stage breakdown. --- **`dynamo_frontend_stage_requests`** `gauge` Requests currently in a given frontend pipeline stage. Labeled by `stage` and `phase` — see [Stage values](/dynamo/reference/observability/metric-labels#stage-values) and [Phase values](/dynamo/reference/observability/metric-labels#phase-values). --- **`dynamo_frontend_inflight_requests`** `gauge` — deprecated Inflight requests. Kept for backward compatibility; prefer `dynamo_frontend_active_requests`, which has identical semantics with a clearer name. --- **`dynamo_frontend_queued_requests`** `gauge` — deprecated Requests in the HTTP processing queue. Kept for backward compatibility; the "waiting for first token" window is now the sum of `dynamo_frontend_stage_requests` across the `preprocess`, `route`, and `dispatch` stages. --- **`dynamo_frontend_disconnected_clients`** `gauge` Number of disconnected clients. --- **`dynamo_frontend_input_sequence_tokens`** `histogram` Input sequence length in tokens. --- **`dynamo_frontend_cached_tokens`** `histogram` Cached tokens (prefix cache hits) per request. --- **`dynamo_frontend_tokenizer_cache_hits_total`** `counter` L1 tokenizer prefix-cache hits across all models. One outcome is recorded per encode operation or batch item. --- **`dynamo_frontend_tokenizer_cache_misses_total`** `counter` L1 tokenizer prefix-cache misses across all models. --- **`dynamo_frontend_tokenizer_cache_cached_tokens_total`** `counter` Tokens returned from the L1 tokenizer prefix cache. Labeled by `model`. --- **`dynamo_frontend_tokenizer_cache_uncached_tokens_total`** `counter` Tokens freshly encoded after an L1 tokenizer cache lookup. Labeled by `model`. --- Use this PromQL expression to calculate the five-minute token reuse ratio by model: ```promql sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m])) / ( sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m])) + sum by (model) (rate(dynamo_frontend_tokenizer_cache_uncached_tokens_total[5m])) ) ``` The ratio is defined only when the model has an active L1 cache and observed tokens. Partial hits increment both token counters. When the cache is disabled with `DYN_TOKENIZER_CACHE=0`, the per-model series are never created, so they are absent from the scrape rather than reported as zero; detect that state with `absent()`, since a `== 0` comparison never matches a missing series. Only the literal value `0` disables the cache; values such as `false` or `off` leave it enabled. With the cache enabled, the counters remain zero when encoding fails or the tokenizer has no registered special-token boundaries. **`dynamo_frontend_inter_token_latency_seconds`** `histogram` Inter-token latency in seconds. --- **`dynamo_frontend_output_sequence_tokens`** `histogram` Output sequence length in tokens. --- **`dynamo_frontend_output_tokens_total`** `counter` Total output tokens generated. --- **`dynamo_frontend_request_duration_seconds`** `histogram` End-to-end LLM request duration in seconds. --- **`dynamo_frontend_requests_total`** `counter` Total LLM requests. --- **`dynamo_frontend_time_to_first_token_seconds`** `histogram` Time to first token in seconds. --- ### Migration Metrics for [request migration](/dynamo/kubernetes/fault-tolerance/request-migration), the mechanism that continues in-flight requests on a healthy worker when the original worker fails. See [Request Migration Architecture](/dynamo/knowledge-base/concepts/fault-tolerance/request-migration-architecture) for the internals. **`dynamo_frontend_model_migration_total`** `counter` Total request migrations due to worker unavailability. Labeled by `model` and [`migration_type`](/dynamo/reference/observability/metric-labels#metric-specific-labels) (`new_request` for an initial connection failure, `ongoing_request` for a mid-stream disconnection). --- **`dynamo_frontend_model_migration_duration_seconds`** `histogram` Time from detecting a migratable worker failure until the replacement response stream is established, migration ends without recovery, or the request is cancelled. Labeled by `model`, [`migration_type`](/dynamo/reference/observability/metric-labels#metric-specific-labels), and [`outcome`](/dynamo/reference/observability/metric-labels#metric-specific-labels). Use the histogram's `_count` series to count completed migration events by outcome. The duration excludes work completed before the failure and generation after the replacement stream is established. --- **`dynamo_frontend_model_migration_max_seq_len_exceeded_total`** `counter` Total times migration was disabled for a request because its sequence length exceeded `--migration-max-seq-len`. A rising value may indicate the limit needs adjustment. Labeled by `model`. --- ### Cancellation and rejection Counters for requests that end early: [cancellations](/dynamo/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture) (the client or frontend aborts an in-flight request) and [rejections](/dynamo/kubernetes/fault-tolerance/request-rejection) (the Frontend sheds load with HTTP 529 by default when all workers are busy). The worker-side counterparts are under [Component metrics](#component-metrics). **`dynamo_frontend_model_cancellation_total`** `counter` Total request cancellations detected by the frontend (client disconnect or stream close). Labeled by `model`, `endpoint` (the API route — `chat_completions`, `completions`, `embeddings`, …), and [`request_type`](/dynamo/reference/observability/metric-labels#metric-specific-labels) (`unary` or `stream`). --- **`dynamo_frontend_model_rejection_total`** `counter` Total overload responses surfaced to the client as HTTP 529 by default. Incremented for worker-scoped `WorkerOverloaded` and pool-scoped `ResourceExhausted` errors. Labeled by `model` and `endpoint` (the API route). --- ### Per-worker load and timing gauges `dynamo_frontend_worker_*` gauges appear once workers register and begin serving. They live on the frontend's local registry (not component-scoped) and do **not** carry `dynamo_namespace` or `dynamo_component` labels. Frontend-only — not available on the standalone router. Labeled by `worker_id`, `dp_rank`, and `worker_type`. **`dynamo_frontend_worker_active_decode_blocks`** `gauge` Active KV cache decode blocks per worker. --- **`dynamo_frontend_worker_active_prefill_tokens`** `gauge` Active prefill tokens queued per worker. --- **`dynamo_frontend_worker_last_time_to_first_token_seconds`** `gauge` Last observed time to first token per worker, in seconds. --- **`dynamo_frontend_worker_last_input_sequence_tokens`** `gauge` Last observed input sequence length per worker. --- **`dynamo_frontend_worker_last_inter_token_latency_seconds`** `gauge` Last observed inter-token latency per worker, in seconds. --- ### Model metrics All `dynamo_frontend_model_*` gauges carry a `model` label. **`dynamo_frontend_model_ready`** `gauge` Whether the frontend can currently route at least one inference request for the model. A value of `1` means at least one complete serving topology with a live worker and serving engine is available; `0` means the model is registered but not currently routable. The frontend evaluates this gauge from its live routing catalog at scrape time, so worker registration and removal are reflected in the next scrape. Unlike request-populated metrics, the series appears when the model registers without requiring an inference request. The series is absent while the model is not registered, and the entire family is absent when no models are registered. Detect this state with `absent(dynamo_frontend_model_ready{model=""})`; a `== 0` comparison does not match an absent series. Aliases get no series of their own: a deployment served under several names emits one series, labeled with its primary name, so query an alias through its primary. `/v1/models` still lists alias names, but only the primary has a readiness series. A LoRA adapter is a distinct model rather than a second name for one, and keeps its own series. --- The remaining model gauges are populated from worker backend registration. When multiple workers register the same model name, only the first instance's configuration is recorded. **`dynamo_frontend_model_total_kv_blocks`** `gauge` Total KV blocks available for a worker serving the model. --- **`dynamo_frontend_model_max_num_seqs`** `gauge` Maximum number of sequences for a worker serving the model. --- **`dynamo_frontend_model_max_num_batched_tokens`** `gauge` Maximum number of batched tokens for a worker serving the model. --- **`dynamo_frontend_model_context_length`** `gauge` Maximum context length for a worker serving the model. Sourced from the Model Deployment Card. --- **`dynamo_frontend_model_kv_cache_block_size`** `gauge` KV cache block size for a worker serving the model. Sourced from the Model Deployment Card. --- **`dynamo_frontend_model_migration_limit`** `gauge` Request migration limit for a worker serving the model. Sourced from the Model Deployment Card. --- ## Component metrics Emitted by backend workers (`python -m dynamo.vllm`, `python -m dynamo.sglang`, etc.) at `/metrics` on `DYN_SYSTEM_PORT`. All carry the hierarchy labels (`dynamo_namespace`, `dynamo_component`, `dynamo_endpoint`) — see [Runtime-injected labels](/dynamo/reference/observability/metric-labels#runtime-injected-labels). **`dynamo_component_inflight_requests`** `gauge` Requests currently being processed by the component. --- **`dynamo_component_request_bytes_total`** `counter` Total bytes received in requests. --- **`dynamo_component_request_duration_seconds`** `histogram` Request processing time in seconds. --- **`dynamo_component_requests_total`** `counter` Total requests processed. --- **`dynamo_component_errors_total`** `counter` Total errors encountered while handling a request. Labeled by `error_type` — see [Component error types](/dynamo/reference/observability/metric-labels#component-error-types). --- **`dynamo_component_response_bytes_total`** `counter` Total bytes sent in responses. --- **`dynamo_component_uptime_seconds`** `gauge` DistributedRuntime uptime. Updated before each Prometheus scrape on both the frontend and the system-status server. --- **`dynamo_component_cancellation_total`** `counter` Total requests cancelled by the work handler. Carries the hierarchy labels. Records cancellation *signals* received by the worker, not whether the engine actually aborted — deduplicated so a control message plus a socket close for the same request counts once. See [Request Cancellation Architecture](/dynamo/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture). --- The following worker-side gauges and counter appear only when a hard concurrency cap is set with `--engine-request-limit`; see [Worker-Side Request Admission](/dynamo/knowledge-base/concepts/fault-tolerance/request-rejection-architecture#worker-side-request-admission). **`dynamo_rejection_request_total`** `counter` Cumulative requests rejected because the worker was at capacity (engine in-flight limit and Dynamo queue both full). --- **`dynamo_engine_request`** `gauge` Current requests being handled by the engine. --- **`dynamo_request_queue`** `gauge` Current requests queued in Dynamo, not yet in the engine. --- Specialized components expose additional families under their own prefix, e.g. `dynamo_preprocessor_*` for preprocessor components. ## Router metrics The router exposes metrics for routing decisions and overhead. Not every metric appears in every deployment — see [Availability by configuration](#availability-by-configuration). ### Router request metrics `dynamo_component_router_*` histograms and counters for aggregate request-level statistics. On the frontend, exposed at `/metrics` on the HTTP port; on the standalone router, exposed on `DYN_SYSTEM_PORT`. Populated per-request when `--router-mode kv` is active; registered with zero values in non-KV modes. **`dynamo_component_router_requests_started_total`** `counter` Requests admitted by the router scheduler. --- **`dynamo_component_router_requests_total`** `counter` Total requests processed by the router. --- **`dynamo_component_router_time_to_first_token_seconds`** `histogram` Time to first token in seconds. --- **`dynamo_component_router_inter_token_latency_seconds`** `histogram` Average inter-token latency in seconds. --- **`dynamo_component_router_input_sequence_tokens`** `histogram` Input sequence length in tokens. --- **`dynamo_component_router_output_sequence_tokens`** `histogram` Output sequence length in tokens. --- **`dynamo_component_router_kv_hit_rate`** `histogram` Predicted KV cache hit rate at routing time (0.0–1.0). --- **`dynamo_component_router_non_max_overlap_selections_total`** `counter` Admitted prefill scheduler selections routed to a worker with less KV cache overlap than another eligible worker. Pinned requests and equal-overlap ties are excluded. Labeled by `worker_type` (`prefill`). --- **`dynamo_component_router_overlap_blocks_lost`** `histogram` Difference in effective KV cache overlap between the highest-overlap eligible prefill worker and the selected worker for each non-max-overlap selection. Labeled by `worker_type` (`prefill`). --- ### Per-request routing overhead `dynamo_router_overhead_*` histograms (milliseconds) track time spent in each phase of the routing decision. Registered on the frontend port with a `router_id` label (the frontend's discovery instance ID). Created only when the frontend has DRT discovery enabled (`--router-mode kv`); absent in non-KV modes and on the standalone router. **`dynamo_router_overhead_block_hashing_ms`** `histogram` Time computing block hashes. --- **`dynamo_router_overhead_indexer_find_matches_ms`** `histogram` Time in indexer `find_matches`. --- **`dynamo_router_overhead_seq_hashing_ms`** `histogram` Time computing sequence hashes. --- **`dynamo_router_overhead_scheduling_ms`** `histogram` Time in scheduler worker selection. --- **`dynamo_router_overhead_total_ms`** `histogram` Total routing overhead per request. --- ### Router queue metrics The frontend registers these metrics when queueing is enabled by `--router-queue-threshold` or a threshold in `--router-policy-config`. They carry `model`, `worker_type`, and `policy_class`. With policy-family or cache-bucket configuration, `policy_class` is the resolved physical queue. **`dynamo_frontend_router_queue_pending_requests`** `gauge` Requests pending in the router scheduler queue. --- **`dynamo_frontend_router_queue_pending_isl_tokens`** `gauge` Raw input-sequence tokens pending in the queue. --- **`dynamo_frontend_router_queue_pending_cached_tokens`** `gauge` Cached-token estimate captured when each request enters the queue. --- **`dynamo_frontend_router_queue_backpressure_total`** `counter` Queue rejections by configured limit reason. Also labeled by `reason`. --- ### KV indexer metrics **`dynamo_component_kv_cache_events_applied`** `counter` KV cache events applied to the router's radix-tree index. Includes events applied to the device tier and lower tiers such as host-pinned memory and disk. Appears only when `--router-kv-overlap-score-credit` is greater than 0 and workers publish KV events. Labeled by `status` and `event_type` — see [Metric-specific labels](/dynamo/reference/observability/metric-labels#metric-specific-labels). --- The standalone indexer exposes the device-tier-only counter `dynamo_kvrouter_kv_cache_events_applied`; see [Standalone KV Indexer](/dynamo/knowledge-base/modular-components/router/standalone-indexer). ### Availability by configuration Not all router metrics appear in every deployment. This matrix shows which groups are **registered** and **populated** in each configuration. | Metric group | Frontend + KV (agg) | Frontend + KV (disagg) | Frontend + non-KV | Standalone router | | ------------------------------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------- | --------------------------------- | | `dynamo_component_router_*` | Registered and populated | Registered and populated | Registered, **always zero** | Populated (on `DYN_SYSTEM_PORT`) | | `dynamo_router_overhead_*` | Registered and populated | Registered and populated | **Not registered** | **Not created** | | `dynamo_frontend_router_queue_*` | Registered; populated when a CLI or policy-class queue threshold is set | Registered; populated when a CLI or policy-class queue threshold is set | **Not registered** | **Not created** | | `dynamo_component_kv_cache_events_applied` | Populated when KV events received | Populated when KV events received | **Not registered** | Populated when KV events received | | `dynamo_frontend_worker_*` | Registered and populated | Registered and populated (`worker_type` = `prefill`/`decode`) | Registered and populated (`worker_type` = `decode`) | **Not created** | **Key:** * **Registered and populated** — the metric appears at `/metrics` with real values. * **Registered, always zero** — the metric appears at `/metrics` but is never incremented (useful for dashboards that expect it to exist). * **Not registered / Not created** — the metric does not appear at `/metrics` at all. ## Endpoint picker metrics Emitted by the endpoint picker (EPP) when routing through the Gateway API Inference Extension, at `/metrics` on `DYN_EPP_METRICS_PORT` (default 9090). Set the port to `0` to disable the endpoint. **`dynamo_epp_cached_tokens`** `histogram` Prompt tokens the model server served from its KV cache per request, read from `usage.prompt_tokens_details.cached_tokens` in the response body. This is the *observed* cache hit, as opposed to the overlap the KV router estimates at selection time. Buckets match `dynamo_frontend_cached_tokens` so the two can share a dashboard. The `model` label is the model the pool serves, resolved once at startup rather than from the request body. --- Requires the client to request usage accounting on streaming calls (`"stream_options": {"include_usage": true}`); without it the response carries no usage block and nothing is recorded. ## Related * [Metric Labels](/dynamo/reference/observability/metric-labels) — the dimensions attached to these metrics. * [Environment Variables](/dynamo/reference/observability/environment-variables) — variables that enable and configure metric emission. * [Operator Metrics](/dynamo/reference/observability/operator-metrics) — Kubernetes operator metrics catalog. * [Metrics Comparison](/dynamo/reference/observability/metrics-comparison) — engine pass-through metrics (`vllm:*`, `sglang:*`, `trtllm_*`).