> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Metric Labels Labels are the **dimensions** attached to Dynamo metric series, not metrics in their own right. A single metric such as `dynamo_component_requests_total` becomes many Prometheus time series once it is split by `dynamo_namespace`, `dynamo_component`, `dynamo_endpoint`, and the labels the scraper adds. This page catalogs those dimensions; for the metrics themselves see the [Metrics Catalog](/dynamo/dev/observability/metrics-catalog). A label reaches a series through one of four origins: * **Auto-injected by the Dynamo runtime** — added to every metric registered through the namespace/component/endpoint hierarchy. * **Added at registration time by backend code** — passed by the worker when it registers an endpoint, so presence depends on the backend. * **Added by the metric itself** — specific to one metric family. * **Injected by Prometheus/Kubernetes** — attached by the scraper, not present in the exposition text the process emits. ## Scope legend Each label is tagged with the scope in which it appears: * `[Shared]` — emitted by the Dynamo process itself; present whether you run locally or on Kubernetes. * `[Kubernetes]` — attached by the Prometheus scraper in a Kubernetes deployment; absent when you scrape a process directly. ## Runtime-injected labels Added by `create_metric()` in `lib/runtime/src/metrics.rs` for every metric registered through the namespace/component/endpoint hierarchy. `[Shared]` The Dynamo runtime namespace — the logical scope shared by every component (router / prefill / decode / encode) in one deployment. **Not** the Kubernetes namespace. Example: `dynamo_cloud_vllm_v1_disagg_router_071de157`. `[Shared]` Service role of the emitting component. See [Component names](#component-names) for the enumeration of values. Example: `backend`, `prefill`, `router`. `[Shared]` The RPC within that component. See [Endpoint names](#endpoint-names) for the enumeration of values. Example: `generate`, `clear_kv_blocks`, `worker_kv_indexer_query_dp0`. `[Shared]` Hex-encoded discovery instance ID of the endpoint, providing a stable per-worker identity that does not depend on Kubernetes. Injected only when the endpoint hierarchy has a connection ID. Example: `1a2b3c4d`. On the per-worker frontend gauges (`dynamo_frontend_worker_*`) this label carries the worker's etcd lease ID instead. Example: `7890`. ## Registration-time labels Passed via `metrics_labels=` when the worker calls `serve_endpoint()`. Not auto-injected, so presence depends on the backend. `[Shared]` The model being served (OpenAI-style label). Added by vLLM, SGLang, and TRT-LLM workers on inference endpoints; absent on internal endpoints like `worker_kv_indexer_query_dp{N}`. Example: `Qwen/Qwen3-0.6B`. `[Shared]` Same model identifier under a second label name, retained for engine-native and dashboard back-compat. Added by vLLM and TRT-LLM workers; **not** added by SGLang. Example: `Qwen/Qwen3-0.6B`. ## Metric-specific labels Added by an individual metric family. Each label appears only on the metrics noted. `[Shared]` On `dynamo_component_errors_total` — the stage of request handling that failed. See [Component error types](#component-error-types) for the enumeration. On `dynamo_operator_reconcile_errors_total` the value set is different; see [Operator Metrics](/dynamo/dev/observability/operator-metrics). `[Shared]` On `dynamo_frontend_stage_requests` — the frontend pipeline stage a request currently occupies. See [Stage values](#stage-values). Allowed values: preprocess route dispatch `[Shared]` On `dynamo_frontend_stage_requests` — the serving phase. See [Phase values](#phase-values). The empty value is used when the stage does not distinguish phases (e.g. `preprocess`). Allowed values: prefill decode aggregated "" (empty) `[Shared]` On `dynamo_frontend_worker_*` and `dynamo_frontend_router_queue_*` — the worker role. In disaggregated mode both values appear; in aggregated mode all workers report as `decode`. Allowed values: prefill decode `[Shared]` On `dynamo_frontend_worker_*` — the data-parallel rank of the worker. Example: `0`. `[Shared]` On `dynamo_frontend_router_queue_*` — the resolved physical scheduler queue. With policy-family or cache-bucket configuration, this can differ from the family requested by the client. `[Shared]` On `dynamo_frontend_router_queue_backpressure_total` — the configured queue limit that rejected the request. `[Shared]` On `dynamo_frontend_model_migration_total` — the category of request migration triggered by worker unavailability. Allowed values: new\_request ongoing\_request `[Shared]` On `dynamo_frontend_model_cancellation_total` — whether the cancelled request was unary or streaming. Allowed values: unary stream `[Shared]` On `dynamo_component_kv_cache_events_applied` — the outcome of applying a KV cache event to the router's radix-tree index. Allowed values: ok parent\_block\_not\_found block\_not\_found invalid\_block capacity\_exhausted indexer\_invariant\_violation `[Shared]` On `dynamo_component_kv_cache_events_applied` — the kind of KV cache event. Allowed values: stored removed cleared ## Scraper-injected labels (Kubernetes) Attached by the Prometheus scraper, not present in the exposition text the process emits. These appear only when Prometheus scrapes the pod in a Kubernetes deployment. `[Kubernetes]` Scrape target as `:`. Example: `192.168.133.236:9090`. `[Kubernetes]` Kubernetes pod name; the per-replica disambiguator. Example: `vllm-v1-disagg-router-vllmdecodeworker-...`. `[Kubernetes]` Container name inside the pod (usually `main`). Example: `main`. `[Kubernetes]` Kubernetes namespace the pod runs in. **Not** the same as `dynamo_namespace`. Example: `dynamo-cloud`. `[Kubernetes]` Prometheus scrape-job name, `/`. Example: `dynamo-cloud/dynamo-worker`. `[Kubernetes]` Named port on the Kubernetes Service that Prometheus scraped. **Not** the same as `dynamo_endpoint`. Example: `system`. ## Label collisions Two pairs of labels share a name but mean different things depending on origin: * `dynamo_namespace` (Dynamo deployment scope, runtime-injected) vs. `namespace` (Kubernetes namespace, scraper-injected). * `dynamo_endpoint` (Dynamo RPC, runtime-injected) vs. `endpoint` (Kubernetes Service port name, scraper-injected). Always qualify with the `dynamo_` prefix when you mean the runtime dimension. ## Enumerations ### Component names Values you will see in the `dynamo_component` label on `dynamo_component_*` series. The HTTP frontend (`python -m dynamo.frontend`) is **not** in this list — it exposes its own `dynamo_frontend_*` metric family, not `dynamo_component_*`. | Value | Meaning | | ----------- | --------------------------------------------------------------------------------------------------------------------- | | `router` | The standalone KV router (`python -m dynamo.router`). | | `Planner` | The planner component (`python -m dynamo.planner`). Note the capital `P`. | | `prefill` | The prefill worker in disaggregated serving (all backends). | | `backend` | The decode worker in disaggregated serving for all backends, **and** the combined worker for vLLM in aggregated mode. | | `encode` | The encode worker for vLLM, SGLang, and TRT-LLM. | | `diffusion` | The diffusion worker for TRT-LLM. | Internal subsystems (e.g. `kvbm` from the block manager, `sequences` from the KV router) also create components and may appear in `dynamo_component_*` series. The default for vLLM/SGLang can be overridden by passing `--endpoint dyn://..` on the worker command line. The name `backend` for the decode worker is historical. The runtime has a TODO to introduce a `decode` constant and migrate to it (see `lib/runtime/src/metrics/prometheus_names.rs::component_names`). ### Endpoint names Values you will see in the `dynamo_endpoint` label on backend workers. | Value | Meaning | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `generate` | Main inference RPC; one increment per request received. On a prefill worker this counts prefill-stage `generate` calls; on a decode worker this counts decode-stage `generate` calls. | | `clear_kv_blocks` | Legacy vLLM distributed admin RPC to flush the worker KV cache. Unified vLLM exposes `/engine/control/clear_kv_blocks` on the system server instead; that route does not emit this component endpoint label. | | `worker_kv_indexer_query_dp{N}` | KV-router queries to the worker's local KV indexer about its cached prefix blocks. One endpoint per data-parallel rank (`_dp0`, `_dp1`, …). Appears on the worker that owns the prefix caches the router consults — in disaggregated serving that is the prefill worker. | ### Component error types The `dynamo_component_errors_total` counter is labeled with `error_type`, identifying which stage of request handling failed. | `error_type` | Stage | Meaning | | ------------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `deserialization` | Ingress | Could not parse the incoming request payload. | | `invalid_message` | Ingress | Wire-format violation in the incoming message. | | `response_stream` | Pre-generate | The worker received the request but could not open the response stream back to the frontend (transport problem before `generate` was called). | | `generate` | Engine | The engine's `generate()` itself returned an error. This is the counter that reflects engine/inference failures. | | `publish_response` | Streaming | The engine produced response chunks but the worker could not push one back to the frontend. **Also fires on client cancellation** — the frontend disconnecting before the stream finishes — so this counter can be inflated by user-aborted requests. | | `publish_final` | Teardown | All response chunks were sent, but the worker could not deliver the final stream-complete marker. The connection died right at the end. | ### Stage values `dynamo_frontend_stage_requests` decomposes the lifetime of an active frontend request into three sequential pipeline stages. A request is counted in exactly one stage at a time. | Stage | What it covers | Enters when | Exits when | | ------------ | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------- | | `preprocess` | Tokenization and chat-template application | The frontend enters `preprocess_request` | Preprocessing returns | | `route` | Worker selection (including parking in the KV-router queue while waiting for a worker) | The router's `generate()` is called | A worker is selected or the request is queued for one | | `dispatch` | Serialization, transport to the chosen worker, and waiting for the backend's first response (includes backend prefill time) | `generate()` is called in `AddressedPushRouter` | The first response is received from the backend | ### Phase values The `phase` label on `dynamo_frontend_stage_requests`. | Phase | Meaning | | ------------ | ----------------------------------------------------------------------------------------- | | `prefill` | The request is being handled by a prefill worker in disaggregated serving. | | `decode` | The request is being handled by a decode worker in disaggregated serving. | | `aggregated` | Aggregated (non-disaggregated) serving — a single worker handles both prefill and decode. | | `""` (empty) | The stage does not distinguish phases (used by `preprocess`). | ## Related * [Metrics Catalog](/dynamo/dev/observability/metrics-catalog) — the `dynamo_*` metrics these labels attach to. * [Operator Metrics](/dynamo/dev/observability/operator-metrics) — Kubernetes operator metrics and their distinct label sets. * [Environment Variables](/dynamo/dev/observability/environment-variables) — variables that enable and configure metric emission.