Metrics Catalog
Dynamo exposes metrics in Prometheus exposition format at the /metrics HTTP endpoint. This page lists every metric that Dynamo’s own code defines; use it to look up a metric name you see in a scrape, dashboard, or alert. Each section lists its metrics in a table, followed by notes and PromQL examples where needed. Metric Labels documents the label values. For setup and dashboards, see the Metrics guide.
This page does not list:
- Inference engine metrics (
vllm:*,sglang:*,trtllm_*) that the engine itself defines. See Engine Metrics. - Kubernetes operator metrics. See Operator Metrics.
- NIXL transfer metrics, which NIXL serves on its own port. See the NIXL telemetry documentation.
- Velo metrics (
velo_*), which the Velo library defines. Dynamo processes expose them whenDYN_RESPONSE_PLANE=velo. See the Velo metrics reference. - KVBM metrics, including the block-pool metrics in the
kvbm-logicalcrate. KVBM was removed in v1.7.0; see the documentation for v1.6 and earlier.
Metric Types
The type of each metric is a Prometheus metric type:
- counter — a cumulative value that only increases (or resets to zero on restart), e.g. total requests.
- gauge — a single value that can go up or down, e.g. in-flight requests.
- histogram — samples observations into configurable buckets and exposes
_bucket,_sum, and_countseries, e.g. request duration. The frontend latency and sequence-length histograms take their bucket ranges from environment variables — see Histogram buckets.
Labeled metrics (HistogramVec, CounterVec, GaugeVec) register a metric family, not individual time series. For request-populated metrics, a series for a given label combination appears at /metrics only after the first matching request. Other families create or remove series from runtime state; their entries below document those semantics.
Every metric table on this page has the same columns:
- Labels lists the labels specific to the metric. Labels shared by every metric in a section are listed in the section introduction.
- Status is empty for a stable metric, which is renamed or removed only after it has been marked Deprecated. Experimental marks a preview metric whose name or meaning may change, and Deprecated marks a metric that is still emitted but scheduled for removal; its description names the replacement.
Metrics defined in Python, such as the planner metrics, also expose a _created series for each counter and histogram.
Where each family is exposed
Backend workers expose dynamo_component_* only when DYN_SYSTEM_PORT is set (disabled by default; the Kubernetes operator typically sets 9090, local examples use 8081). See Environment Variables.
Frontend Metrics
Emitted by python -m dynamo.frontend at /metrics on the HTTP port (--http-port, default 8000). The frontend serves its router and runtime metrics on the same port and does not use DYN_SYSTEM_PORT. Frontend metrics carry no automatic labels.
--metrics-prefix(DYN_METRICS_PREFIX) replaces thedynamo_frontendprefix of thedynamo_frontend_*metrics, exceptdynamo_frontend_stage_*,dynamo_frontend_tokenize_seconds,dynamo_frontend_template_seconds,dynamo_frontend_detokenize_*,dynamo_frontend_tokenizer_cache_*,dynamo_frontend_worker_*,dynamo_frontend_router_*,dynamo_frontend_lora_*,dynamo_frontend_event_loop_*, anddynamo_frontend_metrics_start_time_seconds, which keep their names.- With
--kserve-grpc-server, the frontend serves metrics onDYN_GRPC_METRICS_PORT(default8788) instead. The model configuration and migration metrics, thedynamo_router_*metrics, and the metrics in the runtime registry (dynamo_component_*, including the router request metrics, anddynamo_quic_response_*) are not exposed in that mode; thedynamo_frontend_router_*metrics are.
For endpoints that canonicalize aliases, the model label holds the primary served model name — the first --served-model-name value — so a request sent to an alias is counted under the primary. A name the frontend has not registered is counted under unknown_model.
dynamo_frontend_awaiting_first_token_requests, dynamo_frontend_inflight_input_tokens, and dynamo_frontend_input_tokens_total have a zero-valued series for each registered model, and for unknown_model, before the first request. With --dyn-chat-processor vllm or sglang, chat requests skip Dynamo’s preprocessor, so they record no preprocess stage and no template, tokenization, detokenization, or tokenizer-cache metrics.
Requests producing output tokens are clamp_min(dynamo_frontend_active_requests - dynamo_frontend_awaiting_first_token_requests, 0); the clamp covers the two gauges being sampled a moment apart. Unlike the sum of dynamo_frontend_input_sequence_tokens, which is observed at the first output token, dynamo_frontend_input_tokens_total counts every admitted request when it is tokenized.
Use this PromQL expression to calculate the five-minute token reuse ratio by model:
The ratio is defined only when the model has an active L1 cache and observed tokens. Partial hits increment both token counters. When the cache is disabled with DYN_TOKENIZER_CACHE=0, the per-model series are never created, so they are absent from the scrape rather than reported as zero; detect that state with absent(), since a == 0 comparison never matches a missing series. Only the literal value 0 disables the cache; values such as false or off leave it enabled. With the cache enabled, the counters remain zero when encoding fails, or when the tokenizer has no registered special tokens or has special tokens that can overlap, which turns the cache off.
Multimodal Requests
These histograms carry a model label.
Migration
Metrics for request migration, the mechanism that continues in-flight requests on a healthy worker when the original worker fails. See Request Migration Architecture for the internals. These carry a model label.
Use the _count series of dynamo_frontend_model_migration_duration_seconds to count completed migration events by outcome. The duration excludes work completed before the failure and generation after the replacement stream is established.
Cancellation and Rejection
Counters for requests that end early: cancellations (the client or frontend aborts an in-flight request) and rejections (the Frontend sheds load with HTTP 529 by default when all workers are busy). Both carry model and endpoint labels. The worker-side counterparts are under Component metrics.
Per-Worker Load and Timing Gauges
dynamo_frontend_worker_* gauges appear once workers register and begin serving. They live on the frontend’s local registry (not component-scoped) and do not carry dynamo_namespace or dynamo_component labels. Frontend-only — not available on the standalone router. Labeled by worker_id (the worker’s discovery ID in decimal), dp_rank, and worker_type.
Only the KV router writes the three dynamo_frontend_worker_active_* gauges, so they have no series in other router modes; the three dynamo_frontend_worker_last_* gauges are populated in every mode. At each HTTP scrape, these six gauges include only workers in the current committed router membership that pass hard availability checks, and only their current data-parallel (DP) ranks. The timing gauges also allow dp_rank="none" when the rank is unknown, provided the worker remains available. The worker_type label describes serving attribution: an Encode worker serving Chat/Completions reports its load and timing samples as decode, while the router worker inventory gauges below retain its encode topology role. Removed or excluded workers disappear from the output; late request completions or load updates cannot restore their exposed series.
Router Worker Inventory
These frontend-only metrics describe each Frontend’s independent view of normalized base model-card discovery and router admission. They do not report Kubernetes Pod existence, and LoRA overlays do not count as additional workers. Both metrics carry model, target_namespace, target_component, target_endpoint, worker_type, and state labels. The target labels identify the discovered serving endpoint.
Admission uses first-wins semantics within each Frontend’s model and target group. The first observed checksum cohort remains the incumbent; later incompatible worker IDs are excluded individually and do not remove or replace it. If the incumbent cohort disappears, the next remaining cohort can become its successor after initialization.
The available, pending, and excluded states are disjoint; discovered, which only dynamo_frontend_router_workers reports, is their sum. available requires successfully committed worker-set membership and current hard availability; pending means discovery has observed the worker but initialization has not completed; excluded means discovered but currently unavailable for routing.
Hard availability uses the committed group’s client and, when a prefill or encode hop targets that group, the hop’s local availability. A matching hop that is still activating reports no available workers. If no hop targets the group, its own client availability applies. These counts describe individual worker groups; use dynamo_frontend_model_ready to check whether the frontend has a complete routable serving topology.
Each currently observed group emits all four counts, including zero values for states with no workers. When discovery withdraws a group after its last worker disappears, its count and per-worker state series disappear on the next scrape. Retired rolling-update namespaces are not retained as permanent zero-count groups. Groups never observed by that frontend also have no series. Do not aggregate these counts across Frontends as worker-global readiness; separate Frontends can choose different incumbents from the same discovered workers.
The bounded reason values describe the current admission or availability condition:
When an incumbent is still initializing and an incompatible newcomer is discovered, counts are discovered=2, available=0, pending=1, and excluded=1. The incumbent carries state="pending",reason="initializing"; only the newcomer carries state="excluded",reason="checksum_mismatch". Once the incumbent is healthy, the same group reports discovered=2, available=1, pending=0, and excluded=1. Rejection of the newcomer never makes the incumbent pending or excluded.
To detect an unexpected absence of workers, select one frontend, model, worker role, and the namespaces belonging to one deployment. The following expression alerts when the frontend is being scraped successfully but no selected group has a positive discovered-worker count. It handles both explicit zero counts and series disappearing after group withdrawal:
Replace the selectors with your scrape labels and a namespace pattern that includes only the rolling versions of one deployment. Enable the rule only for deployments expected to have workers, and use an alert duration such as for: 3m to tolerate startup and brief rollout transitions. To alert when no selected worker is available for routing, use state="available" instead of state="discovered". Monitor up == 0 separately for frontend scrape failures. A raw sum(...) == 0 expression does not match when all selected group series have disappeared.
Preserve target_namespace when diagnosing an individual namespace. Do not aggregate independent deployments or different frontend instances.
Inspect workers still discovered but excluded from routing:
Scope these queries to the target frontend and deployment. Evaluate each Frontend separately because discovery order and first-wins admission can produce different incumbents. A missing series does not mean zero workers; check Prometheus up separately for scrape health. These metrics reflect updates after the frontend processes discovery and routing state changes, not at the instant a worker process terminates.
Model Metrics
All dynamo_frontend_model_* gauges carry a model label. The gauges other than dynamo_frontend_model_ready are populated from worker backend registration. Each time a worker group for the model is committed or rebuilt, its configuration overwrites these values, so with several groups, such as prefill and decode, they show the most recently committed one. A worker that joins an existing group does not change them. dynamo_frontend_model_total_kv_blocks, dynamo_frontend_model_max_num_seqs, and dynamo_frontend_model_max_num_batched_tokens change only when the committed configuration reports a value: a group that omits one keeps the previous value, and the series is absent until some group reports it.
The frontend evaluates dynamo_frontend_model_ready from its live routing catalog at scrape time, so worker registration and removal are reflected in the next scrape. Unlike request-populated metrics, the series appears when the model registers, without requiring an inference request. The series is absent while the model is not registered, and the entire family is absent when no models are registered. Detect this state with absent(dynamo_frontend_model_ready{model="<model>"}); a == 0 comparison does not match an absent series. Aliases get no series of their own: a deployment served under several names emits one series, labeled with its primary name, so query an alias through its primary. /v1/models still lists alias names, but only the primary has a readiness series. A LoRA adapter is a distinct model rather than a second name for one, and keeps its own series.
System One Requests
Recorded for the experimental System One (/v1/systemone) and Decisions (/v1/decisions) endpoints, which --enable-systemone-api (DYN_ENABLE_SYSTEMONE_API) enables. These carry model. A request is recorded only after its preflight succeeds, and each phase only when it completes without an error.
LoRA Allocation
Registered on every frontend and populated only when LoRA serving is enabled with DYN_LORA_ENABLED and the allocation controller is not turned off with DYN_LORA_ALLOCATION_ENABLED=false. These carry an endpoint label, namespace/component/endpoint.
Async Runtime and Request Plane
Health of the frontend’s Tokio async runtime and of its request path to workers. Frontend only.
Component Metrics
Emitted by backend workers (python -m dynamo.vllm, python -m dynamo.sglang, etc.) at /metrics on DYN_SYSTEM_PORT. Metrics registered on an endpoint carry the hierarchy labels (dynamo_namespace, dynamo_component, dynamo_endpoint) and worker_id — see Runtime-injected labels. Metrics created on a component rather than an endpoint, such as dynamo_backend_admission_*, have no dynamo_endpoint, and dynamo_work_handler_* has no labels. Workers also expose their inference engine’s own metrics; see Engine Metrics.
The request metrics below are recorded for every endpoint a process serves. vLLM workers also add model and model_name on their inference endpoints, except embedding, classify, and multimodal encode workers, which add only model. SGLang workers add model, except multimodal encode workers, which add model and model_name, and image and video diffusion workers, which add neither. TensorRT-LLM workers add model and model_name only with --publish-kv-events, --publish-metrics, or forward-pass metrics (--fpm-trace or DYN_FORWARDPASS_METRIC_PORT). Internal endpoints, such as the worker_kv_query_source_* endpoint the router uses to recover a worker’s KV index, have no model label.
The shared TCP request plane, the default, exposes the following dynamo_work_handler_* metrics. They are
registered once per process and are independent of backend admission; see Backend Admission Control.
With the NATS request plane, all except dynamo_work_handler_time_to_first_response_seconds stay at 0.
The dynamo_backend_admission_* metrics track engine occupancy, queueing, and request outcomes.
These metrics count requests only when at least one engine limit is configured. With both engine limits
unlimited, they remain at zero.
See Backend Admission Control
for gate behavior.
These carry a dp_rank label: the rank ID, such as "4", or "unspecified" for unranked requests.
Sum across all dp_rank values for worker totals.
KV Event Publishing
Workers that publish KV cache events to the router: vLLM and SGLang with --kv-events-config, TensorRT-LLM with --publish-kv-events, unified backend workers that declare KV event sources, and the mocker. The kv_publisher metrics are created on the component, so they have no dynamo_endpoint.
Engine State
Label sets depend on the worker type:
python -m dynamo.vllm,dynamo.sglang, anddynamo.trtllmaddmodelanddynamo_componentinstead of the automatic labels. TensorRT-LLM updates the cache gauges only with--publish-metricsor forward-pass metrics (--fpm-traceorDYN_FORWARDPASS_METRIC_PORT).- Unified backend workers use the automatic labels plus
modelandmodel_name.
Multimodal and Embedding Caches
These metrics come from several publishers, so their labels differ:
- The image cache metrics, and the embedding cache metrics a worker records in its own process, carry
modelanddynamo_component. - With the
DynamoMultimodalEmbeddingCacheConnector, the vLLM scheduler records the embedding cache metrics once per data-parallel rank. They carrymodel,model_name,dp_rank,dynamo_namespace,dynamo_component,dynamo_endpoint, andworker_id. - The SGLang
dynamo_embedding_*histograms carrymodel,model_name,dynamo_namespace,dynamo_component,dynamo_endpoint, andworker_id.
The embedding cache metrics require --multimodal-embedding-cache-capacity-gb greater than 0; TensorRT-LLM workers also need --publish-kv-events, --publish-metrics, or forward-pass metrics. vLLM encode workers keep an embedding cache but do not export these metrics. The image cache metrics are registered on vLLM multimodal workers, and the shared image cache records samples only with DYN_MM_SHARED_IMAGE_CACHE_ENABLED=1.
TensorRT-LLM Metrics Computed by Dynamo
TensorRT-LLM workers started with --publish-metrics add these to the engine’s own trtllm_* metrics; the trtllm_kv_transfer_* metrics also need --publish-per-request-metrics. They carry model_name, disaggregation_mode, and engine_type, plus the labels Dynamo adds to engine metrics.
Engine Failover
vLLM workers with --gms-shadow-mode (GPU Memory Service shadow failover). These carry engine_id, model, and dynamo_component.
Router Metrics
Emitted by every process that routes requests: the frontend, which serves them on its HTTP port, and the standalone router (python -m dynamo.router), which serves them on DYN_SYSTEM_PORT. The router adds dynamo_namespace and dynamo_component of the first worker component the process routes to (one value per process, even when it routes to both prefill and decode workers), and worker_id, which is the router’s own ID in hex. The request metrics also carry router_id, the same ID in decimal. Not every metric appears in every deployment — see Availability by Configuration.
The dynamo_component_router_worker_registered sample can appear during scheduler preparation, before worker-set materialization commits, so it does not indicate committed model readiness. Late request bookkeeping cannot restore a removed worker’s series. Prefer the Router Worker Inventory metrics for committed available-worker counts and exclusion reasons on the frontend.
Router Request Metrics
dynamo_component_router_* histograms and counters for aggregate request-level statistics. Populated per routing attempt in every router mode. In disaggregated serving, the prefill and decode hops record into the same series, so the request counters count a request once per hop, and the time-to-first-token and output-length histograms also get a prefill-hop sample when the prefill worker returns a token, as vLLM’s does. The KV-specific metrics (kv_hit_rate, the KV-cache reuse counters, non_max_overlap_selections_total, and overlap_blocks_lost) are recorded only with --router-mode kv, and non_max_overlap_selections_total and overlap_blocks_lost only by a router that routes to prefill workers; in aggregated serving they stay at zero.
KV-Cache Reuse Telemetry
The cache reuse funnel shows how much of the incoming prompt could be reused, and how much the worker reports reusing. Each step counts tokens, not requests. The first step, F0, is the tokens the router receives, recorded in every router mode: dynamo_component_router_input_sequence_tokens_sum. These counters record the other steps. They carry router_id, model, and phase, and are recorded only with --router-mode kv; they need no other flag:
Compare token increases over the same time window and matching KV-routing traffic. Use a long enough window: the steps are recorded at different times, and F4 depends on worker reports arriving after routing. Short windows may therefore compare different requests. Divide each step by F0 to express it as a percentage of incoming tokens.
vLLM, SGLang, and TensorRT-LLM workers report F4 on the final response chunk of each aggregated or prefill attempt, as engine_data.kv_cache_hit with prompt_tokens and reused_tokens. reused_tokens is the engine’s per-request cached-token count, so it includes hits on host and storage tiers and KV connectors, the same tiers F2 and F3 count.
- Disaggregated decode attempts do not report, because each engine counts the KV transferred from the prefill worker as cached; read worker reuse from the
prefillphase. The exception is a decode attempt that prefills locally under conditional disaggregation, which reports underphase="decode". - Attempts take their
phasefrom the frontend’s request tracker. A standalone router (python -m dynamo.router) does not set one, so all of its attempts are labeledaggregated. - The router counts a report only when its
prompt_tokensequals the routed prompt length. Engines count multimodal prompts after placeholder expansion, so a multimodal report counts only when the router also routed on expanded tokens; TensorRT-LLM does not report multimodal requests. F2 and F3 count expanded routing tokens while F0 counts the unexpanded prompt, so multimodal ratios to F0 can exceed 100%. - With
n > 1, vLLM and SGLang do not report. Later samples reuse the prompt blocks earlier samples just cached, vLLM keeps the first buffered sample’s count when it merges outputs, and SGLang caches the prompt before running the samples. - Engines recompute the last prompt token, so a full-prefix hit reports at least one token less than F3, and up to one block less for vLLM.
- Preemption, partial-block reuse in TensorRT-LLM, and KV loads that fail under a recompute policy can also make F4 differ from F3 when the router’s estimate is accurate.
Per-Request Routing Overhead
dynamo_router_overhead_* histograms (milliseconds) track time spent in each phase of the routing decision. Registered on the frontend port with only a router_id label (the frontend’s discovery instance ID), without the dynamo_namespace, dynamo_component, and worker_id labels the router adds to its other metrics. Registered on every frontend and recorded only for KV routing decisions, so they stay at zero in non-KV modes. Not created on the standalone router.
Router Queue Metrics
These metrics have series once a KV router starts, whether or not queueing is enabled. The dynamo_frontend_router_queue_* metrics stay at zero unless queueing is enabled by --router-queue-threshold or a threshold in --router-policy-config; the dynamo_frontend_router_admission_* counters are recorded either way. They carry model, worker_type, and policy_class, but not the dynamo_namespace, dynamo_component, and worker_id labels of the other router metrics. policy_class is the resolved class: a request that names no class is reported as the default class, and a request whose class, after any request-classifier override, is unknown fails before it is queued, so these queue metrics do not count it.
Each rejected attempt is counted once in dynamo_frontend_router_admission_rejected_total, labeled due_time_passed when the request’s explicit deadline, or the budget of an edf class with controlled_delay, passes before admission, or queue_size_exceeded when a class queue request or token limit rejects it. An attempt whose response receiver is already closed when the router checks it, because the client went away, is not counted; a receiver that closes after that check can still be counted. A queue_size_exceeded rejection is also counted, by limit, in dynamo_frontend_router_queue_backpressure_total when the frontend’s selection call returns that queue rejection; a cancellation can drop the call before it returns, so the two counts can differ. They describe the same rejections: do not add them together.
KV Indexer Metrics
dynamo_component_kv_cache_events_applied appears, at zero, once the router builds a local KV index: when the worker-selection policy requests WorkerInputs::CACHE or WorkerInputs::KV_TRANSFER, and either --use-remote-indexer is not set or --router-predicted-ttl-secs is set. It counts the KV events workers publish and the router’s own predicted stores and expirations, which it makes with KV events disabled or with --router-predicted-ttl-secs. With --router-event-threads 1, TTL-based predicted events are not counted. The standalone indexer exposes the device-tier-only counter dynamo_kvrouter_kv_cache_events_applied; see Standalone KV Indexer Metrics.
Shared KV Cache
Recorded only with --shared-cache-type hicache. These carry a router_id label.
Active Sequence Synchronization
Track the active-request events each KV router applies to its view of worker load. The active_sequence_ingress metrics carry model, routing_group, and worker_type. The active_sequence_zmq_ingress metrics exist when these events travel over direct ZMQ, the default on the ZMQ event plane.
Approximate Cache
The approximate cache tracks likely-cached blocks when the router does not consume KV events. The two policy gauges are present on every KV router. The LRU metrics update only with --router-approximate-cache-policy lru, KV events disabled, and neither --use-remote-indexer nor --serve-indexer; with either of those, the router falls back to TTL.
Availability by Configuration
Not all router metrics appear in every deployment. This matrix shows which groups are registered and populated in each configuration.
Key:
- Registered and populated — the metric appears at
/metricswith real values. - Registered, always zero — the metric appears at
/metricsbut is never incremented (useful for dashboards that expect it to exist). - Not registered / Not created — the metric does not appear at
/metricsat all.
Every Process
Emitted by every process that runs a Dynamo DistributedRuntime: on the HTTP port for the frontend (not with --kserve-grpc-server), and on DYN_SYSTEM_PORT for other processes.
With the QUIC response plane (--response-plane quic or DYN_RESPONSE_PLANE=quic), every process also records:
Planner Metrics
Exposed by python -m dynamo.planner at /metrics on its own port, set by metric_reporting_prometheus_port. It defaults to the PLANNER_PROMETHEUS_PORT environment variable, or to 0, which disables the server, when that variable is unset. The Kubernetes operator sets the port to 9085. Planner metrics carry no automatic labels. See Planner Configuration.
Replicas, Traffic, and SLA
Scaling Decisions
Plugin Pipeline
Live Retune
Recorded when live retune is configured (retune in the planner configuration), and by a planner without retune while it cleans up after a removed live retune: resetting the routers it changed and deleting its Leases. See Planner Live Retune.
Standalone KV Indexer Metrics
python -m dynamo.indexer serves its own /metrics on its HTTP port (--port), with no automatic labels. See Standalone KV Indexer.
KV State Agent Metrics
python -m dynamo.kv_state_agent serves these on DYN_SYSTEM_PORT, with dynamo_namespace, dynamo_component, and worker_id.
Power Agent Metrics
Emitted by the power agent, a DaemonSet (deploy/power-agent) that enforces the per-GPU power limits set by the dynamo.nvidia.com/gpu-power-limit Pod annotation. It serves them at /metrics on --prometheus-port (PROMETHEUS_PORT); the default, 0, turns the endpoint off, and the Helm chart sets 9100.
Endpoint Picker Metrics
Emitted by the endpoint picker (EPP) when routing through the Gateway API Inference Extension, at /metrics on DYN_EPP_METRICS_PORT (default 9090). Set the port to 0 to disable the endpoint.
Buckets match the default buckets of dynamo_frontend_cached_tokens, so the two can share a dashboard. The model label is the model the pool serves, resolved once at startup rather than from the request body. Recording needs a usage block in the response. On a streaming call the client must set "stream_options": {"include_usage": true}, unless the worker’s Frontend sidecar sets DYN_ENABLE_FORCE_INCLUDE_USAGE; without a usage block, nothing is recorded.
These metrics show where the EPP spends a request’s time and when its async runtime stalls. The settings that control them, and the EPP’s diagnostic log lines, are in EPP Runtime Environment.
Global Frontend Metrics
Emitted by the Global Frontend at /metrics on its serving port. It forwards each request, as received, to a pool frontend, so it sees requests and response bytes but not tokens: inter-token latency and token counts come from the pool frontends’ dynamo_frontend_* metrics. Per-pool series are labeled by pool, the pool’s <cluster>/<namespace>/<DGD name>, and region, the region Global Discovery reports for the pool, or the cluster for a pool given with --pool. No label comes from the request body. The Global Frontend routes to the --pool pools or the pools Global Discovery reports. With --relay-view-port, a pool is routable only while a Relay view reports its model ready; with --pool-relay, this applies only to the pools given a Relay.
For a streaming request, a Dynamo pool frontend sends its headers once a worker has the request, so a request the client gives up on while a worker queues it counts under its status in dynamo_global_frontend_requests_total, not as cancelled. A non-streaming response’s headers arrive with its complete body, so a non-streaming request the client abandons before it completes counts as cancelled. local is true for a pool in the Global Frontend’s --region, false for one elsewhere, and unknown without --region; the share of requests kept in the region is the local="true" rate over the total. A body over DYN_HTTP_BODY_LIMIT_MB gets 413 before it is read and is not counted as a rejection.
If the pool frontend sends SSE keep-alives before the first token, the first keep-alive counts as the first byte: at the DYN_HTTP_SSE_KEEP_ALIVE_INTERVAL_MS interval when set, or after 15 seconds for a request whose output a reasoning parser may hold back. The time-to-first-byte and duration histograms use the dynamo_frontend_time_to_first_token_seconds and dynamo_frontend_request_duration_seconds default buckets.
With --router-policy affinity, the Global Frontend also records how it keeps sessions on their pools. These metrics count requests that carry an x-dynamo-session-id header, and they have no pool or region label. Each scrape adds to the counters what has happened since the previous scrape, so a counter series appears at the first scrape after its first event. The breaker opens after --affinity-breaker-failures consecutive backend errors or timeouts and stays open for --affinity-breaker-cooldown-ms. A request’s time in affinity storage is part of its time to first byte and duration.
Experimental. Time to select a pool for a request, in milliseconds, with either selection path (--use-selection-core or not) and any --router-policy. Counts every request that reaches selection, also one that no pool can take. No labels: a process runs one path and one policy, so compare paths across runs. Unit and buckets match the router’s dynamo_router_overhead_*_ms histograms.
Mocker Metrics
The mocker (python -m dynamo.mocker) simulates an inference engine. It emits imitation engine metrics with simulated values on DYN_SYSTEM_PORT, choosing the set by --engine-type. Every mocker metric carries dynamo_namespace, dynamo_component, dynamo_endpoint, worker_id, and model, plus the engine labels named before each table. Each one means what it means in the engine it imitates. They are Experimental because their names follow the engines’ metric names, which Dynamo does not control.
With --engine-type vllm, the default, the metrics also carry model_name and engine, which holds the DP rank:
With --engine-type sglang, the metrics also carry model_name and engine_type (unified, prefill, or decode). tp_rank, pp_rank, and moe_ep_rank are always 0, dp_rank appears only when --data-parallel-size is above 1, and is_streaming is always true:
With --engine-type trtllm, the metrics also carry model_name and engine_type, which is trtllm:
Engine Metrics
Backend workers also expose their inference engine’s own metrics at /metrics on DYN_SYSTEM_PORT. The engine defines these metrics, so their names, types, and meanings follow the engine version you run. Dynamo passes them through with the engine’s names, types, and values, and adds the dynamo_namespace, dynamo_component, dynamo_endpoint, worker_id, model, and model_name labels.
Some trtllm_* metrics are computed by Dynamo, not by TensorRT-LLM; see TensorRT-LLM Metrics Computed by Dynamo. To compare equivalent metrics across engines, see Metrics Comparison.
Related
- Metric Labels — the dimensions attached to these metrics.
- Environment Variables — variables that enable and configure metric emission.
- Operator Metrics — Kubernetes operator metrics catalog.
- Metrics Comparison — engine pass-through metrics (
vllm:*,sglang:*,trtllm_*). - OTLP Metrics Export — pushing metrics to an OpenTelemetry collector.