Global Router Configuration | NVIDIA Dynamo Documentation

The Global Router (python -m dynamo.global_router) is the cross-pool request routing component of a GlobalPlanner single-endpoint multi-pool deployment. It sits in the control DGD alongside a Frontend and GlobalPlanner, receives inference requests from the Frontend, and routes each one to the appropriate private prefill or decode pool. It is deployed only in raw DGD topologies — there is no DGDR surface for it.

Its configuration has two parts:

CLI flags — identity, addressing, and default SLA fallbacks. Defined in backend_args.py.
JSON config file — pool namespaces and SLA-to-pool routing grids. The schema is the GlobalRouterConfig dataclass in pool_selection.py. Its path is supplied via --config.

How the config is loaded

The Global Router reads a JSON file at the path given to --config. There is no operator-managed surface for this file — you create it as a Kubernetes ConfigMap, mount it into the GlobalRouter container as a volume, and point --config at the mount path.

Kubernetes

Local

Create a ConfigMap containing the JSON, reference it as a volume inside the GlobalRouter service spec of a DynamoGraphDeployment (DGD), and pass --config pointing at the mounted path.

1 # 1. ConfigMap holding the JSON config
2 apiVersion: v1
3 kind: ConfigMap
4 metadata:
5   name: gp-global-router-config
6 data:
7   global_router_config.json: |
8     {
9       "num_prefill_pools": 2,
10       "num_decode_pools": 1,
11       "prefill_pool_dynamo_namespaces": [
12         "${K8S_NAMESPACE}-gp-prefill-0",
13         "${K8S_NAMESPACE}-gp-prefill-1"
14       ],
15       "decode_pool_dynamo_namespaces": [
16         "${K8S_NAMESPACE}-gp-decode-0"
17       ],
18       "prefill_pool_selection_strategy": {
19         "ttft_min_ms": 10,  "ttft_max_ms": 3000, "ttft_resolution": 2,
20         "isl_min": 0,       "isl_max": 32000,    "isl_resolution": 2,
21         "prefill_pool_mapping": [[0, 1], [0, 1]]
22       },
23       "decode_pool_selection_strategy": {
24         "itl_min_ms": 10,  "itl_max_ms": 500,   "itl_resolution": 2,
25         "context_length_min": 0, "context_length_max": 32000,
26         "context_length_resolution": 2,
27         "decode_pool_mapping": [[0, 0], [0, 0]]
28       }
29     }
30 ---
31 # 2. Control DGD: GlobalRouter service with the ConfigMap mounted
32 apiVersion: nvidia.com/v1alpha1
33 kind: DynamoGraphDeployment
34 metadata:
35   name: gp-ctrl
36 spec:
37   services:
38     GlobalRouter:
39       componentType: default
40       replicas: 1
41       extraPodSpec:
42         volumes:
43           - name: global-router-config
44             configMap:
45               name: gp-global-router-config
46         mainContainer:
47           image: ${DYNAMO_IMAGE}
48           command:
49             - python3
50             - -m
51             - dynamo.global_router
52           args:
53             - --config
54             - /config/global_router_config.json
55             - --model-name
56             - ${MODEL_NAME}
57             - --namespace
58             - ${K8S_NAMESPACE}-gp-ctrl
59           volumeMounts:
60             - name: global-router-config
61               mountPath: /config
62               readOnly: true

See examples/global_planner/global-planner-vllm-test.yaml for the complete end-to-end example with two prefill pools and one decode pool.

Command-line flags

--config and --model-name are required. Every flag also has an environment variable fallback; the CLI value takes precedence when both are set.

--config

stringDefaults to null

Path to the JSON configuration file defining pool namespaces and selection strategy. Must be set via CLI or environment variable; startup fails if unset or empty.

Environment variable: DYN_GLOBAL_ROUTER_CONFIG

--model-name

stringDefaults to null

Model name for router registration. Must match the model name used by the pool workers. Must be set via CLI or environment variable; startup fails if unset or empty.

Environment variable: DYN_GLOBAL_ROUTER_MODEL_NAME

--namespace

stringDefaults to dynamo

Dynamo namespace the Global Router registers under. Must match the namespace used by the control DGD Frontend so that the Frontend can discover the router.

Environment variable: DYN_NAMESPACE

--component-name

stringDefaults to global_router

Component name for the Global Router’s Dynamo service registration.

Environment variable: DYN_GLOBAL_ROUTER_COMPONENT_NAME

--default-ttft-target-ms

numberDefaults to null

Default TTFT target in milliseconds for prefill pool selection. Applied to requests that carry no ttft_target in the typed router field, which the frontend derives from nvext.router. When unset, the prefill selection strategy falls back to the midpoint of its configured TTFT range. Replaces the deprecated --default-ttft-target flag.

Environment variable: DYN_GLOBAL_ROUTER_DEFAULT_TTFT_TARGET_MS

--default-itl-target-ms

numberDefaults to null

Default ITL target in milliseconds for decode pool selection. Applied to requests that carry no itl_target in the typed router field, which the frontend derives from nvext.router. When unset, the decode selection strategy falls back to the midpoint of its configured ITL range. Replaces the deprecated --default-itl-target flag.

Environment variable: DYN_GLOBAL_ROUTER_DEFAULT_ITL_TARGET_MS

JSON configuration

The file passed to --config is a JSON object matching the GlobalRouterConfig schema. Two modes are supported:

disagg (default): separate prefill and decode pools. Prefill routing uses the grid (ISL, TTFT target) → prefill pool index; decode routing currently uses (request token count, ITL target) → decode pool index through the strategy’s context_length_* fields.
agg: unified pools handling both prefill and decode. Routing uses (TTFT target, ITL target) → agg pool index, optionally extended with ISL as the leading dimension.

Both modes support optional priority-based pool overrides and priority retry.

Common fields

mode

stringDefaults to disagg

Operating mode. Determines which pool fields are required and which selection grid is active.

Allowed values: disagg agg

enable_priority_retry

booleanDefaults to false

When true, if a pool attempt raises an exception before producing its first streamed output, the router retries through progressively faster pools (lower *_pool_priorities values), from slowest to fastest. Once an attempt has streamed output, later failures are returned without retry. When false, only the grid-selected pool is attempted.

In disaggregated mode, the router can attempt decode retries, but current Dynamo backends do not support them safely: a failed decode attempt may already have caused the prefill engine to retire or drop that request’s KV cache.

Disaggregated mode fields

The following fields are required when mode is "disagg" and are ignored in "agg" mode.

num_prefill_pools

integerDefaults to null

Number of prefill pools. Must equal the length of prefill_pool_dynamo_namespaces. Required for disagg mode.

num_decode_pools

integerDefaults to null

Number of decode pools. Must equal the length of decode_pool_dynamo_namespaces. Required for disagg mode.

prefill_pool_dynamo_namespaces

[]stringDefaults to null

Ordered list of Dynamo namespaces for the prefill pool LocalRouters. Index i corresponds to pool i. Length must equal num_prefill_pools. Required for disagg mode.

decode_pool_dynamo_namespaces

[]stringDefaults to null

Ordered list of Dynamo namespaces for the decode pool LocalRouters. Index i corresponds to pool i. Length must equal num_decode_pools. Required for disagg mode.

prefill_pool_priorities

[]integerDefaults to [0, 1, ..., num_prefill_pools-1]

Priority value for each prefill pool, parallel to prefill_pool_dynamo_namespaces. Lower values indicate faster pools (higher priority). Length must equal num_prefill_pools. When omitted, defaults to pool order: pool 0 gets priority 0, pool 1 gets priority 1, and so on. Used by enable_priority_retry to build the retry order.

decode_pool_priorities

[]integerDefaults to [0, 1, ..., num_decode_pools-1]

Priority value for each decode pool. Same semantics as prefill_pool_priorities. Defaults to pool order when omitted.

prefill_pool_selection_strategy

PrefillPoolSelectionStrategyDefaults to null

Grid mapping (ISL, TTFT target) to a prefill pool index. Required for disagg mode. See PrefillPoolSelectionStrategy.

decode_pool_selection_strategy

DecodePoolSelectionStrategyDefaults to null

Grid mapping (request token count, ITL target) to a decode pool index. Required for disagg mode. See DecodePoolSelectionStrategy.

Aggregated mode fields

The following fields are required when mode is "agg" and are ignored in "disagg" mode.

num_agg_pools

integerDefaults to null

Number of aggregated pools. Must equal the length of agg_pool_dynamo_namespaces. Required for agg mode.

agg_pool_dynamo_namespaces

[]stringDefaults to null

Ordered list of Dynamo namespaces for the aggregated pool LocalRouters. Index i corresponds to pool i. Length must equal num_agg_pools. Required for agg mode.

agg_pool_priorities

[]integerDefaults to [0, 1, ..., num_agg_pools-1]

Priority value for each agg pool. Same semantics as prefill_pool_priorities. Defaults to pool order when omitted.

agg_pool_selection_strategy

AggPoolSelectionStrategyDefaults to null

Grid mapping (TTFT target, ITL target) to an agg pool index, or (ISL, TTFT target, ITL target) when all three optional isl_* fields are set. Required for agg mode. See AggPoolSelectionStrategy.

Additional types

Nested object types referenced by the JSON configuration fields above.

PrefillPoolSelectionStrategy

Routes a request to a prefill pool based on its input sequence length (ISL) and TTFT target. The routing space is a 2D grid of isl_resolution × ttft_resolution cells. At routing time the router computes isl_idx and ttft_idx by mapping the request’s ISL and TTFT target into the grid (with clamping at the boundaries), then reads prefill_pool_mapping[isl_idx][ttft_idx] to get the pool index.

ttft_min_ms

numberRequired

Lower bound of the TTFT target range, in milliseconds. Requests with a TTFT target below this value are clamped to the first TTFT column of the grid. Also accepts the deprecated key ttft_min (no _ms suffix); use ttft_min_ms in new configurations.

ttft_max_ms

numberRequired

Upper bound of the TTFT target range, in milliseconds. Must be strictly greater than ttft_min_ms. Also accepts the deprecated key ttft_max.

ttft_resolution

integerRequired

Number of TTFT grid columns (the second dimension of prefill_pool_mapping). Must be a positive integer. Each row of prefill_pool_mapping must have exactly ttft_resolution entries.

isl_min

integerRequired

Lower bound of the ISL range (input tokens). Requests with fewer tokens are clamped to the first ISL row of the grid.

isl_max

integerRequired

Upper bound of the ISL range (input tokens). Must be strictly greater than isl_min.

isl_resolution

integerRequired

Number of ISL grid rows (the first dimension of prefill_pool_mapping). Must be a positive integer. prefill_pool_mapping must have exactly isl_resolution rows.

prefill_pool_mapping

[][]integerRequired

2D array of pool indices with shape [isl_resolution][ttft_resolution]. Each entry is a zero-based prefill pool index in [0, num_prefill_pools). Row i covers the i-th ISL bucket; column j covers the j-th TTFT bucket. Out-of-range ISL or TTFT values are clamped to the nearest boundary row or column.

priority_overrides

[]PriorityPoolOverrideDefaults to []

Optional list of priority-based pool overrides. When a request carries a priority value from its agent context that matches a rule, the override’s target_pool replaces the grid result. Rules are evaluated in list order; first match wins. See PriorityPoolOverride.

DecodePoolSelectionStrategy

Routes a request to a decode pool based on its context_length bucket and ITL target. The current handler computes context_length as the number of request token_ids at decode routing time; it does not add subsequently generated tokens. The routing space is a 2D grid of context_length_resolution × itl_resolution cells. decode_pool_mapping[ctx_idx][itl_idx] gives the pool index.

itl_min_ms

numberRequired

Lower bound of the ITL target range, in milliseconds. Also accepts the deprecated key itl_min.

itl_max_ms

numberRequired

Upper bound of the ITL target range, in milliseconds. Must be strictly greater than itl_min_ms. Also accepts the deprecated key itl_max.

itl_resolution

integerRequired

Number of ITL grid columns (the second dimension of decode_pool_mapping). Must be a positive integer.

context_length_min

integerRequired

Lower bound of the request-token-count range used by the current handler as context_length. Smaller requests are clamped to the first row.

context_length_max

integerRequired

Upper bound of the request-token-count range used as context_length. Must be strictly greater than context_length_min.

context_length_resolution

integerRequired

Number of context-length grid rows (the first dimension of decode_pool_mapping). Must be a positive integer.

decode_pool_mapping

[][]integerRequired

2D array of pool indices with shape [context_length_resolution][itl_resolution]. Each entry is a zero-based decode pool index in [0, num_decode_pools). Row i covers the i-th context-length bucket; column j covers the j-th ITL bucket.

priority_overrides

[]PriorityPoolOverrideDefaults to []

Optional priority-based pool overrides. See PriorityPoolOverride.

AggPoolSelectionStrategy

Routes a request to an aggregated pool based on both its TTFT and ITL targets. Used in mode: "agg" where each pool handles both prefill and decode. By default, the routing space is a 2D grid of ttft_resolution × itl_resolution cells. Configuring all three optional isl_* fields adds a leading ISL dimension. This dimension works whether or not chunked prefill is enabled.

ttft_min_ms

numberRequired

Lower bound of the TTFT target range, in milliseconds. Also accepts the deprecated key ttft_min.

ttft_max_ms

numberRequired

Upper bound of the TTFT target range, in milliseconds. Must be strictly greater than ttft_min_ms. Also accepts the deprecated key ttft_max.

ttft_resolution

integerRequired

Number of TTFT grid rows (the first dimension of agg_pool_mapping). Must be a positive integer.

itl_min_ms

numberRequired

Lower bound of the ITL target range, in milliseconds. Also accepts the deprecated key itl_min.

itl_max_ms

numberRequired

Upper bound of the ITL target range, in milliseconds. Must be strictly greater than itl_min_ms. Also accepts the deprecated key itl_max.

itl_resolution

integerRequired

Number of ITL grid columns (the second dimension of agg_pool_mapping). Must be a positive integer.

isl_min

integerDefaults to null

Optional lower bound for aggregated-mode input sequence length. Set this together with isl_max and isl_resolution to enable 3D selection.

isl_max

integerDefaults to null

Optional upper bound for aggregated-mode input sequence length. Must be greater than isl_min when ISL selection is enabled.

isl_resolution

integerDefaults to null

Optional number of aggregated-mode ISL buckets. Must be a positive integer when ISL selection is enabled.

agg_pool_mapping

[][]integer | [][][]integerRequired

Without isl_*, a 2D array with shape [ttft_resolution][itl_resolution]. With all three isl_* fields, a 3D array with shape [isl_resolution][ttft_resolution][itl_resolution]. Each entry is a zero-based agg pool index in [0, num_agg_pools). ISL, TTFT, and ITL values outside their configured ranges are clamped to the nearest bucket.

priority_overrides

[]PriorityPoolOverrideDefaults to []

Optional priority-based pool overrides. See PriorityPoolOverride.

PriorityPoolOverride

A single priority-range rule in a strategy’s priority_overrides list. When a request carries a priority value from its agent context, each rule is evaluated in list order; the first matching rule’s target_pool overrides the grid result. If no rule matches, the grid result is used.

min_priority

integerRequired

Inclusive lower bound of the priority range to match.

max_priority

integerRequired

Inclusive upper bound of the priority range to match. Must be greater than or equal to min_priority.

target_pool

integerRequired

Zero-based pool index to route to when the request priority falls within [min_priority, max_priority]. Must be a valid pool index for the enclosing strategy (0 to num_prefill_pools - 1, num_decode_pools - 1, or num_agg_pools - 1).

Validation rules

The router validates the configuration at startup and exits if any rule is violated:

mode must be "disagg" or "agg".
disagg mode: num_prefill_pools, num_decode_pools, prefill_pool_dynamo_namespaces, decode_pool_dynamo_namespaces, prefill_pool_selection_strategy, and decode_pool_selection_strategy are all required.
agg mode: num_agg_pools, agg_pool_dynamo_namespaces, and agg_pool_selection_strategy are all required.
Namespace list lengths must equal their corresponding pool counts (prefill_pool_dynamo_namespaces length = num_prefill_pools, and so on).
*_pool_priorities lists must have the same length as their pool count when provided.
ttft_min_ms must be strictly less than ttft_max_ms; likewise for itl_min_ms/itl_max_ms and context_length_min/context_length_max, and isl_min/isl_max.
All *_resolution values must be positive integers.
prefill_pool_mapping must have exactly isl_resolution rows, each of length ttft_resolution; all pool indices must be in [0, num_prefill_pools).
decode_pool_mapping must have exactly context_length_resolution rows, each of length itl_resolution; all pool indices must be in [0, num_decode_pools).
Without aggregated isl_* fields, agg_pool_mapping must have exactly ttft_resolution rows, each of length itl_resolution; all pool indices must be in [0, num_agg_pools).
For aggregated selection, isl_min, isl_max, and isl_resolution must be omitted together or set together. When set, isl_min < isl_max, isl_resolution is positive, and agg_pool_mapping must have shape [isl_resolution][ttft_resolution][itl_resolution] instead of the 2D shape above.
In each priority_overrides entry: min_priority must be ≤ max_priority, and target_pool must be a valid pool index for the enclosing strategy.