> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Global Router Configuration The Global Router (`python -m dynamo.global_router`) is the cross-pool request routing component of a [GlobalPlanner single-endpoint multi-pool deployment](/dynamo/dev/knowledge-base/modular-components/planner/global-planner-guide). It sits in the control DGD alongside a Frontend and GlobalPlanner, receives inference requests from the Frontend, and routes each one to the appropriate private prefill or decode pool. It is deployed only in raw DGD topologies — there is no DGDR surface for it. Its configuration has two parts: * **CLI flags** — identity, addressing, and default SLA fallbacks. Defined in [`backend_args.py`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/global_router/backend_args.py). * **JSON config file** — pool namespaces and SLA-to-pool routing grids. The schema is the [`GlobalRouterConfig`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/global_router/pool_selection.py) dataclass in `pool_selection.py`. Its path is supplied via `--config`. ## How the config is loaded The Global Router reads a JSON file at the path given to `--config`. There is no operator-managed surface for this file — you create it as a Kubernetes ConfigMap, mount it into the GlobalRouter container as a volume, and point `--config` at the mount path. #### Kubernetes Create a ConfigMap containing the JSON, reference it as a volume inside the GlobalRouter service spec of a [DynamoGraphDeployment](/dynamo/dev/reference/api/kubernetes/dynamo-graph-deployment) (DGD), and pass `--config` pointing at the mounted path. ```yaml # 1. ConfigMap holding the JSON config apiVersion: v1 kind: ConfigMap metadata: name: gp-global-router-config data: global_router_config.json: | { "num_prefill_pools": 2, "num_decode_pools": 1, "prefill_pool_dynamo_namespaces": [ "${K8S_NAMESPACE}-gp-prefill-0", "${K8S_NAMESPACE}-gp-prefill-1" ], "decode_pool_dynamo_namespaces": [ "${K8S_NAMESPACE}-gp-decode-0" ], "prefill_pool_selection_strategy": { "ttft_min_ms": 10, "ttft_max_ms": 3000, "ttft_resolution": 2, "isl_min": 0, "isl_max": 32000, "isl_resolution": 2, "prefill_pool_mapping": [[0, 1], [0, 1]] }, "decode_pool_selection_strategy": { "itl_min_ms": 10, "itl_max_ms": 500, "itl_resolution": 2, "context_length_min": 0, "context_length_max": 32000, "context_length_resolution": 2, "decode_pool_mapping": [[0, 0], [0, 0]] } } --- # 2. Control DGD: GlobalRouter service with the ConfigMap mounted apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeployment metadata: name: gp-ctrl spec: services: GlobalRouter: componentType: default replicas: 1 extraPodSpec: volumes: - name: global-router-config configMap: name: gp-global-router-config mainContainer: image: ${DYNAMO_IMAGE} command: - python3 - -m - dynamo.global_router args: - --config - /config/global_router_config.json - --model-name - ${MODEL_NAME} - --namespace - ${K8S_NAMESPACE}-gp-ctrl volumeMounts: - name: global-router-config mountPath: /config readOnly: true ``` See [examples/global\_planner/global-planner-vllm-test.yaml](https://github.com/ai-dynamo/dynamo/blob/main/examples/global_planner/global-planner-vllm-test.yaml) for the complete end-to-end example with two prefill pools and one decode pool. #### Local Write a JSON file and pass `--config` on the command line: ```bash python -m dynamo.global_router \ --config global_router_config.json \ --model-name meta-llama/Llama-3.3-70B-Instruct \ --namespace dynamo-ctrl ``` ## Command-line flags `--config` and `--model-name` are required. Every flag also has an environment variable fallback; the CLI value takes precedence when both are set. **`--config`** `string` — default: null Path to the JSON configuration file defining pool namespaces and selection strategy. Must be set via CLI or environment variable; startup fails if unset or empty. Environment variable: `DYN_GLOBAL_ROUTER_CONFIG` --- **`--model-name`** `string` — default: null Model name for router registration. Must match the model name used by the pool workers. Must be set via CLI or environment variable; startup fails if unset or empty. Environment variable: `DYN_GLOBAL_ROUTER_MODEL_NAME` --- **`--namespace`** `string` — default: dynamo Dynamo namespace the Global Router registers under. Must match the namespace used by the control DGD Frontend so that the Frontend can discover the router. Environment variable: `DYN_NAMESPACE` --- **`--component-name`** `string` — default: global\_router Component name for the Global Router's Dynamo service registration. Environment variable: `DYN_GLOBAL_ROUTER_COMPONENT_NAME` --- **`--default-ttft-target-ms`** `number` — default: null Default TTFT target in milliseconds for prefill pool selection. Applied to requests that carry no `ttft_target` in the typed `router` field, which the frontend derives from `nvext.router`. When unset, the prefill selection strategy falls back to the midpoint of its configured TTFT range. Replaces the deprecated `--default-ttft-target` flag. Environment variable: `DYN_GLOBAL_ROUTER_DEFAULT_TTFT_TARGET_MS` --- **`--default-itl-target-ms`** `number` — default: null Default ITL target in milliseconds for decode pool selection. Applied to requests that carry no `itl_target` in the typed `router` field, which the frontend derives from `nvext.router`. When unset, the decode selection strategy falls back to the midpoint of its configured ITL range. Replaces the deprecated `--default-itl-target` flag. Environment variable: `DYN_GLOBAL_ROUTER_DEFAULT_ITL_TARGET_MS` --- ## JSON configuration The file passed to `--config` is a JSON object matching the `GlobalRouterConfig` schema. Two modes are supported: * **`disagg`** (default): separate prefill and decode pools. Prefill routing uses the grid `(ISL, TTFT target) → prefill pool index`; decode routing currently uses `(request token count, ITL target) → decode pool index` through the strategy's `context_length_*` fields. * **`agg`**: unified pools handling both prefill and decode. Routing uses `(TTFT target, ITL target) → agg pool index`, optionally extended with ISL as the leading dimension. Both modes support optional priority-based pool overrides and priority retry. ### Common fields **`mode`** `string` — default: disagg Operating mode. Determines which pool fields are required and which selection grid is active. Allowed values: disagg agg --- **`enable_priority_retry`** `boolean` — default: false When `true`, if a pool attempt raises an exception before producing its first streamed output, the router retries through progressively faster pools (lower `*_pool_priorities` values), from slowest to fastest. Once an attempt has streamed output, later failures are returned without retry. When `false`, only the grid-selected pool is attempted. In disaggregated mode, the router can attempt decode retries, but current Dynamo backends do not support them safely: a failed decode attempt may already have caused the prefill engine to retire or drop that request's KV cache. --- ### Disaggregated mode fields The following fields are required when `mode` is `"disagg"` and are ignored in `"agg"` mode. **`num_prefill_pools`** `integer` — default: null Number of prefill pools. Must equal the length of `prefill_pool_dynamo_namespaces`. Required for `disagg` mode. --- **`num_decode_pools`** `integer` — default: null Number of decode pools. Must equal the length of `decode_pool_dynamo_namespaces`. Required for `disagg` mode. --- **`prefill_pool_dynamo_namespaces`** `[]string` — default: null Ordered list of Dynamo namespaces for the prefill pool LocalRouters. Index `i` corresponds to pool `i`. Length must equal `num_prefill_pools`. Required for `disagg` mode. --- **`decode_pool_dynamo_namespaces`** `[]string` — default: null Ordered list of Dynamo namespaces for the decode pool LocalRouters. Index `i` corresponds to pool `i`. Length must equal `num_decode_pools`. Required for `disagg` mode. --- **`prefill_pool_priorities`** `[]integer` — default: \[0, 1, ..., num\_prefill\_pools-1] Priority value for each prefill pool, parallel to `prefill_pool_dynamo_namespaces`. Lower values indicate faster pools (higher priority). Length must equal `num_prefill_pools`. When omitted, defaults to pool order: pool 0 gets priority 0, pool 1 gets priority 1, and so on. Used by `enable_priority_retry` to build the retry order. --- **`decode_pool_priorities`** `[]integer` — default: \[0, 1, ..., num\_decode\_pools-1] Priority value for each decode pool. Same semantics as `prefill_pool_priorities`. Defaults to pool order when omitted. --- **`prefill_pool_selection_strategy`** `PrefillPoolSelectionStrategy` — default: null Grid mapping `(ISL, TTFT target)` to a prefill pool index. Required for `disagg` mode. See [PrefillPoolSelectionStrategy](#prefillpoolselectionstrategy). --- **`decode_pool_selection_strategy`** `DecodePoolSelectionStrategy` — default: null Grid mapping `(request token count, ITL target)` to a decode pool index. Required for `disagg` mode. See [DecodePoolSelectionStrategy](#decodepoolselectionstrategy). --- ### Aggregated mode fields The following fields are required when `mode` is `"agg"` and are ignored in `"disagg"` mode. **`num_agg_pools`** `integer` — default: null Number of aggregated pools. Must equal the length of `agg_pool_dynamo_namespaces`. Required for `agg` mode. --- **`agg_pool_dynamo_namespaces`** `[]string` — default: null Ordered list of Dynamo namespaces for the aggregated pool LocalRouters. Index `i` corresponds to pool `i`. Length must equal `num_agg_pools`. Required for `agg` mode. --- **`agg_pool_priorities`** `[]integer` — default: \[0, 1, ..., num\_agg\_pools-1] Priority value for each agg pool. Same semantics as `prefill_pool_priorities`. Defaults to pool order when omitted. --- **`agg_pool_selection_strategy`** `AggPoolSelectionStrategy` — default: null Grid mapping `(TTFT target, ITL target)` to an agg pool index, or `(ISL, TTFT target, ITL target)` when all three optional `isl_*` fields are set. Required for `agg` mode. See [AggPoolSelectionStrategy](#aggpoolselectionstrategy). --- ## Additional types Nested object types referenced by the JSON configuration fields above. ### PrefillPoolSelectionStrategy Routes a request to a prefill pool based on its input sequence length (ISL) and TTFT target. The routing space is a 2D grid of `isl_resolution × ttft_resolution` cells. At routing time the router computes `isl_idx` and `ttft_idx` by mapping the request's ISL and TTFT target into the grid (with clamping at the boundaries), then reads `prefill_pool_mapping[isl_idx][ttft_idx]` to get the pool index. **`ttft_min_ms`** `number` — required Lower bound of the TTFT target range, in milliseconds. Requests with a TTFT target below this value are clamped to the first TTFT column of the grid. Also accepts the deprecated key `ttft_min` (no `_ms` suffix); use `ttft_min_ms` in new configurations. --- **`ttft_max_ms`** `number` — required Upper bound of the TTFT target range, in milliseconds. Must be strictly greater than `ttft_min_ms`. Also accepts the deprecated key `ttft_max`. --- **`ttft_resolution`** `integer` — required Number of TTFT grid columns (the second dimension of `prefill_pool_mapping`). Must be a positive integer. Each row of `prefill_pool_mapping` must have exactly `ttft_resolution` entries. --- **`isl_min`** `integer` — required Lower bound of the ISL range (input tokens). Requests with fewer tokens are clamped to the first ISL row of the grid. --- **`isl_max`** `integer` — required Upper bound of the ISL range (input tokens). Must be strictly greater than `isl_min`. --- **`isl_resolution`** `integer` — required Number of ISL grid rows (the first dimension of `prefill_pool_mapping`). Must be a positive integer. `prefill_pool_mapping` must have exactly `isl_resolution` rows. --- **`prefill_pool_mapping`** `[][]integer` — required 2D array of pool indices with shape `[isl_resolution][ttft_resolution]`. Each entry is a zero-based prefill pool index in `[0, num_prefill_pools)`. Row `i` covers the `i`-th ISL bucket; column `j` covers the `j`-th TTFT bucket. Out-of-range ISL or TTFT values are clamped to the nearest boundary row or column. --- **`priority_overrides`** `[]PriorityPoolOverride` — default: \[] Optional list of priority-based pool overrides. When a request carries a priority value from its agent context that matches a rule, the override's `target_pool` replaces the grid result. Rules are evaluated in list order; first match wins. See [PriorityPoolOverride](#prioritypooloverride). --- ### DecodePoolSelectionStrategy Routes a request to a decode pool based on its `context_length` bucket and ITL target. The current handler computes `context_length` as the number of request `token_ids` at decode routing time; it does not add subsequently generated tokens. The routing space is a 2D grid of `context_length_resolution × itl_resolution` cells. `decode_pool_mapping[ctx_idx][itl_idx]` gives the pool index. **`itl_min_ms`** `number` — required Lower bound of the ITL target range, in milliseconds. Also accepts the deprecated key `itl_min`. --- **`itl_max_ms`** `number` — required Upper bound of the ITL target range, in milliseconds. Must be strictly greater than `itl_min_ms`. Also accepts the deprecated key `itl_max`. --- **`itl_resolution`** `integer` — required Number of ITL grid columns (the second dimension of `decode_pool_mapping`). Must be a positive integer. --- **`context_length_min`** `integer` — required Lower bound of the request-token-count range used by the current handler as `context_length`. Smaller requests are clamped to the first row. --- **`context_length_max`** `integer` — required Upper bound of the request-token-count range used as `context_length`. Must be strictly greater than `context_length_min`. --- **`context_length_resolution`** `integer` — required Number of context-length grid rows (the first dimension of `decode_pool_mapping`). Must be a positive integer. --- **`decode_pool_mapping`** `[][]integer` — required 2D array of pool indices with shape `[context_length_resolution][itl_resolution]`. Each entry is a zero-based decode pool index in `[0, num_decode_pools)`. Row `i` covers the `i`-th context-length bucket; column `j` covers the `j`-th ITL bucket. --- **`priority_overrides`** `[]PriorityPoolOverride` — default: \[] Optional priority-based pool overrides. See [PriorityPoolOverride](#prioritypooloverride). --- ### AggPoolSelectionStrategy Routes a request to an aggregated pool based on both its TTFT and ITL targets. Used in `mode: "agg"` where each pool handles both prefill and decode. By default, the routing space is a 2D grid of `ttft_resolution × itl_resolution` cells. Configuring all three optional `isl_*` fields adds a leading ISL dimension. This dimension works whether or not chunked prefill is enabled. **`ttft_min_ms`** `number` — required Lower bound of the TTFT target range, in milliseconds. Also accepts the deprecated key `ttft_min`. --- **`ttft_max_ms`** `number` — required Upper bound of the TTFT target range, in milliseconds. Must be strictly greater than `ttft_min_ms`. Also accepts the deprecated key `ttft_max`. --- **`ttft_resolution`** `integer` — required Number of TTFT grid rows (the first dimension of `agg_pool_mapping`). Must be a positive integer. --- **`itl_min_ms`** `number` — required Lower bound of the ITL target range, in milliseconds. Also accepts the deprecated key `itl_min`. --- **`itl_max_ms`** `number` — required Upper bound of the ITL target range, in milliseconds. Must be strictly greater than `itl_min_ms`. Also accepts the deprecated key `itl_max`. --- **`itl_resolution`** `integer` — required Number of ITL grid columns (the second dimension of `agg_pool_mapping`). Must be a positive integer. --- **`isl_min`** `integer` — default: null Optional lower bound for aggregated-mode input sequence length. Set this together with `isl_max` and `isl_resolution` to enable 3D selection. --- **`isl_max`** `integer` — default: null Optional upper bound for aggregated-mode input sequence length. Must be greater than `isl_min` when ISL selection is enabled. --- **`isl_resolution`** `integer` — default: null Optional number of aggregated-mode ISL buckets. Must be a positive integer when ISL selection is enabled. --- **`agg_pool_mapping`** `[][]integer | [][][]integer` — required Without `isl_*`, a 2D array with shape `[ttft_resolution][itl_resolution]`. With all three `isl_*` fields, a 3D array with shape `[isl_resolution][ttft_resolution][itl_resolution]`. Each entry is a zero-based agg pool index in `[0, num_agg_pools)`. ISL, TTFT, and ITL values outside their configured ranges are clamped to the nearest bucket. --- **`priority_overrides`** `[]PriorityPoolOverride` — default: \[] Optional priority-based pool overrides. See [PriorityPoolOverride](#prioritypooloverride). --- ### PriorityPoolOverride A single priority-range rule in a strategy's `priority_overrides` list. When a request carries a priority value from its agent context, each rule is evaluated in list order; the first matching rule's `target_pool` overrides the grid result. If no rule matches, the grid result is used. **`min_priority`** `integer` — required Inclusive lower bound of the priority range to match. --- **`max_priority`** `integer` — required Inclusive upper bound of the priority range to match. Must be greater than or equal to `min_priority`. --- **`target_pool`** `integer` — required Zero-based pool index to route to when the request priority falls within `[min_priority, max_priority]`. Must be a valid pool index for the enclosing strategy (0 to `num_prefill_pools - 1`, `num_decode_pools - 1`, or `num_agg_pools - 1`). --- ## Validation rules The router validates the configuration at startup and exits if any rule is violated: * `mode` must be `"disagg"` or `"agg"`. * **`disagg` mode**: `num_prefill_pools`, `num_decode_pools`, `prefill_pool_dynamo_namespaces`, `decode_pool_dynamo_namespaces`, `prefill_pool_selection_strategy`, and `decode_pool_selection_strategy` are all required. * **`agg` mode**: `num_agg_pools`, `agg_pool_dynamo_namespaces`, and `agg_pool_selection_strategy` are all required. * Namespace list lengths must equal their corresponding pool counts (`prefill_pool_dynamo_namespaces` length = `num_prefill_pools`, and so on). * `*_pool_priorities` lists must have the same length as their pool count when provided. * `ttft_min_ms` must be strictly less than `ttft_max_ms`; likewise for `itl_min_ms`/`itl_max_ms` and `context_length_min`/`context_length_max`, and `isl_min`/`isl_max`. * All `*_resolution` values must be positive integers. * `prefill_pool_mapping` must have exactly `isl_resolution` rows, each of length `ttft_resolution`; all pool indices must be in `[0, num_prefill_pools)`. * `decode_pool_mapping` must have exactly `context_length_resolution` rows, each of length `itl_resolution`; all pool indices must be in `[0, num_decode_pools)`. * Without aggregated `isl_*` fields, `agg_pool_mapping` must have exactly `ttft_resolution` rows, each of length `itl_resolution`; all pool indices must be in `[0, num_agg_pools)`. * For aggregated selection, `isl_min`, `isl_max`, and `isl_resolution` must be omitted together or set together. When set, `isl_min < isl_max`, `isl_resolution` is positive, and `agg_pool_mapping` must have shape `[isl_resolution][ttft_resolution][itl_resolution]` instead of the 2D shape above. * In each `priority_overrides` entry: `min_priority` must be ≤ `max_priority`, and `target_pool` must be a valid pool index for the enclosing strategy. ## Related pages #### [Global Planner Guide](/dynamo/dev/knowledge-base/modular-components/planner/global-planner-guide) How to deploy a single-endpoint multi-pool topology with GlobalRouter and GlobalPlanner, including the ConfigMap + DGD assembly workflow. #### [Router Guide](/dynamo/dev/knowledge-base/modular-components/router/router-guide) Concepts, routing modes, and operational details for the Dynamo router family. #### [Planner Configuration](/dynamo/dev/reference/components/planner-configuration) Field reference for the PlannerConfig JSON object used by pool-local Planners in a GlobalPlanner deployment. #### [Router Configuration](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning) CLI flags and tuning knobs for the LocalRouter component used inside each pool DGD.