> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server.

# LLM Request Router Load Balancing

Use this guide to deploy and validate Stargate load-balancer configuration in
self-managed NVCF. For the complete `lb-config.json` schema, algorithm
behavior, defaults, and tuning fields, see the
[Stargate load balancer configuration](https://github.com/NVIDIA/nvcf/blob/main/src/libraries/rust/stargate/docs/load-balancer-configuration.md).

This guide covers NVCF deployment ownership and trusted request metadata. It
does not redefine the Stargate schema.

## Configure the self-managed stack

Set the request-router configuration in the Helmfile environment:

```yaml
addons:
  llm:
    enabled: true
    requestRouter:
      loadBalancer:
        config: |
          {
            "default": "power-of-two",
            "request_algorithms": {
              "round-robin": "round-robin"
            }
          }
```

The self-managed stack passes
`addons.llm.requestRouter.loadBalancer` to the request-router chart as
`llmRequestRouter.loadBalancer`.

The chart supports two configuration sources:

| Value | Owner and behavior |
| --- | --- |
| `loadBalancer.config` | The chart creates the `llm-request-router-lb` ConfigMap, stores the JSON under `lb-config.json`, mounts it read-only at `/etc/llm-request-router`, and passes `--lb-config-path=/etc/llm-request-router/lb-config.json`. |
| `loadBalancer.configPath` | The chart only passes `--lb-config-path=<path>`. The operator must add and maintain the file mount by another mechanism. |

Inline `config` takes precedence when both values are set. When neither value
is set, Stargate uses its built-in `power-of-two` default.

Stargate reads and validates the file only during process startup. The
StatefulSet does not include a load-balancer ConfigMap checksum in its pod
template. After a ConfigMap-only update, restart the StatefulSet so every
replica loads the same configuration.

## Distinguish router algorithms from nvcf-cli routing methods

Algorithm availability is enforced at separate layers:

| Layer | Input contract | `pulsar-multiregion` support |
| --- | --- | --- |
| `lb-config.json` | Canonical Stargate algorithm names | Supported as the default, a model algorithm, or a `request_algorithms` entry. |
| Current `nvcf-cli` `llmConfig.routingMethod` | `round_robin`, `power_of_two`, `groq_multiregion`, `pulsar`, or `random` | `pulsar_multiregion` is not accepted. |
| LLM API Gateway | Nonblank routing method from authenticated model metadata | Trimmed, then forwarded as `x-routing-method` without algorithm validation. |
| Stargate `x-routing-method` | Case-insensitive algorithm name; multiword names may use hyphens or underscores | Normalized, then accepted only when it matches the effective algorithm or a model or top-level `request_algorithms` entry. Otherwise, Stargate returns HTTP `400`. |

For example, when `power-of-two` is the effective algorithm,
`groq_multiregion` requires a `groq-multiregion` entry in
`request_algorithms`.

Stargate can use `pulsar-multiregion` as its default or a model algorithm when
no request override is sent. Selecting it through function model metadata
requires a control-plane API and client path that accepts
`pulsar_multiregion`. The current `nvcf-cli` does not provide that path.

## Keep router headers trusted

The LLM API Gateway resolves router-facing headers from authenticated function
metadata and the parsed request:

| Header | NVCF source |
| --- | --- |
| `x-request-id` | Gateway request context. |
| `x-routing-key` | Authorized function routing key. |
| `x-model` | Routed model name normalized from the request. |
| `x-routing-method` | Function model `llmConfig.routingMethod`. |
| `x-input-tokens` | Gateway input-token estimate. |
| `x-cache-affinity-key` | Gateway session-affinity derivation. |
| `x-priority` | Gateway-owned priority when one is resolved. |
| `x-request-slo-ms` | Not generated by the stock gateway. A trusted upstream can supply the queue-SLO duration on native proxy paths. |
| `x-max-wait-ms` | Not generated by the stock gateway. A trusted upstream can supply the routing wait budget on native proxy paths. |
| `x-stargate-max-wait-ms` | Not generated by the stock gateway. A trusted upstream can supply the proxy retry budget on native proxy paths. |

The chat-completions provider creates a new outbound request from gateway
context, so it drops all three request deadline headers. Responses and
embeddings use native proxy paths that clone inbound headers and overwrite the
routing headers for which the gateway resolved a value. These paths always
remove caller-supplied `x-priority`, but do not unconditionally remove every
other router-facing header.

<Warning>
The stock LLM invocation HTTPRoute does not strip the other router-facing
headers. Do not expose the route to untrusted callers until a managed ingress
policy removes them before the request reaches the LLM API Gateway.
</Warning>

For a separately managed Gateway API route, add this filter to the rule that
forwards to `llm-api-gateway`:

```yaml
filters:
  - type: RequestHeaderModifier
    requestHeaderModifier:
      remove:
        - x-request-id
        - x-routing-key
        - x-model
        - x-routing-method
        - x-input-tokens
        - x-cache-affinity-key
        - x-priority
        - x-request-slo-ms
        - x-max-wait-ms
        - x-stargate-max-wait-ms
```

The stock gateway-routes chart does not expose a value for this filter. Use an
equivalent policy at an external edge or maintain a route override. Preserve
`x-multi-turn-session-id`; it is the supported client-facing session header.
See the
[Gateway API header modifier guide](https://gateway-api.sigs.k8s.io/guides/user-guides/http-header-modifier/)
for filter semantics.

The gateway derives `x-cache-affinity-key` for chat-completions and Responses
requests when affinity applies. It does not derive affinity for embeddings.
Do not set `require_cache_affinity_key` on a model that serves
`/v1/embeddings` unless another trusted gateway supplies the key.

Stargate returns HTTP `400` for a blank, unknown, or configured-but-unavailable
`x-routing-method`. It also returns HTTP `400` when a required router header is
missing or a numeric header is invalid.

## Apply and roll out

Render the chart before applying it:

```bash
helm template llm-request-router \
  deploy/helm/llm-request-router/llm-request-router \
  --values <request-router-values.yaml>
```

Confirm that the rendered StatefulSet has the expected
`--lb-config-path` argument and that inline JSON creates one ConfigMap with the
`lb-config.json` key.

If the LLM route accepts untrusted traffic, inspect the rendered or live
HTTPRoute and confirm that the trusted-header filter is present:

```bash
kubectl get httproute -n <gateway-namespace> llm-invocation -o yaml
```

Apply the self-managed environment from the unpacked stack directory:

```bash
cd path/to/nvcf-self-managed-stack
make apply HELMFILE_ENV=<environment-name>
```

For an inline configuration, inspect the live file and start argument:

```bash
kubectl get configmap -n nvcf llm-request-router-lb \
  -o jsonpath='{.data.lb-config\.json}'
kubectl get statefulset -n nvcf llm-request-router \
  -o jsonpath='{.spec.template.spec.containers[0].args}'
```

Restart after a ConfigMap-only change, then wait for all replicas:

```bash
kubectl rollout restart statefulset/llm-request-router -n nvcf
kubectl rollout status statefulset/llm-request-router -n nvcf
kubectl get pods -n nvcf -l app.kubernetes.io/name=llm-request-router
```

Confirm every listed pod was recreated after the ConfigMap update. For each
pod, check for the `load balancer config loaded` startup log and compare its
reported default and model-override count:

```bash
kubectl logs -n nvcf <request-router-pod> \
  | grep 'load balancer config loaded'
```

Do not continue if a pod predates the update, lacks a successful load log, or
reports a different configuration summary. Stable affinity requires the same
mounted configuration, seed, and candidate view across replicas.

## Validate routing

Use a deployed function whose `routingMethod` is the effective configured
algorithm or is present in `request_algorithms`.

1. Invoke without changing the function routing method and confirm success.
2. Update the function to a method present in `request_algorithms` and confirm
   success.
3. Try a method accepted by `nvcf-cli` that is neither the configured
   algorithm nor present in `request_algorithms`; confirm that Stargate returns
   HTTP `400`.
4. For an affinity-aware method, repeat a supported multi-turn request with
   the returned `x-multi-turn-session-id`.
5. Exercise a failed or saturated backend and confirm selection and retry
   counters change.

## Observe the request router

The chart exposes `llm-request-router:9090/metrics` when request-router metrics
are enabled. The current chart passes `--metrics-port` and uses Stargate's
default `stargate_` metric prefix.

See [LLM Request Router Metrics](/nvcf/dev/llm-request-router) for
metric names, labels, and scrape configuration.

## Troubleshoot

| Symptom | Check |
| --- | --- |
| Pod does not start after a configuration change | Inspect request-router logs for file read, JSON parse, unknown field, or algorithm factory errors. |
| HTTP `400` before backend selection | Compare the function `routingMethod` with the configured algorithm and `request_algorithms`. Check required and numeric gateway headers. |
| HTTP `400` for affinity-aware routing | Confirm the gateway generated a nonblank affinity key when `require_cache_affinity_key` is enabled. |
| HTTP `503` with no eligible candidates | Confirm pylons are registered and publish the capacity, queue, and optional KV-cache statistics required by the algorithm. |
| New ConfigMap value has no effect | Confirm the pod creation time. Restart the StatefulSet because Stargate does not reload the file. |
| Unexpected fallback or retries | Compare routing selection, proxy attempt, retry, and retry-exhaustion metrics. Check pylon retry reasons. |
| Affinity differs between replicas | Compare the ConfigMap, pod creation times, startup summaries, seed values, and registered candidate set. |

Use these logs together:

```bash
kubectl logs -n nvcf statefulset/llm-request-router \
  --all-pods=true --tail=100
kubectl logs -n nvcf deploy/llm-api-gateway --tail=100
kubectl logs -n nvcf-backend <function-pod> -c llm-worker --tail=100
```