LLM Request Router Load Balancing

Use this guide to deploy and validate Stargate load-balancer configuration in self-managed NVCF. For the complete lb-config.json schema, algorithm behavior, defaults, and tuning fields, see the Stargate load balancer configuration.

This guide covers NVCF deployment ownership and trusted request metadata. It does not redefine the Stargate schema.

Configure the self-managed stack

Set the request-router configuration in the Helmfile environment:

1
addons:
2
  llm:
3
    enabled: true
4
    requestRouter:
5
      loadBalancer:
6
        config: |
7
          {
8
            "default": "power-of-two",
9
            "request_algorithms": {
10
              "round-robin": "round-robin"
11
            }
12
          }

The self-managed stack passes addons.llm.requestRouter.loadBalancer to the request-router chart as llmRequestRouter.loadBalancer.

The chart supports two configuration sources:

Inline config takes precedence when both values are set. When neither value is set, Stargate uses its built-in power-of-two default.

Stargate reads and validates the file only during process startup. The StatefulSet does not include a load-balancer ConfigMap checksum in its pod template. After a ConfigMap-only update, restart the StatefulSet so every replica loads the same configuration.

Distinguish router algorithms from nvcf-cli routing methods

Algorithm availability is enforced at separate layers:

For example, when power-of-two is the effective algorithm, groq_multiregion requires a groq-multiregion entry in request_algorithms.

Stargate can use pulsar-multiregion as its default or a model algorithm when no request override is sent. Selecting it through function model metadata requires a control-plane API and client path that accepts pulsar_multiregion. The current nvcf-cli does not provide that path.

Keep router headers trusted

The LLM API Gateway resolves router-facing headers from authenticated function metadata and the parsed request:

The chat-completions provider creates a new outbound request from gateway context, so it drops all three request deadline headers. Responses and embeddings use native proxy paths that clone inbound headers and overwrite the routing headers for which the gateway resolved a value. These paths always remove caller-supplied x-priority, but do not unconditionally remove every other router-facing header.

For a separately managed Gateway API route, add this filter to the rule that forwards to llm-api-gateway:

1
filters:
2
  - type: RequestHeaderModifier
3
    requestHeaderModifier:
4
      remove:
5
        - x-request-id
6
        - x-routing-key
7
        - x-model
8
        - x-routing-method
9
        - x-input-tokens
10
        - x-cache-affinity-key
11
        - x-priority
12
        - x-request-slo-ms
13
        - x-max-wait-ms
14
        - x-stargate-max-wait-ms

The stock gateway-routes chart does not expose a value for this filter. Use an equivalent policy at an external edge or maintain a route override. Preserve x-multi-turn-session-id; it is the supported client-facing session header. See the Gateway API header modifier guide for filter semantics.

The gateway derives x-cache-affinity-key for chat-completions and Responses requests when affinity applies. It does not derive affinity for embeddings. Do not set require_cache_affinity_key on a model that serves /v1/embeddings unless another trusted gateway supplies the key.

Stargate returns HTTP 400 for a blank, unknown, or configured-but-unavailable x-routing-method. It also returns HTTP 400 when a required router header is missing or a numeric header is invalid.

Apply and roll out

Render the chart before applying it:

$
helm template llm-request-router \
>
  deploy/helm/llm-request-router/llm-request-router \
>
  --values <request-router-values.yaml>

Confirm that the rendered StatefulSet has the expected --lb-config-path argument and that inline JSON creates one ConfigMap with the lb-config.json key.

If the LLM route accepts untrusted traffic, inspect the rendered or live HTTPRoute and confirm that the trusted-header filter is present:

$
kubectl get httproute -n <gateway-namespace> llm-invocation -o yaml

Apply the self-managed environment from the unpacked stack directory:

$
cd path/to/nvcf-self-managed-stack
$
make apply HELMFILE_ENV=<environment-name>

For an inline configuration, inspect the live file and start argument:

$
kubectl get configmap -n nvcf llm-request-router-lb \
>
  -o jsonpath='{.data.lb-config\.json}'
$
kubectl get statefulset -n nvcf llm-request-router \
>
  -o jsonpath='{.spec.template.spec.containers[0].args}'

Restart after a ConfigMap-only change, then wait for all replicas:

$
kubectl rollout restart statefulset/llm-request-router -n nvcf
$
kubectl rollout status statefulset/llm-request-router -n nvcf
$
kubectl get pods -n nvcf -l app.kubernetes.io/name=llm-request-router

Confirm every listed pod was recreated after the ConfigMap update. For each pod, check for the load balancer config loaded startup log and compare its reported default and model-override count:

$
kubectl logs -n nvcf <request-router-pod> \
>
  | grep 'load balancer config loaded'

Do not continue if a pod predates the update, lacks a successful load log, or reports a different configuration summary. Stable affinity requires the same mounted configuration, seed, and candidate view across replicas.

Validate routing

Use a deployed function whose routingMethod is the effective configured algorithm or is present in request_algorithms.

1. Invoke without changing the function routing method and confirm success.
2. Update the function to a method present in request_algorithms and confirm success.
3. Try a method accepted by nvcf-cli that is neither the configured algorithm nor present in request_algorithms; confirm that Stargate returns HTTP 400.
4. For an affinity-aware method, repeat a supported multi-turn request with the returned x-multi-turn-session-id.
5. Exercise a failed or saturated backend and confirm selection and retry counters change.

Observe the request router

The chart exposes llm-request-router:9090/metrics when request-router metrics are enabled. The current chart passes --metrics-port and uses Stargate’s default stargate_ metric prefix.

See LLM Request Router Metrics for metric names, labels, and scrape configuration.

Troubleshoot

Use these logs together:

$
kubectl logs -n nvcf statefulset/llm-request-router \
>
  --all-pods=true --tail=100
$
kubectl logs -n nvcf deploy/llm-api-gateway --tail=100
$
kubectl logs -n nvcf-backend <function-pod> -c llm-worker --tail=100