LLM Request Router Load Balancing

Use this guide to deploy and validate Stargate load-balancer configuration in self-managed NVCF. For the complete lb-config.json schema, algorithm behavior, defaults, and tuning fields, see the Stargate load balancer configuration.

This guide covers NVCF deployment ownership and trusted request metadata. It does not redefine the Stargate schema.

Configure the self-managed stack

Set the request-router configuration in the Helmfile environment:

addons:
  llm:
    enabled: true
    requestRouter:
      loadBalancer:
        config: |
          {
            "default": "power-of-n",
            "request_algorithms": {
              "round-robin": "round-robin"
            },
            "models": {
              "model-a": {
                "algorithm": "wait-and-widen",
                "cache_affinity_backend_selection_count": 2,
                "cache_affinity_wait_ms": 200
              }
            }
          }

The example uses these settings in the model-a configuration:

Replace model-a with the exact routed model name. For requests with an affinity key, this example keeps global buckets closed for 200 ms. Stargate can select an available affine candidate immediately. Set cache_affinity_wait_ms in the model configuration.

After the wait, global buckets include all backends, including the affinity group. Global selection uses full prefill cost for every backend and retains the queue-admission, capacity, and retry-exclusion checks. Stargate still tries affinity selection first on each routing attempt.

When an engine reports a positive concurrency limit and has a free request slot, Stargate estimates zero queue delay. It counts prefill, decode, and pending assignments against that limit. The request’s own prefill time still contributes to estimated TTFT. Pylon uses the same rule for its local queue admission check. Engines with an unknown limit retain the existing queue-delay estimate.

The self-managed stack passes addons.llm.requestRouter.loadBalancer to the request-router chart as llmRequestRouter.loadBalancer.

The chart supports two configuration sources:

Inline config takes precedence when both values are set. When neither value is set, Stargate uses its built-in power-of-n default and accepts a routing-method override when it is in the allowlist of built-in algorithms.

Stargate reads and validates the file only during process startup. The request-router workload does not include a load-balancer ConfigMap checksum in its pod template. New installations default to a Deployment; existing installations can pin a StatefulSet. After a ConfigMap-only update, restart the selected workload so every replica loads the same configuration.

Existing StatefulSet installations must set addons.llm.requestRouter.workload.kind=StatefulSet before upgrading. Changing the workload kind is a controlled migration, not an in-place Kubernetes kind mutation. A plain Helm upgrade across workload kinds can temporarily run both the Deployment and StatefulSet; use the chart migration procedure to remove or rename the old workload and verify that only the selected kind owns the router Pods before scaling it.

Distinguish router algorithms from nvcf-cli routing methods

Algorithm availability is enforced at separate layers:

For example, when a configuration is set and power-of-n is the effective algorithm, wait_and_widen requires a wait-and-widen entry in request_algorithms. The legacy groq_multiregion value remains accepted and resolves to the same algorithm.

Use wait-and-widen and pulsar-wait-and-widen in new function metadata, lb-config.json files, request-algorithm maps, and deployment manifests. Existing groq-multiregion and pulsar-multiregion values continue to work through the Stargate and control-plane compatibility aliases.

Algorithm names

Stargate accepts these algorithm names and aliases:

Tune a model with a routing expression

A function owner can tune the selected algorithm for one model. Set llmConfig.routingMethod to the algorithm name followed by ;name=value parameters:

pulsar;seed=stable-a;consider_kv_free_tokens=true
wait-and-widen;n=3;max_queue_time_floor_ms=200;max_queue_time_ceil_ms=2000

The parameters override the router configuration of that algorithm for that model only. Omitted parameters keep the lb-config.json values: the model entry when its algorithm matches, else the request_algorithms entry, else the default. The algorithm must be available to the model, as for a value without parameters. A value without parameters, such as pulsar, routes as before.

With nvcf-cli, put the whole model spec in single quotes. Without quotes, the shell ends the command at the first ; and the CLI sends the algorithm name without its parameters:

nvcf-cli function update --function-id=<function-id> --version-id=<version-id> \
  --llm-model-update='name=my-model,routingMethod=pulsar;seed=stable-a'

In an update JSON file, set the same value under modelUpdates[].llmConfig:

{
  "modelUpdates": [
    {
      "modelName": "my-model",
      "llmConfig": { "routingMethod": "pulsar;seed=stable-a" }
    }
  ]
}

Format

The value is an RFC 8941 Item with Parameters, with these rules:

• At most 1024 bytes and 32 parameters.
• No commas anywhere.
• Each parameter is name=value. A name starts with a lowercase letter, uses lowercase letters, digits, and underscores, and appears once.
• A space is allowed after ;, not around =.
• A value is an integer, a decimal with at most three fractional digits, a token, or a quoted string. Write booleans as true or false, not ?1 or ?0. A semicolon is not allowed inside a quoted string.
• A quoted string is read as the parameter type. Quote a number that needs more than three fractional digits, such as next_bucket_unlock_factor="0.0625". A quoted number must be a plain decimal number, without an exponent, inf, or NaN.

The function API checks this format on create and update and returns HTTP 400 for a malformed value. It stores an accepted value as written, apart from leading and trailing spaces. The router checks parameter names and values on each request.

Parameters

Each parameter overrides the lb-config.json field with the same name. For field behavior and built-in defaults, see the Stargate load balancer configuration.

round-robin and random accept only the parameters for all algorithms.

Rejections

Stargate rejects an invalid value with HTTP 400 before backend selection. The response has the body {"error":"<message>","code":"<code>"} and the header x-stargate-error-code: <code>. The message names the rejected parameter or value.

Values without parameters use the same response for unknown_method and unavailable.

Through the LLM API Gateway, clients receive the message in the endpoint’s error format, not the router body. The gateway removes the x-stargate-error-code header. It returns the code in error.code only when llmApiGateway.config.exposeRouterErrorCodes is true. See Error Responses.

How updates apply

• An update needs no router restart or ConfigMap change.
• The LLM API Gateway caches function metadata for up to 60 seconds. An update reaches the router within that time, and until then requests can carry the old or the new value.
• For each routing key and model, each router keeps the configurations of the two most recent values, compared byte for byte. Both values reuse their configuration and load-balancer state while an update spreads. A third value replaces the older one.
• A new value starts with new load-balancer state, such as Pulsar rankings or the round-robin position.
• An entry expires after 15 idle minutes. The router removes expired entries within a minute, even without further requests. Each router keeps at most 16384 entries.

Logs and traces

• The span field routing.requested_algorithm carries the header value as received on every request.

• When the router builds or replaces a configuration, it writes the info log routing expression configuration built with the algorithm, outcome, and value. The expression field is the value as received.

• The effective field on the same line lists every value the algorithm routes with, as a JSON object in the load-balancer configuration format. It includes values inherited from the static configuration and the router defaults. Unset optional values, such as seed or the queue-time pair, are omitted. For example, the value wait-and-widen;n=3 logs:

  effective={"algorithm":"wait-and-widen","comparator":"ttft","cache_affinity_virtual_nodes":150,"cache_affinity_input_tokens_scale":1.0,"cache_affinity_wait_ms":0,"ttft_bucket_size_ms":20,"next_bucket_unlock_factor":0.25,"n":3,"max_queued":0,"ignore_queue_time":false,"ignore_input_processing_time":false,"require_cache_affinity_key":false,"require_input_tokens":false}

• Each rejected request writes the warn log invalid routing algorithm header with the code and message.

Routers without expression support

A request router release without routing expression support reads the whole value as an algorithm name. It returns HTTP 400 for every value with parameters, and values without parameters keep working. The function API accepts parameters regardless of the router release. In that case, a model with parameters gets HTTP 400 on every request until the router is upgraded or the parameters are removed.

Keep router headers trusted

The gateway can send the following headers to Stargate. Derive or validate their values from authenticated function metadata and the request.

An explicit x-max-wait-ms shorter than the configured affinity wait can end routing with HTTP 503 before global buckets become eligible. Set the affinity wait within the latency budget for the workload.

For a separately managed Gateway API route, add this filter to the rule that forwards to llm-api-gateway:

filters:
  - type: RequestHeaderModifier
    requestHeaderModifier:
      remove:
        - x-request-id
        - x-routing-key
        - x-model
        - x-routing-method
        - x-input-tokens
        - x-cache-affinity-key
        - x-priority
        - x-request-slo-ms
        - x-max-wait-ms
        - x-stargate-max-wait-ms

The gateway-routes chart does not expose a value for this filter. Use an equivalent policy at an external edge or maintain a route override. Preserve x-multi-turn-session-id; clients can use it for session affinity. Chat Completions and Responses request bodies can also supply prompt_cache_key. See the Gateway API header modifier guide for filter semantics.

For affinity routing, the gateway can supply x-cache-affinity-key as a stable session or prefix identifier. When using prompt_cache_key, place its SHA-256-derived value in the header. The request body can retain the raw value for the model backend. Enable require_cache_affinity_key only when the gateway supplies a key for every endpoint served by the model.

Stargate returns HTTP 400 for a blank, unknown, or configured-but-unavailable x-routing-method, and for an invalid routing expression. The response carries the code described in Rejections. It also returns HTTP 400 when a required router header is missing or a numeric header is invalid.

Apply and roll out

Render the chart before applying it:

helm template llm-request-router \
  deploy/helm/llm-request-router/llm-request-router \
  --values <request-router-values.yaml>

Confirm that the rendered request-router workload (a Deployment by default) has the expected --lb-config-path argument and that inline JSON creates one ConfigMap with the lb-config.json key.

If the LLM route accepts untrusted traffic, inspect the rendered or live HTTPRoute and confirm that the trusted-header filter is present:

kubectl get httproute -n <gateway-namespace> llm-invocation -o yaml

Apply the self-managed environment from the unpacked stack directory:

cd "$BUNDLE_DIR/deploy/stacks/self-managed"
make apply HELMFILE_ENV=<environment-name>

For an inline configuration, inspect the live file and start argument:

kubectl get configmap -n nvcf llm-request-router-lb \
  -o jsonpath='{.data.lb-config\.json}'
kubectl get deployment -n nvcf llm-request-router \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

Replace deployment with statefulset in these commands when addons.llm.requestRouter.workload.kind is pinned to StatefulSet.

Restart after a ConfigMap-only change, then wait for all replicas:

kubectl rollout restart deployment/llm-request-router -n nvcf
kubectl rollout status deployment/llm-request-router -n nvcf
kubectl get pods -n nvcf -l app.kubernetes.io/name=llm-request-router

Use statefulset/llm-request-router instead when the StatefulSet workload is explicitly selected.

Confirm every listed pod was recreated after the ConfigMap update. For each pod, check for the load balancer config loaded startup log and compare its reported default and model-override count:

kubectl logs -n nvcf <request-router-pod> \
  | grep 'load balancer config loaded'

Do not continue if a pod predates the update, lacks a successful load log, or reports a different configuration summary. Stable affinity requires the same mounted configuration, seed, and candidate view across replicas.

Validate routing

Use a deployed function whose routingMethod is the effective configured algorithm or is present in request_algorithms.

1. Invoke without changing the function routing method and confirm success.
2. Update the function to a method present in request_algorithms and confirm success.
3. Update the function to a method that is neither the configured algorithm nor present in request_algorithms. Invoke it and confirm that Stargate returns HTTP 400.
4. Update the function to a method with a parameter that applies to it, such as pulsar;seed=test for pulsar. Invoke it and confirm success. Then update it with an unknown parameter, such as pulsar;widen=2. The update succeeds. Invoke it and confirm HTTP 400 with the unknown_parameter code.
5. For an affinity-aware method, repeat a supported multi-turn request with the same prompt_cache_key or the returned x-multi-turn-session-id.
6. Exercise a failed or saturated backend and confirm selection and retry counters change.

Observe the request router

The chart exposes llm-request-router:9090/metrics when request-router metrics are enabled. The current chart passes --metrics-port and uses Stargate’s default stargate_ metric prefix.

See LLM Request Router Metrics for metric names, labels, and scrape configuration.

Troubleshoot

Use these logs together:

kubectl logs -n nvcf deployment/llm-request-router \
  --all-pods=true --tail=100
kubectl logs -n nvcf deploy/llm-api-gateway --tail=100
kubectl logs -n nvcf-backend <function-pod> -c llm-worker --tail=100

Use statefulset/llm-request-router for the first command when the StatefulSet workload is explicitly selected.