LLM Gateway

The LLM Gateway handles OpenAI-compatible, native Anthropic Messages, and beta decision model invocation for NVCF LLM functions in self-managed deployments. Use LLM functions when requests should enter through the LLM invocation route and NVCF should route them by function and model.

Self-managed deployments usually expose the LLM invocation route through the gateway load balancer and use the Host header for routing:

export GATEWAY_ADDR=<gateway-address>

Requests use the model field in this format:

<function-id>/<model-name>

The gateway uses <function-id> as the routing key for NVCF authorization and backend selection. It forwards <model-name> to the upstream model server.

Requests must use the native protocol for the selected endpoint when they reach the LLM invocation route. Any model-specific prompt formatting or tokenizer behavior belongs in the upstream model server.

Request Flow

The LLM Gateway path has these runtime components:

1. Client sends a native protocol request to the LLM invocation route.
2. LLM API Gateway extracts the routing key from the model field, validates authorization, applies request and token rate limits, and validates endpoint-specific request fields.
3. LLM API Gateway forwards the request to LLM request router with routing metadata such as request ID, routing key, model name, routing method, token estimate, and cache affinity key when present.
4. LLM request router selects a healthy backend for the requested function and model.
5. The pylon sidecar on the selected workload forwards the request to the user container through the configured inference port.
6. The user container handles the selected route and returns the response through the same path.

The function container must expose the declared protocol paths on its inference port. Use inferenceUrl: "/" for LLM functions unless the container needs a different base path.

Multi-Cluster View

In a global deployment, DNS or a custom front door selects a regional llm.invocation.<domain> endpoint. The LLM Gateway can use NATS for cross-cluster usage-state chatter. LLM worker request streams still use the request router and worker gateway path, not NATS worker streams. The worker-gateway arrows show that routers can target local or remote worker gateways when those workers are registered into the router mesh.

Function Configuration

Set functionType to LLM and define model routing metadata under models[].llmConfig.

{
  "name": "sample-llm-function",
  "containerImage": "nvcr.io/example/openai-compatible:latest",
  "inferenceUrl": "/",
  "inferencePort": 8000,
  "functionType": "LLM",
  "apiBodyFormat": "CUSTOM",
  "models": [
    {
      "name": "dummy-model",
      "llmConfig": {
        "uris": ["/v1/chat/completions", "/v1/responses", "/v1/embeddings"],
        "routingMethod": "power_of_two",
        "tokenRateLimit": "1000-S"
      }
    }
  ]
}

apiBodyFormat accepts CUSTOM and PREDICT_V2; if omitted, it defaults to CUSTOM. Use CUSTOM for LLM functions with native OpenAI or Anthropic Messages requests. OPENAI_CHAT is not an accepted value and the create-function API rejects it with 400 Bad Request.

The body-format field does not select the LLM protocol or an OpenAI endpoint. functionType: "LLM" selects the LLM invocation path, and llmConfig.uris declares the paths implemented by the container. The client request body uses the native protocol for the selected path; the gateway does not wrap it in a second NVCF envelope.

llmConfig.uris declares the paths the model supports. Supported LLM paths are:

llmConfig.routingMethod is a routing algorithm name, optionally followed by tuning parameters, such as pulsar;seed=stable-a;consider_kv_free_tokens=true. nvcf-cli forwards the value unchanged. The function API checks its format, and the request router checks the algorithm and parameters on each request.

For the mapping to Stargate algorithms, the request-router allowlist, and the tuning parameters, see LLM Request Router Load Balancing.

llmConfig.tokenRateLimit applies a per-model token limit. Use one or more comma-separated limits in <value>-<unit> format, where <value> is a positive integer and <unit> is one of S (seconds), M (minutes), H (hours), D (days), W (weeks), or MO (30 days). A single limit is one token budget over one time window, such as 1000-S. The budget refills continuously over the window rather than resetting at a fixed time, so 3000000-MO returns about 100000 tokens per day. A combined limit is multiple token budgets over distinct time windows, such as 1000-S,5000-M,100000-H,500000-D,1000000-W; do not repeat a unit in the same value.

The same configuration can be provided with CLI flags:

./nvcf-cli function create \
  --name "sample-llm-function" \
  --image "nvcr.io/example/openai-compatible:latest" \
  --inference-url "/" \
  --inference-port 8000 \
  --function-type LLM \
  --llm-model "name=dummy-model,uris=/v1/chat/completions|/v1/responses|/v1/embeddings,routingMethod=power_of_two,tokenRateLimit=1000-S"

These per-model routing fields are mutable. Use nvcf-cli function update --llm-model-update "name=<model>,routingMethod=<method>,tokenRateLimit=<limit>" or JSON modelUpdates to change them without recreating the function version.

llmConfig.uris and llmConfig.tokenizer cannot be changed after creation. A modelUpdates request that sets either one is rejected with 400 Bad Request.

For request admission and rate limiting, the gateway uses request estimates until the upstream service returns usage data. Do not depend on gateway-side exact token counts.

Endpoint Behavior

Chat Completions

Use /v1/chat/completions for OpenAI-compatible chat requests:

curl -sS -X POST "http://${GATEWAY_ADDR}/v1/chat/completions" \
  -H "Host: llm.invocation.${GATEWAY_ADDR}" \
  -H "Authorization: Bearer ${NVCF_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<function-id>/dummy-model",
    "stream": true,
    "prompt_cache_key": "nvcf-summary-session",
    "messages": [
      {
        "role": "user",
        "content": "Write a one sentence summary of NVCF."
      }
    ]
  }'

When stream is true, the gateway relays server-sent events. When stream is false or omitted, the gateway returns one JSON chat completion.

For a non-streaming request, the gateway returns an OpenAI-compatible chat completion. For example:

{
  "id": "chatcmpl-example",
  "object": "chat.completion",
  "created": 1787745600,
  "model": "dummy-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "NVCF routes GPU-backed inference workloads."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 7,
    "total_tokens": 19
  }
}

The gateway always sends stream: true to the upstream container, also when the client asks for a non-streaming response. The container must return OpenAI-compatible SSE chunks and end every stream with data: [DONE]. The gateway relays the upstream response; it does not synthesize model output or token usage.

The upstream stream fails when it ends before data: [DONE], the connection fails, the gateway request timeout expires, or the upstream sends an error event such as data: {"error":{...}}. A non-streaming client receives HTTP 502. A streaming client already has HTTP 200, so the stream ends with this event instead of data: [DONE], with the upstream message when there is one: data: {"error":{"message":"upstream stream ended before completion","type":"server_error","param":null,"code":null}}. The gateway records these failures as 502.

Routing checks versus token-generation tests

An echo or fixed-response workload is useful for verifying function creation, authentication, worker registration, routing, streaming transport, and failover. It does not generate tokens. Repeated or canned output from such a workload cannot measure model token throughput, time to first token, or token-generation latency. Capacity and latency claims require a genuine token-generating model, with input and output token counts recorded alongside concurrency, duration, error rate, throughput, and latency percentiles.

Responses

Use /v1/responses for native Responses API requests:

curl -sS -X POST "http://${GATEWAY_ADDR}/v1/responses" \
  -H "Host: llm.invocation.${GATEWAY_ADDR}" \
  -H "Authorization: Bearer ${NVCF_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<function-id>/dummy-model",
    "input": "Write a one sentence summary of NVCF.",
    "stream": false
  }'

The gateway requires the routed model to declare /v1/responses in llmConfig.uris when model specs are available. The gateway proxies native Responses requests upstream instead of converting them to chat completions.

For upstream compatibility, the gateway sends native Responses requests to the router as streaming requests. If the client requested streaming, the gateway relays SSE. If the client requested a non-streaming response, the gateway consumes the upstream SSE stream and returns the terminal Responses JSON object.

The gateway treats the upstream stream as failed when it closes without a response.completed, response.failed, or response.incomplete event, when the upstream connection fails, or when the upstream sends an error event before a terminal event. A non-streaming client receives HTTP 502. A streaming client has already received HTTP 200, so the gateway ends the stream with a response.failed event whose error.code is server_error. The gateway does not relay an upstream error event; its message becomes the response.failed message. The gateway records these cases as 502 in its logs and metrics.

Embeddings

Use /v1/embeddings for OpenAI-compatible embeddings requests:

curl -sS -X POST "http://${GATEWAY_ADDR}/v1/embeddings" \
  -H "Host: llm.invocation.${GATEWAY_ADDR}" \
  -H "Authorization: Bearer ${NVCF_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<function-id>/dummy-model",
    "input": "NVCF embeddings check"
  }'

The input must not be empty and can include up to 2048 entries. The backend validates the entry types.

Embeddings requests do not use session stickiness.

What the gateway checks on OpenAI endpoints

The gateway checks only the fields that it needs to route, admit, and pin a request. A failed check returns a gateway 400. The backend validates all other fields, so an invalid value there returns the backend’s status and error body.

The body must be a JSON object. A checked field that appears twice in different letter cases returns 400.

The gateway forwards the body unchanged except for these fields:

• Chat: sets model and stream, merges stream_options, and removes debug.
• Responses: sets model and stream.
• Embeddings: sets model.
• Decisions and System One: replaces only the value of model. Every other byte of the body, including key order and spacing, is unchanged.

The gateway does not add service_tier. The backend receives it only when the client sends it.

Decision models (beta)

Decision models answer typed questions about an input and return probabilities instead of generated text. Two beta routes pass these requests through:

• POST /v1/decisions for the OpenAI Decisions format.
• POST /v1/systemone for the TypeSafe System One format.

The gateway reads only model and replaces its value with the backend model name. The body is forwarded unchanged except for the model value. The gateway does not translate between the two formats, so the backend must implement each route that it declares. The routes are meant for non-streaming responses: the gateway does not read stream and does not ask for an event stream. Whether a backend streams depends on the request it receives, which is forwarded as described above.

curl -sS -X POST "http://${GATEWAY_ADDR}/v1/decisions" \
  -H "Host: llm.invocation.${GATEWAY_ADDR}" \
  -H "Authorization: Bearer ${NVCF_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<function-id>/decision-model",
    "input": "The sky is clear.",
    "questions": [{"type": "predicate", "name": "raining", "instructions": "Is it raining?"}]
  }'

The gateway returns 400 for a body it cannot route:

• The body is not valid JSON (invalid_json).
• The body is not a JSON object, or model is missing, not a string, or has no <function-id>/ prefix (invalid_request).
• The body has two model keys that differ only in letter case, such as model and Model (invalid_request).
• The body is nested more than 64 levels deep (invalid_request).

A model is enabled for a route only when its llmConfig.uris lists that path. A model with no uris, or without the route in them, returns 404 in the error shape of the route. These routes have no default-allow, and MODEL_URI_ALLOWLIST_ENABLED does not change this. llmConfig.uris is fixed when the function is created, so declare the decision routes at creation.

Headers

Client headers, including TypeSafe-Danger-Zone-Images, reach the backend, with these exceptions:

• Credentials are not forwarded. The gateway removes Authorization, X-Api-Key, Proxy-Authorization, and Cookie.
• The gateway owns the request router control headers. It removes X-Max-Wait-Ms, X-Request-Slo-Ms, X-Cache-Affinity-Key, X-Input-Tokens, and X-Token-Estimate sent by the client, and sets X-Input-Tokens and X-Token-Estimate to its own estimate, at least 1.

Token estimate

Token admission counts one request and an estimate of the input tokens. The estimate counts every string in the body as text, with these exceptions for images:

• Each image counts a flat number of tokens, 1024 by default, set with llmApiGateway.config.decisionImageTokens in the Helm chart (NVCF_DECISION_IMAGE_TOKENS on the gateway).
• An image is an element of images, a value of assets, or a string that starts with data:image/<type>;base64,. Its base64 data is not counted.
• A nested object with type set to input_image counts one image in place of its image_url and base64 fields. Its other strings still count as text. The top-level object is never counted as an image.

Base64 data in any other field, or in a data URL that is not an image, counts as text.

When a successful response reports usage.input_tokens, the gateway records it on llm_api_gateway_llm_tokens_total and counts the response in llm_api_gateway_decision_usage_observations_total with status="observed". A response without it counts as absent. The estimate remains the admission value.

Request size

By default the gateway does not limit request body size, and it holds the whole body in memory while it reads model and estimates tokens. Image requests can be large, so limit request body size in front of the gateway when you enable these routes.

Errors

Errors that the gateway, the request router, or the Vanity Gateway generate use the shape of the route:

• /v1/decisions uses the OpenAI error shape, {"error":{"message","type","param","code"}}. type follows the status, as for the other OpenAI endpoints.
• /v1/systemone uses {"error":{"code","message","request_id"},"request_id"}. request_id equals the X-Request-ID response header.

An error from the backend is relayed unchanged, whatever its shape, size, or encoding. The request router marks the errors it generates with the X-Stargate-Error-Code response header. For older request router builds that do not set the marker, the gateway also treats a 413, 499, 502, 503, or 504 with an empty body as a router error and rewrites it in the route’s shape. A backend response with one of these statuses and an empty body is therefore rewritten too.

Vanity Gateway mapping

To serve decision models through the Vanity Gateway, map them under v2config.openai.decisions and v2config.typesafeSystemOne on the shared API host. A model is enabled for a route only when it is listed in that route’s section. An entry with functionType: LLM reaches the function through this gateway, which keeps the path:

{
  "v2config": {
    "openai": {
      "host": "api.example.com",
      "decisions": {
        "decision-model": {
          "modelName": "decision-model",
          "functionID": "<function-id>",
          "functionType": "LLM"
        }
      }
    },
    "typesafeSystemOne": {
      "decision-model": {
        "modelName": "decision-model",
        "functionID": "<function-id>",
        "functionType": "LLM"
      }
    }
  }
}

The Vanity Gateway rejects usePexec on functionType: LLM entries. For a direct entry, one without functionType: LLM, of a function that serves both routes, set usePexec: false. A direct usePexec: true entry reaches only the function’s default invocation path. On the decision routes the Vanity Gateway returns 415 when a request sets a Content-Type other than application/json, and accepts request bodies up to its configured limit, 25 MiB by default. In the self-managed stack, set it with addons.vanityGateway.config.maxRequestSizeBytes. The LLM Gateway itself rejects request bodies over 50 MiB by default.

Anthropic Messages

Use /v1/messages with a backend that implements the native Anthropic Messages API. Add /v1/messages to that model’s llmConfig.uris. The gateway does not translate between Anthropic and OpenAI protocols.

curl -N -sS -X POST "http://${GATEWAY_ADDR}/v1/messages" \
  -H "Host: llm.invocation.${GATEWAY_ADDR}" \
  -H "Authorization: Bearer ${NVCF_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-multi-turn-session-id: messages-example" \
  -d '{
    "model": "<function-id>/dummy-model",
    "max_tokens": 256,
    "stream": true,
    "system": "Answer briefly.",
    "messages": [{"role": "user", "content": "Say hello."}]
  }'

Set stream to false or omit it for a native JSON response. The gateway requires a positive max_tokens and a non-empty messages array. The backend validates content blocks, tools, thinking settings, and beta feature support. Upstream status codes and SSE event bytes pass through. Backend error bodies use the Anthropic error envelope, as described below.

A streaming response must end with message_stop. The upstream stream fails when it ends without message_stop, the connection fails, or the gateway request timeout expires. A streaming client already has HTTP 200, so the gateway ends the stream with this event: event: error and data: {"type":"error","error":{"type":"api_error","message":"upstream stream ended before completion"},"request_id":"<id>"}. If the stream stops inside an event, the gateway first closes that event with a ping event, which clients skip. A whole upstream error event passes through unchanged. If it is cut, the gateway adds its own error event after the ping. A non-streaming or compressed response gets no added bytes. The gateway records these failures as 502.

The gateway preserves anthropic-version, all anthropic-beta values, Content-Type, Accept, User-Agent, and W3C trace headers. NVCF sets X-Request-Id from its request context and sends it to the backend. Every Messages response, including gateway errors, carries the same ID in the Anthropic request-id header, which Anthropic SDKs report as the request ID. NVCF replaces any request-id or X-Request-Id the backend returns. Error bodies also include it as request_id, including backend error bodies, where it replaces any backend value. Use this ID to find the request in gateway logs. The gateway requests identity response encoding so usage can be observed without decoding or changing the response. Send the NVCF API key in Authorization: Bearer or in x-api-key, the Anthropic SDK default. When a request carries an Authorization header, the gateway uses it and ignores x-api-key. Other endpoints accept only Authorization: Bearer. Credentials, internal routing headers, and other unlisted headers stop before the model API. The gateway removes top-level extra-headers body members, including letter-case and underscore variants. Native Anthropic metadata and backend extension fields retain their JSON meaning. Request rewriting can change key order and JSON escaping; request bytes are not preserved.

Admission fields (model, max_tokens, stream, messages, system, and tools) require their canonical lowercase spelling. The gateway rejects ambiguous duplicates so the backend and admission logic use the same values. Messages use the existing authorization-resolved priority. Pylon strips inbound engine priority headers before generating its own Dynamo priority headers.

What the gateway checks and what it passes through

The gateway checks only what it needs to authenticate, route, and admit a request. Everything else goes to the backend unchanged, so validation of the remaining fields depends on the backend. A backend that accepts an invalid value returns 200, where the Anthropic API would return 400.

The gateway relays a backend error body when it already uses the Anthropic envelope with a lowercase snake_case error type, as the Anthropic API does, and sets its request_id. Any other backend error body, such as an OpenAI-style or FastAPI validation error, is rendered in the Anthropic envelope with the type that matches the status code and the backend’s message. The message text is the backend’s. Function owners are responsible for what their container puts in error messages.

Token admission estimates system, messages, and tools and reserves max_tokens for output. X-Input-Tokens and X-Token-Estimate both describe estimated input, as on other endpoints. JSON usage and streaming message_start and message_delta usage reconcile the reservation. Streaming counts are cumulative. Cached input counts are included in input accounting.

The observer skips JSON content without buffering it and reads top-level usage even after a large content block. JSON usage, SSE line, and event data buffers are bounded at 1 MiB; JSON nesting and key buffers are also bounded. Unsupported encoding, oversized usage, malformed usage, and missing usage do not change response delivery. Explicit identity encoding and LF, CRLF, and CR SSE line endings are supported.

Only complete, valid usage releases unused output tokens. Streaming accounting requires final output usage in message_delta, message_stop, and a clean copy. After an accepted request, an interrupted or incomplete response retains at least the full output reservation; observed input usage replaces its estimate when valid. Reservations expire with the configured rate-limit window. A request rejected before generation releases its output reservation. A transport failure before response headers also retains output because the gateway cannot prove that generation never started. Estimates are not reported as actual tokens. Usage outcome counters and warnings expose accounting fallbacks, and in-band SSE errors mark the stream as failed.

The existing MODEL_URI_ALLOWLIST_ENABLED mode applies to Messages: an undeclared path is logged in the default mode and rejected in enforce mode. An absent or empty URI list keeps the existing allowlist behavior.

For Vanity Gateway aliases, configure v2config.anthropicMessages on the existing shared API host, configured by v2config.openai.host. The section supports the same model mapping, custom-header validation, discovery, and shadow settings as other JSON endpoints. Shared /v1/models discovery includes Messages aliases and adds supported_endpoints to these aliases so clients can distinguish Messages-only models:

{
  "v2config": {
    "openai": {
      "host": "api.example.com"
    },
    "anthropicMessages": {
      "native-model": {
        "modelName": "dummy-model",
        "functionID": "<function-id>",
        "functionType": "LLM"
      }
    }
  }
}

A request to Vanity Gateway that carries anthropic-version, as Anthropic SDKs and Claude Code send, gets the Anthropic Models API instead. GET /v1/models lists only anthropicMessages aliases, each with type, id, display_name, and created_at, plus has_more, first_id, and last_id. Use limit (1 to 1000, default 20) with after_id or before_id to page. GET /v1/models/{model} returns one alias, or 404 not_found_error for a model that is not served on Messages. An offline model returns 503 api_error with Retry-After: 10800, and an expired model returns 410 not_found_error. Requests without anthropic-version get the shared OpenAI-shaped list.

Messages shadows support sticky sampling methods. On Messages, promptCacheKey reads only headers, by default x-multi-turn-session-id and then x-claude-code-session-id, so a Claude Code conversation stays on one side of the shadow. firstMessageHash hashes system and the first user message. List both, for example [promptCacheKey, firstMessageHash], to fall back to the first message when no session header is present.

For Claude Code, configure the invocation origin and pin both foreground and background requests to a declared model:

export ANTHROPIC_BASE_URL="https://llm.invocation.example.com"
export ANTHROPIC_AUTH_TOKEN="${NVCF_API_KEY}"
export ANTHROPIC_MODEL="<function-id>/<model-name>"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="${ANTHROPIC_MODEL}"

A Vanity Gateway alias uses its public model name. The gateway consumes the native x-claude-code-session-id header for stable affinity across turns. Clients without that header can set ANTHROPIC_CUSTOM_HEADERS to x-multi-turn-session-id: <unique-conversation-id>. Use a different ID for each conversation. See the Claude Code gateway compatibility guide.

The backend must accept the actual requests produced by the selected Claude Code version, including its role layout, tool blocks, thinking settings, and beta features. Claude Code 2.1.287 can include a system role inside messages, which stock vLLM 0.20.1 rejects. The gateway preserves these roles. A hand-written native tool loop passing against a backend does not prove Claude Code compatibility. Qualify the actual CLI and SDK against the deployed backend. /v1/messages/count_tokens is not added by this change.

This feature requires compatible LLM API Gateway, Stargate router, Pylon worker, and Vanity Gateway builds. Upgrade Pylon workers first, then the router and gateway builds, before enabling Messages routes. The new gateway removes caller credentials before Stargate, including when an older worker is present; older workers do not enforce the new backend metadata allowlist. Source support does not establish availability in a published stack release. Release owners must publish images and charts and update the self-managed and compute-plane inventories and catalog after qualification.

Model Routing And Upstream Paths

The client-facing model value must include a routing key and model name:

<function-id>/<model-name>

The gateway uses <function-id> to authorize the request and select the NVCF function deployment. It rewrites the upstream model to <model-name> before forwarding the request.

The model server never receives caller credentials. On every route, the gateway removes Authorization, X-Api-Key, Proxy-Authorization, and Cookie before forwarding, because they authorize NVCF, not the model server.

The configured llmConfig.uris must match the paths served by the container. For example, a model that declares /v1/responses must have a container backend that can answer POST /v1/responses.

Session Stickiness

The LLM Gateway supports sticky routing for multi-turn requests on /v1/chat/completions, /v1/responses, and /v1/messages.

Sticky routing is not supported on /v1/embeddings.

For Messages, affinity uses x-multi-turn-session-id, then the native x-claude-code-session-id, then a messages payload hash. The payload hash changes when a turn is appended and does not ensure multi-turn stickiness. Send a stable session header or reuse the returned x-multi-turn-session-id when later requests need the same affinity.

For OpenAI endpoints, set prompt_cache_key in the request body. You can also send the x-multi-turn-session-id response header value back as the x-multi-turn-session-id request header on the next request.

The gateway chooses the sticky routing key in this order:

For Responses API follow-up calls, previous_response_id does not override the sticky routing key. Continue sending prompt_cache_key, conversation.id, or the returned x-multi-turn-session-id header when the next request needs the same backend affinity.

The gateway accepts a nonempty prompt_cache_key of up to 256 bytes without control characters. An empty value is ignored. The gateway preserves the raw value in the upstream request body and returns it in x-multi-turn-session-id. The gateway derives a SHA-256 value for the internal x-cache-affinity-key header. Clients must not send x-cache-affinity-key.

Sticky routing only affects backend selection when the LLM request router is configured with a cache-affinity-aware routing method for the target model.

Last-Cluster Hints

The gateway can send the LLM request router the cluster that served a session’s last successful request, in the trusted x-stargate-last-cluster-id header. When the model’s wait-and-widen or pulsar-wait-and-widen configuration sets last_cluster_affinity: true, a returning session stays on its last cluster and a new session skips the affinity wait.

Sessions from prompt_cache_key, conversation.id, x-multi-turn-session-id, or x-claude-code-session-id are eligible. Sessions that fall back to a messages or input hash are also eligible, with no extra setting.

A messages or input hash changes every turn, so for those sessions the gateway keys the store by a prefix chain:

• It writes the cluster under a hash of the full conversation.
• It looks up under a hash of the conversation before the last contiguous run of model-output items. For chat and Messages these are assistant messages. For Responses they are assistant message, reasoning, function_call, custom_tool_call, web_search_call, code_interpreter_call, mcp_call, mcp_list_tools, and mcp_approval_request items.
• Messages hashes system first. Responses hashes instructions and prompt first.
• Values come from the raw request JSON. Object key order and whitespace outside strings do not change them. They are scoped by endpoint, org and project, routing key, and model.

Clients that append to their history and resend it verbatim, including tool loops, find the cluster that served the previous turn. Edited, truncated, or summarized history misses and routes as a new session when the change falls in the part hashed for lookup. Assistant prefill continuation is outside this guarantee and usually misses. Responses requests with previous_response_id or an item_reference item are skipped. A first turn skips the store read, counts as a miss, and is written on a 2xx response. Delivery is best-effort: writes are asynchronous, entries expire after the TTL, and with Olric off each replica has its own store. The x-cache-affinity-key header and router placement do not change.

With OLRIC_ENABLED=true, all replicas share entries through the embedded Olric cluster, which then starts even with rate limiting off.

Roll out in this order:

1. Upgrade the LLM request router with last_cluster_affinity off. It accepts the header and ignores it.
2. Set STARGATE_LAST_CLUSTER_ENABLED=true on the gateway.
3. Turn on last_cluster_affinity per model in the router configuration.

Do not turn on last_cluster_affinity before step 2. Without the hint, every request is treated as a new session and skips the affinity wait.

Metrics

LLM API Gateway request metrics include a function_id label. The value is the function ID extracted from the request routing key. Requests without a routing key, such as health checks, use function_id="none".

Infrastructure metrics for authentication, rate limit synchronization, pub/sub, the last-cluster store, and the distributed cache do not include function_id because they are not associated with a single routed function.

Use the label to calculate request rate by function:

sum by (function_id) (
  rate(llm_api_gateway_http_requests_total[5m])
)

Use the histogram label to calculate p95 request latency by function:

histogram_quantile(
  0.95,
  sum by (le, function_id) (
    rate(llm_api_gateway_http_request_duration_seconds_bucket[5m])
  )
)

Operational Notes

• The LLM invocation route is served by the llm.invocation.<domain> hostname.
• The LLM API Gateway chart and LLM request router chart are installed as self-managed stack components.
• The Gateway Routes chart creates the external HTTPRoute for LLM invocation.
• Token rate limits are evaluated per model when llmConfig.tokenRateLimit is set.
• The LLM request router and pylon worker sidecar images must be from a stack release that includes the endpoint support documented here.