> This page is for version 1.1.0. > For other versions, use one of these documentation indexes: > - 1.0.4 (default): https://docs.nvidia.com/nvcf/1.0.4/llms.txt > - 1.1.0: https://docs.nvidia.com/nvcf/1.1.0/llms.txt > - 1.0.3: https://docs.nvidia.com/nvcf/1.0.3/llms.txt > - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt > - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt > - Development: https://docs.nvidia.com/nvcf/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. # LLM Gateway The LLM Gateway handles OpenAI-compatible, native Anthropic Messages, and beta decision model invocation for NVCF LLM functions in self-managed deployments. Use LLM functions when requests should enter through the LLM invocation route and NVCF should route them by function and model. Self-managed deployments usually expose the LLM invocation route through the gateway load balancer and use the `Host` header for routing: ```bash export GATEWAY_ADDR= ``` Requests use the `model` field in this format: ```text / ``` The gateway uses `` as the routing key for NVCF authorization and backend selection. It forwards `` to the upstream model server. Requests must use the native protocol for the selected endpoint when they reach the LLM invocation route. Any model-specific prompt formatting or tokenizer behavior belongs in the upstream model server. ## Request Flow The LLM Gateway path has these runtime components: ![LLM invocation path](/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/a79e448a760bc74f47486811f8ff5625b722ca8ba88477cb750fe643a9184e29/versions/1.1.0/docs/overview/images/nvcf-llm-invocation-path.svg) 1. Client sends a native protocol request to the LLM invocation route. 2. LLM API Gateway extracts the routing key from the `model` field, validates authorization, applies request and token rate limits, and validates endpoint-specific request fields. 3. LLM API Gateway forwards the request to LLM request router with routing metadata such as request ID, routing key, model name, routing method, token estimate, and cache affinity key when present. 4. LLM request router selects a healthy backend for the requested function and model. 5. The `pylon` sidecar on the selected workload forwards the request to the user container through the configured inference port. 6. The user container handles the selected route and returns the response through the same path. The function container must expose the declared protocol paths on its inference port. Use `inferenceUrl: "/"` for LLM functions unless the container needs a different base path. ### Multi-Cluster View In a global deployment, DNS or a custom front door selects a regional `llm.invocation.` endpoint. The LLM Gateway can use NATS for cross-cluster usage-state chatter. LLM worker request streams still use the request router and worker gateway path, not NATS worker streams. The worker-gateway arrows show that routers can target local or remote worker gateways when those workers are registered into the router mesh. ![LLM multi-cluster invocation path](/nvcf/_fern-files/nvidia-nvcf.docs.buildwithfern.com/087fb7ec64708a9af1c550540953d0a8e214d7dd1c1a51e120f298d1d122ddf3/versions/1.1.0/docs/overview/images/nvcf-llm-multicluster-invocation.svg) ## Function Configuration Set `functionType` to `LLM` and define model routing metadata under `models[].llmConfig`. ```json { "name": "sample-llm-function", "containerImage": "nvcr.io/example/openai-compatible:latest", "inferenceUrl": "/", "inferencePort": 8000, "functionType": "LLM", "apiBodyFormat": "CUSTOM", "models": [ { "name": "dummy-model", "llmConfig": { "uris": ["/v1/chat/completions", "/v1/responses", "/v1/embeddings"], "routingMethod": "power_of_two", "tokenRateLimit": "1000-S" } } ] } ``` `apiBodyFormat` accepts `CUSTOM` and `PREDICT_V2`; if omitted, it defaults to `CUSTOM`. Use `CUSTOM` for LLM functions with native OpenAI or Anthropic Messages requests. `OPENAI_CHAT` is not an accepted value and the create-function API rejects it with `400 Bad Request`. The body-format field does not select the LLM protocol or an OpenAI endpoint. `functionType: "LLM"` selects the LLM invocation path, and `llmConfig.uris` declares the paths implemented by the container. The client request body uses the native protocol for the selected path; the gateway does not wrap it in a second NVCF envelope. `llmConfig.uris` declares the paths the model supports. Supported LLM paths are: | Path | Behavior | | --- | --- | | `/v1/chat/completions` | Supports streaming and non-streaming chat completion requests. | | `/v1/responses` | Supports native Responses API requests. Streaming clients receive server-sent events (SSE). Non-streaming clients receive the terminal Responses JSON object. | | `/v1/embeddings` | Supports embeddings requests with string or string array input. | | `/v1/messages` | Preserves native Anthropic Messages JSON responses and server-sent events, including tool and thinking content blocks. | | `/v1/decisions` | Beta. Forwards OpenAI Decisions requests unchanged except for the model value, and returns responses unchanged. Meant for non-streaming responses. | | `/v1/systemone` | Beta. Forwards TypeSafe System One requests unchanged except for the model value, and returns responses unchanged. Meant for non-streaming responses. | `llmConfig.routingMethod` is a routing algorithm name, optionally followed by tuning parameters, such as `pulsar;seed=stable-a;consider_kv_free_tokens=true`. `nvcf-cli` forwards the value unchanged. The function API checks its format, and the request router checks the algorithm and parameters on each request. For the mapping to Stargate algorithms, the request-router allowlist, and the tuning parameters, see [LLM Request Router Load Balancing](/nvcf/1.1.0/self-managed/llm-request-router-load-balancing). `llmConfig.tokenRateLimit` applies a per-model token limit. Use one or more comma-separated limits in `-` format, where `` is a positive integer and `` is one of `S` (seconds), `M` (minutes), `H` (hours), `D` (days), `W` (weeks), or `MO` (30 days). A single limit is one token budget over one time window, such as `1000-S`. The budget refills continuously over the window rather than resetting at a fixed time, so `3000000-MO` returns about 100000 tokens per day. A combined limit is multiple token budgets over distinct time windows, such as `1000-S,5000-M,100000-H,500000-D,1000000-W`; do not repeat a unit in the same value. The same configuration can be provided with CLI flags: ```bash ./nvcf-cli function create \ --name "sample-llm-function" \ --image "nvcr.io/example/openai-compatible:latest" \ --inference-url "/" \ --inference-port 8000 \ --function-type LLM \ --llm-model "name=dummy-model,uris=/v1/chat/completions|/v1/responses|/v1/embeddings,routingMethod=power_of_two,tokenRateLimit=1000-S" ``` These per-model routing fields are mutable. Use `nvcf-cli function update --llm-model-update "name=,routingMethod=,tokenRateLimit="` or JSON `modelUpdates` to change them without recreating the function version. `llmConfig.uris` and `llmConfig.tokenizer` cannot be changed after creation. A `modelUpdates` request that sets either one is rejected with `400 Bad Request`. For request admission and rate limiting, the gateway uses request estimates until the upstream service returns usage data. Do not depend on gateway-side exact token counts. ## Endpoint Behavior ### Chat Completions Use `/v1/chat/completions` for OpenAI-compatible chat requests: ```bash curl -sS -X POST "http://${GATEWAY_ADDR}/v1/chat/completions" \ -H "Host: llm.invocation.${GATEWAY_ADDR}" \ -H "Authorization: Bearer ${NVCF_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "/dummy-model", "stream": true, "prompt_cache_key": "nvcf-summary-session", "messages": [ { "role": "user", "content": "Write a one sentence summary of NVCF." } ] }' ``` When `stream` is `true`, the gateway relays server-sent events. When `stream` is false or omitted, the gateway returns one JSON chat completion. For a non-streaming request, the gateway returns an OpenAI-compatible chat completion. For example: ```json { "id": "chatcmpl-example", "object": "chat.completion", "created": 1787745600, "model": "dummy-model", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "NVCF routes GPU-backed inference workloads." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 12, "completion_tokens": 7, "total_tokens": 19 } } ``` The gateway always sends `stream: true` to the upstream container, also when the client asks for a non-streaming response. The container must return OpenAI-compatible SSE chunks and end every stream with `data: [DONE]`. The gateway relays the upstream response; it does not synthesize model output or token usage. The upstream stream fails when it ends before `data: [DONE]`, the connection fails, the gateway request timeout expires, or the upstream sends an error event such as `data: {"error":{...}}`. A non-streaming client receives HTTP 502. A streaming client already has HTTP 200, so the stream ends with this event instead of `data: [DONE]`, with the upstream message when there is one: `data: {"error":{"message":"upstream stream ended before completion","type":"server_error","param":null,"code":null}}`. The gateway records these failures as 502. ### Routing checks versus token-generation tests An echo or fixed-response workload is useful for verifying function creation, authentication, worker registration, routing, streaming transport, and failover. It does not generate tokens. Repeated or canned output from such a workload cannot measure model token throughput, time to first token, or token-generation latency. Capacity and latency claims require a genuine token-generating model, with input and output token counts recorded alongside concurrency, duration, error rate, throughput, and latency percentiles. ### Responses Use `/v1/responses` for native Responses API requests: ```bash curl -sS -X POST "http://${GATEWAY_ADDR}/v1/responses" \ -H "Host: llm.invocation.${GATEWAY_ADDR}" \ -H "Authorization: Bearer ${NVCF_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "/dummy-model", "input": "Write a one sentence summary of NVCF.", "stream": false }' ``` The gateway requires the routed model to declare `/v1/responses` in `llmConfig.uris` when model specs are available. The gateway proxies native Responses requests upstream instead of converting them to chat completions. For upstream compatibility, the gateway sends native Responses requests to the router as streaming requests. If the client requested streaming, the gateway relays SSE. If the client requested a non-streaming response, the gateway consumes the upstream SSE stream and returns the terminal Responses JSON object. The gateway treats the upstream stream as failed when it closes without a `response.completed`, `response.failed`, or `response.incomplete` event, when the upstream connection fails, or when the upstream sends an `error` event before a terminal event. A non-streaming client receives HTTP 502. A streaming client has already received HTTP 200, so the gateway ends the stream with a `response.failed` event whose `error.code` is `server_error`. The gateway does not relay an upstream `error` event; its message becomes the `response.failed` message. The gateway records these cases as 502 in its logs and metrics. ### Embeddings Use `/v1/embeddings` for OpenAI-compatible embeddings requests: ```bash curl -sS -X POST "http://${GATEWAY_ADDR}/v1/embeddings" \ -H "Host: llm.invocation.${GATEWAY_ADDR}" \ -H "Authorization: Bearer ${NVCF_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "/dummy-model", "input": "NVCF embeddings check" }' ``` The `input` must not be empty and can include up to 2048 entries. The backend validates the entry types. Embeddings requests do not use session stickiness. ### What the gateway checks on OpenAI endpoints The gateway checks only the fields that it needs to route, admit, and pin a request. A failed check returns a gateway `400`. The backend validates all other fields, so an invalid value there returns the backend's status and error body. | Endpoint | Checked by the gateway | | --- | --- | | `/v1/chat/completions` | `model` with the `/` prefix, `stream`, `max_tokens`, `max_completion_tokens`, `prompt_cache_key`, `stream_options`, and a non-empty `messages` array | | `/v1/responses` | `model` with the `/` prefix, `stream`, `max_output_tokens`, `prompt_cache_key`, `conversation.id`, and a non-empty `input` unless `prompt`, `previous_response_id`, or `conversation` is present | | `/v1/embeddings` | `model` with the `/` prefix, and a non-empty `input` with at most 2048 entries | | `/v1/decisions`, `/v1/systemone` | `model` with the `/` prefix, and a body nested at most 64 levels deep. No other field. | The body must be a JSON object. A checked field that appears twice in different letter cases returns `400`. The gateway forwards the body unchanged except for these fields: - Chat: sets `model` and `stream`, merges `stream_options`, and removes `debug`. - Responses: sets `model` and `stream`. - Embeddings: sets `model`. - Decisions and System One: replaces only the value of `model`. Every other byte of the body, including key order and spacing, is unchanged. The gateway does not add `service_tier`. The backend receives it only when the client sends it. ### Decision models (beta) Decision models answer typed questions about an input and return probabilities instead of generated text. Two beta routes pass these requests through: - `POST /v1/decisions` for the OpenAI Decisions format. - `POST /v1/systemone` for the TypeSafe System One format. The gateway reads only `model` and replaces its value with the backend model name. The body is forwarded unchanged except for the model value. The gateway does not translate between the two formats, so the backend must implement each route that it declares. The routes are meant for non-streaming responses: the gateway does not read `stream` and does not ask for an event stream. Whether a backend streams depends on the request it receives, which is forwarded as described above. ```bash curl -sS -X POST "http://${GATEWAY_ADDR}/v1/decisions" \ -H "Host: llm.invocation.${GATEWAY_ADDR}" \ -H "Authorization: Bearer ${NVCF_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "/decision-model", "input": "The sky is clear.", "questions": [{"type": "predicate", "name": "raining", "instructions": "Is it raining?"}] }' ``` The gateway returns `400` for a body it cannot route: - The body is not valid JSON (`invalid_json`). - The body is not a JSON object, or `model` is missing, not a string, or has no `/` prefix (`invalid_request`). - The body has two `model` keys that differ only in letter case, such as `model` and `Model` (`invalid_request`). - The body is nested more than 64 levels deep (`invalid_request`). A model is enabled for a route only when its `llmConfig.uris` lists that path. A model with no `uris`, or without the route in them, returns `404` in the error shape of the route. These routes have no default-allow, and `MODEL_URI_ALLOWLIST_ENABLED` does not change this. `llmConfig.uris` is fixed when the function is created, so declare the decision routes at creation. #### Headers Client headers, including `TypeSafe-Danger-Zone-Images`, reach the backend, with these exceptions: - Credentials are not forwarded. The gateway removes `Authorization`, `X-Api-Key`, `Proxy-Authorization`, and `Cookie`. - The gateway owns the request router control headers. It removes `X-Max-Wait-Ms`, `X-Request-Slo-Ms`, `X-Cache-Affinity-Key`, `X-Input-Tokens`, and `X-Token-Estimate` sent by the client, and sets `X-Input-Tokens` and `X-Token-Estimate` to its own estimate, at least `1`. #### Token estimate Token admission counts one request and an estimate of the input tokens. The estimate counts every string in the body as text, with these exceptions for images: - Each image counts a flat number of tokens, `1024` by default, set with `llmApiGateway.config.decisionImageTokens` in the Helm chart (`NVCF_DECISION_IMAGE_TOKENS` on the gateway). - An image is an element of `images`, a value of `assets`, or a string that starts with `data:image/;base64,`. Its base64 data is not counted. - A nested object with `type` set to `input_image` counts one image in place of its `image_url` and `base64` fields. Its other strings still count as text. The top-level object is never counted as an image. Base64 data in any other field, or in a data URL that is not an image, counts as text. When a successful response reports `usage.input_tokens`, the gateway records it on `llm_api_gateway_llm_tokens_total` and counts the response in `llm_api_gateway_decision_usage_observations_total` with `status="observed"`. A response without it counts as `absent`. The estimate remains the admission value. #### Request size By default the gateway does not limit request body size, and it holds the whole body in memory while it reads `model` and estimates tokens. Image requests can be large, so limit request body size in front of the gateway when you enable these routes. #### Errors Errors that the gateway, the request router, or the Vanity Gateway generate use the shape of the route: - `/v1/decisions` uses the OpenAI error shape, `{"error":{"message","type","param","code"}}`. `type` follows the status, as for the other OpenAI endpoints. - `/v1/systemone` uses `{"error":{"code","message","request_id"},"request_id"}`. `request_id` equals the `X-Request-ID` response header. | Status | `/v1/decisions` code | `/v1/systemone` code | | --- | --- | --- | | `400`, body is not valid JSON | `invalid_json` | `invalid_json` | | `400`, any other reason | `invalid_request` | `invalid_request` | | `401` | `invalid_api_key` | `invalid_api_key` | | `403` | `forbidden` | `forbidden` | | `404` | `model_not_found` | `not_found` | | `410`, model past end of life | `model_expired` | `not_found` | | `413` | `request_too_large` | `request_too_large` | | `415` | `unsupported_media_type` | `unsupported_media_type` | | `422` | `invalid_request` | `invalid_request` | | `429` | `rate_limit_exceeded` | `rate_limit_exceeded` | | `499` | `request_cancelled` | `request_cancelled` | | `502`, `503`, `529` | `provider_unavailable` | `provider_unavailable` | | `504` | `provider_timeout` | `provider_timeout` | | Other `4xx` | `invalid_request` | `invalid_request` | | Other `5xx` | `internal_error` | `internal_error` | An error from the backend is relayed unchanged, whatever its shape, size, or encoding. The request router marks the errors it generates with the `X-Stargate-Error-Code` response header. For older request router builds that do not set the marker, the gateway also treats a `413`, `499`, `502`, `503`, or `504` with an empty body as a router error and rewrites it in the route's shape. A backend response with one of these statuses and an empty body is therefore rewritten too. #### Vanity Gateway mapping To serve decision models through the Vanity Gateway, map them under `v2config.openai.decisions` and `v2config.typesafeSystemOne` on the shared API host. A model is enabled for a route only when it is listed in that route's section. An entry with `functionType: LLM` reaches the function through this gateway, which keeps the path: ```json { "v2config": { "openai": { "host": "api.example.com", "decisions": { "decision-model": { "modelName": "decision-model", "functionID": "", "functionType": "LLM" } } }, "typesafeSystemOne": { "decision-model": { "modelName": "decision-model", "functionID": "", "functionType": "LLM" } } } } ``` The Vanity Gateway rejects `usePexec` on `functionType: LLM` entries. For a direct entry, one without `functionType: LLM`, of a function that serves both routes, set `usePexec: false`. A direct `usePexec: true` entry reaches only the function's default invocation path. On the decision routes the Vanity Gateway returns `415` when a request sets a `Content-Type` other than `application/json`, and accepts request bodies up to its configured limit, 25 MiB by default. In the self-managed stack, set it with `addons.vanityGateway.config.maxRequestSizeBytes`. The LLM Gateway itself rejects request bodies over 50 MiB by default. ### Anthropic Messages Use `/v1/messages` with a backend that implements the native Anthropic Messages API. Add `/v1/messages` to that model's `llmConfig.uris`. The gateway does not translate between Anthropic and OpenAI protocols. ```bash curl -N -sS -X POST "http://${GATEWAY_ADDR}/v1/messages" \ -H "Host: llm.invocation.${GATEWAY_ADDR}" \ -H "Authorization: Bearer ${NVCF_API_KEY}" \ -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -H "x-multi-turn-session-id: messages-example" \ -d '{ "model": "/dummy-model", "max_tokens": 256, "stream": true, "system": "Answer briefly.", "messages": [{"role": "user", "content": "Say hello."}] }' ``` Set `stream` to `false` or omit it for a native JSON response. The gateway requires a positive `max_tokens` and a non-empty `messages` array. The backend validates content blocks, tools, thinking settings, and beta feature support. Upstream status codes and SSE event bytes pass through. Backend error bodies use the Anthropic error envelope, as described below. A streaming response must end with `message_stop`. The upstream stream fails when it ends without `message_stop`, the connection fails, or the gateway request timeout expires. A streaming client already has HTTP 200, so the gateway ends the stream with this event: `event: error` and `data: {"type":"error","error":{"type":"api_error","message":"upstream stream ended before completion"},"request_id":""}`. If the stream stops inside an event, the gateway first closes that event with a `ping` event, which clients skip. A whole upstream `error` event passes through unchanged. If it is cut, the gateway adds its own error event after the `ping`. A non-streaming or compressed response gets no added bytes. The gateway records these failures as 502. The gateway preserves `anthropic-version`, all `anthropic-beta` values, `Content-Type`, `Accept`, `User-Agent`, and W3C trace headers. NVCF sets `X-Request-Id` from its request context and sends it to the backend. Every Messages response, including gateway errors, carries the same ID in the Anthropic `request-id` header, which Anthropic SDKs report as the request ID. NVCF replaces any `request-id` or `X-Request-Id` the backend returns. Error bodies also include it as `request_id`, including backend error bodies, where it replaces any backend value. Use this ID to find the request in gateway logs. The gateway requests identity response encoding so usage can be observed without decoding or changing the response. Send the NVCF API key in `Authorization: Bearer` or in `x-api-key`, the Anthropic SDK default. When a request carries an `Authorization` header, the gateway uses it and ignores `x-api-key`. Other endpoints accept only `Authorization: Bearer`. Credentials, internal routing headers, and other unlisted headers stop before the model API. The gateway removes top-level `extra-headers` body members, including letter-case and underscore variants. Native Anthropic `metadata` and backend extension fields retain their JSON meaning. Request rewriting can change key order and JSON escaping; request bytes are not preserved. Admission fields (`model`, `max_tokens`, `stream`, `messages`, `system`, and `tools`) require their canonical lowercase spelling. The gateway rejects ambiguous duplicates so the backend and admission logic use the same values. Messages use the existing authorization-resolved priority. Pylon strips inbound engine priority headers before generating its own Dynamo priority headers. #### What the gateway checks and what it passes through The gateway checks only what it needs to authenticate, route, and admit a request. Everything else goes to the backend unchanged, so validation of the remaining fields depends on the backend. A backend that accepts an invalid value returns `200`, where the Anthropic API would return `400`. | Field or header | Checked by the gateway | Otherwise | | --- | --- | --- | | `model` | Yes. Requires the `/` prefix. | Rewritten to the backend model name. | | `max_tokens` | Yes. Must be a positive integer. | Passed through. | | `messages` | Yes. Must be a non-empty array. Content is not checked. | Passed through. | | Spelling of `model`, `max_tokens`, `stream`, `messages`, `system`, `tools` | Yes. Must use the canonical lowercase name. | Passed through. | | `anthropic-version`, `anthropic-beta` | No. Not required, validated, or interpreted. | Forwarded to the backend. | | Content blocks, images, documents | No. | Backend validates. | | `tools`, `tool_choice`, `tool_result` | No. Tools are only counted for token admission. | Backend validates. | | `thinking`, `cache_control`, `metadata`, `service_tier` | No. | Backend validates and decides support. | | Backend error bodies | Rendered in the Anthropic error envelope with `request_id`. | Message text comes from the backend. | The gateway relays a backend error body when it already uses the Anthropic envelope with a lowercase snake_case error type, as the Anthropic API does, and sets its `request_id`. Any other backend error body, such as an OpenAI-style or FastAPI validation error, is rendered in the Anthropic envelope with the type that matches the status code and the backend's message. The message text is the backend's. Function owners are responsible for what their container puts in error messages. Token admission estimates system, messages, and tools and reserves `max_tokens` for output. `X-Input-Tokens` and `X-Token-Estimate` both describe estimated input, as on other endpoints. JSON usage and streaming `message_start` and `message_delta` usage reconcile the reservation. Streaming counts are cumulative. Cached input counts are included in input accounting. The observer skips JSON content without buffering it and reads top-level usage even after a large content block. JSON usage, SSE line, and event data buffers are bounded at 1 MiB; JSON nesting and key buffers are also bounded. Unsupported encoding, oversized usage, malformed usage, and missing usage do not change response delivery. Explicit identity encoding and LF, CRLF, and CR SSE line endings are supported. Only complete, valid usage releases unused output tokens. Streaming accounting requires final output usage in `message_delta`, `message_stop`, and a clean copy. After an accepted request, an interrupted or incomplete response retains at least the full output reservation; observed input usage replaces its estimate when valid. Reservations expire with the configured rate-limit window. A request rejected before generation releases its output reservation. A transport failure before response headers also retains output because the gateway cannot prove that generation never started. Estimates are not reported as actual tokens. Usage outcome counters and warnings expose accounting fallbacks, and in-band SSE errors mark the stream as failed. The existing `MODEL_URI_ALLOWLIST_ENABLED` mode applies to Messages: an undeclared path is logged in the default mode and rejected in enforce mode. An absent or empty URI list keeps the existing allowlist behavior. For Vanity Gateway aliases, configure `v2config.anthropicMessages` on the existing shared API host, configured by `v2config.openai.host`. The section supports the same model mapping, custom-header validation, discovery, and shadow settings as other JSON endpoints. Shared `/v1/models` discovery includes Messages aliases and adds `supported_endpoints` to these aliases so clients can distinguish Messages-only models: ```json { "v2config": { "openai": { "host": "api.example.com" }, "anthropicMessages": { "native-model": { "modelName": "dummy-model", "functionID": "", "functionType": "LLM" } } } } ``` A request to Vanity Gateway that carries `anthropic-version`, as Anthropic SDKs and Claude Code send, gets the Anthropic Models API instead. `GET /v1/models` lists only `anthropicMessages` aliases, each with `type`, `id`, `display_name`, and `created_at`, plus `has_more`, `first_id`, and `last_id`. Use `limit` (1 to 1000, default 20) with `after_id` or `before_id` to page. `GET /v1/models/{model}` returns one alias, or `404` `not_found_error` for a model that is not served on Messages. An offline model returns `503` `api_error` with `Retry-After: 10800`, and an expired model returns `410` `not_found_error`. Requests without `anthropic-version` get the shared OpenAI-shaped list. Messages shadows support sticky sampling methods. On Messages, `promptCacheKey` reads only headers, by default `x-multi-turn-session-id` and then `x-claude-code-session-id`, so a Claude Code conversation stays on one side of the shadow. `firstMessageHash` hashes `system` and the first user message. List both, for example `[promptCacheKey, firstMessageHash]`, to fall back to the first message when no session header is present. For Claude Code, configure the invocation origin and pin both foreground and background requests to a declared model: ```bash export ANTHROPIC_BASE_URL="https://llm.invocation.example.com" export ANTHROPIC_AUTH_TOKEN="${NVCF_API_KEY}" export ANTHROPIC_MODEL="/" export ANTHROPIC_DEFAULT_HAIKU_MODEL="${ANTHROPIC_MODEL}" ``` A Vanity Gateway alias uses its public model name. The gateway consumes the native `x-claude-code-session-id` header for stable affinity across turns. Clients without that header can set `ANTHROPIC_CUSTOM_HEADERS` to `x-multi-turn-session-id: `. Use a different ID for each conversation. See the [Claude Code gateway compatibility guide](https://code.claude.com/docs/en/llm-gateway-protocol). The backend must accept the actual requests produced by the selected Claude Code version, including its role layout, tool blocks, thinking settings, and beta features. Claude Code 2.1.287 can include a `system` role inside `messages`, which stock vLLM 0.20.1 rejects. The gateway preserves these roles. A hand-written native tool loop passing against a backend does not prove Claude Code compatibility. Qualify the actual CLI and SDK against the deployed backend. `/v1/messages/count_tokens` is not added by this change. This feature requires compatible LLM API Gateway, Stargate router, Pylon worker, and Vanity Gateway builds. Upgrade Pylon workers first, then the router and gateway builds, before enabling Messages routes. The new gateway removes caller credentials before Stargate, including when an older worker is present; older workers do not enforce the new backend metadata allowlist. Source support does not establish availability in a published stack release. Release owners must publish images and charts and update the self-managed and compute-plane inventories and catalog after qualification. ## Model Routing And Upstream Paths The client-facing `model` value must include a routing key and model name: ```text / ``` The gateway uses `` to authorize the request and select the NVCF function deployment. It rewrites the upstream model to `` before forwarding the request. The model server never receives caller credentials. On every route, the gateway removes `Authorization`, `X-Api-Key`, `Proxy-Authorization`, and `Cookie` before forwarding, because they authorize NVCF, not the model server. The configured `llmConfig.uris` must match the paths served by the container. For example, a model that declares `/v1/responses` must have a container backend that can answer `POST /v1/responses`. ## Session Stickiness The LLM Gateway supports sticky routing for multi-turn requests on `/v1/chat/completions`, `/v1/responses`, and `/v1/messages`. Sticky routing is not supported on `/v1/embeddings`. For Messages, affinity uses `x-multi-turn-session-id`, then the native `x-claude-code-session-id`, then a messages payload hash. The payload hash changes when a turn is appended and does not ensure multi-turn stickiness. Send a stable session header or reuse the returned `x-multi-turn-session-id` when later requests need the same affinity. For OpenAI endpoints, set `prompt_cache_key` in the request body. You can also send the `x-multi-turn-session-id` response header value back as the `x-multi-turn-session-id` request header on the next request. The gateway chooses the sticky routing key in this order: | Endpoint | Precedence | | --- | --- | | `/v1/messages` | `x-multi-turn-session-id`, `x-claude-code-session-id`, messages hash fallback | | `/v1/responses` | `prompt_cache_key`, `conversation.id`, `x-multi-turn-session-id`, input hash fallback (none when `input` is empty) | | `/v1/chat/completions` | `prompt_cache_key`, `x-multi-turn-session-id`, messages hash fallback | For Responses API follow-up calls, `previous_response_id` does not override the sticky routing key. Continue sending `prompt_cache_key`, `conversation.id`, or the returned `x-multi-turn-session-id` header when the next request needs the same backend affinity. The gateway accepts a nonempty `prompt_cache_key` of up to 256 bytes without control characters. An empty value is ignored. The gateway preserves the raw value in the upstream request body and returns it in `x-multi-turn-session-id`. The gateway derives a SHA-256 value for the internal `x-cache-affinity-key` header. Clients must not send `x-cache-affinity-key`. ```mermaid sequenceDiagram participant Client participant Gateway as LLM API Gateway participant Router as LLM Request Router participant Backend as Model backend Client->>Gateway: Chat request with raw prompt_cache_key Note over Gateway: Validate key and derive SHA-256 affinity value Gateway->>Router: Chat JSON and hashed X-Cache-Affinity-Key Router->>Backend: Chat JSON with raw prompt_cache_key Backend-->>Router: Completion response Router-->>Gateway: Completion response Gateway-->>Client: Response and raw x-multi-turn-session-id ``` Sticky routing only affects backend selection when the LLM request router is configured with a cache-affinity-aware routing method for the target model. ### Last-Cluster Hints The gateway can send the LLM request router the cluster that served a session's last successful request, in the trusted `x-stargate-last-cluster-id` header. When the model's `wait-and-widen` or `pulsar-wait-and-widen` configuration sets `last_cluster_affinity: true`, a returning session stays on its last cluster and a new session skips the affinity wait. Sessions from `prompt_cache_key`, `conversation.id`, `x-multi-turn-session-id`, or `x-claude-code-session-id` are eligible. Sessions that fall back to a messages or input hash are also eligible, with no extra setting. A messages or input hash changes every turn, so for those sessions the gateway keys the store by a prefix chain: - It writes the cluster under a hash of the full conversation. - It looks up under a hash of the conversation before the last contiguous run of model-output items. For chat and Messages these are assistant messages. For Responses they are assistant `message`, `reasoning`, `function_call`, `custom_tool_call`, `web_search_call`, `code_interpreter_call`, `mcp_call`, `mcp_list_tools`, and `mcp_approval_request` items. - Messages hashes `system` first. Responses hashes `instructions` and `prompt` first. - Values come from the raw request JSON. Object key order and whitespace outside strings do not change them. They are scoped by endpoint, org and project, routing key, and model. Clients that append to their history and resend it verbatim, including tool loops, find the cluster that served the previous turn. Edited, truncated, or summarized history misses and routes as a new session when the change falls in the part hashed for lookup. Assistant prefill continuation is outside this guarantee and usually misses. Responses requests with `previous_response_id` or an `item_reference` item are skipped. A first turn skips the store read, counts as a `miss`, and is written on a 2xx response. Delivery is best-effort: writes are asynchronous, entries expire after the TTL, and with Olric off each replica has its own store. The `x-cache-affinity-key` header and router placement do not change. | Env var | Helm value | Default | Effect | | --- | --- | --- | --- | | `STARGATE_LAST_CLUSTER_ENABLED` | `llmApiGateway.config.lastCluster.enabled` | `false` | Turns lookup, the header, and store writes on. | | `STARGATE_LAST_CLUSTER_TTL` | `llmApiGateway.config.lastCluster.ttl` | `10m` | Entry lifetime, reset on every write. | | `STARGATE_LAST_CLUSTER_LOOKUP_TIMEOUT` | `llmApiGateway.config.lastCluster.lookupTimeout` | `20ms` | Lookup deadline on the request path. The request proceeds without the hint after it. | | `STARGATE_LAST_CLUSTER_LOCAL_MAX_ENTRIES` | `llmApiGateway.config.lastCluster.localMaxEntries` | `100000` | Per-replica LRU cap when Olric is off. | With `OLRIC_ENABLED=true`, all replicas share entries through the embedded Olric cluster, which then starts even with rate limiting off. Roll out in this order: 1. Upgrade the LLM request router with `last_cluster_affinity` off. It accepts the header and ignores it. 2. Set `STARGATE_LAST_CLUSTER_ENABLED=true` on the gateway. 3. Turn on `last_cluster_affinity` per model in the router configuration. Do not turn on `last_cluster_affinity` before step 2. Without the hint, every request is treated as a new session and skips the affinity wait. ## Metrics LLM API Gateway request metrics include a `function_id` label. The value is the function ID extracted from the request routing key. Requests without a routing key, such as health checks, use `function_id="none"`. | Metric | Type | Labels | Description | | --- | --- | --- | --- | | `llm_api_gateway_http_requests_total` | Counter | `method`, `route`, `status`, `function_id` | Inbound HTTP requests. | | `llm_api_gateway_http_request_duration_seconds` | Histogram | `method`, `route`, `status`, `function_id` | Inbound HTTP request latency. | | `llm_api_gateway_http_active_requests` | Up-down counter | `method`, `route`, `function_id` | In-flight inbound HTTP requests. | | `llm_api_gateway_upstream_requests_total` | Counter | `upstream`, `result`, `status`, `function_id` | Requests sent to an upstream provider. | | `llm_api_gateway_upstream_request_duration_seconds` | Histogram | `upstream`, `result`, `status`, `function_id` | Upstream provider request latency. | | `llm_api_gateway_messages_usage_observations_total` | Counter | `status`, `stream`, `function_id` | Messages usage outcomes: complete, partial, missing, oversized, invalid, encoded, error, or rejected. | | `llm_api_gateway_llm_tokens_total` | Counter | `endpoint`, `token_type`, `stream`, `function_id` | Token counts reported by upstream providers. | | `llm_api_gateway_provider_time_seconds` | Histogram | `endpoint`, `phase`, `stream`, `function_id` | Provider-reported timing phases. | | `llm_api_gateway_stream_first_token_seconds` | Histogram | `endpoint`, `function_id` | Time from stream request start to the first token. | | `llm_api_gateway_stream_duration_seconds` | Histogram | `endpoint`, `status`, `function_id` | Total stream duration. | Infrastructure metrics for authentication, rate limit synchronization, pub/sub, the last-cluster store, and the distributed cache do not include `function_id` because they are not associated with a single routed function. Use the label to calculate request rate by function: ```promql sum by (function_id) ( rate(llm_api_gateway_http_requests_total[5m]) ) ``` Use the histogram label to calculate p95 request latency by function: ```promql histogram_quantile( 0.95, sum by (le, function_id) ( rate(llm_api_gateway_http_request_duration_seconds_bucket[5m]) ) ) ``` ## Operational Notes - The LLM invocation route is served by the `llm.invocation.` hostname. - The LLM API Gateway chart and LLM request router chart are installed as self-managed stack components. - The Gateway Routes chart creates the external HTTPRoute for LLM invocation. - Token rate limits are evaluated per model when `llmConfig.tokenRateLimit` is set. - The LLM request router and `pylon` worker sidecar images must be from a stack release that includes the endpoint support documented here.