> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# NVIDIA Request Extensions (nvext)

`nvext` is a top-level JSON object on the request body that provides NVIDIA-specific extensions to the OpenAI-compatible API. `nvext` fields are consumed by the Dynamo frontend, preprocessor, router, and backend workers to control routing, preprocessing, response metadata, scheduling, and engine-level priority.

## Usage

Include `nvext` as a top-level field alongside standard OpenAI-compatible fields:

```json
{
    "model": "my-model",
    "messages": [{"role": "user", "content": "Hello"}],
    "nvext": {
        "greed_sampling": true,
        "extra_fields": ["worker_id", "timing"],
        "agent_hints": {
            "osl": 1024,
            "priority": 5,
            "strict_priority": 1
        }
    }
}
```

## Field Reference

| Field | Type | Default | Consumed By | Description |
|-------|------|---------|-------------|-------------|
| `greed_sampling` | `bool` | `None` | Preprocessor | Forces greedy sampling regardless of other sampling parameters. |
| `use_raw_prompt` | `bool` | `None` | Preprocessor | Bypasses the prompt template and passes the prompt directly to the tokenizer. |
| `annotations` | `string[]` | `None` | Preprocessor | Triggers out-of-band information in the SSE stream via the `event:` field. |
| `backend_instance_id` | `u64` | `None` | Router | Routes the request to a specific backend instance. |
| `token_data` | `u32[]` | `None` | Preprocessor | Pre-tokenized prompt tokens. When provided with `backend_instance_id`, tokenization is skipped. |
| `max_thinking_tokens` | `u32` | `None` | Backend | Maximum thinking tokens allowed (passed through to backends). |
| `cache_salt` | `string` | `None` | Router / supported backends | Namespaces Dynamo KV routing. vLLM and TensorRT-LLM also isolate backend KV-cache reuse; see [Backend support](#backend-support). This is the recommended cache-isolation input. |
| `extra_fields` | `string[]` | `None` | Response builder | Fields to include in the response `nvext`. Supported: `"worker_id"`, `"timing"`, `"routed_experts"`, `"engine_data"`, `"stop_reason"`. |
| `prefill_worker_id` | `u64` | `None` | Router | Routes the request to a specific prefill worker (disaggregated serving). |
| `decode_worker_id` | `u64` | `None` | Router | Routes the request to a specific decode worker (disaggregated serving). |
| `dp_rank` | `u32` | `None` | Router/backend | Data-parallel rank for the decode worker. Typically set by EPP routing headers. |
| `prefill_dp_rank` | `u32` | `None` | Router/backend | Data-parallel rank for the prefill worker in disaggregated serving. Typically set by EPP routing headers. |
| `agent_hints` | object | `None` | Router | Per-request hints for scheduling and load balancing. See [Agent Hints](#agent-hints). |

Related root-level Dynamo output option:

| Field | Type | Default | Consumed By | Description |
|-------|------|---------|-------------|-------------|
| `return_tokens_as_token_ids` | `bool` | `false` | Response builder | Formats logprob token strings as `token_id:<id>` instead of decoded text. |

`return_tokens_as_token_ids` only changes returned logprob token display. To stop on
token IDs, pass integer IDs in the normal `stop` array, for example
`"stop": [576]`. Strings such as `"token_id:576"` remain literal string stop
sequences and are not parsed as token IDs.

### Header Overrides

Routing fields can also be set via HTTP headers, which take priority over `nvext` values:

| Header | Overrides |
|--------|-----------|
| `x-dynamo-worker-instance-id` | `backend_instance_id` and `decode_worker_id` |
| `x-dynamo-prefill-instance-id` | `prefill_worker_id` |
| `x-dynamo-dp-rank` | `dp_rank` |
| `x-dynamo-prefill-dp-rank` | `prefill_dp_rank` |
| `x-tenant-id` | `cache_salt` |

<Warning>
The unprefixed forms (`x-worker-instance-id`, `x-prefill-instance-id`, `x-dp-rank`,
`x-data-parallel-rank`, and `x-prefill-dp-rank`) are compatibility aliases planned for future
deprecation. Use the `x-dynamo-*` headers for new integrations.
</Warning>

### Cache salt and tenant isolation

Use `nvext.cache_salt` to namespace KV-cache routing. Dynamo also forwards the salt to supported
backend engines so identical prompts in different namespaces cannot reuse the same backend
KV-cache entries:

```json
{
    "model": "my-model",
    "messages": [{"role": "user", "content": "Hello"}],
    "nvext": {
        "cache_salt": "tenant-a"
    }
}
```

#### Backend support

| Backend | Support | Behavior |
|---------|---------|----------|
| vLLM | Supported | Router matching and backend KV-cache reuse are isolated by salt. |
| TensorRT-LLM | Supported | Router matching and backend KV-cache reuse are isolated by salt. |
| SGLang | Not supported end to end | Dynamo request hashes are namespaced, but the embedded SGLang engine does not receive the salt. SGLang KV events and radix-cache reuse remain unsalted. Do not rely on `cache_salt` for tenant cache isolation with SGLang. |

Dynamo accepts three inputs, in descending precedence:

1. The non-empty `x-tenant-id` HTTP header, intended for gateway-controlled tenant identity.
2. The recommended `nvext.cache_salt` request field.
3. The compatibility top-level `cache_salt` field on chat and completion requests.

Empty strings are treated as absent. In particular, an empty `nvext.cache_salt` falls back to a
non-empty top-level compatibility value. Requests without a salt retain the unsalted hashing and
cache-reuse behavior.

`DYN_ENABLE_FRONTEND_NVEXT=false` disables both the `nvext` form and routing-header overrides,
including `x-tenant-id`. The top-level backend-compatibility field is not part of the NvExt
protocol. Cache salt is an isolation key, not an authentication or authorization mechanism;
gateways must still authenticate the tenant identity they place in `x-tenant-id`.

Session identity is header-only. Use the coding-agent headers or Dynamo
session headers described in [Session IDs](/dynamo/v1.3.0/user-guides/agents/session-i-ds);
`nvext` does not accept session identity fields.

When session affinity is enabled with `--router-session-affinity-ttl-secs`, the
router also uses `X-Dynamo-Session-ID` for router-local affinity. See
[Configuration and Tuning](/dynamo/v1.3.0/components/router/configuration-and-tuning#session-affinity)
for routing behavior and TTL settings.

For trace sink configuration and JSONL schema details, see
[Agent Tracing](/dynamo/v1.3.0/user-guides/agents/agent-tracing).

## Agent Hints

The `agent_hints` sub-object carries per-request hints that the router uses for scheduling, load balancing, and KV cache optimization.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `priority` | `i32` | `None` | Unified soft request priority. Used for router policy scoring and backend scheduling/eviction. |
| `strict_priority` | `u32` | `None` | Router pending-queue tier. Higher values always precede lower values. Unset is equivalent to `0`. |
| `osl` | `u32` | `None` | Expected output sequence length (tokens). Used for output block tracking and resource estimation. |
| `speculative_prefill` | `bool` | `false` | When `true`, speculatively prefills the predicted next-turn prompt after the current turn completes to warm the KV cache. |

### `priority`

`priority` is the cross-layer scheduling hint. Higher values mean "more
important" across Dynamo.

When `--router-queue-threshold` is set and the queue is active, higher-priority requests are shifted earlier in the router queue. Once dispatched, Dynamo forwards the same semantic priority to the backend engine for queue ordering, preemption, and KV cache eviction. Dynamo normalizes backend-specific polarity internally, including vLLM's lower-is-higher convention.

For layer-by-layer behavior and backend requirements, see
[Priority Scheduling](/dynamo/v1.3.0/components/router/priority-scheduling).

```json
{
    "nvext": {
        "agent_hints": {
            "priority": 5
        }
    }
}
```

### `strict_priority`

`strict_priority` is an unsigned router-only tier for requests waiting in a
router scheduler queue. The queue orders requests by
`(strict_priority, configured_policy_key)`, so FCFS, LCFS, or WSPT still orders
requests within the same tier.

This field does not change backend engine priority, preempt running work, or
provide ordering across router replicas. It also does not prevent an eligible
new arrival from being admitted directly while other requests are parked.

```json
{
    "nvext": {
        "agent_hints": {
            "strict_priority": 2
        }
    }
}
```

### `osl`

Expected output sequence length — the estimated number of output tokens the request will generate. The router uses this hint in two ways:

1. **Output block tracking**: When `--router-track-output-blocks` is enabled, the router adds placeholder blocks during generation and applies fractional decay based on progress toward `osl`.
2. **Resource estimation**: Helps the router estimate total resource requirements when making routing decisions.

```json
{
    "nvext": {
        "agent_hints": {
            "osl": 1024
        }
    }
}
```

### `speculative_prefill`

When set to `true`, the system speculatively prefills the predicted next-turn prompt after the current assistant turn completes. This is designed for multi-turn agentic workloads where the next request's prefix is predictable.

How it works:

1. As the assistant response streams, the system accumulates the full response text.
2. Once the response finishes, a background task constructs the next-turn prompt by appending the assistant response to the conversation history (with thinking content stripped for non-last turns).
3. The constructed prompt is tokenized and sent as a `max_tokens=1` request to warm the KV cache on a worker.
4. When the actual next request arrives, it benefits from the already-warm KV cache, reducing TTFT.

```json
{
    "nvext": {
        "agent_hints": {
            "speculative_prefill": true
        }
    }
}
```

Backend details:

- **SGLang**: Requires [`--enable-priority-scheduling`](/dynamo/v1.3.0/backends/sg-lang/agentic-workloads#priority-scheduling) for queue ordering and [`--radix-eviction-policy priority`](/dynamo/v1.3.0/backends/sg-lang/agentic-workloads#priority-based-kv-cache-eviction) for priority-based eviction.
- **vLLM**: Requires [`--scheduling-policy priority`](/dynamo/v1.3.0/backends/v-llm/reference-guide#priority-scheduling).
- **TensorRT-LLM**: Does not currently support per-request priority.

```json
{
    "nvext": {
        "agent_hints": {
            "priority": 5
        }
    }
}
```

## Response Extensions

When the client requests response metadata via `extra_fields`, the response includes an `nvext` object with the requested fields:

| Field | Requested Via | Description |
|-------|---------------|-------------|
| `worker_id` | `extra_fields: ["worker_id"]` | Prefill/decode worker IDs and data parallel ranks that processed the request. |
| `timing` | `extra_fields: ["timing"]` | Per-request timing information (TTFT, ITL, queue time, etc.). |
| `routed_experts` | `extra_fields: ["routed_experts"]` | Routed expert capture payload returned by SGLang-backed requests. |
| `engine_data` | `extra_fields: ["engine_data"]` | Opaque backend-provided engine metadata. |
| `stop_reason` | `extra_fields: ["stop_reason"]` | Backend-specific matched stop condition, returned under `nvext` because it is not part of the OpenAI completions schema. Dynamo currently serves this as a response-level field for single-choice requests; supporting `n > 1` will require an indexed per-choice shape. |
| `token_ids` | Automatic (GAIE Stage 1) | Tokenized prompt for reuse in Stage 2 query-only mode. |

### Example response `nvext`

```json
{
    "nvext": {
        "worker_id": {
            "prefill_worker_id": 1,
            "prefill_dp_rank": 0,
            "decode_worker_id": 2,
            "decode_dp_rank": 0
        },
        "timing": {
            "ttft_ms": 45.2,
            "itl_ms": 12.1
        }
    }
}
```

## See Also

| Document | Description |
|----------|-------------|
| [Frontend Guide](/dynamo/v1.3.0/components/frontend/frontend-guide) | KServe gRPC configuration and integration |
| [Configuration and Tuning](/dynamo/v1.3.0/components/router/configuration-and-tuning) | Full router configuration and CLI arguments |
| [Session IDs](/dynamo/v1.3.0/user-guides/agents/session-i-ds) | Passive session identity |
| [Agent Tracing](/dynamo/v1.3.0/user-guides/agents/agent-tracing) | JSONL request traces, inferred tool-call metadata, and harness tool-event ingestion |
| [Agent Hints](/dynamo/v1.3.0/user-guides/agents/agent-hints) | Per-request serving hints for routing, scheduling, and cache behavior |
| [SGLang for Agentic Workloads](/dynamo/v1.3.0/backends/sg-lang/agentic-workloads) | SGLang engine flags for priority scheduling and KV eviction policies |