NVIDIA Request Extensions (nvext)

nvext is a top-level JSON object on the request body that provides NVIDIA-specific extensions to the OpenAI-compatible API. nvext fields are consumed by the Dynamo frontend, preprocessor, router, and backend workers to control routing, preprocessing, response metadata, scheduling, and engine-level priority.

Usage

Include nvext as a top-level field alongside standard OpenAI-compatible fields:

1
{
2
    "model": "my-model",
3
    "messages": [{"role": "user", "content": "Hello"}],
4
    "nvext": {
5
        "greed_sampling": true,
6
        "extra_fields": ["worker_id", "timing"],
7
        "agent_hints": {
8
            "osl": 1024,
9
            "priority": 5,
10
            "strict_priority": 1
11
        }
12
    }
13
}

Field Reference

Related root-level Dynamo output option:

return_tokens_as_token_ids only changes returned logprob token display. To stop on token IDs, pass integer IDs in the normal stop array, for example "stop": [576]. Strings such as "token_id:576" remain literal string stop sequences and are not parsed as token IDs.

Header Overrides

Routing fields can also be set via HTTP headers, which take priority over nvext values:

Cache salt and tenant isolation

Use nvext.cache_salt to namespace KV-cache routing. Dynamo also forwards the salt to supported backend engines so identical prompts in different namespaces cannot reuse the same backend KV-cache entries:

1
{
2
    "model": "my-model",
3
    "messages": [{"role": "user", "content": "Hello"}],
4
    "nvext": {
5
        "cache_salt": "tenant-a"
6
    }
7
}

Backend support

Dynamo accepts three inputs, in descending precedence:

1. The non-empty x-tenant-id HTTP header, intended for gateway-controlled tenant identity.
2. The recommended nvext.cache_salt request field.
3. The compatibility top-level cache_salt field on chat and completion requests.

Empty strings are treated as absent. In particular, an empty nvext.cache_salt falls back to a non-empty top-level compatibility value. Requests without a salt retain the unsalted hashing and cache-reuse behavior.

DYN_ENABLE_FRONTEND_NVEXT=false disables both the nvext form and routing-header overrides, including x-tenant-id. The top-level backend-compatibility field is not part of the NvExt protocol. Cache salt is an isolation key, not an authentication or authorization mechanism; gateways must still authenticate the tenant identity they place in x-tenant-id.

Session identity is header-only. Use the coding-agent headers or Dynamo session headers described in Session IDs; nvext does not accept session identity fields.

When session affinity is enabled with --router-session-affinity-ttl-secs, the router also uses X-Dynamo-Session-ID for router-local affinity. See Configuration and Tuning for routing behavior and TTL settings.

For trace sink configuration and JSONL schema details, see Agent Tracing.

Agent Hints

The agent_hints sub-object carries per-request hints that the router uses for scheduling, load balancing, and KV cache optimization.

priority

priority is the cross-layer scheduling hint. Higher values mean “more important” across Dynamo.

When --router-queue-threshold is set and the queue is active, higher-priority requests are shifted earlier in the router queue. Once dispatched, Dynamo forwards the same semantic priority to the backend engine for queue ordering, preemption, and KV cache eviction. Dynamo normalizes backend-specific polarity internally, including vLLM’s lower-is-higher convention.

For layer-by-layer behavior and backend requirements, see Priority Scheduling.

1
{
2
    "nvext": {
3
        "agent_hints": {
4
            "priority": 5
5
        }
6
    }
7
}

strict_priority

strict_priority is an unsigned router-only tier for requests waiting in a router scheduler queue. The queue orders requests by (strict_priority, configured_policy_key), so FCFS, LCFS, or WSPT still orders requests within the same tier.

This field does not change backend engine priority, preempt running work, or provide ordering across router replicas. It also does not prevent an eligible new arrival from being admitted directly while other requests are parked.

1
{
2
    "nvext": {
3
        "agent_hints": {
4
            "strict_priority": 2
5
        }
6
    }
7
}

osl

Expected output sequence length — the estimated number of output tokens the request will generate. The router uses this hint in two ways:

1. Output block tracking: When --router-track-output-blocks is enabled, the router adds placeholder blocks during generation and applies fractional decay based on progress toward osl.
2. Resource estimation: Helps the router estimate total resource requirements when making routing decisions.

1
{
2
    "nvext": {
3
        "agent_hints": {
4
            "osl": 1024
5
        }
6
    }
7
}

speculative_prefill

When set to true, the system speculatively prefills the predicted next-turn prompt after the current assistant turn completes. This is designed for multi-turn agentic workloads where the next request’s prefix is predictable.

How it works:

1. As the assistant response streams, the system accumulates the full response text.
2. Once the response finishes, a background task constructs the next-turn prompt by appending the assistant response to the conversation history (with thinking content stripped for non-last turns).
3. The constructed prompt is tokenized and sent as a max_tokens=1 request to warm the KV cache on a worker.
4. When the actual next request arrives, it benefits from the already-warm KV cache, reducing TTFT.

1
{
2
    "nvext": {
3
        "agent_hints": {
4
            "speculative_prefill": true
5
        }
6
    }
7
}

Backend details:

• SGLang: Requires --enable-priority-scheduling for queue ordering and --radix-eviction-policy priority for priority-based eviction.
• vLLM: Requires --scheduling-policy priority.
• TensorRT-LLM: Does not currently support per-request priority.

1
{
2
    "nvext": {
3
        "agent_hints": {
4
            "priority": 5
5
        }
6
    }
7
}

Response Extensions

When the client requests response metadata via extra_fields, the response includes an nvext object with the requested fields:

Example response nvext

1
{
2
    "nvext": {
3
        "worker_id": {
4
            "prefill_worker_id": 1,
5
            "prefill_dp_rank": 0,
6
            "decode_worker_id": 2,
7
            "decode_dp_rank": 0
8
        },
9
        "timing": {
10
            "ttft_ms": 45.2,
11
            "itl_ms": 12.1
12
        }
13
    }
14
}

See Also