Agents

Agent Hints

Per-request serving hints for agentic workloads

以 Markdown 格式查看

Agent hints are optional per-request metadata that a harness sends under nvext.agent_hints. Dynamo parses these hints in the frontend and passes them to the router and, where supported, backend runtimes.

Use hints only for serving-relevant intent. Use session IDs for passive trace identity.

Send a Dynamo request with agent hints
{
    "model": "my-model",
    "messages": [
        { "role": "user", "content": "Continue the report." }
    ],
    "nvext": {
        "agent_hints": {
            "priority": 5,
            "strict_priority": 1,
            "osl": 1024,
            "speculative_prefill": true
        }
    }
}
HintDescription
priorityUnified request priority. Higher values mean higher priority at the Dynamo API layer; see Priority Scheduling for router and backend requirements.
strict_priorityRouter pending-queue tier. Higher values always precede lower values before the configured queue policy is applied; see Priority Scheduling for due-time ordering within a tier.
oslExpected output sequence length in tokens. Used by the router for output block tracking and load-balancing accuracy when --router-track-output-blocks is enabled.
speculative_prefillWhen true, Dynamo can prefill the predicted next-turn prefix after the current turn completes to warm the KV cache for the next request.

Request Flow

nvext.agent_hints Agent harness Dynamo frontend Router Backend worker

The frontend parses nvext.agent_hints, the router uses hints for queueing and worker selection, and supported backends use forwarded hints for engine-level scheduling and cache policy. For priority-specific semantics, see Priority Scheduling.

Backend Support

Backend support is runtime-specific. For SGLang flags and behavior, see SGLang for Agentic Workloads.

FeaturevLLMSGLangTensorRT-LLM
Priority-aware routingYesYesYes
Priority-based cache evictionPlannedYesPlanned
Speculative prefillYesYesYes

agent_hints is separate from session identity:

  • Session IDs are passive identity for traces and joins.
  • agent_hints is active serving intent for routing, scheduling, and cache behavior.

Neither the presence of a session ID nor agent_hints enables sticky sessions. Configure any session-aware routing policy separately.