> This page is for NeMo Relay, version v0.8.4. > For other versions, use one of these documentation indexes: > - Latest (v0.10.0) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt > - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt > - v0.10.0: https://docs.nvidia.com/nemo/relay/v0.10.0/llms.txt > - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt > - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt > - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt > - v0.6.0: https://docs.nvidia.com/nemo/relay/v0.6.0/llms.txt > - v0.5.0: https://docs.nvidia.com/nemo/relay/v0.5.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Response Cache > Configure exact-match response caching for managed LLM calls and tool results. Use the response cache when a repeatable managed LLM request or explicitly classified tool call should be served from a store instead of running live. An eligible request or tool call that matches an unexpired cache entry can be served without calling the provider or tool. Buffered LLM hits preserve the stored response shape and usage fields; streaming hits replay an equivalent provider-native stream. The cache is an optional `response_cache` section of the [Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin kind. It is off until the section is present. Its LLM surface applies to [managed LLM calls](/instrument-applications/instrument-llm-call) without changing the execution API; its tool-result surface is separately opt-in. By default, only requests with an explicit numeric `temperature = 0` are eligible; set `cache_nondeterministic = true` to opt sampled requests into caching. Runtime backend errors fail open to a normal live call, while invalid configuration is rejected during validation. `namespace` is required and defines one trusted cache-sharing domain. Do not use one namespace across mutually untrusted tenants or upstreams. `header_allowlist` does not replace this trust boundary. ## When to Use It * Development and test loops that replay the same prompts — eligible repeats can reuse stored responses. * CI suites that exercise real models — cached repeats can avoid additional provider calls. * Demos and workshops — reuse stable stored answers while reducing provider calls. * A shared team cache — callers in one trusted sharing domain can use the same Redis store, namespace, and key prefix across processes and machines. * Gateway deployments — enable caching for an agent without touching its code. Reuse requires the request to match exactly after normalization, so prompts that embed volatile content (timestamps, random IDs) will not repeat. The TTL is the maximum entry age, not guaranteed residency; a backend can evict or delete an entry sooner. Use `bypass_rate` to re-run a sample of cacheable calls live and refresh an entry when the new response is stored. ## `plugins.toml` Example ```toml version = 1 [[components]] kind = "adaptive" enabled = true [components.config] version = 1 [components.config.response_cache] ttl_seconds = 3600 # maximum reuse age namespace = "dev-harness" # identifies one trusted cache-sharing domain bypass_rate = 0.0 # 0.1 would re-run 10% of cacheable calls live cache_nondeterministic = false # set true to cache nondeterministic requests [components.config.response_cache.backend] kind = "in_memory" # or "redis" for a shared cache ``` With this configuration the first eligible occurrence of a request runs the provider and attempts to store a complete answer. A later eligible, identical request within the TTL can be served from the store when that write succeeds and the entry remains present. The request's provider surface is auto-detected from its shape, so there is nothing else to configure. For file discovery and precedence rules, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ## Plugin Configuration Use plugin configuration when the application should let NeMo Relay own the cache lifecycle. #### Python ```python import asyncio import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( response_cache=nemo_relay.adaptive.ResponseCacheConfig( ttl_seconds=3600, namespace="dev-harness", ), ) plugin_config = nemo_relay.plugin.PluginConfig( components=[nemo_relay.adaptive.ComponentSpec(adaptive_config)] ) report = nemo_relay.plugin.validate(plugin_config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) def call_model(_request): return { "model": "gpt-4o", "choices": [{ "message": {"role": "assistant", "content": "NeMo Relay instruments agent calls."}, "finish_reason": "stop", }], } async def main(): await nemo_relay.plugin.initialize(plugin_config) try: # Managed calls need no changes. The first call runs the provider; # an identical repeat can be served from the cache. request = nemo_relay.LLMRequest( {}, { "model": "gpt-4o", "messages": [{"role": "user", "content": "What is NeMo Relay?"}], "temperature": 0, }, ) await nemo_relay.llm.execute("openai", request, call_model) await nemo_relay.llm.execute("openai", request, call_model) finally: await nemo_relay.plugin.clear_async() asyncio.run(main()) ``` #### Node.js ```js const adaptive = require("nemo-relay-node/adaptive"); const plugin = require("nemo-relay-node/plugin"); const adaptiveConfig = adaptive.defaultConfig(); adaptiveConfig.responseCache = adaptive.responseCacheConfig({ ttlSeconds: 3600, namespace: "dev-harness", }); const pluginConfig = plugin.defaultConfig(); pluginConfig.components = [adaptive.ComponentSpec(adaptiveConfig)]; const report = plugin.validate(pluginConfig); if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { throw new Error(JSON.stringify(report.diagnostics)); } void (async () => { await plugin.initialize(pluginConfig); try { // Eligible deterministic repeats can be served from the cache. } finally { plugin.clear(); } })().catch((error) => { console.error(error); process.exitCode = 1; }); ``` #### Rust ```rust use nemo_relay::plugin::{ clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, }; use nemo_relay_adaptive::plugin_component::{register_adaptive_component, ComponentSpec}; use nemo_relay_adaptive::{AdaptiveConfig, ResponseCacheConfig}; #[tokio::main] async fn main() -> Result<(), Box> { let mut adaptive = AdaptiveConfig::default(); adaptive.response_cache = Some(ResponseCacheConfig { ttl_seconds: 3600, namespace: "dev-harness".into(), ..ResponseCacheConfig::default() }); let mut plugin_config = PluginConfig::default(); plugin_config.components.push(ComponentSpec::new(adaptive).into()); register_adaptive_component()?; let report = validate_plugin_config(&plugin_config); assert!(!report.has_errors()); let _active = initialize_plugins(plugin_config).await?; // Eligible deterministic repeats can now be served from the cache. clear_plugin_configuration()?; Ok(()) } ``` > **Note** > > Canonical plugin documents and `plugins.toml` use `snake_case`. Python > response-cache fields follow that convention, while Node.js uses `camelCase` at > its response-cache helper surface. `ComponentSpec` and `AdaptiveRuntime` > serialize those fields to canonical keys. Python uses > `BackendSpec.redis(url, key_prefix=...)`; Node.js uses > `adaptive.redisBackend(url, keyPrefix)`. Both Redis helpers default the prefix > to `"nemo_relay:"`, overriding the value in the fields table. Configure the > same `key_prefix` in every process and binding that should share entries. ## Manual API Use the manual runtime API when an integration needs to own the adaptive lifecycle directly instead of activating the top-level plugin component. #### Python ```python import asyncio import nemo_relay adaptive_config = nemo_relay.adaptive.AdaptiveConfig( response_cache=nemo_relay.adaptive.ResponseCacheConfig(namespace="dev-harness"), ) async def main(): runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) await runtime.register() try: # Run instrumented application work here. runtime.wait_for_idle() finally: await runtime.shutdown() asyncio.run(main()) ``` #### Node.js ```js const adaptive = require("nemo-relay-node/adaptive"); const adaptiveConfig = adaptive.defaultConfig(); adaptiveConfig.responseCache = adaptive.responseCacheConfig({ namespace: "dev-harness", }); const runtime = new adaptive.AdaptiveRuntime(adaptiveConfig); void (async () => { await runtime.register(); try { // Run instrumented application work here. runtime.waitForIdle(); } finally { await runtime.shutdown(); } })().catch((error) => { console.error(error); process.exitCode = 1; }); ``` #### Rust ```rust use nemo_relay_adaptive::{AdaptiveConfig, AdaptiveRuntime, ResponseCacheConfig}; #[tokio::main] async fn main() -> Result<(), Box> { let mut adaptive = AdaptiveConfig::default(); adaptive.response_cache = Some(ResponseCacheConfig { namespace: "dev-harness".into(), ..ResponseCacheConfig::default() }); let mut runtime = AdaptiveRuntime::new(adaptive).await?; runtime.register().await?; // Run instrumented application work here. runtime.wait_for_idle(); runtime.shutdown().await?; Ok(()) } ``` ## LLM Responses Only complete, replayable LLM answers are stored: * A response with a non-null `error` or a `status` such as `failed`, `cancelled`, `incomplete`, or `in_progress` is never stored. * Stateful requests bypass the cache entirely: a request that opts into server-side persistence (a non-null `store` value other than `false`, or an OpenAI Responses call without an explicit `store = false`), continues a stored interaction (`previous_response_id`), or references server-side state (`conversation`, `container`). Their answers depend on state the cache key cannot see. * Streaming calls are cached too. On a miss the live chunks are forwarded to the consumer while being assembled into one aggregate response — the same shape a buffered call stores, so buffered and streaming calls share one keyspace. On a hit the stored answer is replayed as provider-native chunks (OpenAI Chat deltas, OpenAI Responses lifecycle events, Anthropic Messages events, or Gemini `generateContent` response chunks), so strict streaming clients parse it like a live stream. Provider-terminal token-limited Chat (`finish_reason = "length"`) and Anthropic (`stop_reason = "max_tokens"`) answers can be stored. OpenAI Responses answers with `status = "incomplete"`, streams without a terminal event, and streams whose content cannot be replayed faithfully are never stored. A streaming request whose surface cannot be inferred runs live, uncached. Streaming publication is write-behind: Relay can report end-of-stream before the backend write finishes. `wait_for_idle()` does not wait for cache writes. An immediate repeat can therefore run live, and concurrent identical misses can each call the provider because the cache does not coalesce them. A buffered hit returns the stored response unchanged. A streaming hit replays an equivalent provider-native stream. Available saved-token and estimated-cost values are reported on the `response_cache` mark, never by editing a buffered response body. ## Cache Keys Two requests hit the same entry when they are the same request after normalization, under the default `key_strategy = "exact_request"`: * The request is decoded to its normalized form and fingerprinted with SHA-256, so provider-shaped differences that mean the same thing collapse to one key. When a request cannot be decoded faithfully, the raw body is fingerprinted instead — that fallback can only cost a miss, never a wrong hit. * Field order and whitespace never matter (RFC 8785 canonicalization). * Only the built-in noise fields `stream`, `user`, `metadata`, and `store` are dropped. All other normalized or raw provider controls remain in the key, including `service_tier`; there is no configurable skip list. * OpenAI Chat requests preserve whether the caller used `max_tokens` or `max_completion_tokens`, because providers can treat the two fields differently. * Tool-call IDs are normalized, so randomly generated per-call IDs do not fragment the keyspace. * Request headers stay out of the key unless named in `header_allowlist`. Allowlist every trusted, non-secret response-affecting header; an omitted header contributes no value. A non-empty normalized allowlist policy also partitions the key, even when a listed header is absent; an empty allowlist preserves the version-1 key shape. Known auth headers are rejected, but validation cannot recognize every custom credential name, so never allowlist credentials. * The provider name and required namespace partition every key. An internal dispatch backend supplied by a routing plugin is also partitioned automatically. A provider name alone cannot distinguish upstreams that reuse that name, so use separate configurations and namespaces for different trusted upstream domains. * Requests containing integers outside the exactly representable RFC 8785 range (less than `-2^53` or greater than `2^53`) bypass the cache. ## Tool-Result Cache The same `response_cache` section can also cache results from [managed tool calls](/instrument-applications/instrument-tool-call). This is a separate, opt-in surface: it shares the configured store and namespace with the LLM cache, but tool keys carry a distinct surface tag and cannot collide with LLM keys. Caching a tool call suppresses the real call. Cache only tools that are read-only and stable for their TTL; do not cache a tool merely because it is idempotent. A cache hit skips even an idempotent write. Classify each tool explicitly before enabling the tool surface: ```toml [components.config.response_cache.tools] enabled = true cache_errors = false # default: do not store conventional in-band error results [components.config.response_cache.tools.classes.docs] cacheable = true ttl_seconds = 300 tool_version = "docs-v1" arg_skip = ["trace_id"] members = ["docs_search", "docs_lookup"] [components.config.response_cache.tools.overrides.docs_search] tool_version = "docs-search-v2" # identifies the deployed tool contract ``` Prefer exact names for cacheable tools. Wildcards are appropriate only for a controlled namespace whose current and future matching tools are guaranteed to remain read-only and TTL-stable. Supported wildcard forms are `prefix*`, `*suffix`, `*contains*`, and the catch-all `*` for noncacheable policies; other placements are rejected. An exact class member wins over a wildcard, and an exact override wins over a wildcard override. Among wildcard matches, Relay selects the most-specific pattern. Configuration validation rejects overlapping wildcard class patterns or wildcard overrides that disagree on `cacheable`, so a broad deny policy cannot silently become a cacheable result. Unmatched tools use `tools.default`, which must remain uncacheable. ### Tool Keys and Identity A tool key includes the namespace, tool name, optional `tool_version`, effective arguments, `arg_skip` policy, and `cache_errors` policy. It has no automatic tenant, scope, caller-identity, or request-header partition. In particular, `header_allowlist` applies only to LLM keys. Do not cache a tool whose result depends on tenant identity, caller identity, the active scope, permissions, ambient state, or another input absent from its arguments. If a cacheable tool needs a caller or tenant partition, add a trusted discriminator to the real arguments in a tool request interceptor before the cache runs. A scope-local execution interceptor is not a cache partition. `arg_skip` removes only top-level argument keys before keying. Skip only fields that cannot change the result, such as tracing metadata. The normalized `arg_skip` policy itself is part of key identity, so changing it starts a separate keyspace instead of reusing values created under a different policy. ### Tool Errors and Middleware Order Tool callbacks that return an actual execution error are never stored. By default, Relay also does not store a JSON object with a non-null `error` field, `isError = true`, or `is_error = true`; these are conventional in-band error signals rather than a universal tool-result schema. Set `tools.cache_errors = true` only when such results are stable and safe to reuse. With the default, a sampled refresh that returns one of these error-shaped values leaves a previously stored successful result in place. Tool conditional-execution guardrails and tool request interceptors run before the cache derives its key. Sanitize guardrails affect emitted observability payloads only; they do not change the real arguments, result, or cache key. `tools.priority` controls the tool execution interceptor: lower priorities run outermost. A hit returns before later execution interceptors and the managed callback. ## Observability Every cache decision emits a `response_cache` mark with `data.status` set to one of: | Status | Meaning | | -------- | ------------------------------------------------------------------------------------------------------------------ | | `hit` | Served the stored result; the provider or tool callback was skipped. | | `miss` | No entry was served; the call ran live. After an ordinary lookup miss, Relay attempts to store a cacheable result. | | `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. | Cache mark attributes use `nemo_relay.response_cache.*`. They can include `backend`, `surface`, `key_hash`, `ttl_ms`, and `age_ms`. LLM hit marks can also include `saved_tokens` and `saved_cost_usd`. Tool hit marks can include `saved_invocations`. Tool marks use `surface = "tool"`. Unclassified tools that use the default policy and explicitly uncacheable tools emit a `bypass` mark with `reason = "uncacheable"` before Relay runs the tool. Other bypass reasons include `sampled`, `stateful_store`, and `stream_no_codec`. A failed cache-store operation produces a `miss` with `reason = "store_error"`. Cache marks do not contain prompts, answers, or credentials. Treat `key_hash` as sensitive because an observer might identify an input by guessing and hashing it. `nemo-relay doctor` reports the cache state: `not configured` when the section is absent, `configured but disabled (adaptive plugin disabled)` when the adaptive component is off, `on; backend '' reachable` when healthy, and a failure when the config is invalid or the backend is unreachable. When the tool surface is configured, `Response cache (tools)` reports `configured but disabled` when its switch is off. When it is on, the line reports the number of cacheable classes and cacheable overrides, plus the default policy. ## Fields | Field | Default | Notes | | --------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ttl_seconds` | `3600` | Maximum entry age in seconds; a backend can evict an entry sooner. | | `namespace` | `""` (unconfigured) | Required non-empty trust-domain partition folded into every key. Empty or whitespace-only values are rejected. | | `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | | `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call attempts to refresh the entry when its result is cacheable and the write succeeds. | | `cache_nondeterministic` | `false` | Only requests with an explicit numeric `temperature = 0` are eligible. Set `true` to cache and reuse sampled responses. | | `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | | `header_allowlist` | `[]` | Trusted, non-secret response-affecting headers folded into the key (case-insensitive). A non-empty normalized allowlist policy also partitions the key. Known auth headers are rejected. | | `backend.kind` | `"in_memory"` | `"in_memory"`, or `"redis"` (requires building with the `redis-backend` feature). | | `backend.config.max_bytes` | 256 MiB | In-memory size budget; the oldest entries are evicted first. | | `backend.config.url` | — | Redis connection URL. Required for the `redis` backend. | | `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. | ### Tool Cache Fields The following fields configure the separately opt-in tool-result surface: | Field | Default | Notes | | ------------------------ | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tools.enabled` | `false` | Master switch. Classes and overrides are validated even when the surface is disabled. | | `tools.priority` | `100` | Tool execution-intercept priority. Lower values run earlier and outermost. | | `tools.cache_errors` | `false` | Store conventional in-band error objects only when `true`; callback errors are never stored. | | `tools.default` | uncacheable | Policy for tools that match no class. Keep it uncacheable; classify every cacheable tool through a named class or override. | | `tools.classes.` | — | Named policy with `cacheable`, optional `ttl_seconds`, `bypass_rate`, and `tool_version`, top-level `arg_skip`, and `members` containing exact names or supported wildcard forms. An omitted TTL or bypass rate inherits the response-cache value. | | `tools.overrides.` | — | Per-tool refinement after class resolution. `cacheable`, `ttl_seconds`, `bypass_rate`, and `tool_version` override when supplied; `arg_skip` replaces the class list, including when set to `[]`. | A routing plugin that sets `x-nemo-relay-internal-dispatch-backend` must use a priority lower than `response_cache.priority` so backend selection runs before cache key derivation. To derive keys before ACG rewrites requests, set `response_cache.priority` lower than `acg.priority`. > **Warning** > > Cached LLM responses and tool results are stored unredacted. PII sanitize > guardrails rewrite emitted telemetry, never payloads, so the store holds full > result bodies. Cache entries can also store provider and model diagnostics plus > the key fingerprint; they do not store full LLM request bodies or headers. A > shared Redis backend must be trusted and access-controlled. Use a separate > configuration and namespace for each mutually untrusted tenant or upstream > domain. `backend.config.max_bytes` limits only the in-memory backend; configure > Redis capacity and eviction in Redis itself. ## Common Validation Failures * `namespace` is empty or whitespace-only, `ttl_seconds` is `0`, or `bypass_rate` is outside `[0.0, 1.0]`. * `key_strategy` is not `"exact_request"`. * `header_allowlist` names an auth header such as `authorization` or `x-api-key`. * `in_memory` `max_bytes` is zero or is not an integer. * `backend.kind` is unknown; or `redis` has a missing, non-string, or whitespace-only `backend.config.url`, uses a non-string `key_prefix`, or is unavailable because Relay was built without the `redis-backend` feature. * A tool policy sets `ttl_seconds = 0` or a `bypass_rate` outside `[0.0, 1.0]`, or the same member name or pattern appears in two classes. * A tool class member or override key uses an unsupported wildcard form. * Overlapping wildcard class patterns or wildcard overrides disagree on `cacheable`. `tools.default.cacheable` and a cacheable catch-all `*` member or override are rejected; use named classes to limit caching to explicitly read-only tools. Tool-cache diagnostics use the following error codes. | Diagnostic | Condition | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `response_cache.tool_default_members` | `tools.default.members` is non-empty. The default bucket is not a matcher. | | `response_cache.tool_cacheable_default` | `tools.default.cacheable` is true. | | `response_cache.tool_multiple_classes` | The same member or pattern appears in more than one named class. | | `response_cache.tool_invalid_pattern` | A class member or override key is not exact, `*`, `prefix*`, `*suffix`, or `*contains*`. | | `response_cache.tool_invalid_ttl` | A class, default policy, or override sets `ttl_seconds = 0`. | | `response_cache.tool_invalid_bypass_rate` | A class, default policy, or override sets `bypass_rate` outside `[0.0, 1.0]`. | | `response_cache.tool_catch_all_member` | A cacheable class uses a catch-all `*` member. | | `response_cache.tool_catch_all_override` | A cacheable `*` override applies to every tool. | | `response_cache.tool_conflicting_classes` | Overlapping wildcard members in different classes have different `cacheable` values. | | `response_cache.tool_conflicting_overrides` | Overlapping wildcard overrides have different `cacheable` declarations. An omitted value inherits policy, so it cannot safely overlap an explicit value. |