Response Cache

Use the response cache when a repeatable managed LLM request or explicitly classified tool call should be served from a store instead of running live. An eligible request or tool call that matches an unexpired cache entry can be served without calling the provider or tool. Buffered LLM hits preserve the stored response shape and usage fields; streaming hits replay an equivalent provider-native stream.

The cache is an optional response_cache section of the Adaptive plugin, not a standalone plugin kind. It is off until the section is present. Its LLM surface applies to managed LLM calls without changing the execution API; its tool-result surface is separately opt-in. By default, only requests with an explicit numeric temperature = 0 are eligible; set cache_nondeterministic = true to opt sampled requests into caching. Runtime backend errors fail open to a normal live call, while invalid configuration is rejected during validation.

namespace is required and defines one trusted cache-sharing domain. Do not use one namespace across mutually untrusted tenants or upstreams. header_allowlist does not replace this trust boundary.

When to Use It

• Development and test loops that replay the same prompts — eligible repeats can reuse stored responses.
• CI suites that exercise real models — cached repeats can avoid additional provider calls.
• Demos and workshops — reuse stable stored answers while reducing provider calls.
• A shared team cache — callers in one trusted sharing domain can use the same Redis store, namespace, and key prefix across processes and machines.
• Gateway deployments — enable caching for an agent without touching its code.

Reuse requires the request to match exactly after normalization, so prompts that embed volatile content (timestamps, random IDs) will not repeat. The TTL is the maximum entry age, not guaranteed residency; a backend can evict or delete an entry sooner. Use bypass_rate to re-run a sample of cacheable calls live and refresh an entry when the new response is stored.

plugins.toml Example

version = 1
[[components]]
kind = "adaptive"
enabled = true
[components.config]
version = 1
[components.config.response_cache]
ttl_seconds = 3600        # maximum reuse age
namespace = "dev-harness" # identifies one trusted cache-sharing domain
bypass_rate = 0.0         # 0.1 would re-run 10% of cacheable calls live
cache_nondeterministic = false # set true to cache nondeterministic requests
[components.config.response_cache.backend]
kind = "in_memory"        # or "redis" for a shared cache

With this configuration the first eligible occurrence of a request runs the provider and attempts to store a complete answer. A later eligible, identical request within the TTL can be served from the store when that write succeeds and the entry remains present. The request’s provider surface is auto-detected from its shape, so there is nothing else to configure. For file discovery and precedence rules, refer to Plugin Configuration Files.

Plugin Configuration

Use plugin configuration when the application should let NeMo Relay own the cache lifecycle.

import asyncio
import nemo_relay
adaptive_config = nemo_relay.adaptive.AdaptiveConfig(
    response_cache=nemo_relay.adaptive.ResponseCacheConfig(
        ttl_seconds=3600,
        namespace="dev-harness",
    ),
)
plugin_config = nemo_relay.plugin.PluginConfig(
    components=[nemo_relay.adaptive.ComponentSpec(adaptive_config)]
)
report = nemo_relay.plugin.validate(plugin_config)["config"]
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
    raise RuntimeError(report["diagnostics"])
def call_model(_request: nemo_relay.LLMRequest) -> dict[str, object]:
    return {
        "model": "gpt-4o",
        "choices": [{
            "message": {"role": "assistant", "content": "NeMo Relay instruments agent calls."},
            "finish_reason": "stop",
        }],
    }
async def main() -> None:
    async with nemo_relay.plugin.activate(plugin_config):
        # Managed calls need no changes. The first call runs the provider;
        # an identical repeat can be served from the cache.
        request = nemo_relay.LLMRequest(
            {},
            {
                "model": "gpt-4o",
                "messages": [{"role": "user", "content": "What is NeMo Relay?"}],
                "temperature": 0,
            },
        )
        await nemo_relay.llm.execute("openai", request, call_model)
        await nemo_relay.llm.execute("openai", request, call_model)
asyncio.run(main())

Manual API

Use the manual runtime API when an integration needs to own the adaptive lifecycle directly instead of activating the top-level plugin component.

import asyncio
import nemo_relay
adaptive_config = nemo_relay.adaptive.AdaptiveConfig(
    response_cache=nemo_relay.adaptive.ResponseCacheConfig(namespace="dev-harness"),
)
async def main() -> None:
    runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict())
    await runtime.register()
    try:
        # Run instrumented application work here.
        runtime.wait_for_idle()
    finally:
        await runtime.shutdown()
asyncio.run(main())

LLM Responses

Only complete, replayable LLM answers are stored:

• A response with a non-null error or a status such as failed, cancelled, incomplete, or in_progress is never stored.
• Stateful requests bypass the cache entirely: a request that opts into server-side persistence (a non-null store value other than false, or an OpenAI Responses call without an explicit store = false), continues a stored interaction (previous_response_id), or references server-side state (conversation, container). Their answers depend on state the cache key cannot see.
• Streaming calls are cached too. On a miss the live chunks are forwarded to the consumer while being assembled into one aggregate response — the same shape a buffered call stores, so buffered and streaming calls share one keyspace. On a hit the stored answer is replayed as provider-native chunks (OpenAI Chat deltas, OpenAI Responses lifecycle events, Anthropic Messages events, or Gemini generateContent response chunks), so strict streaming clients parse it like a live stream. Provider-terminal token-limited Chat (finish_reason = "length") and Anthropic (stop_reason = "max_tokens") answers can be stored. OpenAI Responses answers with status = "incomplete", streams without a terminal event, and streams whose content cannot be replayed faithfully are never stored. A streaming request whose surface cannot be inferred runs live, uncached.

Streaming publication is write-behind: Relay can report end-of-stream before the backend write finishes. wait_for_idle() does not wait for cache writes. An immediate repeat can therefore run live, and concurrent identical misses can each call the provider because the cache does not coalesce them.

A buffered hit returns the stored response unchanged. A streaming hit replays an equivalent provider-native stream. Available saved-token and estimated-cost values are reported on the response_cache mark, never by editing a buffered response body.

Key Strategies

key_strategy selects what counts as “the same request”. Strategies never share cache entries: changing the strategy starts from an empty keyspace.

exact_request (default)

Two requests hit the same entry when they are the same request after normalization:

• The request is decoded to its normalized form and fingerprinted with SHA-256, so provider-shaped differences that mean the same thing collapse to one key. When a request cannot be decoded faithfully, the raw body is fingerprinted instead — that fallback can only cost a miss, never a wrong hit.
• Field order and whitespace never matter (RFC 8785 canonicalization).
• Only the built-in noise fields stream, user, metadata, and store are dropped. All other normalized or raw provider controls remain in the key, including service_tier; there is no configurable skip list.
• OpenAI Chat requests preserve whether the caller used max_tokens or max_completion_tokens, because providers can treat the two fields differently.
• Tool-call IDs are normalized, so randomly generated per-call IDs do not fragment the keyspace.
• Request headers stay out of the key unless named in header_allowlist. Allowlist every trusted, non-secret response-affecting header; an omitted header contributes no value. A non-empty normalized allowlist policy also partitions the key, even when a listed header is absent; an empty allowlist preserves the version-1 key shape. Known auth headers are rejected, but validation cannot recognize every custom credential name, so never allowlist credentials.
• The provider name and required namespace partition every key. An internal dispatch backend supplied by a routing plugin is also partitioned automatically. A provider name alone cannot distinguish upstreams that reuse that name, so use separate configurations and namespaces for different trusted upstream domains.
• Requests containing integers outside the exactly representable RFC 8785 range (less than -2^53 or greater than 2^53) bypass the cache.

logical

Everything keys exactly as exact_request except the tools array: each tool is keyed on its full definition with human-readable description text removed, and the array is sorted. Rewording a tool description or reordering the tools array no longer busts the cache; any other change to a tool definition — its name, its parameter schema (including constraints such as enum and required), its type, or its settings — still does.

[components.config.response_cache]
key_strategy = "logical"

Use logical when prompt-engineering iterations on tool descriptions keep invalidating entries whose behavior did not change.

Tool-Result Cache

The same response_cache section can also cache results from managed tool calls. This is a separate, opt-in surface: it shares the configured store and namespace with the LLM cache, but tool keys carry a distinct surface tag and cannot collide with LLM keys.

Caching a tool call suppresses the real call. Cache only tools that are read-only and stable for their TTL; do not cache a tool merely because it is idempotent. A cache hit skips even an idempotent write. Classify each tool explicitly before enabling the tool surface:

[components.config.response_cache.tools]
enabled = true
cache_errors = false # default: do not store conventional in-band error results
[components.config.response_cache.tools.classes.docs]
cacheable = true
ttl_seconds = 300
tool_version = "docs-v1"
arg_skip = ["trace_id"]
members = ["docs_search", "docs_lookup"]
[components.config.response_cache.tools.overrides.docs_search]
tool_version = "docs-search-v2" # identifies the deployed tool contract

Prefer exact names for cacheable tools. Wildcards are appropriate only for a controlled namespace whose current and future matching tools are guaranteed to remain read-only and TTL-stable. Supported wildcard forms are prefix*, *suffix, *contains*, and the catch-all * for noncacheable policies; other placements are rejected. An exact class member wins over a wildcard, and an exact override wins over a wildcard override. Among wildcard matches, Relay selects the most-specific pattern. Configuration validation rejects overlapping wildcard class patterns or wildcard overrides that disagree on cacheable, so a broad deny policy cannot silently become a cacheable result. Unmatched tools use tools.default, which must remain uncacheable.

Tool Keys and Identity

A tool key includes the namespace, tool name, optional tool_version, effective arguments, arg_skip policy, and cache_errors policy. It has no automatic tenant, scope, caller-identity, or request-header partition. In particular, header_allowlist applies only to LLM keys.

Do not cache a tool whose result depends on tenant identity, caller identity, the active scope, permissions, ambient state, or another input absent from its arguments. If a cacheable tool needs a caller or tenant partition, add a trusted discriminator to the real arguments in a tool request interceptor before the cache runs. A scope-local execution interceptor is not a cache partition.

arg_skip removes only top-level argument keys before keying. Skip only fields that cannot change the result, such as tracing metadata. The normalized arg_skip policy itself is part of key identity, so changing it starts a separate keyspace instead of reusing values created under a different policy.

Tool Errors and Middleware Order

Tool callbacks that return an actual execution error are never stored. By default, Relay also does not store a JSON object with a non-null error field, isError = true, or is_error = true; these are conventional in-band error signals rather than a universal tool-result schema. Set tools.cache_errors = true only when such results are stable and safe to reuse. With the default, a sampled refresh that returns one of these error-shaped values leaves a previously stored successful result in place.

Tool conditional-execution guardrails and tool request interceptors run before the cache derives its key. Sanitize guardrails affect emitted observability payloads only; they do not change the real arguments, result, or cache key. tools.priority controls the tool execution interceptor: lower priorities run outermost. A hit returns before later execution interceptors and the managed callback.

Observability

Every cache decision emits a response_cache mark with data.status set to one of:

Cache mark attributes use nemo_relay.response_cache.*. They can include backend, surface, key_hash, ttl_ms, and age_ms. LLM hit marks can also include saved_tokens and saved_cost_usd. Tool hit marks can include saved_invocations.

Tool marks use surface = "tool". Unclassified tools that use the default policy and explicitly uncacheable tools emit a bypass mark with reason = "uncacheable" before Relay runs the tool.

Other bypass reasons include sampled, stateful_store, and stream_no_codec. A failed cache-store operation produces a miss with reason = "store_error".

Cache marks do not contain prompts, answers, or credentials. Treat key_hash as sensitive because an observer might identify an input by guessing and hashing it.

nemo-relay doctor reports the cache state: not configured when the section is absent, configured but disabled (adaptive plugin disabled) when the adaptive component is off, on; backend '<kind>' reachable when healthy, and a failure when the config is invalid or the backend is unreachable. When the tool surface is configured, Response cache (tools) reports configured but disabled when its switch is off. When it is on, the line reports the number of cacheable classes and cacheable overrides, plus the default policy.

Fields

Tool Cache Fields

The following fields configure the separately opt-in tool-result surface:

A routing plugin that sets x-nemo-relay-internal-dispatch-backend must use a priority lower than response_cache.priority so backend selection runs before cache key derivation. To derive keys before ACG rewrites requests, set response_cache.priority lower than acg.priority.

Common Validation Failures

• namespace is empty or whitespace-only, ttl_seconds is 0, or bypass_rate is outside [0.0, 1.0].
• key_strategy is neither "exact_request" nor "logical".
• header_allowlist names an auth header such as authorization or x-api-key.
• in_memory max_bytes is zero or is not an integer.
• backend.kind is unknown; or redis has a missing, non-string, or whitespace-only backend.config.url, uses a non-string key_prefix, or is unavailable because Relay was built without the redis-backend feature.
• A tool policy sets ttl_seconds = 0 or a bypass_rate outside [0.0, 1.0], or the same member name or pattern appears in two classes.
• A tool class member or override key uses an unsupported wildcard form.
• Overlapping wildcard class patterns or wildcard overrides disagree on cacheable. tools.default.cacheable and a cacheable catch-all * member or override are rejected; use named classes to limit caching to explicitly read-only tools.

Tool-cache diagnostics use the following error codes.