Module response_cache

This page is for NeMo Relay, version dev. For other versions, use one of these documentation indexes: - Latest (v0.11.1) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt - v0.11.1: https://docs.nvidia.com/nemo/relay/v0.11.1/llms.txt - v0.10.4: https://docs.nvidia.com/nemo/relay/v0.10.4/llms.txt - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt

Generated from cargo doc --no-deps -p nemo-relay-builtins -p nemo-relay-diagnostics -p nemo-relay-logging -p nemo-relay-plugin-host -p nemo-relay-codecs -p nemo-relay -p nemo-relay-runtime -p nemo-relay-model-pricing -p nemo-relay-resource-metrics -p nemo-relay-observability -p nemo-relay-adaptive -p nemo-relay-pii-redaction -p nemo-relay-ffi -p nemo-relay-types -p nemo-relay-plugin -p nemo-relay-worker-proto -p nemo-relay-worker.

Opt-in exact-match LLM response and tool-result cache. Opt-in cache for LLM responses and tool results: a feature of the adaptive plugin, configured through crate::config::AdaptiveConfig::response_cache.

The two surfaces share storage but use disjoint keys.

intercept holds the execution intercepts and storage rules, key the cache-key derivation, store the backends, replay the streaming replay, and mark the observability surface.

Re-exports

• pub use crate::config::ResponseCacheConfig;
• pub use crate::response_cache::config::BackendConfig;
• pub use crate::response_cache::config::ResponseCacheKeyStrategy;
• pub use crate::response_cache::config::ToolCacheConfig;

Modules

• config: Backend selection for the adaptive plugin’s response_cache feature.

Constants

• RESPONSE_CACHE_MARK: Mark-event name emitted on every cache decision.