nat.middleware.cache.cache_middleware_config#
Configuration for cache middleware.
Classes#
Configuration for cache middleware. |
Module Contents#
- class CacheMiddlewareConfig#
Bases:
nat.data_models.middleware.FunctionMiddlewareBaseConfigConfiguration for cache middleware.
The cache middleware memoizes function outputs based on input similarity, with support for both exact and fuzzy matching.
- Args:
- enabled_mode: Controls when caching is active:
“always”: Cache is always enabled
“eval”: Cache only active when Context.is_evaluating is True
- similarity_threshold: Float in [0, 1.0] for input matching:
1.0: Exact string matching (fastest, recommended)
< 1.0: Fuzzy matching via difflib. Note that difflib is quadratic in the worst case, so large caches with low thresholds may have a performance cost. Values near 0 increase the risk of cache collisions where different inputs return the same cached response.
- max_entries: Upper bound on cached entries. When exceeded, the
least-recently-used entry is evicted. Must be a positive int; defaults to 1024.
- enabled_mode: Literal['always', 'eval'] = None#