> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/llms.txt. # nemoguardrails.llm.cache General-purpose caching utilities for NeMo Guardrails. ## Submodules * **[`nemoguardrails.llm.cache.interface`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/interface)** * **[`nemoguardrails.llm.cache.lfu`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/lfu)** * **[`nemoguardrails.llm.cache.utils`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/utils)** ## Package Contents ### Classes | Name | Description | | ---------------------------------------------------------------------- | --------------------------------------------------------------------- | | [`CacheInterface`](#nemoguardrails-llm-cache-interface-CacheInterface) | Abstract base class defining the interface for cache implementations. | | [`LFUCache`](#nemoguardrails-llm-cache-lfu-LFUCache) | Least Frequently Used (LFU) Cache implementation. | ### API ```python class nemoguardrails.llm.cache.CacheInterface() ``` Abstract Abstract base class defining the interface for cache implementations. All cache implementations must inherit from this class and implement the required methods to ensure compatibility with the caching system. **`maxsize`** `int` Get the maximum size of the cache. --- ```python nemoguardrails.llm.cache.CacheInterface.clear() -> None ``` abstract Remove all items from the cache. After calling this method, the cache should be empty. ```python nemoguardrails.llm.cache.CacheInterface.contains( key: typing.Any ) -> bool ``` Check if a key exists in the cache. This is an optional method that can be overridden for efficiency. The default implementation uses get() to check existence. **Parameters:** **`key`** `Any` The key to check. --- **Returns:** `bool` True if the key exists in the cache, False otherwise. ```python nemoguardrails.llm.cache.CacheInterface.get( key: typing.Any, default: typing.Any = None ) -> typing.Any ``` abstract Retrieve an item from the cache. **Parameters:** **`key`** `Any` The key to look up in the cache. --- **`default`** `Any` — default: None Value to return if key is not found (default: None). --- **Returns:** `Any` The value associated with the key, or default if not found. ```python nemoguardrails.llm.cache.CacheInterface.get_or_compute( key: typing.Any, compute_fn: typing.Callable[[], typing.Any], default: typing.Any = None ) -> typing.Any ``` async Atomically get a value from the cache or compute it if not present. This method ensures that the compute function is called at most once even in the presence of concurrent requests for the same key. This is an optional method with a default implementation. Cache implementations should override this for better thread-safety guarantees. **Parameters:** **`key`** `Any` The key to look up --- **`compute_fn`** `Callable[[], Any]` Async function to compute the value if key is not found --- **`default`** `Any` — default: None Value to return if compute_fn raises an exception --- **Returns:** `Any` The cached value or the computed value ```python nemoguardrails.llm.cache.CacheInterface.get_stats() -> dict ``` Get cache statistics. The default implementation returns a message indicating that statistics tracking is not supported. **Returns:** `dict` Dictionary with cache statistics. The format and contents ```python nemoguardrails.llm.cache.CacheInterface.is_empty() -> bool ``` abstract Check if the cache is empty. **Returns:** `bool` True if the cache contains no items, False otherwise. ```python nemoguardrails.llm.cache.CacheInterface.log_stats_now() -> None ``` Force immediate logging of cache statistics. This is an optional method that cache implementations can override if they support statistics logging. The default implementation does nothing. Implementations that support statistics logging should output the current cache statistics to their configured logging backend. ```python nemoguardrails.llm.cache.CacheInterface.put( key: typing.Any, value: typing.Any ) -> None ``` abstract Store an item in the cache. If the cache is at maxsize, this method should evict an item according to the cache's eviction policy (e.g., LFU, LRU, etc.). **Parameters:** **`key`** `Any` The key to store. --- **`value`** `Any` The value to associate with the key. --- ```python nemoguardrails.llm.cache.CacheInterface.reset_stats() -> None ``` Reset cache statistics. This is an optional method that cache implementations can override if they support statistics tracking. The default implementation does nothing. ```python nemoguardrails.llm.cache.CacheInterface.size() -> int ``` abstract Get the current number of items in the cache. **Returns:** `int` The number of items currently stored in the cache. ```python nemoguardrails.llm.cache.CacheInterface.supports_stats_logging() -> bool ``` Check if this cache implementation supports statistics logging. The default implementation returns False. Cache implementations that support statistics logging should override this to return True when logging is enabled. **Returns:** `bool` True if the cache supports statistics logging, False otherwise. ```python class nemoguardrails.llm.cache.LFUCache( maxsize: int, track_stats: bool = False, stats_logging_interval: typing.Optional[float] = None ) ``` **Bases:** [CacheInterface](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/interface#nemoguardrails-llm-cache-interface-CacheInterface) Least Frequently Used (LFU) Cache implementation. When the cache reaches maxsize, it evicts the least frequently used item. If there are ties in frequency, it evicts the least recently used among them. **`_computing`** `dict[Any, Future] = {}` --- **`_lock`** `= threading.RLock()` --- **`freq_map`** `dict[int, DoublyLinkedList] = {}` --- **`key_map`** `dict[Any, LFUNode] = {}` --- **`maxsize`** `int` Get the maximum size of the cache. --- **`min_freq`** `= 0` --- **`stats`** --- ```python nemoguardrails.llm.cache.LFUCache._check_and_log_stats() -> None ``` Check if enough time has passed and log stats if needed. ```python nemoguardrails.llm.cache.LFUCache._evict_lfu() -> None ``` Evict the least frequently used item from the cache. ```python nemoguardrails.llm.cache.LFUCache._log_stats() -> None ``` Log current cache statistics. ```python nemoguardrails.llm.cache.LFUCache._update_node_freq( node: nemoguardrails.llm.cache.lfu.LFUNode ) -> None ``` Update the frequency of a node and move it to the appropriate frequency list. ```python nemoguardrails.llm.cache.LFUCache.clear() -> None ``` Clear all items from the cache. ```python nemoguardrails.llm.cache.LFUCache.contains( key: typing.Any ) -> bool ``` Check if a key exists in the cache without updating its frequency. This is more efficient than the default implementation which uses get() and has the side effect of updating frequency counts. **Parameters:** **`key`** `Any` The key to check --- **Returns:** `bool` True if the key exists in the cache, False otherwise ```python nemoguardrails.llm.cache.LFUCache.get( key: typing.Any, default: typing.Any = None ) -> typing.Any ``` Get an item from the cache. **Parameters:** **`key`** `Any` The key to look up --- **`default`** `Any` — default: None Value to return if key is not found --- **Returns:** `Any` The value associated with the key, or default if not found ```python nemoguardrails.llm.cache.LFUCache.get_or_compute( key: typing.Any, compute_fn: typing.Callable[[], typing.Any], default: typing.Any = None ) -> typing.Any ``` async Atomically get a value from the cache or compute it if not present. This method ensures that the compute function is called at most once even in the presence of concurrent requests for the same key. **Parameters:** **`key`** `Any` The key to look up --- **`compute_fn`** `Callable[[], Any]` Async function to compute the value if key is not found --- **`default`** `Any` — default: None Value to return if compute_fn raises an exception --- **Returns:** `Any` The cached value or the computed value ```python nemoguardrails.llm.cache.LFUCache.get_stats() -> dict ``` Get cache statistics. **Returns:** `dict` Dictionary with cache statistics (if tracking is enabled) ```python nemoguardrails.llm.cache.LFUCache.is_empty() -> bool ``` Check if the cache is empty. ```python nemoguardrails.llm.cache.LFUCache.log_stats_now() -> None ``` Force immediate logging of cache statistics. ```python nemoguardrails.llm.cache.LFUCache.put( key: typing.Any, value: typing.Any ) -> None ``` Put an item into the cache. **Parameters:** **`key`** `Any` The key to store --- **`value`** `Any` The value to associate with the key --- ```python nemoguardrails.llm.cache.LFUCache.reset_stats() -> None ``` Reset cache statistics. ```python nemoguardrails.llm.cache.LFUCache.size() -> int ``` Return the current size of the cache. ```python nemoguardrails.llm.cache.LFUCache.supports_stats_logging() -> bool ``` Check if this cache instance supports stats logging. ## Docs - [nemoguardrails.llm.cache.interface](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/interface.md) - [nemoguardrails.llm.cache.lfu](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/lfu.md) - [nemoguardrails.llm.cache.utils](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/utils.md)