Skip to navigation

General-purpose caching utilities for NeMo Guardrails.

Submodules

Package Contents

Classes

NameDescription
CacheInterfaceAbstract base class defining the interface for cache implementations.
LFUCacheLeast Frequently Used (LFU) Cache implementation.

API

class nemoguardrails.llm.cache.CacheInterface()
Abstract

Abstract base class defining the interface for cache implementations.

All cache implementations must inherit from this class and implement the required methods to ensure compatibility with the caching system.

maxsize
int

Get the maximum size of the cache.

nemoguardrails.llm.cache.CacheInterface.clear() -> None
abstract

Remove all items from the cache.

After calling this method, the cache should be empty.

nemoguardrails.llm.cache.CacheInterface.contains(
key: typing.Any
) -> bool

Check if a key exists in the cache.

This is an optional method that can be overridden for efficiency. The default implementation uses get() to check existence.

Parameters:

key
Any

The key to check.

Returns: bool

True if the key exists in the cache, False otherwise.

nemoguardrails.llm.cache.CacheInterface.get(
key: typing.Any,
default: typing.Any = None
) -> typing.Any
abstract

Retrieve an item from the cache.

Parameters:

key
Any

The key to look up in the cache.

default
AnyDefaults to None

Value to return if key is not found (default: None).

Returns: Any

The value associated with the key, or default if not found.

nemoguardrails.llm.cache.CacheInterface.get_or_compute(
key: typing.Any,
compute_fn: typing.Callable[[], typing.Any],
default: typing.Any = None
) -> typing.Any
async

Atomically get a value from the cache or compute it if not present.

This method ensures that the compute function is called at most once even in the presence of concurrent requests for the same key.

This is an optional method with a default implementation. Cache implementations should override this for better thread-safety guarantees.

Parameters:

key
Any

The key to look up

compute_fn
Callable[[], Any]

Async function to compute the value if key is not found

default
AnyDefaults to None

Value to return if compute_fn raises an exception

Returns: Any

The cached value or the computed value

nemoguardrails.llm.cache.CacheInterface.get_stats() -> dict

Get cache statistics.

The default implementation returns a message indicating that statistics tracking is not supported.

Returns: dict

Dictionary with cache statistics. The format and contents

nemoguardrails.llm.cache.CacheInterface.is_empty() -> bool
abstract

Check if the cache is empty.

Returns: bool

True if the cache contains no items, False otherwise.

nemoguardrails.llm.cache.CacheInterface.log_stats_now() -> None

Force immediate logging of cache statistics.

This is an optional method that cache implementations can override if they support statistics logging. The default implementation does nothing.

Implementations that support statistics logging should output the current cache statistics to their configured logging backend.

nemoguardrails.llm.cache.CacheInterface.put(
key: typing.Any,
value: typing.Any
) -> None
abstract

Store an item in the cache.

If the cache is at maxsize, this method should evict an item according to the cache’s eviction policy (e.g., LFU, LRU, etc.).

Parameters:

key
Any

The key to store.

value
Any

The value to associate with the key.

nemoguardrails.llm.cache.CacheInterface.reset_stats() -> None

Reset cache statistics.

This is an optional method that cache implementations can override if they support statistics tracking. The default implementation does nothing.

nemoguardrails.llm.cache.CacheInterface.size() -> int
abstract

Get the current number of items in the cache.

Returns: int

The number of items currently stored in the cache.

nemoguardrails.llm.cache.CacheInterface.supports_stats_logging() -> bool

Check if this cache implementation supports statistics logging.

The default implementation returns False. Cache implementations that support statistics logging should override this to return True when logging is enabled.

Returns: bool

True if the cache supports statistics logging, False otherwise.

class nemoguardrails.llm.cache.LFUCache(
maxsize: int,
track_stats: bool = False,
stats_logging_interval: typing.Optional[float] = None
)

Bases: CacheInterface

Least Frequently Used (LFU) Cache implementation.

When the cache reaches maxsize, it evicts the least frequently used item. If there are ties in frequency, it evicts the least recently used among them.

_computing
dict[Any, Future] = {}
_lock
= threading.RLock()
freq_map
dict[int, DoublyLinkedList] = {}
key_map
dict[Any, LFUNode] = {}
maxsize
int

Get the maximum size of the cache.

min_freq
= 0
stats
nemoguardrails.llm.cache.LFUCache._check_and_log_stats() -> None

Check if enough time has passed and log stats if needed.

nemoguardrails.llm.cache.LFUCache._evict_lfu() -> None

Evict the least frequently used item from the cache.

nemoguardrails.llm.cache.LFUCache._log_stats() -> None

Log current cache statistics.

nemoguardrails.llm.cache.LFUCache._update_node_freq(
) -> None

Update the frequency of a node and move it to the appropriate frequency list.

nemoguardrails.llm.cache.LFUCache.clear() -> None

Clear all items from the cache.

nemoguardrails.llm.cache.LFUCache.contains(
key: typing.Any
) -> bool

Check if a key exists in the cache without updating its frequency.

This is more efficient than the default implementation which uses get() and has the side effect of updating frequency counts.

Parameters:

key
Any

The key to check

Returns: bool

True if the key exists in the cache, False otherwise

nemoguardrails.llm.cache.LFUCache.get(
key: typing.Any,
default: typing.Any = None
) -> typing.Any

Get an item from the cache.

Parameters:

key
Any

The key to look up

default
AnyDefaults to None

Value to return if key is not found

Returns: Any

The value associated with the key, or default if not found

nemoguardrails.llm.cache.LFUCache.get_or_compute(
key: typing.Any,
compute_fn: typing.Callable[[], typing.Any],
default: typing.Any = None
) -> typing.Any
async

Atomically get a value from the cache or compute it if not present.

This method ensures that the compute function is called at most once even in the presence of concurrent requests for the same key.

Parameters:

key
Any

The key to look up

compute_fn
Callable[[], Any]

Async function to compute the value if key is not found

default
AnyDefaults to None

Value to return if compute_fn raises an exception

Returns: Any

The cached value or the computed value

nemoguardrails.llm.cache.LFUCache.get_stats() -> dict

Get cache statistics.

Returns: dict

Dictionary with cache statistics (if tracking is enabled)

nemoguardrails.llm.cache.LFUCache.is_empty() -> bool

Check if the cache is empty.

nemoguardrails.llm.cache.LFUCache.log_stats_now() -> None

Force immediate logging of cache statistics.

nemoguardrails.llm.cache.LFUCache.put(
key: typing.Any,
value: typing.Any
) -> None

Put an item into the cache.

Parameters:

key
Any

The key to store

value
Any

The value to associate with the key

nemoguardrails.llm.cache.LFUCache.reset_stats() -> None

Reset cache statistics.

nemoguardrails.llm.cache.LFUCache.size() -> int

Return the current size of the cache.

nemoguardrails.llm.cache.LFUCache.supports_stats_logging() -> bool

Check if this cache instance supports stats logging.