nemoguardrails.rails.llm.buffer
Module Contents
Classes
Functions
API
Abstract base class for buffer strategies in streaming output rails.
This class defines the interface for buffer strategies that manage how streaming chunks are buffered and processed for output rails. Concrete implementations should handle the accumulation and yielding of chunks in a way that optimizes output rails processing while maintaining streaming performance.
The interface separates concerns:
- Buffer management logic (process_stream)
- Chunk representation formatting (format_chunks)
Note:
All concrete implementations must implement from_config, process_stream,
and format_chunks methods to provide configuration-based
instantiation, chunk processing, and string representation capabilities.
Callable interface that delegates to process_stream.
It delegates to the process_stream method and can
be extended to add common functionality like validation, logging,
or error handling.
Example
>>> strategy = SomeBufferStrategy() >>> # both of these work: >>> async for batch in strategy.process_stream(handler): … context_formatted = strategy.format_chunks(batch.processing_context) >>> async for batch in strategy(handler): # delegates to process_stream … user_formatted = strategy.format_chunks(batch.user_output_chunks)
Parameters:
An async iterator that yields individual string chunks from the LLM stream.
Format chunks into a string representation for user consumption.
This method defines how chunks should be formatted into a string representation. Different strategies might join chunks differently (e.g., preserving spaces, adding separators, etc.).
Example
>>> strategy = SomeBufferStrategy() >>> chunks = [“Hello”, ” ”, “world”] >>> result = strategy.format_chunks(chunks) >>> print(result) # “Hello world”
Parameters:
List of chunk tokens to be formatted.
Returns: str
String representation of the chunks ready for consumers.
Create a buffer strategy instance from configuration.
Parameters:
OutputRailsStreamingConfig — Configuration object containing
buffer strategy parameters.
Returns: BufferStrategy
A configured buffer strategy instance.
Process streaming chunks and yield chunk batches.
This is the main method that concrete buffer strategies must implement. It defines how chunks from the streaming handler should be buffered, processed, and yielded as ChunkBatch objects.
Example
>>> strategy = SomeBufferStrategy() >>> async for chunk_batch in strategy.process_stream(handler): … # for output rails processing (needs context): … context_formatted = strategy.format_chunks(chunk_batch.processing_context) … # for user output (new content only): … user_formatted = strategy.format_chunks(chunk_batch.user_output_chunks) … print(f”Processing: {context_formatted}”) … print(f”User: {user_formatted}”)
Parameters:
An async iterator that yields individual string chunks from the LLM stream.
Bases: NamedTuple
Represents a batch of processed chunks from a buffer strategy.
This class contains the raw chunk data from buffer processing. For string representation of chunks, use the buffer strategy’s format_chunks() method.
Example
>>> async for chunk_batch in buffer_strategy.process_stream(handler): … # for output rails processing (needs context): … context_str = buffer_strategy.format_chunks(chunk_batch.processing_context) … analyze_content(context_str) … … # for user output (only new content): … user_output = buffer_strategy.format_chunks(chunk_batch.user_output_chunks) … yield_to_user(user_output) … … # or iterate over raw chunks: … for chunk in chunk_batch.user_output_chunks: … process_individual_chunk(chunk)
Chunks to be used for output rails processing, including context from previous chunks.
New chunks to be streamed to the end user in their original token format. Use this for user output or when you only need the newly processed content.
Bases: BufferStrategy
A rolling buffer strategy for streaming output rails processing.
This strategy accumulates incoming chunks in a buffer and yields them in batches when the buffer reaches the specified chunk size. It maintains context from previous chunks to ensure continuity in processing output rails.
The buffer operates by:
- Accumulating incoming chunks until reaching the chunk size threshold
- Yielding a processing buffer (with context) and new chunks to process
- Retaining context tokens for the next processing round
- Yielding any remaining chunks at the end of the stream
Example
>>> config = OutputRailsStreamingConfig(context_size=2, chunk_size=4) >>> buffer = RollingBuffer.from_config(config) >>> async for chunk_batch in buffer.process_stream(stream_handler): … # for output rails processing (needs context) … processing_text = buffer.format_chunks(chunk_batch.processing_context) … # For user output (new content only) … user_text = buffer.format_chunks(chunk_batch.user_output_chunks) … pass >>> # or use the callable interface: >>> async for chunk_batch in buffer(stream_handler): … # same as above, delegates to process_stream … processing_text = buffer.format_chunks(chunk_batch.processing_context) … pass
Parameters:
Number of tokens carried over from previous chunks to provide context for continuity. Defaults to 5.
Number of tokens in each processing chunk. This determines the size of token blocks on which output rails are applied. Defaults to 10.
The processing buffer includes context from previous chunks, while user_output_chunks contains only the tokens to be yielded to the user.
Tracks the total number of chunks yielded to the user.
Generate string representation of chunks preserving original token format.
The RollingBuffer strategy preserves the original token format by joining chunks without modification, maintaining spaces and formatting as they appeared in the original LLM output.
Example
>>> buffer = RollingBuffer() >>> chunks = [“Hello”, ” ”, “world”, ”!”] >>> result = buffer.format_chunks(chunks) >>> print(result) # “Hello world!”
Parameters:
List of chunk tokens to be formatted.
Returns: str
String representation preserving original token spacing and format.
Create a RollingBuffer instance from a streaming configuration.
Example
>>> config = OutputRailsStreamingConfig(context_size=3, chunk_size=6) >>> buffer = RollingBuffer.from_config(config)
Parameters:
OutputRailsStreamingConfig — Configuration object containing
context_size and chunk_size parameters.
Returns:
A new RollingBuffer instance configured with the provided parameters.
Process streaming chunks using rolling buffer strategy.
This method implements the rolling buffer logic, accumulating chunks and yielding them in batches with context for output rails processing. The buffer maintains a sliding window of context tokens for continuity.
Example
>>> async def stream_handler(): … for chunk in [“Hello”, ” ”, “world”, ”!”]: … yield chunk >>> >>> buffer = RollingBuffer(context_size=1, chunk_size=2) >>> async for chunk_batch in buffer.process_stream(stream_handler()): … print(f”Processing buffer: {chunk_batch.processing_context}”) … print(f”New chunks: {chunk_batch.user_output_chunks}”) … # for output rails processing (with context): … context_str = buffer.format_chunks(chunk_batch.processing_context) … # for user output (new content only): … user_str = buffer.format_chunks(chunk_batch.user_output_chunks) … print(f”Processing: ‘{context_str}’, User: ‘{user_str}’”)
Parameters:
An async iterator that yields individual string chunks from the LLM stream.
The method resets the total_yielded counter at the start of each streaming session to ensure accurate tracking.
Create a buffer strategy from the given configuration.
Example
>>> config = OutputRailsStreamingConfig(context_size=2, chunk_size=4) >>> strategy = get_buffer_strategy(config) >>> isinstance(strategy, RollingBuffer) True
Parameters:
OutputRailsStreamingConfig — Configuration object specifying
the buffer strategy parameters.
Returns: BufferStrategy
A configured buffer strategy instance. Currently returns a RollingBuffer instance.
This is currently a simple factory that only returns RollingBuffer instances. Future versions may support multiple buffer strategies with a registry pattern.