Skip to navigation

nemoguardrails.rails.llm.buffer

View as MarkdownOpen in Claude

Module Contents

Classes

NameDescription
BufferStrategyAbstract base class for buffer strategies in streaming output rails.
ChunkBatchRepresents a batch of processed chunks from a buffer strategy.
RollingBufferA rolling buffer strategy for streaming output rails processing.

Functions

NameDescription
get_buffer_strategyCreate a buffer strategy from the given configuration.

API

class nemoguardrails.rails.llm.buffer.BufferStrategy()
Abstract

Abstract base class for buffer strategies in streaming output rails.

This class defines the interface for buffer strategies that manage how streaming chunks are buffered and processed for output rails. Concrete implementations should handle the accumulation and yielding of chunks in a way that optimizes output rails processing while maintaining streaming performance.

The interface separates concerns:

  • Buffer management logic (process_stream)
  • Chunk representation formatting (format_chunks)

Note: All concrete implementations must implement from_config, process_stream, and format_chunks methods to provide configuration-based instantiation, chunk processing, and string representation capabilities.

nemoguardrails.rails.llm.buffer.BufferStrategy.__call__(
streaming_handler
) -> typing.AsyncGenerator[nemoguardrails.rails.llm.buffer.ChunkBatch, None]
async

Callable interface that delegates to process_stream.

It delegates to the process_stream method and can be extended to add common functionality like validation, logging, or error handling.

Example

>>> strategy = SomeBufferStrategy() >>> # both of these work: >>> async for batch in strategy.process_stream(handler): … context_formatted = strategy.format_chunks(batch.processing_context) >>> async for batch in strategy(handler): # delegates to process_stream … user_formatted = strategy.format_chunks(batch.user_output_chunks)

Parameters:

streaming_handler

An async iterator that yields individual string chunks from the LLM stream.

nemoguardrails.rails.llm.buffer.BufferStrategy.format_chunks(
chunks: typing.List[str]
) -> str
abstract

Format chunks into a string representation for user consumption.

This method defines how chunks should be formatted into a string representation. Different strategies might join chunks differently (e.g., preserving spaces, adding separators, etc.).

Example

>>> strategy = SomeBufferStrategy() >>> chunks = [“Hello”, ” ”, “world”] >>> result = strategy.format_chunks(chunks) >>> print(result) # “Hello world”

Parameters:

chunks
List[str]

List of chunk tokens to be formatted.

Returns: str

String representation of the chunks ready for consumers.

classmethodabstract

Create a buffer strategy instance from configuration.

Parameters:

config
OutputRailsStreamingConfig

OutputRailsStreamingConfig — Configuration object containing buffer strategy parameters.

Returns: BufferStrategy

A configured buffer strategy instance.

nemoguardrails.rails.llm.buffer.BufferStrategy.process_stream(
streaming_handler
) -> typing.AsyncGenerator[nemoguardrails.rails.llm.buffer.ChunkBatch, None]
asyncabstract

Process streaming chunks and yield chunk batches.

This is the main method that concrete buffer strategies must implement. It defines how chunks from the streaming handler should be buffered, processed, and yielded as ChunkBatch objects.

Example

>>> strategy = SomeBufferStrategy() >>> async for chunk_batch in strategy.process_stream(handler): … # for output rails processing (needs context): … context_formatted = strategy.format_chunks(chunk_batch.processing_context) … # for user output (new content only): … user_formatted = strategy.format_chunks(chunk_batch.user_output_chunks) … print(f”Processing: {context_formatted}”) … print(f”User: {user_formatted}”)

Parameters:

streaming_handler

An async iterator that yields individual string chunks from the LLM stream.

class nemoguardrails.rails.llm.buffer.ChunkBatch()

Bases: NamedTuple

Represents a batch of processed chunks from a buffer strategy.

This class contains the raw chunk data from buffer processing. For string representation of chunks, use the buffer strategy’s format_chunks() method.

Example

>>> async for chunk_batch in buffer_strategy.process_stream(handler): … # for output rails processing (needs context): … context_str = buffer_strategy.format_chunks(chunk_batch.processing_context) … analyze_content(context_str) … … # for user output (only new content): … user_output = buffer_strategy.format_chunks(chunk_batch.user_output_chunks) … yield_to_user(user_output) … … # or iterate over raw chunks: … for chunk in chunk_batch.user_output_chunks: … process_individual_chunk(chunk)

processing_context
List[str]

Chunks to be used for output rails processing, including context from previous chunks.

user_output_chunks
List[str]

New chunks to be streamed to the end user in their original token format. Use this for user output or when you only need the newly processed content.

class nemoguardrails.rails.llm.buffer.RollingBuffer(
buffer_context_size: int = 5,
buffer_chunk_size: int = 10
)

Bases: BufferStrategy

A rolling buffer strategy for streaming output rails processing.

This strategy accumulates incoming chunks in a buffer and yields them in batches when the buffer reaches the specified chunk size. It maintains context from previous chunks to ensure continuity in processing output rails.

The buffer operates by:

  1. Accumulating incoming chunks until reaching the chunk size threshold
  2. Yielding a processing buffer (with context) and new chunks to process
  3. Retaining context tokens for the next processing round
  4. Yielding any remaining chunks at the end of the stream

Example

>>> config = OutputRailsStreamingConfig(context_size=2, chunk_size=4) >>> buffer = RollingBuffer.from_config(config) >>> async for chunk_batch in buffer.process_stream(stream_handler): … # for output rails processing (needs context) … processing_text = buffer.format_chunks(chunk_batch.processing_context) … # For user output (new content only) … user_text = buffer.format_chunks(chunk_batch.user_output_chunks) … pass >>> # or use the callable interface: >>> async for chunk_batch in buffer(stream_handler): … # same as above, delegates to process_stream … processing_text = buffer.format_chunks(chunk_batch.processing_context) … pass

Parameters:

buffer_context_size
intDefaults to 5

Number of tokens carried over from previous chunks to provide context for continuity. Defaults to 5.

buffer_chunk_size
intDefaults to 10

Number of tokens in each processing chunk. This determines the size of token blocks on which output rails are applied. Defaults to 10.

The processing buffer includes context from previous chunks, while user_output_chunks contains only the tokens to be yielded to the user.

total_yielded
int = 0

Tracks the total number of chunks yielded to the user.

nemoguardrails.rails.llm.buffer.RollingBuffer.format_chunks(
chunks: typing.List[str]
) -> str

Generate string representation of chunks preserving original token format.

The RollingBuffer strategy preserves the original token format by joining chunks without modification, maintaining spaces and formatting as they appeared in the original LLM output.

Example

>>> buffer = RollingBuffer() >>> chunks = [“Hello”, ” ”, “world”, ”!”] >>> result = buffer.format_chunks(chunks) >>> print(result) # “Hello world!”

Parameters:

chunks
List[str]

List of chunk tokens to be formatted.

Returns: str

String representation preserving original token spacing and format.

nemoguardrails.rails.llm.buffer.RollingBuffer.from_config(
)
classmethod

Create a RollingBuffer instance from a streaming configuration.

Example

>>> config = OutputRailsStreamingConfig(context_size=3, chunk_size=6) >>> buffer = RollingBuffer.from_config(config)

Parameters:

config
OutputRailsStreamingConfig

OutputRailsStreamingConfig — Configuration object containing context_size and chunk_size parameters.

Returns:

A new RollingBuffer instance configured with the provided parameters.

nemoguardrails.rails.llm.buffer.RollingBuffer.process_stream(
streaming_handler
) -> typing.AsyncGenerator[nemoguardrails.rails.llm.buffer.ChunkBatch, None]
async

Process streaming chunks using rolling buffer strategy.

This method implements the rolling buffer logic, accumulating chunks and yielding them in batches with context for output rails processing. The buffer maintains a sliding window of context tokens for continuity.

Example

>>> async def stream_handler(): … for chunk in [“Hello”, ” ”, “world”, ”!”]: … yield chunk >>> >>> buffer = RollingBuffer(context_size=1, chunk_size=2) >>> async for chunk_batch in buffer.process_stream(stream_handler()): … print(f”Processing buffer: {chunk_batch.processing_context}”) … print(f”New chunks: {chunk_batch.user_output_chunks}”) … # for output rails processing (with context): … context_str = buffer.format_chunks(chunk_batch.processing_context) … # for user output (new content only): … user_str = buffer.format_chunks(chunk_batch.user_output_chunks) … print(f”Processing: ‘{context_str}’, User: ‘{user_str}’”)

Parameters:

streaming_handler

An async iterator that yields individual string chunks from the LLM stream.

The method resets the total_yielded counter at the start of each streaming session to ensure accurate tracking.

Create a buffer strategy from the given configuration.

Example

>>> config = OutputRailsStreamingConfig(context_size=2, chunk_size=4) >>> strategy = get_buffer_strategy(config) >>> isinstance(strategy, RollingBuffer) True

Parameters:

config
OutputRailsStreamingConfig

OutputRailsStreamingConfig — Configuration object specifying the buffer strategy parameters.

Returns: BufferStrategy

A configured buffer strategy instance. Currently returns a RollingBuffer instance.

This is currently a simple factory that only returns RollingBuffer instances. Future versions may support multiple buffer strategies with a registry pattern.