> This page is for version v0.25.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo/guardrails/latest/llms.txt
> - v0.25.0: https://docs.nvidia.com/nemo/guardrails/v0.25.0/llms.txt
> - v0.24.1: https://docs.nvidia.com/nemo/guardrails/v0.24.1/llms.txt
> - v0.24.0: https://docs.nvidia.com/nemo/guardrails/v0.24.0/llms.txt
> - v0.23.0: https://docs.nvidia.com/nemo/guardrails/v0.23.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/nemo/guardrails/v0.22.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt.

# nemoguardrails.rails.llm.buffer

## Module Contents

### Classes

| Name                                                                | Description                                                          |
| ------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`BufferStrategy`](#nemoguardrails-rails-llm-buffer-BufferStrategy) | Abstract base class for buffer strategies in streaming output rails. |
| [`ChunkBatch`](#nemoguardrails-rails-llm-buffer-ChunkBatch)         | Represents a batch of processed chunks from a buffer strategy.       |
| [`RollingBuffer`](#nemoguardrails-rails-llm-buffer-RollingBuffer)   | A rolling buffer strategy for streaming output rails processing.     |

### Functions

| Name                                                                          | Description                                            |
| ----------------------------------------------------------------------------- | ------------------------------------------------------ |
| [`get_buffer_strategy`](#nemoguardrails-rails-llm-buffer-get_buffer_strategy) | Create a buffer strategy from the given configuration. |

### API

```python
class nemoguardrails.rails.llm.buffer.BufferStrategy()
```

Abstract

Abstract base class for buffer strategies in streaming output rails.

This class defines the interface for buffer strategies that manage how
streaming chunks are buffered and processed for output rails.
Concrete implementations should handle the accumulation and yielding of
chunks in a way that optimizes output rails processing while maintaining
streaming performance.

The interface separates concerns:

* Buffer management logic (process_stream)
* Chunk representation formatting (format_chunks)

Note:
All concrete implementations must implement `from_config`, `process_stream`,
and `format_chunks` methods to provide configuration-based
instantiation, chunk processing, and string representation capabilities.

```python
nemoguardrails.rails.llm.buffer.BufferStrategy.__call__(
    streaming_handler
) -> typing.AsyncGenerator[nemoguardrails.rails.llm.buffer.ChunkBatch, None]
```

async

Callable interface that delegates to process_stream.

It delegates to the `process_stream` method and can
be extended to add common functionality like validation, logging,
or error handling.

**Example**

\>>> strategy = SomeBufferStrategy()
\>>> # both of these work:
\>>> async for batch in strategy.process_stream(handler):
...     context_formatted = strategy.format_chunks(batch.processing_context)
\>>> async for batch in strategy(handler):  # delegates to process_stream
...     user_formatted = strategy.format_chunks(batch.user_output_chunks)

**Parameters:**

**`streaming_handler`**

An async iterator that yields individual string
chunks from the LLM stream.

---

```python
nemoguardrails.rails.llm.buffer.BufferStrategy.format_chunks(
    chunks: typing.List[str]
) -> str
```

abstract

Format chunks into a string representation for user consumption.

This method defines how chunks should be formatted into a string
representation. Different strategies might join chunks differently
(e.g., preserving spaces, adding separators, etc.).

**Example**

\>>> strategy = SomeBufferStrategy()
\>>> chunks = \["Hello", " ", "world"]
\>>> result = strategy.format_chunks(chunks)
\>>> print(result)  # "Hello world"

**Parameters:**

**`chunks`** `List[str]`

List of chunk tokens to be formatted.

---

**Returns:** `str`

String representation of the chunks ready for consumers.

```python
nemoguardrails.rails.llm.buffer.BufferStrategy.from_config(
    config: nemoguardrails.rails.llm.config.OutputRailsStreamingConfig
) -> nemoguardrails.rails.llm.buffer.BufferStrategy
```

classmethod

abstract

Create a buffer strategy instance from configuration.

**Parameters:**

**`config`** `OutputRailsStreamingConfig`

[`OutputRailsStreamingConfig`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/rails/llm/config#nemoguardrails-rails-llm-config-OutputRailsStreamingConfig) — Configuration object containing
buffer strategy parameters.

---

**Returns:** [`BufferStrategy`](#nemoguardrails-rails-llm-buffer-BufferStrategy)

A configured buffer strategy instance.

```python
nemoguardrails.rails.llm.buffer.BufferStrategy.process_stream(
    streaming_handler
) -> typing.AsyncGenerator[nemoguardrails.rails.llm.buffer.ChunkBatch, None]
```

async

abstract

Process streaming chunks and yield chunk batches.

This is the main method that concrete buffer strategies must implement.
It defines how chunks from the streaming handler should be buffered,
processed, and yielded as ChunkBatch objects.

**Example**

\>>> strategy = SomeBufferStrategy()
\>>> async for chunk_batch in strategy.process_stream(handler):
...     # for output rails processing (needs context):
...     context_formatted = strategy.format_chunks(chunk_batch.processing_context)
...     # for user output (new content only):
...     user_formatted = strategy.format_chunks(chunk_batch.user_output_chunks)
...     print(f"Processing: \{context_formatted}")
...     print(f"User: \{user_formatted}")

**Parameters:**

**`streaming_handler`**

An async iterator that yields individual string
chunks from the LLM stream.

---

```python
class nemoguardrails.rails.llm.buffer.ChunkBatch()
```

**Bases:** `NamedTuple`

Represents a batch of processed chunks from a buffer strategy.

This class contains the raw chunk data from buffer processing. For string
representation of chunks, use the buffer strategy's format_chunks() method.

**Example**

\>>> async for chunk_batch in buffer_strategy.process_stream(handler):
...     # for output rails processing (needs context):
...     context_str = buffer_strategy.format_chunks(chunk_batch.processing_context)
...     analyze_content(context_str)
...
...     # for user output (only new content):
...     user_output = buffer_strategy.format_chunks(chunk_batch.user_output_chunks)
...     yield_to_user(user_output)
...
...     # or iterate over raw chunks:
...     for chunk in chunk_batch.user_output_chunks:
...         process_individual_chunk(chunk)

**`processing_context`** `List[str]`

Chunks to be used for output rails processing,
including context from previous chunks.

---

**`user_output_chunks`** `List[str]`

New chunks to be streamed to the end user
in their original token format. Use this for user output or when you
only need the newly processed content.

---

```python
class nemoguardrails.rails.llm.buffer.RollingBuffer(
    buffer_context_size: int = 5,
    buffer_chunk_size: int = 10
)
```

**Bases:** [BufferStrategy](#nemoguardrails-rails-llm-buffer-BufferStrategy)

A rolling buffer strategy for streaming output rails processing.

This strategy accumulates incoming chunks in a buffer and yields them in
batches when the buffer reaches the specified chunk size. It maintains
context from previous chunks to ensure continuity in processing output rails.

The buffer operates by:

1. Accumulating incoming chunks until reaching the chunk size threshold
2. Yielding a processing buffer (with context) and new chunks to process
3. Retaining context tokens for the next processing round
4. Yielding any remaining chunks at the end of the stream

**Example**

\>>> config = OutputRailsStreamingConfig(context_size=2, chunk_size=4)
\>>> buffer = RollingBuffer.from_config(config)
\>>> async for chunk_batch in buffer.process_stream(stream_handler):
...     # for output rails processing (needs context)
...     processing_text = buffer.format_chunks(chunk_batch.processing_context)
...     # For user output (new content only)
...     user_text = buffer.format_chunks(chunk_batch.user_output_chunks)
...     pass
\>>> # or use the callable interface:
\>>> async for chunk_batch in buffer(stream_handler):
...     # same as above, delegates to process_stream
...     processing_text = buffer.format_chunks(chunk_batch.processing_context)
...     pass

**Parameters:**

**`buffer_context_size`** `int` — default: 5

Number of tokens carried over from
previous chunks to provide context for continuity. Defaults to 5.

---

**`buffer_chunk_size`** `int` — default: 10

Number of tokens in each processing
chunk. This determines the size of token blocks on which output
rails are applied. Defaults to 10.

---

> **Note**
>
> The processing buffer includes context from previous chunks, while
> user_output_chunks contains only the tokens to be yielded to the user.

**`total_yielded`** `int = 0`

Tracks the total number of chunks yielded to the user.

---

```python
nemoguardrails.rails.llm.buffer.RollingBuffer.format_chunks(
    chunks: typing.List[str]
) -> str
```

Generate string representation of chunks preserving original token format.

The RollingBuffer strategy preserves the original token format by
joining chunks without modification, maintaining spaces and formatting
as they appeared in the original LLM output.

**Example**

\>>> buffer = RollingBuffer()
\>>> chunks = \["Hello", " ", "world", "!"]
\>>> result = buffer.format_chunks(chunks)
\>>> print(result)  # "Hello world!"

**Parameters:**

**`chunks`** `List[str]`

List of chunk tokens to be formatted.

---

**Returns:** `str`

String representation preserving original token spacing and format.

```python
nemoguardrails.rails.llm.buffer.RollingBuffer.from_config(
    config: nemoguardrails.rails.llm.config.OutputRailsStreamingConfig
)
```

classmethod

Create a RollingBuffer instance from a streaming configuration.

**Example**

\>>> config = OutputRailsStreamingConfig(context_size=3, chunk_size=6)
\>>> buffer = RollingBuffer.from_config(config)

**Parameters:**

**`config`** `OutputRailsStreamingConfig`

[`OutputRailsStreamingConfig`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/rails/llm/config#nemoguardrails-rails-llm-config-OutputRailsStreamingConfig) — Configuration object containing
context_size and chunk_size parameters.

---

**Returns:**

A new RollingBuffer instance configured with the
provided parameters.

```python
nemoguardrails.rails.llm.buffer.RollingBuffer.process_stream(
    streaming_handler
) -> typing.AsyncGenerator[nemoguardrails.rails.llm.buffer.ChunkBatch, None]
```

async

Process streaming chunks using rolling buffer strategy.

This method implements the rolling buffer logic, accumulating chunks
and yielding them in batches with context for output rails processing.
The buffer maintains a sliding window of context tokens for continuity.

**Example**

\>>> async def stream_handler():
...     for chunk in \["Hello", " ", "world", "!"]:
...         yield chunk
\>>>
\>>> buffer = RollingBuffer(context_size=1, chunk_size=2)
\>>> async for chunk_batch in buffer.process_stream(stream_handler()):
...     print(f"Processing buffer: \{chunk_batch.processing_context}")
...     print(f"New chunks: \{chunk_batch.user_output_chunks}")
...     # for output rails processing (with context):
...     context_str = buffer.format_chunks(chunk_batch.processing_context)
...     # for user output (new content only):
...     user_str = buffer.format_chunks(chunk_batch.user_output_chunks)
...     print(f"Processing: '\{context_str}', User: '\{user_str}'")

**Parameters:**

**`streaming_handler`**

An async iterator that yields individual string
chunks from the LLM stream.

---

> **Note**
>
> The method resets the total_yielded counter at the start of each
> streaming session to ensure accurate tracking.

```python
nemoguardrails.rails.llm.buffer.get_buffer_strategy(
    config: nemoguardrails.rails.llm.config.OutputRailsStreamingConfig
) -> nemoguardrails.rails.llm.buffer.BufferStrategy
```

Create a buffer strategy from the given configuration.

**Example**

\>>> config = OutputRailsStreamingConfig(context_size=2, chunk_size=4)
\>>> strategy = get_buffer_strategy(config)
\>>> isinstance(strategy, RollingBuffer)
True

**Parameters:**

**`config`** `OutputRailsStreamingConfig`

[`OutputRailsStreamingConfig`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/rails/llm/config#nemoguardrails-rails-llm-config-OutputRailsStreamingConfig) — Configuration object specifying
the buffer strategy parameters.

---

**Returns:** [`BufferStrategy`](#nemoguardrails-rails-llm-buffer-BufferStrategy)

A configured buffer strategy instance. Currently
returns a RollingBuffer instance.

> **Note**
>
> This is currently a simple factory that only returns RollingBuffer
> instances. Future versions may support multiple buffer strategies
> with a registry pattern.