Skip to navigation

Module Contents

Classes

NameDescription
StreamingHandlerProvider-agnostic streaming handler with prefix/suffix/stop handling.
_EndOfStream-

Data

END_OF_STREAM

log

API

class nemoguardrails.streaming.StreamingHandler(
enable_print: bool = False,
enable_buffer: bool = False,
include_metadata: typing.Optional[bool] = False,
include_generation_metadata: typing.Optional[bool] = None,
allow_tool_calls: bool = True
)

Bases: AsyncIterator

Provider-agnostic streaming handler with prefix/suffix/stop handling.

Implements AsyncIterator interface so it can be used directly to stream back the response. Chunks are pushed via push_chunk() and consumed via async iteration.

_producer_task
Optional[Task] = None
_stop
= []
buffer
= ''
completion
= ''
current_chunk
= ''
current_metadata
= {}
k
= 0
queue
= asyncio.Queue()
stop
List[str]
streaming_finished_event
= asyncio.Event()
top_k_nonempty_lines_event
= asyncio.Event()
uid
= new_uuid()
nemoguardrails.streaming.StreamingHandler.__aiter__()
nemoguardrails.streaming.StreamingHandler.__anext__()
async
nemoguardrails.streaming.StreamingHandler._flush_current_chunk()
async
nemoguardrails.streaming.StreamingHandler._notify_buffer_ready()
nemoguardrails.streaming.StreamingHandler._observe_producer(
task: asyncio.Task
)
nemoguardrails.streaming.StreamingHandler._process(
metadata: typing.Optional[typing.Dict[str, typing.Any]] = None
)
async

Process a chunk of text.

If we’re in buffering mode, record the text. Otherwise, update the full completion, check for stop tokens, and enqueue the chunk.

nemoguardrails.streaming.StreamingHandler._validate_chunk(
chunk
)
nemoguardrails.streaming.StreamingHandler._wait_for_event_or_producer(
event: asyncio.Event
)
async
nemoguardrails.streaming.StreamingHandler.disable_buffering()
async

When we disable the buffer, we process the buffer as a chunk.

nemoguardrails.streaming.StreamingHandler.enable_buffering()
async
nemoguardrails.streaming.StreamingHandler.finish()
async

Signal end of stream.

nemoguardrails.streaming.StreamingHandler.push_chunk(
metadata: typing.Optional[typing.Dict[str, typing.Any]] = None
)
async

Push text or a structured tool-call event to the stream.

Parameters:

chunk
Union[str, ToolCallChunk, ToolCallDeltaChunk, _EndOfStream, None]

Union[str, ToolCallChunk , ToolCallDeltaChunk , _EndOfStream, None] — Text, completed tool calls, tool-call deltas, None, or the END_OF_STREAM sentinel.

metadata
Optional[Dict[str, Any]]Defaults to None

Optional metadata about the generation.

nemoguardrails.streaming.StreamingHandler.set_pattern(
prefix: typing.Optional[str] = None,
suffix: typing.Optional[str] = None
)

Sets the pattern that is expected.

If a prefix or a suffix are specified, they will be removed from the output.

nemoguardrails.streaming.StreamingHandler.set_producer_task(
task: asyncio.Task
)
nemoguardrails.streaming.StreamingHandler.wait()
async

Waits until the stream finishes and returns the full completion.

nemoguardrails.streaming.StreamingHandler.wait_top_k_nonempty_lines(
k: int
)
async

Waits for top k non-empty lines from the LLM.

When k lines have been received (and k+1 has been started) it will return and remove them from the buffer.

class nemoguardrails.streaming._EndOfStream()
nemoguardrails.streaming.END_OF_STREAM = _EndOfStream()
nemoguardrails.streaming.log = logging.getLogger(__name__)