`nemo_microservices.types.shared_params.output_rails_streaming_config`#

Module Contents#

class nemo_microservices.types.shared_params.output_rails_streaming_config.OutputRailsStreamingConfig#

Bases: typing_extensions.TypedDict

chunk_size: int#

None

The number of tokens in each processing chunk.

This is the size of the token block on which output rails are applied.

context_size: int#

None

The number of tokens carried over from the previous chunk to provide context for continuity in processing.

enabled: bool#

None

Enables streaming mode when True.

stream_first: bool#

None

If True, token chunks are streamed immediately before output rails are applied.