ReferenceAPIsRust Library Referencenemo-relay

Module stream

View as Markdown

Generated from cargo doc --no-deps -p nemo-relay -p nemo-relay-adaptive -p nemo-relay-pii-redaction -p nemo-relay-ffi -p nemo-relay-types -p nemo-relay-plugin -p nemo-relay-worker-proto -p nemo-relay-worker.

Streaming LLM response wrapper.

This module provides LlmStreamWrapper, a [Stream] adapter that sits between the raw stream from an LLM API and the consumer. It feeds chunks to a user-supplied collector, and automatically emits lifecycle events when the stream ends.

Pipeline

1raw chunk (Json) -> collector(chunk) -> Ok(()) -> yield chunk
2                                     -> Err(e) -> terminate stream with error
3upstream error -> terminate stream with error -> finalizer() -> queue END sanitization
4stream ends -> finalizer() -> queue END sanitization

The collector receives each chunk (Json) and can accumulate state (e.g., concatenating tokens). If the collector returns Err, the stream terminates immediately with that error. Upstream stream errors also terminate the stream immediately. The finalizer is called once when the stream terminates and returns the aggregated response as Json. That aggregated response is queued for sanitize response guardrails before being included in the END event. Stream termination does not await that queued observability work.

Structs

  • LlmStreamWrapper: Wraps an inner Stream<Item = Result<Json>> of raw chunks and: