ReferenceAPIsRust Library Referencenemo-relaycodec

Module streaming

View as Markdown

Generated from cargo doc --no-deps -p nemo-relay -p nemo-relay-adaptive -p nemo-relay-ffi.

Streaming response codecs for the managed LLM execution pipeline.

crate::codec::traits::LlmResponseCodec decodes a complete provider response into a normalized AnnotatedLlmResponse. For streaming providers, the analogous job is to:

  1. consume per-chunk events as they arrive on a streaming HTTP response, and
  2. assemble a single non-streaming-shape JSON payload at end of stream.

Once assembled, the payload can be fed back through the matching crate::codec::traits::LlmResponseCodec to produce an AnnotatedLlmResponse - meaning streaming and non-streaming requests converge on the same observability output without per-route shape duplication.

StreamingCodec is the trait that bundles the two functions (LlmCollectorFn, LlmFinalizerFn) used by crate::api::llm::llm_stream_call_execute. Each provider supplies one impl whose internal state holds whatever incremental information is needed to materialize the final payload.

Structs

  • SseEvent: One decoded SSE frame, paired with the parsed data: payload.
  • SseEventDecoder: Incremental decoder for text/event-stream byte streams that yields one JSON object per complete data: payload.

Traits