Module middleware

This page is for NeMo Relay, version dev. For other versions, use one of these documentation indexes: - Latest (v0.11.1) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt - v0.11.1: https://docs.nvidia.com/nemo/relay/v0.11.1/llms.txt - v0.10.4: https://docs.nvidia.com/nemo/relay/v0.10.4/llms.txt - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt

Generated from cargo doc --no-deps -p nemo-relay-builtins -p nemo-relay-diagnostics -p nemo-relay-logging -p nemo-relay-plugin-host -p nemo-relay-codecs -p nemo-relay -p nemo-relay-runtime -p nemo-relay-model-pricing -p nemo-relay-resource-metrics -p nemo-relay-observability -p nemo-relay-adaptive -p nemo-relay-pii-redaction -p nemo-relay-ffi -p nemo-relay-types -p nemo-relay-plugin -p nemo-relay-worker-proto -p nemo-relay-worker.

Shared middleware callback, stream, and invocation-context contracts.

Modules

• callbacks: Middleware callback and stream contracts. Callback type aliases used by the runtime middleware pipeline.
• llm_execution_context: Invocation-scoped codec context. Invocation-scoped codec context for LLM execution intercepts.

Structs

• LlmExecutionContext: Codec access for one LLM execution.
• LlmJsonStream: Stream of JSON chunks produced by the managed streaming LLM pipeline.
• LlmSanitizeRequestContext: Per-call codec identity and capability for an LLM request.
• LlmSanitizeResponseContext: Per-call codec identity and capability for an LLM response.
• ToolExecutionContext: Per-call context delivered to a tool execution intercept.

Enums

• BuiltinLlmCodec: Relay’s built-in LLM codec identities.
• LlmCodecIdentity: Per-call LLM codec identity supplied to sanitizer and SDK callbacks.
• LlmCollector: Collector used to accumulate streaming response chunks.
• LlmFinalizer: Finalizer used to synthesize the aggregate streaming response payload.

Traits

• LlmStreamInner: Internal close-aware stream implementation.

Type Aliases

• ConditionalMiddlewareGuardrailFn: Decide whether matching global runtime registrations remain eligible.
• EventMetadataInjectorFn: Add flat metadata attributes to a fully constructed event.
• EventSanitizeFn: Sanitize mutable observability fields on a fully constructed event.
• EventSubscriberFn: Consume runtime lifecycle events after they are emitted.
• LlmAsyncCollectorFn: Collector that is awaited; see LlmCollector.
• LlmAsyncFinalizerFn: Finalizer that is awaited; see LlmFinalizer.
• LlmBorrowedCollectorFn: Collector that reads each provider JSON chunk without copying it.
• LlmCollectorFn: Per-chunk collector used by the streaming LLM runtime.
• LlmCollectorFuture: Future returned by an asynchronous collector.
• LlmConditionalFn: Decide whether an LLM call is allowed to continue.
• LlmExecutionFn: Wrap or replace non-streaming LLM execution.
• LlmExecutionNextFn: Continuation type invoked by non-streaming LLM execution intercepts.
• LlmFallibleFinalizerFn: Finalizer that can fail; see LlmFinalizer.
• LlmFinalizerFn: Finalizer used to synthesize the aggregate streaming response payload.
• LlmFinalizerFuture: Future returned by an asynchronous finalizer.
• LlmRequestContext: Request codec context exposed to an LLM execution intercept.
• LlmRequestInterceptFn: Rewrite or annotate an LLM request before execution.
• LlmResponseContext: Response codec context exposed to a non-streaming LLM execution intercept.
• LlmSanitizeRequestFn: Sanitize an LLM request before the runtime records it.
• LlmSanitizeResponseFn: Sanitize an LLM response before the runtime records it.
• LlmStreamExecutionFn: Wrap or replace streaming LLM execution.
• LlmStreamExecutionNextFn: Continuation type invoked by streaming LLM execution intercepts.
• ToolConditionalFn: Decide whether a tool call is allowed to continue.
• ToolExecutionFn: Wrap or replace tool execution with access to the full call context.
• ToolExecutionNextFn: Continuation type invoked by tool execution intercepts.
• ToolInterceptFn: Rewrite tool arguments before execution.
• ToolSanitizeFn: Sanitize a tool request payload before the runtime records it.