nemo_relay.llm

Generated from python/nemo_relay/llm.py.

Module nemo_relay.llm.

LLM lifecycle helpers for non-streaming and streaming calls.

Functions

`call`

def call(name: str, request: LLMRequest, *, handle: ScopeHandle | None = None, attributes: LLMAttributes | None = None, data: Json | None = None, metadata: Json | None = None, model_name: str | None = None, timestamp: datetime | None = None) -> LLMHandle

Start a manual LLM span and return its LLMHandle.

`call_end`

def call_end(handle: LLMHandle, response: Json, *, data: Json | None = None, metadata: Json | None = None, annotated_response: AnnotatedLLMResponse | Mapping[str, Json] | None = None, response_codec: LlmResponseCodec | None = None, timestamp: datetime | None = None) -> None

Finish a manual LLM span started by call().

`execute`

def execute(name: str, request: LLMRequest, func: Callable[[LLMRequest], Json | Awaitable[Json]], *, handle: ScopeHandle | None = None, attributes: LLMAttributes | None = None, data: Json | None = None, metadata: Json | None = None, model_name: str | None = None, codec: LlmCodec | None = None, response_codec: LlmResponseCodec | None = None) -> Awaitable[Json]

Run an LLM call through the managed middleware pipeline.

`stream_execute`

def stream_execute(name: str, request: LLMRequest, func: Callable[[LLMRequest], AsyncIterator[Json]], collector: Callable[[Json], None], finalizer: Callable[[], Json], *, handle: ScopeHandle | None = None, attributes: LLMAttributes | None = None, data: Json | None = None, metadata: Json | None = None, model_name: str | None = None, codec: LlmCodec | None = None, response_codec: LlmResponseCodec | None = None) -> Awaitable[LlmStream]

Run a streaming LLM call through the managed middleware pipeline.

`request_intercepts`

def request_intercepts(name: str, request: LLMRequest) -> LLMRequestInterceptOutcome | Awaitable[LLMRequestInterceptOutcome]

Apply global LLM request intercepts to request.

`conditional_execution`

def conditional_execution(request: LLMRequest) -> Awaitable[None] | None

Run LLM conditional-execution guardrails for request.