Skip to navigation

Generated from cargo doc --no-deps -p nemo-relay-builtins -p nemo-relay-diagnostics -p nemo-relay-logging -p nemo-relay-plugin-host -p nemo-relay-codecs -p nemo-relay -p nemo-relay-runtime -p nemo-relay-model-pricing -p nemo-relay-resource-metrics -p nemo-relay-observability -p nemo-relay-adaptive -p nemo-relay-pii-redaction -p nemo-relay-ffi -p nemo-relay-types -p nemo-relay-plugin -p nemo-relay-worker-proto -p nemo-relay-worker.

pub type LlmExecutionFn = Arc<dyn Fn(&str, LlmRequest, LlmExecutionContext, Arc<dyn Fn(LlmRequest) -> Pin<Box<dyn Future<Output = Result<Value, FlowError>> + Send>> + Sync + Send>) -> Pin<Box<dyn Future<Output = Result<Value, FlowError>> + Send>> + Sync + Send>;

Wrap or replace non-streaming LLM execution.

A non-streaming execution intercept receives the logical provider name, the current request, the invocation codec context, and the continuation representing the rest of the chain.

Parameters

  • First argument: Logical provider or model family name.
  • Second argument: Current LLM request.
  • Third argument: Request and unary-response codec context.
  • Fourth argument: Continuation for the remaining execution chain.

Returns

A future resolving to the provider response JSON.

Errors

The future resolves to an error when the intercept or remaining execution chain fails.

Aliased Type

pub struct LlmExecutionFn { /* private fields */ }