> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Native Dynamic Plugins (Rust) > Build in-process Rust shared-library plugins against the NeMo Relay Native ABI v3. Native dynamic plugins are trusted in-process shared libraries. Use them when plugin behavior needs host-process access while still following the Relay plugin contract: a stable kind, JSON component configuration, validation diagnostics, and registration through a component-scoped context. Native plugins are not sandboxed. They run in the gateway process, must not unwind across ABI callbacks, and remain loaded until Relay removes their registered callbacks. ## Manifest Use the following manifest: ```toml manifest_version = 1 [plugin] id = "acme.native_policy" kind = "rust_dynamic" [compat] relay = ">=0.5,<1.0" native_api = "1" [defaults] enabled = false [capabilities] items = ["plugin_native"] [source] artifact = "target/release/libacme_native_policy.dylib" [integrity] sha256 = "sha256:" [load] library = "target/release/libacme_native_policy.dylib" symbol = "nemo_relay_register_plugin" ``` Set `plugin.kind` to `rust_dynamic`, `compat.native_api` to `"1"`, and `capabilities.items` to include `plugin_native`. Relay resolves relative paths from the manifest. The exported symbol must return a descriptor whose `plugin_kind` matches `plugin.id` exactly. If the plugin registers an LLM request intercept, set `compat.relay = ">=0.6,<1.0"`, or another range that excludes Relay 0.5. Relay rejects the registration when the manifest admits a host that predates the version 2 request annotation envelope. ## Create a Native Plugin Create a Rust library project with the following `Cargo.toml` file: ```toml [package] name = "acme-native-policy" version = "0.1.0" edition = "2024" [lib] crate-type = ["cdylib"] [dependencies] nemo-relay-plugin = "0.5.0" serde_json = "1" ``` Add the following implementation to `src/lib.rs`: ```rust use nemo_relay_plugin::{Json, NativePlugin, PluginContext, Result}; use serde_json::Map; struct NativePolicy; impl NativePlugin for NativePolicy { fn plugin_kind(&self) -> &str { "acme.native_policy" } fn register( &mut self, _config: &Map, context: &mut PluginContext<'_>, ) -> Result<()> { context.register_subscriber("audit", |_| {})?; Ok(()) } } nemo_relay_plugin::nemo_relay_plugin!(nemo_relay_register_plugin, || NativePolicy); ``` Build the library with the following command: ```bash cargo build --release ``` Update `source.artifact` and `load.library` with the resulting platform library path, then replace `` with that library's SHA-256 digest. Use `.dylib` on macOS, `.so` on Linux, or `.dll` on Windows. Refer to [Build a Rust Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example) for a complete example with validation, middleware, scopes, and configuration schema support. ## Native ABI v3 The entry symbol receives a `*const NemoRelayNativeHostApiV1` pointer. It points at the v1 prefix of a v3 `NemoRelayNativeHostApiV3` table; check `abi_version` and `struct_size` before casting. The plugin returns a `NemoRelayNativePluginV1` descriptor: ```rust extern "C" fn nemo_relay_register_plugin( host: *const NemoRelayNativeHostApiV1, out: *mut NemoRelayNativePluginV1, ) -> NemoRelayStatus ``` The v3 host table retains the frozen legacy prefix and appends a completion-based asynchronous middleware extension. An entry that rejects the v3 table with `InvalidArg` is retried with the legacy table. Rust plugins using the typed `NativePlugin` APIs continue to work unchanged. Raw ABI plugins can use `PluginContext::register_async_middleware_raw` when a callback must complete later. The callback receives a JSON invocation, an optional continuation for execution intercepts, and a one-shot completion handle. Return `Complete` after resolving or rejecting the completion before the callback returns. Return `Pending` only when retaining the completion; settle it exactly once, call `async_completion_release`, and release an async `next` handle after use. The host marks a completion cancelled when the awaiting runtime work is dropped; late and duplicate settlement is rejected safely. Invoke and finish every `next` operation before resolving or rejecting the owning completion. Relay rejects new calls and cancels unfinished calls after the completion settles. For unary execution intercepts, use `async_next_invoke_result` to call `next` repeatedly or concurrently with an independent result callback for each call. The older completion-coupled `async_next_invoke` is one-shot because its result settles the middleware completion. For incremental stream intercepts, the output stream owns the callback lifetime: `async_next_invoke_stream` may run repeatedly or concurrently until the output stream finishes, rejects, or is cancelled. Each call requires independent callback state. Relay rejects later calls and cancels unfinished calls when the output stream settles. Relay invokes an asynchronous middleware callback synchronously on the Tokio runtime worker that is polling that middleware invocation. There is no stable OS-thread affinity, and separate invocations can run concurrently. After returning `Pending`, a plugin can use its retained completion, stream, or `next` handle from another plugin-owned thread. The host synchronizes those opaque handles, schedules `next` on the captured Relay runtime, and can invoke an incremental downstream-stream callback on a Relay runtime worker. The plugin must synchronize shared `user_data` and callback state, and must keep them alive until the corresponding host-owned registration and all callback-owned handle references are released. Do not race a handle's release operation against settlement, cancellation inspection, stream operations, or `next` invocation using that same reference; serialize the final release after the last such call returns. Event sanitizers registered through this extension still run on Relay's serial publication dispatcher. Scope and mark emission remain synchronous and their sanitized events are delivered later in emission order. The generic v3 completion registration settles one JSON value and rejects the `LlmStreamExecutionIntercept` kind. Register asynchronous stream intercepts with `plugin_context_register_async_stream_middleware` instead. Its dedicated `async_next_invoke_stream` continuation forwards downstream chunks incrementally, so Relay does not buffer the provider stream into an array. The incremental output queue is bounded. `async_stream_push_json` and `async_stream_reject` never block a native callback thread. If either returns `Internal` and the host last-error contains `backpressured`, retain the logical chunk or rejection and retry after the consumer advances. `InvalidArg` means the stream is already closed or cancelled and must not be retried. Check `async_stream_is_cancelled` during longer producer work. For `async_next_invoke_stream`, Relay reports downstream failure or consumer cancellation through one terminal callback with a non-null error, and reports clean completion with `done = true`. Reclaim the callback's `user_data` in that terminal callback. If a chunk callback returns `false`, reclaim `user_data` before returning because Relay does not invoke another callback afterward. Cancelling the one-shot completion supplied to a non-stream execution intercept also aborts any pending `async_next_invoke` continuation. Plugins must still release their callback-owned completion and `next` references exactly once; cancellation only stops the host-side continuation and prevents it from retaining the plugin indefinitely. Legacy v1/v2 middleware callbacks are synchronous and run on the runtime's execution path. They must not block on I/O; use the v3 completion-based API for long-running work. Text and JSON data cross this boundary as host-owned `NemoRelayNativeString` handles. ABI structs also carry scalars, opaque handles, callback pointers, and plugin-owned `user_data`. Do not pass Rust runtime types, trait objects, futures, `serde_json::Value`, or allocator-owned strings across the ABI. ABI callbacks can register these runtime surfaces: * Subscribers * Tool and LLM guardrails or intercepts, including stream intercepts * Marks, scopes, and isolated scope stacks Relay keeps the library alive while those registrations exist and deregisters them before unloading it. LLM sanitize callbacks receive their request or response JSON first, followed by `NemoRelayNativeLlmSanitizeRequestContext` or `NemoRelayNativeLlmSanitizeResponseContext`. Each context contains structured codec identity and a borrowed, callback-lifetime codec handle. `codec_kind` is `None`, `BuiltIn`, `Runtime`, or `Opaque`. `codec_id` is present for `BuiltIn` (one of `openai_chat`, `openai_responses`, or `anthropic_messages`) and `Runtime`, and null for `None` and `Opaque`. The request handle supports host operations to decode an `LlmRequest` into an `AnnotatedLlmRequest` and encode normalized changes onto the original request. The response handle supports decoding response JSON into an `AnnotatedLlmResponse`. A null handle means no codec is active. Runtime and opaque codecs still have non-null handles and support the same operations as a built-in codec. Do not retain a handle or resolved SDK facade after the sanitizer callback returns. The Rust SDK converts the raw structures into `LlmSanitizeRequestContext` and `LlmSanitizeResponseContext`. Call `resolve_codec()` to obtain the safe directional facade. Raw ABI consumers use the `llm_request_codec_decode`, `llm_request_codec_encode`, and `llm_response_codec_decode` host-table functions. The host owns all returned JSON string handles, which callers release with the ordinary host string release operation. A successful null sanitizer output omits the LLM observability payload and its annotation. Returning an error also omits the payload and annotation; Relay records the callback error. Neither case changes the client-visible request or response. Use the `nemo-relay-plugin` crate rather than the host `nemo-relay` runtime crate. Refer to [Build a Rust Native Plugin](/build-plugins/dynamic-plugins/native-dynamic/rust-native-plugin-example) for the SDK-backed example. ## Register the Plugin After you package the manifest and library, register and validate the plugin: ```bash nemo-relay plugins add ./relay-plugin.toml nemo-relay plugins enable acme.native_policy nemo-relay plugins validate acme.native_policy ``` Refer to [Configure Discoverable Plugins](/configure-plugins/discoverable-plugins) for lifecycle and trust-policy configuration.