Streaming
For examples that reuse config, start with Start With One Run.
Native OpenAI Streaming
Use Runtime.invoke_openai_stream() when the selected adapter provides native
OpenAI Chat Completions chunks. Runtime.supports_openai_streaming reflects the
selected descriptor’s existing capabilities.streaming claim. If the claim is
false, invoke_openai_stream() raises FabricCapabilityError with code
openai_streaming_unavailable.
Run the following example from a NeMo Fabric source checkout after completing the NVIDIA NeMo Agent Toolkit reference adapter bootstrap. Export NVIDIA_API_KEY for the NVIDIA-hosted model selected by its maintained calculator configuration. That configuration discovers the source-only adapter and registered target, and starts its calculator MCP server over stdio; no separately managed agent endpoint is required. The reference adapter supports native OpenAI chunks when the workflow declares the required ChatResponseChunk output schema.
invoke_openai_stream(...) is synchronous and starts exactly one adapter
invocation in the background. Its OpenAIInvokeStream yields only JSON mappings
whose object field is chat.completion.chunk. An empty stream is valid. The
terminal normalized RunResult is not a final chunk; obtain it separately with
await stream.result() and treat it as authoritative.
Only one turn can be active on a runtime. Fully consume the stream before
starting another turn. If iteration stops early, call await stream.aclose();
it drains the invocation and discards unread chunks so the runtime can be reused.
It does not cancel the target invocation. Awaiting stream.result() also drains
and discards any unread chunks before it returns, so consume the iterator first
when the application needs every chunk.
The SDK owns an authenticated listener on the loopback interface and supplies the adapter host with a single-invocation HTTP sink that uses chunked NDJSON framing. It validates chunk shape, framing, authentication, and invocation correlation. This transport is an implementation detail; consumers provide no listener address, token, or protocol configuration.
Native OpenAI streaming does not require NeMo Relay or
start_runtime(..., streaming=True). It is independent of Relay ATOF streaming:
supports_openai_streaming and invoke_openai_stream() describe adapter-native
OpenAI chunks, while supports_streaming and invoke_stream() describe Relay
ATOF. Enabling one does not enable or change the other.
NeMo Relay Streaming
Enable NeMo Relay before starting a runtime to consume raw ATOF records while a
turn runs. Runtime.supports_streaming reports whether this NeMo Relay path is
available. It is separate from RuntimeCapabilities.streaming, which describes
adapter-native progressive output. This separation is intentional:
Runtime.invoke_stream() exposes only ATOF records generated by NeMo Relay; it
does not call the adapter’s native OpenAI stream operation.
Install the matching collector for the default embedded streaming path:
This extra is not required when launch_collector=False selects an externally
managed collector.
The following example streams one invocation and collects its terminal result:
invoke_stream(...) is synchronous and starts the invocation in the background.
The returned InvokeStream is an async iterator of raw NeMo Relay ATOF
dictionaries. The terminal RunResult stays out of band and is available only
through await stream.result(). Treat that terminal result as authoritative:
replace any provisional rendering instead of appending it. Calling
stream.result() does not finalize unread stream records. Calling
Runtime.stop() after the result completes finalizes the stream and discards
those unread records.
Streaming has the following v0.1 constraints:
- Streaming requires two explicit settings: enable NeMo Relay in
FabricConfigand passstreaming=Truetostart_runtime(...). Thestreaming=Trueflag does not enable NeMo Relay by itself; without NeMo Relay telemetry,start_runtime(...)raisesFabricConfigError. By default, streaming starts an embedded collector and configures Relay to publish NDJSON to it. Setlaunch_collector=Falseto use an externally managed collector. In that case, configure exactly onenemo-fabric-streamsink with anhttporhttpscollector base URL andtransport="ndjson"; NVIDIA NeMo Fabric sends Relay to<base-url>/v1/atof. Pi is the exception as Pi streaming supports only the default embedded collector. Do not setlaunch_collector=Falsefor Pi because its extension does not attach the NeMo Fabric request IDs required for external correlation. The embedded collector waits one second for a late Piagent_settledmarker by default. Increasecompletion_wait_timeoutwhen Relay delivery can take longer. - The default
streaming=Falseleaves the existing NeMo Relay configuration unchanged; it neither enables nor disables ATOF. It also avoids collector startup and streaming control traffic for runtimes that use only ATIF, OpenTelemetry, OpenInference, or application-configured ATOF sinks. - One invocation can be active on a runtime. Fully consume the stream or call
await stream.aclose()before starting another turn. - Breaking an
async forloop does not finalize the stream. Callawait stream.aclose()explicitly. It waits for the invocation to finish and discards unread records; it does not cancel the harness turn. - The end of async iteration only means that no more ATOF records are available.
It does not indicate invocation success. Always await
stream.result(). Invocation exceptions raise from that call, while harness-reported failures are represented by the returned result’s status and error fields. - The collector limits each request queue to 1,024 records and 16 MiB of encoded record data. It rejects individual records larger than 1 MiB and applies TCP backpressure for up to 30 seconds. If the queue remains full, the collector drops the record and logs a warning. NeMo Relay can drop records if a consumer stalls longer than its delivery timeout.
- ATOF granularity depends on the NeMo Relay integration. Gateway harnesses expose per-delta event structure, but current ATOF records keep token text in the terminal scope. In-process harnesses expose scope-level progress.
- Reconstruct nested and parallel work with
uuidandparent_uuid. Stream order alone does not define the Deep Agents scope tree. - For request-correlated adapters, the runtime registers its request ID with
the collector before invoking the agent. The collector routes the ATOF root
scope carrying that request ID and its descendants, based on UUID ancestry,
to that invocation’s private queue. With
relay_streaming: true, Remote Agent services receive the ID asmetadata.nemo_fabric_request_idin the mapped request body and must carry it into Relay metadata. Records that cannot be correlated to a registered request are discarded. Pi instead serializes both streaming and plain invocations behind one embedded-collector lease. Streaming capture begins at the firstturn_startand closes atagent_settled. If output is interrupted or late, the collector discards the remainder through that same terminal marker before allowing the next invocation to start. - An external collector can be shared by multiple NeMo Fabric runtimes whose adapters propagate request IDs. It must be reachable by both Relay publishers and NeMo Fabric clients. Use HTTPS and its configured authentication when it is deployed beyond a trusted local boundary.
Claude and Codex streaming use the NeMo Relay nemo-relay gateway CLI and
require a release in the >=0.9,<0.12 range. Their adapter harness and
full extras install the supported CLI. Deep Agents uses its in-process NeMo
Relay integration with Relay 0.11. Hermes Agent v0.22.7 uses Relay 0.11; v0.22.5 with Relay 0.10 remains supported. Hermes runs on Python 3.14, with ADAPTER_PYTHON selecting a separate environment when the application uses another interpreter.
Remote Agent services own their Relay installation and publish to the same
configured nemo-fabric-stream collector
base URL. The local adapter does not install or start Relay for the service.