Skip to navigation

Streaming

View as MarkdownOpen in Claude

For examples that reuse config, start with Start With One Run.

Native OpenAI Streaming

Use Runtime.invoke_openai_stream() when the selected adapter provides native OpenAI Chat Completions chunks. Runtime.supports_openai_streaming reflects the selected descriptor’s existing capabilities.streaming claim. If the claim is false, invoke_openai_stream() raises FabricCapabilityError with code openai_streaming_unavailable.

Run the following example from a NeMo Fabric source checkout after completing the NVIDIA NeMo Agent Toolkit reference adapter bootstrap. Export NVIDIA_API_KEY for the NVIDIA-hosted model selected by its maintained calculator configuration. That configuration discovers the source-only adapter and registered target, and starts its calculator MCP server over stdio; no separately managed agent endpoint is required. The reference adapter supports native OpenAI chunks when the workflow declares the required ChatResponseChunk output schema.

from external.nat.examples.calculator import build_config
from nemo_fabric import Fabric
native_streaming_config = build_config()
async with await Fabric().start_runtime(native_streaming_config) as runtime:
if not runtime.supports_openai_streaming:
raise RuntimeError("the selected adapter does not support OpenAI streaming")
stream = runtime.invoke_openai_stream(input="What is 21 multiplied by 2?")
async for chunk in stream:
print(chunk)
result = await stream.result()
print(result.status, result.output)

invoke_openai_stream(...) is synchronous and starts exactly one adapter invocation in the background. Its OpenAIInvokeStream yields only JSON mappings whose object field is chat.completion.chunk. An empty stream is valid. The terminal normalized RunResult is not a final chunk; obtain it separately with await stream.result() and treat it as authoritative.

Only one turn can be active on a runtime. Fully consume the stream before starting another turn. If iteration stops early, call await stream.aclose(); it drains the invocation and discards unread chunks so the runtime can be reused. It does not cancel the target invocation. Awaiting stream.result() also drains and discards any unread chunks before it returns, so consume the iterator first when the application needs every chunk.

The SDK owns an authenticated listener on the loopback interface and supplies the adapter host with a single-invocation HTTP sink that uses chunked NDJSON framing. It validates chunk shape, framing, authentication, and invocation correlation. This transport is an implementation detail; consumers provide no listener address, token, or protocol configuration.

Native OpenAI streaming does not require NeMo Relay or start_runtime(..., streaming=True). It is independent of Relay ATOF streaming: supports_openai_streaming and invoke_openai_stream() describe adapter-native OpenAI chunks, while supports_streaming and invoke_stream() describe Relay ATOF. Enabling one does not enable or change the other.

NeMo Relay Streaming

Enable NeMo Relay before starting a runtime to consume raw ATOF records while a turn runs. Runtime.supports_streaming reports whether this NeMo Relay path is available. It is separate from RuntimeCapabilities.streaming, which describes adapter-native progressive output. This separation is intentional: Runtime.invoke_stream() exposes only ATOF records generated by NeMo Relay; it does not call the adapter’s native OpenAI stream operation.

Install the matching collector for the default embedded streaming path:

pip install "nemo-fabric[streaming]"

This extra is not required when launch_collector=False selects an externally managed collector.

The following example streams one invocation and collects its terminal result:

import asyncio
from nemo_fabric import (
EnvironmentConfig,
Fabric,
FabricConfig,
HarnessConfig,
MetadataConfig,
ModelConfig,
)
async def main() -> None:
config = FabricConfig(
metadata=MetadataConfig(name="streaming-agent"),
harness=HarnessConfig(adapter_id="nvidia.fabric.codex"),
models={
"default": ModelConfig(provider="openai", model="openai/gpt-5.4")
},
environment=EnvironmentConfig(provider="local", workspace="."),
).enable_relay()
async with await Fabric().start_runtime(config, streaming=True) as runtime:
stream = runtime.invoke_stream(input="Review the latest patch")
async for record in stream:
print(record)
result = await stream.result()
print(result.status, result.output)
asyncio.run(main())

invoke_stream(...) is synchronous and starts the invocation in the background. The returned InvokeStream is an async iterator of raw NeMo Relay ATOF dictionaries. The terminal RunResult stays out of band and is available only through await stream.result(). Treat that terminal result as authoritative: replace any provisional rendering instead of appending it. Calling stream.result() does not finalize unread stream records. Calling Runtime.stop() after the result completes finalizes the stream and discards those unread records.

Streaming has the following v0.1 constraints:

  • Streaming requires two explicit settings: enable NeMo Relay in FabricConfig and pass streaming=True to start_runtime(...). The streaming=True flag does not enable NeMo Relay by itself; without NeMo Relay telemetry, start_runtime(...) raises FabricConfigError. By default, streaming starts an embedded collector and configures Relay to publish NDJSON to it. Set launch_collector=False to use an externally managed collector. In that case, configure exactly one nemo-fabric-stream sink with an http or https collector base URL and transport="ndjson"; NVIDIA NeMo Fabric sends Relay to <base-url>/v1/atof. Pi is the exception as Pi streaming supports only the default embedded collector. Do not set launch_collector=False for Pi because its extension does not attach the NeMo Fabric request IDs required for external correlation. The embedded collector waits one second for a late Pi agent_settled marker by default. Increase completion_wait_timeout when Relay delivery can take longer.
  • The default streaming=False leaves the existing NeMo Relay configuration unchanged; it neither enables nor disables ATOF. It also avoids collector startup and streaming control traffic for runtimes that use only ATIF, OpenTelemetry, OpenInference, or application-configured ATOF sinks.
  • One invocation can be active on a runtime. Fully consume the stream or call await stream.aclose() before starting another turn.
  • Breaking an async for loop does not finalize the stream. Call await stream.aclose() explicitly. It waits for the invocation to finish and discards unread records; it does not cancel the harness turn.
  • The end of async iteration only means that no more ATOF records are available. It does not indicate invocation success. Always await stream.result(). Invocation exceptions raise from that call, while harness-reported failures are represented by the returned result’s status and error fields.
  • The collector limits each request queue to 1,024 records and 16 MiB of encoded record data. It rejects individual records larger than 1 MiB and applies TCP backpressure for up to 30 seconds. If the queue remains full, the collector drops the record and logs a warning. NeMo Relay can drop records if a consumer stalls longer than its delivery timeout.
  • ATOF granularity depends on the NeMo Relay integration. Gateway harnesses expose per-delta event structure, but current ATOF records keep token text in the terminal scope. In-process harnesses expose scope-level progress.
  • Reconstruct nested and parallel work with uuid and parent_uuid. Stream order alone does not define the Deep Agents scope tree.
  • For request-correlated adapters, the runtime registers its request ID with the collector before invoking the agent. The collector routes the ATOF root scope carrying that request ID and its descendants, based on UUID ancestry, to that invocation’s private queue. With relay_streaming: true, Remote Agent services receive the ID as metadata.nemo_fabric_request_id in the mapped request body and must carry it into Relay metadata. Records that cannot be correlated to a registered request are discarded. Pi instead serializes both streaming and plain invocations behind one embedded-collector lease. Streaming capture begins at the first turn_start and closes at agent_settled. If output is interrupted or late, the collector discards the remainder through that same terminal marker before allowing the next invocation to start.
  • An external collector can be shared by multiple NeMo Fabric runtimes whose adapters propagate request IDs. It must be reachable by both Relay publishers and NeMo Fabric clients. Use HTTPS and its configured authentication when it is deployed beyond a trusted local boundary.

Claude and Codex streaming use the NeMo Relay nemo-relay gateway CLI and require a release in the >=0.9,<0.12 range. Their adapter harness and full extras install the supported CLI. Deep Agents uses its in-process NeMo Relay integration with Relay 0.11. Hermes Agent v0.22.7 uses Relay 0.11; v0.22.5 with Relay 0.10 remains supported. Hermes runs on Python 3.14, with ADAPTER_PYTHON selecting a separate environment when the application uses another interpreter. Remote Agent services own their Relay installation and publish to the same configured nemo-fabric-stream collector base URL. The local adapter does not install or start Relay for the service.