> This page is for NeMo Fabric, version v0.7.0.
> For other versions, use one of these documentation indexes:
> - Latest (v0.7.0) (default): https://docs.nvidia.com/nemo/fabric/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/fabric/dev/llms.txt
> - v0.7.0: https://docs.nvidia.com/nemo/fabric/v0.7.0/llms.txt
> - v0.6.1: https://docs.nvidia.com/nemo/fabric/v0.6.1/llms.txt
> - v0.6.0: https://docs.nvidia.com/nemo/fabric/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/nemo/fabric/v0.5.0/llms.txt
> - v0.4.0: https://docs.nvidia.com/nemo/fabric/v0.4.0/llms.txt
> - v0.3.0: https://docs.nvidia.com/nemo/fabric/v0.3.0/llms.txt
> - v0.2.0: https://docs.nvidia.com/nemo/fabric/v0.2.0/llms.txt
> - v0.1.1: https://docs.nvidia.com/nemo/fabric/v0.1.1/llms.txt
> - v0.1.0: https://docs.nvidia.com/nemo/fabric/v0.1.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/fabric/llms.txt.

# Streaming

> Use native OpenAI streaming and NeMo Relay event streaming.

For examples that reuse `config`, start with [Start With One Run](/nemo/fabric/sdk/python-sdk#start-with-one-run).

## Native OpenAI Streaming

Use `Runtime.invoke_openai_stream()` when the selected adapter provides native
OpenAI Chat Completions chunks. `Runtime.supports_openai_streaming` reflects the
selected descriptor's existing `capabilities.streaming` claim. If the claim is
false, `invoke_openai_stream()` raises `FabricCapabilityError` with code
`openai_streaming_unavailable`.

Run the following example from a NeMo Fabric source checkout after completing the [NVIDIA NeMo Agent Toolkit reference adapter bootstrap](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.7.0/external/nat/README.md#development-bootstrap). Export `NVIDIA_API_KEY` for the NVIDIA-hosted model selected by its maintained calculator configuration. That configuration discovers the source-only adapter and registered target, and starts its calculator MCP server over stdio; no separately managed agent endpoint is required. The reference adapter supports native OpenAI chunks when the workflow declares the required `ChatResponseChunk` output schema.

```python
from external.nat.examples.calculator import build_config
from nemo_fabric import Fabric

native_streaming_config = build_config()

async with await Fabric().start_runtime(native_streaming_config) as runtime:
    if not runtime.supports_openai_streaming:
        raise RuntimeError("the selected adapter does not support OpenAI streaming")

    stream = runtime.invoke_openai_stream(input="What is 21 multiplied by 2?")
    async for chunk in stream:
        print(chunk)

    result = await stream.result()
    print(result.status, result.output)
```

`invoke_openai_stream(...)` is synchronous and starts exactly one adapter
invocation in the background. Its `OpenAIInvokeStream` yields only JSON mappings
whose `object` field is `chat.completion.chunk`. An empty stream is valid. The
terminal normalized `RunResult` is not a final chunk; obtain it separately with
`await stream.result()` and treat it as authoritative.

Only one turn can be active on a runtime. Fully consume the stream before
starting another turn. If iteration stops early, call `await stream.aclose()`;
it drains the invocation and discards unread chunks so the runtime can be reused.
It does not cancel the target invocation. Awaiting `stream.result()` also drains
and discards any unread chunks before it returns, so consume the iterator first
when the application needs every chunk.

The SDK owns an authenticated listener on the loopback interface and supplies
the adapter host with a single-invocation HTTP sink that uses chunked NDJSON
framing. It validates chunk shape, framing, authentication, and invocation
correlation. This transport is an implementation detail; consumers provide no
listener address, token, or protocol configuration.

Native OpenAI streaming does not require NeMo Relay or
`start_runtime(..., streaming=True)`. It is independent of Relay ATOF streaming:
`supports_openai_streaming` and `invoke_openai_stream()` describe adapter-native
OpenAI chunks, while `supports_streaming` and `invoke_stream()` describe Relay
ATOF. Enabling one does not enable or change the other.

## NeMo Relay Streaming

Enable NeMo Relay before starting a runtime to consume raw ATOF records while a
turn runs. `Runtime.supports_streaming` reports whether this NeMo Relay path is
available. It is separate from `RuntimeCapabilities.streaming`, which describes
adapter-native progressive output. This separation is intentional:
`Runtime.invoke_stream()` exposes only ATOF records generated by NeMo Relay; it
does not call the adapter's native OpenAI stream operation.

Install the matching collector for the default embedded streaming path:

```bash
pip install "nemo-fabric[streaming]"
```

This extra is not required when `launch_collector=False` selects an externally
managed collector.

The following example streams one invocation and collects its terminal result:

```python
import asyncio

from nemo_fabric import (
    EnvironmentConfig,
    Fabric,
    FabricConfig,
    HarnessConfig,
    MetadataConfig,
    ModelConfig,
)

async def main() -> None:
    config = FabricConfig(
        metadata=MetadataConfig(name="streaming-agent"),
        harness=HarnessConfig(adapter_id="nvidia.fabric.codex"),
        models={
            "default": ModelConfig(provider="openai", model="openai/gpt-5.4")
        },
        environment=EnvironmentConfig(provider="local", workspace="."),
    ).enable_relay()

    async with await Fabric().start_runtime(config, streaming=True) as runtime:
        stream = runtime.invoke_stream(input="Review the latest patch")
        async for record in stream:
            print(record)
        result = await stream.result()
        print(result.status, result.output)

asyncio.run(main())
```

`invoke_stream(...)` is synchronous and starts the invocation in the background.
The returned `InvokeStream` is an async iterator of raw NeMo Relay ATOF
dictionaries. The terminal `RunResult` stays out of band and is available only
through `await stream.result()`. Treat that terminal result as authoritative:
replace any provisional rendering instead of appending it. Calling
`stream.result()` does not finalize unread stream records. Calling
`Runtime.stop()` after the result completes finalizes the stream and discards
those unread records.

Streaming has the following v0.1 constraints:

* Streaming requires two explicit settings: enable NeMo Relay in `FabricConfig`
  and pass `streaming=True` to `start_runtime(...)`. The `streaming=True` flag
  does not enable NeMo Relay by itself; without NeMo Relay telemetry,
  `start_runtime(...)` raises `FabricConfigError`. By default, streaming starts
  an embedded collector and configures Relay to publish NDJSON to it. Set
  `launch_collector=False` to use an externally managed collector. In that
  case, configure exactly one `nemo-fabric-stream` sink with an `http` or
  `https` collector base URL and `transport="ndjson"`; NVIDIA NeMo Fabric sends Relay to
  `<base-url>/v1/atof`. Pi is the exception as Pi streaming supports only the
  default embedded collector. Do not set `launch_collector=False` for Pi
  because its extension does not attach the NeMo Fabric request IDs required
  for external correlation. The embedded collector waits one second for a late
  Pi `agent_settled` marker by default. Increase `completion_wait_timeout` when
  Relay delivery can take longer.
* The default `streaming=False` leaves the existing NeMo Relay configuration
  unchanged; it neither enables nor disables ATOF. It also avoids collector
  startup and streaming control traffic for runtimes that use only ATIF,
  OpenTelemetry, OpenInference, or application-configured ATOF sinks.
* One invocation can be active on a runtime. Fully consume the stream or call
  `await stream.aclose()` before starting another turn.
* Breaking an `async for` loop does not finalize the stream. Call
  `await stream.aclose()` explicitly. It waits for the invocation to finish and
  discards unread records; it does not cancel the harness turn.
* The end of async iteration only means that no more ATOF records are available.
  It does not indicate invocation success. Always await `stream.result()`.
  Invocation exceptions raise from that call, while harness-reported failures
  are represented by the returned result's status and error fields.
* The collector limits each request queue to 1,024 records and 16 MiB of
  encoded record data. It rejects individual records larger than 1 MiB and
  applies TCP backpressure for up to 30 seconds. If the queue remains full,
  the collector drops the record and logs a warning.
  NeMo Relay can drop records if a consumer stalls longer than its delivery
  timeout.
* ATOF granularity depends on the NeMo Relay integration. Gateway harnesses expose
  per-delta event structure, but current ATOF records keep token text in the
  terminal scope. In-process harnesses expose scope-level progress.
* Reconstruct nested and parallel work with `uuid` and `parent_uuid`. Stream
  order alone does not define the Deep Agents scope tree.
* For request-correlated adapters, the runtime registers its request ID with
  the collector before invoking the agent. The collector routes the ATOF root
  scope carrying that request ID and its descendants, based on UUID ancestry,
  to that invocation's private queue. With `relay_streaming: true`, Remote
  Agent services receive the ID as `metadata.nemo_fabric_request_id` in the
  mapped request body and must carry it into Relay metadata. Records that
  cannot be correlated to a registered request are discarded. Pi instead
  serializes both streaming and plain invocations behind one embedded-collector
  lease. Streaming capture begins at the first `turn_start` and closes at
  `agent_settled`. If output is interrupted or late, the collector discards the
  remainder through that same terminal marker before allowing the next
  invocation to start.
* An external collector can be shared by multiple NeMo Fabric runtimes whose
  adapters propagate request IDs. It must be reachable by both Relay
  publishers and NeMo Fabric clients. Use HTTPS and its configured authentication
  when it is deployed beyond a trusted local boundary.

Claude and Codex streaming use the NeMo Relay `nemo-relay` gateway CLI and
require a release in the `>=0.9,<0.12` range. Their adapter `harness` and
`full` extras install the supported CLI. Deep Agents uses its in-process NeMo
Relay integration with Relay 0.11. Hermes Agent v0.22.7 uses Relay 0.11; v0.22.5 with Relay 0.10 remains supported. Hermes runs on Python 3.14, with `ADAPTER_PYTHON` selecting a separate environment when the application uses another interpreter.
Remote Agent services own their Relay installation and publish to the same
configured `nemo-fabric-stream` collector
base URL. The local adapter does not install or start Relay for the service.