> This page is for NeMo Fabric, version v0.6.0.
> For other versions, use one of these documentation indexes:
> - Latest (v0.6.0) (default): https://docs.nvidia.com/nemo/fabric/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/fabric/dev/llms.txt
> - v0.6.0: https://docs.nvidia.com/nemo/fabric/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/nemo/fabric/v0.5.0/llms.txt
> - v0.4.0: https://docs.nvidia.com/nemo/fabric/v0.4.0/llms.txt
> - v0.3.0: https://docs.nvidia.com/nemo/fabric/v0.3.0/llms.txt
> - v0.2.0: https://docs.nvidia.com/nemo/fabric/v0.2.0/llms.txt
> - v0.1.1: https://docs.nvidia.com/nemo/fabric/v0.1.1/llms.txt
> - v0.1.0: https://docs.nvidia.com/nemo/fabric/v0.1.0/llms.txt
> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/fabric/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/fabric/_mcp/server.
# NVIDIA NeMo Fabric Python SDK
> Use the NeMo Fabric Python SDK from applications, services, and evaluation harnesses.
The Python SDK is the application-facing interface for NeMo Fabric. Use it to
configure an agent harness, inspect the resolved plan, run one request or a
multi-turn runtime, and collect normalized results, events, artifacts, and
telemetry references.
The SDK is config-first. Applications should construct a Pydantic
`FabricConfig` from their own job, deployment, or evaluation config.
Examples, CI jobs, and applications construct the typed config directly and
can use ordinary Python functions to create variants.
Generated API reference pages remain the source of truth for exact signatures.
This guide explains how the pieces are intended to fit together.
For installation and a package-backed example, start with the
[NeMo Fabric overview](/nemo/fabric/about-nemo-fabric/overview).
## Start With One Run
This example uses an NVIDIA-hosted model. Export your NVIDIA API key before
running it:
```bash
export NVIDIA_API_KEY=...
```
Construct a typed config, then pass it to `Fabric.run(...)`. The SDK starts a
runtime, invokes it once, collects the result, and stops the runtime.
```python
import asyncio
from nemo_fabric import (
Fabric,
FabricConfig,
HarnessConfig,
InstructionConfig,
InstructionsConfig,
MetadataConfig,
ModelConfig,
RuntimeConfig,
)
config = FabricConfig(
metadata=MetadataConfig(name="review-agent"),
harness=HarnessConfig(adapter_id="nvidia.fabric.hermes"),
instructions=InstructionsConfig(
system=InstructionConfig(
content="Review code for concrete correctness risks.",
mode="replace",
)
),
runtime=RuntimeConfig(max_turns=8),
models={
"default": ModelConfig(
provider="nvidia",
model="nvidia/nemotron-3-nano-omni-30b-a3b-reasoning",
api_key_env="NVIDIA_API_KEY",
)
},
)
async def main() -> None:
fabric = Fabric()
result = await fabric.run(
config,
input="Review the workspace changes.",
)
print(result.status)
print(result.output.response)
asyncio.run(main())
```
Omit `instructions.system` to preserve the harness's native system instruction.
When present, `InstructionConfig.mode` defaults to `replace`. Use `append` only
with an adapter whose descriptor advertises that mode; planning rejects other
combinations before the runtime starts.
Use `base_dir` to resolve relative paths in an in-memory config. Before running
in a new environment, call `plan(...)` to inspect adapter selection and
`doctor(...)` to check runtime requirements.
The remaining examples reuse this `config` value.
### Inspect Metadata On A Separate Host
Use `inspect_adapter(config, descriptor)` when the admission host cannot access
the task's installed adapter. Supply canonical metadata from the matching
`nemo-fabric-adapter-catalog` release or an external adapter:
```python
from nemo_fabric import inspect_adapter
from nemo_fabric_adapter_catalog import get_adapter_descriptor
profile = inspect_adapter(config, get_adapter_descriptor(config.harness.adapter_id))
print(profile.skills, profile.mcp, profile.atif)
# In the task environment, resolve its own executable descriptor.
result = await Fabric().run(
config,
input="Review the changes.",
expected_descriptor_sha256=profile.descriptor_sha256,
)
```
The immutable `AdapterCapabilityProfile` reports declared skills and MCP support
and ATIF availability for the selected telemetry configuration. Inspection uses
the existing native planner without importing the harness SDK or reading
task-local discovery paths. Unsupported requested configuration fails admission.
Passing `None` as the descriptor permits basic execution with false capability
claims, but rejects requested skills, MCP, or telemetry.
The optional `expected_descriptor_sha256` argument on `run()` and
`start_runtime()` verifies the normalized descriptor in the actual execution
plan before startup. Catalog metadata never becomes an execution origin. A
fingerprint confirms descriptor correspondence, not installed software versions
or successful evidence production. ATIF requires a selected provider declaring
that output; Relay additionally requires one explicitly enabled observability
component with ATIF enabled. Unconfirmed ATIF requests, including ambiguous
duplicate components, fail admission.
For a registered workflow, also supply its canonical target metadata:
`inspect_adapter(config, descriptor, target_descriptor=get_target_descriptor(config.workflow.target_id))`.
Import `get_target_descriptor` from `nemo_fabric_adapter_catalog`. The fingerprint
then covers both normalized descriptors, including the target factory reference
and settings schema. Missing or mismatched target metadata fails admission.
Task execution still resolves its own installed target and verifies both
descriptors before starting the factory. Standalone fingerprints are unchanged.
Attached services require separate route-specific admission.
## Execution Model
NeMo Fabric separates configuration, planning, runtime lifecycle, and individual
invocations. It does not expose a separate portable session layer.
```mermaid
flowchart TB
Config["FabricConfig"]
Plan["RunPlan"]
Runtime["Runtime"]
Invoke1["Invocation
turn 1"]
Invoke2["Invocation
turn 2"]
Result1["RunResult
turn 1"]
Result2["RunResult
turn 2"]
Config --> Plan
Plan --> Runtime
Runtime --> Invoke1 --> Result1
Runtime --> Invoke2 --> Result2
```
Most application code works with five objects:
| Concept | What It Represents | How Consumers Use It |
| -------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `FabricConfig` | Typed configuration for the harness, models, runtime, and capabilities. | Construct it from application config before planning or running. |
| `RunPlan` | The canonical config and path context, selected adapter, and declared capabilities. | Inspect it before starting work when adapter selection or feature support matters. |
| `Service` | A NeMo Fabric-owned or caller-owned adapter service shared by one or more runtimes. | Prepare or attach it, pass it to `start_runtime(...)`, then release it after all connected runtimes stop. |
| `Runtime` | The Python object for one logical, stateful harness execution. | Use it to send ordered invocations and stop the execution. Use it as an async context manager for cleanup. |
| `RunResult` | The normalized outcome of one invocation. | Read its status, output, error, usage, events, artifacts, and correlation IDs. |
`Fabric` is a lightweight, reusable SDK facade. It resolves configuration,
creates plans and runtimes, and provides a single-invocation convenience API,
but it does not represent a started execution and does not require cleanup. A
`Runtime` owns stateful execution and shutdown, so it is the object used as an
async context manager.
`RuntimeHandle` and `InvocationHandle` carry lifecycle identity across the
native boundary. Most Python callers use their `runtime_id` and
`invocation_id` through `Runtime` and `RunResult` rather than manipulating the
handles directly.
A runtime is a logical execution boundary, not necessarily an operating-system
process. An adapter may use an in-process SDK, a process, or shared service
infrastructure while preserving isolated state for each NeMo Fabric runtime.
Runtime hosting is selected by the adapter kind, not by a public `FabricConfig`
setting. Every local Process or Python adapter implements the persistent
local-host wire protocol. NeMo Fabric starts one host during
`start_runtime(...)`, reuses its adapter-owned native resources across ordered
invocations, and attempts to release them during `stop()`. The lifecycle start
operation carries the resolved configuration and capability plan. Each
subsequent `AdapterInvocation` carries only `runtime_context` and `request`.
Third-party local adapters must implement the persistent local-host contract.
Adapters that advertise service support can also prepare a NeMo Fabric-owned service
or attach to a caller-owned service. Multiple runtimes can connect to one
service while retaining distinct runtime IDs and harness sessions. Service
handles are process-local in this release and must not be persisted for cold
resume. A crashed persistent host is terminal for its runtime or service. The
same applies when the host exceeds the protocol response timeout. NeMo Fabric
does not silently respawn the host or replay a request.
Harness-native threads, sessions, and conversations remain adapter-owned state
associated with the NeMo Fabric runtime. They are not additional NeMo Fabric lifecycle
objects.
NeMo Fabric provides the runtime contract. Applications own scheduling, queues,
retries, worker scaling, and the number of runtimes to run.
### Explicit Environment Lifecycle
`prepare_environment(config)` returns an immutable `EnvironmentHandle` without
starting an agent. Pass it to `start_runtime_in(config, environment)` to start a
runtime, then invoke and stop that runtime normally. Call
`release_environment(environment)` separately after collecting results. Local
release preserves caller-owned files and supports multiple runtimes.
`attach_environment(config, EnvironmentReference.from_mapping(...))` asks the
selected provider to verify an existing resource. Providers that do not support
attachment return an unsupported-provider error. Attachment retains caller
ownership; release detaches without deleting that resource. Provider availability
and runtime reuse limits depend on the selected environment implementation.
Non-local workspace and artifact paths refer to the task environment and are
not resolved against the host's `base_dir`. The OpenShell provider starts the
agent through the task SDK command bridge. One sandbox can host multiple
independent runtime IDs; invoke and stop target a specific ID, and stopping one
does not stop the others. Streaming and shared-service hosting remain local SDK
capabilities.
The optional [OpenShell provider](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.6.0/environment-providers/openshell/README.md)
uses the task command bridge and returns the task-generated SDK runtime ID and
normalized results. Declared files are collected under the host's
`runtime.artifacts///` directory. The task artifact
root is separate. OpenShell invocation is buffered; streaming and shared-service
hosting remain local SDK capabilities. Commands with uncertain outcomes are
never replayed; an uncertain start blocks further starts in that sandbox.
### Runtime Health
Use `Fabric.check(environment=..., request=...)` to check runtime health. Refer to the [health-check overview](/nemo/fabric/sdk/health-checks) for scopes, outcomes, and deadlines, and the [adapter support table](/nemo/fabric/sdk/health-checks#native-evidence-by-adapter) for supported levels and native evidence.
The runtime ID selects the exact session inside the supplied environment:
```python
from nemo_fabric import CheckRequest, Fabric
fabric = Fabric()
environment = await fabric.prepare_environment(config)
runtime = await fabric.start_runtime_in(config, environment)
try:
report = await fabric.check(
environment=environment,
request=CheckRequest(
scope="runtime", runtime_id=runtime.runtime_id, level="ready"
),
)
print(report.outcome)
finally:
await runtime.stop()
await fabric.release_environment(environment)
```
The default level is `live`, and the default timeout is three seconds. A request
for `ready` evaluates `live`, `active`, and `ready` in order, stopping at the
first nonpassing result. One deadline covers routing, every probe, and response
decoding. Checks can run during an invocation; a busy session can pass all
three levels. A timeout returns `unknown/probe_timeout`. Stop or environment release during a probe returns
`unknown/observation_changed`.
`CheckReport` is an immutable mapping with outcomes `passed`, `failed`, `unknown`,
and `unsupported`. Reports contain fixed evidence-source IDs, UTC timestamps,
and elapsed milliseconds. An unavailable runtime returns
`unknown/runtime_unavailable`. Adapters without health support return
`unsupported/unsupported_adapter`.
Cline `live` checks verify native session residency, identity, and closure. `active` checks also inspect request-interface state. `ready` checks additionally call `ClineCore.get()` to read a fresh native session snapshot. Cline checks can run during the first invocation and do not require model access. Refer to the [Cline health contract](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.6.0/adapters/typescript/cline/README.md#runtime-health).
Codex checks its owned app-server process and a fresh loaded-thread response; native `idle` or `active` status is the readiness equivalent. Pi checks native session liveness, initialization and request-interface guards, and required model, tool, and MCP connection state. OpenClaw checks Gateway liveness, completed startup, selected-agent registration, the enabled Chat Completions endpoint, and native readiness including configured channels. Deep Agents checks its owned checkpoint connection, compiled message interface, and native state for the selected LangGraph thread. Hermes checks its owned native session, initialized invocation interface, and a fresh `get_activity_summary()` diagnostic response. Runtime activity covers NeMo Fabric invocations. Refer to the [OpenClaw runtime health guide](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.6.0/adapters/python/openclaw/README.md#runtime-health).
A local `CheckRequest(scope="server")` returns `unsupported/unsupported_scope`. For OpenShell environments,
`Fabric.check()` maps to `fabric-agent-ctl check` in the sandbox. Server requests
stop at the runtime server; runtime requests select the owned agent and use the
local SDK health path inside the sandbox. Health uses an independent provider
worker during remote invocations.
The diagram shows the local runtime adapter path. OpenShell adds the provider
and command bridge before this path; server checks stop at the runtime server.
```mermaid
flowchart LR
SDK["Fabric SDK"] --> Core["Rust core"]
Core -->|"start / invoke / stop
stdin / stdout"| Lifecycle
Core -->|"check
authenticated loopback TCP"| Health
Endpoint["health.endpoint.json
port and shared secret"] -.->|"endpoint discovery"| Core
subgraph Host["Adapter host process"]
Lifecycle["Lifecycle dispatcher"] -->|"await operation"| Adapter["Adapter runtime instance"]
Health["Health listener"] -->|"check native state"| Adapter
Adapter --> Harness["Native harness SDK"]
end
```
The lifecycle dispatcher awaits each start, invocation, or stop operation. The
health listener checks the same adapter runtime through a separate connection,
so checks can run while an invocation is pending. The host publishes the port
and a per-process shared secret in a private `health.endpoint.json` file under
`FABRIC_HOME`; the Rust core reads that file and authenticates health requests
and responses. Each health request carries the remaining timeout budget, which the host applies
to the probe.
### Gateway Service Health
Check a prepared or attached OpenClaw Gateway with `Fabric.check(service=..., request=...)`. Service checks use the selected process-local `ServiceHandle`:
```python
from nemo_fabric import CheckRequest, Fabric
fabric = Fabric()
async with await fabric.prepare_service(config) as service:
report = await fabric.check(
service=service.handle,
request=CheckRequest(
scope="service", service_id=service.service_id, level="ready"
),
)
print(report.outcome)
```
For OpenClaw, `live` validates `/healthz`, `active` validates an authenticated `/v1/models` response, and `ready` follows native `/readyz`, including configured channel health. A failed channel can fail Gateway readiness while `live` and `active` pass. Refer to the [OpenClaw Gateway health guide](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.6.0/adapters/python/openclaw/README.md#gateway-health).
Service checks evaluate earlier levels in order under the same three-second default deadline. They can run during invocations. Reports have `scope="service"`, the selected `service_id`, and null runtime evidence. Release during a check invalidates evidence with `unknown/observation_changed`; a released or unavailable binding returns `unknown/service_unavailable`. Service hosting and checks remain local SDK capabilities; the sandbox command bridge continues to support server and runtime checks.
### Sandbox Command Bridge
`fabric-agent-ctl check` checks server responsiveness with a three-second default deadline. Use `--timeout-seconds SECONDS` to set another positive, finite budget. Server checks work before a runtime exists and during an invocation. Completed checks return a `fabric.check/v1alpha1` report in `result`, with `changed: false`; passed health exits 0, while failed, unknown, or unsupported health exits 1. Use `fabric-agent-ctl check --agent researcher --runtime-id R1 --ready` to check one runtime and return a runtime-scoped report. Runtime checks require both target flags and default to `--live`; `--active` and `--ready` are mutually exclusive alternatives. OpenShell SDK requests map to these named commands. Each layer forwards the remaining deadline, and failed health evidence stays in the typed SDK report.
`fabric.check/v1alpha1` identifies the shared report schema returned by
`Fabric.check()` and `fabric-agent-ctl check`. The executable ships with
`nemo-fabric-runtime` and follows the SDK package release.
`fabric-agent-ctl serve` starts one runtime server per sandbox. The `start`, `invoke`, and `stop` commands call the existing `Fabric.start_runtime()`, `Runtime.invoke()`, and `Runtime.stop()` methods. Start accepts `FabricConfig` and returns the SDK `RuntimeHandle` mapping. Invoke accepts a full `RunRequest` through `--request` or any JSON SDK input through `--input`, and returns the SDK `RunResult` mapping. Stop returns JSON null. These payloads occupy the response envelope's `result` field; command diagnostics remain separate. Targeted commands require the returned runtime ID and exact agent name. The caller waits for the private socket before start. Caller-side preflight uses `Fabric.plan()` and `Fabric.doctor()`; startup validates configuration against installed descriptors.
One sandbox can own multiple runtimes, each addressed by its returned runtime ID. Invoke and stop operations are independent across IDs; stopping one runtime leaves the others usable, and another runtime can be started after a stop. An uncertain start blocks further starts because the server cannot safely determine whether an unaddressable runtime exists. Ordinary SDK stop does not confirm descendant cleanup, so the runtime server retains `fabric.unclean` after runtime use and reports `cleanup_unconfirmed` if previous sandbox process cleanup cannot be confirmed. Refer to the [runtime server command guide](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.6.0/sdk/python/nemo-fabric-runtime/src/nemo_fabric/runtime_server/README.md).
## Configure Agents In Code
### Normalized Configuration Compatibility
NeMo Fabric-owned fields such as workspace, environment variables, and the
invocation deadline apply to every local adapter. Adapter-translated fields are
validated against the selected adapter descriptor during planning.
`Core` means NeMo Fabric owns the behavior and applies it uniformly before or
around adapter execution. `Yes` means the adapter translates the normalized
field into its harness. `No` means an explicitly configured value fails
planning instead of being ignored. The following table groups provider-specific
Relay subfields:
| `FabricConfig` Field | Claude | Codex | Deep Agents | Hermes Agent | mini-SWE-agent |
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------ |
| `schema_version` | Core | Core | Core | Core | Core |
| `metadata.name`, `.description` | Core | Core | Core | Core | Core |
| `harness.adapter_id`, `.resolution` | Core | Core | Core | Core | Core |
| `harness.settings` | Closed adapter schema | Closed adapter schema | Closed adapter schema | Closed adapter schema | Closed `timeout` schema |
| `workflow.target_id`, `.settings` | No | No | No | No | No |
| `models..provider` | `anthropic` uses native auth; custom names require an Anthropic Messages-compatible `base_url` and `api_key_env` | `openai` uses native auth; custom names require a Responses-compatible `base_url` and `api_key_env` | Dynamic LangChain provider; custom OpenAI-compatible endpoints require `base_url` and `api_key_env` | Dynamic Hermes provider | Configured provider |
| `models..model` | Yes | Yes | Yes | Yes | Yes |
| `models..api_key_env` | Yes | Yes | Yes | Yes | Yes |
| `models..base_url` | Yes | Yes | Yes | Yes | Yes |
| `models..temperature` | No | No | Yes | Yes | Yes |
| `models..settings.` | No keys declared | No keys declared | No keys declared | No keys declared | No keys declared |
| `models..top_p`, `.max_tokens` | No | No | Yes | Yes | Yes; passed through LiteLLM |
| `instructions.system` | `replace`, `append` | `replace`; maps to Codex base instructions | `replace` | `replace` | `replace` |
| `runtime.input_schema`, `.output_schema` | Core | Core | Core | Core | Core |
| `runtime.artifacts`, `.timeout_seconds` | Core | Core | Core | Core | Core |
| `runtime.max_turns` | Yes | No | Yes; maps to LangGraph supersteps | Yes; maps to Hermes iterations | Yes |
| `environment.provider`, `.control_location`, `.ownership` | Core | Core | Core | Core | Core |
| `environment.workspace`, `.artifacts`, `.env`, `.env_names` | Core | Core | Core | Core | Core |
| `environment.connection`, `.metadata`, `.settings` | Environment-provider-owned | Environment-provider-owned | Environment-provider-owned | Environment-provider-owned | Environment-provider-owned |
| `tools.definitions` | No | No | No | No | No |
| `tools.enabled`, `.blocked` | Yes | No | Yes | Yes; native selectors are Hermes toolset names | No |
| `skills.paths` | Yes | Yes | Yes | Yes | No |
| `mcp.servers..transport`, `.url` with `harness_native` exposure | Yes | Yes | Yes | Yes | No |
| `mcp.servers..allowed_tools`, `.blocked_tools` | No | No | No | No | No |
| `mcp.servers..exposure = "fabric_managed"` | No; not implemented | No; not implemented | No; not implemented | No; not implemented | No |
| `telemetry.providers.relay` | Yes | Yes | Yes | Yes | Yes |
| `telemetry.providers.native` | No | Yes; OpenTelemetry | Yes; OpenTelemetry and OpenInference | No | No |
| `telemetry.providers..config` | Declared-provider pass-through | Declared-provider pass-through | Declared-provider pass-through | Declared-provider pass-through | Declared-provider pass-through |
| `relay.project`, `.output_dir`, `.observability` | Yes | Yes | Yes | Yes | Yes |
| `relay.components`, `.policy` | Yes | Yes | Yes | Yes | Yes |
| Other additive `extensions` on typed config objects | Rejected unless declared by the descriptor | Rejected unless declared by the descriptor | Rejected unless declared by the descriptor | Rejected unless declared by the descriptor | Rejected unless declared by the descriptor |
Model selection is deterministic: the `default` role wins; otherwise a single
named role is selected. More than one role without `default` fails planning.
All bundled adapters except Hermes publish a descriptor-owned `model_schema`
for every configured model role. The Claude and Codex native providers
(`anthropic` and `openai`, respectively) keep the existing authentication path.
Other Claude and Codex providers remain valid only with an explicit `base_url`
and `api_key_env`. Deep Agents keeps dynamic LangChain provider selection. Each
published schema rejects undeclared `ModelConfig.settings` during planning and
reports each issue through `doctor(...)` before adapter startup. Existing
configurations that supplied `top_p` and `max_tokens` as flattened model
extensions retain the same wire shape when loaded as normalized fields. Legacy
adapter descriptors that accept either name through `extension_schemas.model`
continue receiving it in `AgentModelConfig.extensions`; adapters that advertise
the normalized capability receive the typed field instead.
If a normalized field has no complete mapping, `plan(...)` and runtime start
fail with a configuration compatibility error naming the adapter and field.
`doctor(...)` retains the incompatibility and returns a failed diagnostic check
so callers can inspect all preflight findings. NeMo Fabric does not silently
drop the field. `runtime.max_turns` is optional, so a config that omits it
remains portable across adapters with different native turn-limit support.
Build the complete nested `FabricConfig` directly, or start with a base
config and use helpers to add capabilities. For example, extend the config from
the first example with skills, MCP, and telemetry:
```python
capability_config = config.model_copy(deep=True)
capability_config.add_skill_path("./skills/code-review")
capability_config.add_mcp_server(
"github",
transport="stdio",
url="github-mcp",
args=["stdio"],
env={"GITHUB_TOKEN": "${GITHUB_TOKEN}"},
exposure="harness_native",
)
capability_config.enable_relay(
project="fabric-review",
output_dir="./artifacts/relay",
)
```
### Set Environment Values Or Forward Them By Name
Use `environment.env` to set literal values. Use `environment.env_names` to forward variables that are already available to the NeMo Fabric process.
| Field | What You Supply | What NeMo Fabric Saves In The Config And Run Plan |
| ----------- | -------------------------------------------------------- | ------------------------------------------------- |
| `env` | A name and its literal value, such as `LOG_LEVEL: debug` | The name and value |
| `env_names` | A name, such as `GITHUB_TOKEN` | Only the name |
For example, set `GITHUB_TOKEN` in the shell that launches your application:
```bash
export GITHUB_TOKEN="example-token"
```
Then extend the configuration with a literal log level and the token's name:
```python
from nemo_fabric import EnvironmentConfig
config.environment = EnvironmentConfig(
env={"LOG_LEVEL": "debug"},
env_names=["GITHUB_TOKEN"],
)
```
NeMo Fabric looks up `GITHUB_TOKEN` in its own process environment and passes the actual value, `example-token`, to the adapter in `RuntimeContext.environment.env`. The adapter receives the value, not a name that it must look up. The configuration and run plan contain only the name; NeMo Fabric redacts environment values in the saved adapter invocation. Execution fails if a listed variable is unset. A name cannot appear in both `env` and `env_names`.
`environment.env` does not expand `$GITHUB_TOKEN` or `${GITHUB_TOKEN}`. For example, `env={"GITHUB_TOKEN": "$GITHUB_TOKEN"}` passes the literal text `$GITHUB_TOKEN`. Use `env_names=["GITHUB_TOKEN"]` to request lookup. If your application instead reads the token and places its value in `env`, that value becomes part of the saved configuration and run plan.
A `.env` file stores names and values, such as `GITHUB_TOKEN=example-token`. Neither `env` nor `env_names` loads that file. Your launcher or application must load it into the process environment before NeMo Fabric starts the adapter. The same forwarding mechanism works with variables supplied by a shell, CI secrets, or a container.
With Harbor, the NeMo Fabric process runs inside the task environment. `env_names` reads that process's variables; listing a name does not copy its value from the Harbor host into the task. Harbor owns delivery through its managed execution environment. The existing Harbor bridge uses a separate `environment_env_names` payload field and resolves those names into `environment.env` inside the task runner. That bridge behavior is unchanged by `env_names`. Refer to [Harbor Environment Forwarding](/nemo/fabric/integrations/consumer-integrations/harbor#environment-forwarding) for the two-stage flow.
NeMo Fabric reads named values at adapter start and checks them before each invocation. Persistent harnesses and shell backends can retain their startup environment, so NeMo Fabric rejects an invocation if a named value has changed instead of sending a context that disagrees with that environment. The error identifies the variable without exposing its value. Restart the runtime and any associated service after changing a forwarded value. Connecting a new runtime to a service also fails if the named values differ from the service's startup values.
### Configure MCP Authentication And Headers
MCP authentication and custom-header support vary by adapter:
| Adapter | OAuth 2.0 Authentication | HTTP `custom_headers` |
| ------------ | ------------------------ | --------------------- |
| Claude | Unsupported | Supported |
| Codex | Supported | Supported |
| Deep Agents | Unsupported | Supported |
| Hermes Agent | Supported | Supported |
Claude and Deep Agents return a configuration error when `authentication` is
set. The bundled adapters do not support `service_account` MCP authentication.
Configure OAuth 2.0 with `McpAuthenticationConfig`. The adapter and MCP server
complete the authorization flow before the first agent turn. The flow can
require interactive browser authorization.
```python
from nemo_fabric import McpAuthenticationConfig
capability_config.add_mcp_server(
"github",
transport="streamable-http",
url="https://mcp.example.com/mcp",
authentication=McpAuthenticationConfig(
type="oauth2",
scopes=["repo:read"],
),
exposure="harness_native",
)
```
Use `custom_headers` to send HTTP headers with an `sse` or `streamable-http`
server. Header names and values must be valid HTTP header text. To provide a
sensitive value, set it in the process environment and use a `${NAME}`
reference in the header value:
```python
import getpass
import os
os.environ["MCP_ACCESS_TOKEN"] = getpass.getpass("MCP access token: ")
capability_config.add_mcp_server(
"private-api",
transport="streamable-http",
url="https://mcp.example.com/mcp",
authentication=None,
custom_headers={
"Authorization": "Bearer ${MCP_ACCESS_TOKEN}",
"X-Tenant": "example-team",
},
)
```
Set referenced environment variables before calling `run(...)` or
`start_runtime(...)`. Do not use `McpServerConfig.env` for HTTP header
variables; `env` applies only to `stdio` MCP server processes.
> **Warning**
>
> The Claude adapter stages its MCP configuration in a file for the Claude SDK.
> A sensitive value in `custom_headers` can therefore be written to
> disk. Store sensitive values in environment variables and use `${NAME}` in
> `custom_headers`. The adapter stages the reference and supplies the value
> through the scoped child-process environment instead of writing the value to
> the MCP configuration file.
MCP tool policy is scoped to one server and uses the tool names that server
advertises. `allowed_tools=None` exposes every discovered tool, while an empty
allowlist exposes none. `blocked_tools` is then removed from the allowed or
discovered set, and the same tool cannot appear in both lists:
```python
capability_config.add_mcp_server(
"github",
transport="streamable-http",
url="${GITHUB_MCP_URL}",
exposure="harness_native",
allowed_tools=["issues.read", "pull_requests.read"],
blocked_tools=["issues.delete"],
)
```
An adapter must declare both `mcp` and `mcp.tool_filters` support before
planning accepts these fields. The bundled adapters currently support
unfiltered MCP servers only. Per-server MCP policy is distinct from the root
`ToolsConfig`, which controls adapter-native tools across the harness.
An MCP server requiring authentication also requires `mcp.auth.oauth2` or
`mcp.auth.service_account`, matching its configured authentication type.
`tools.definitions` names tools and tool groups that the adapter constructs.
Each definition has a portable `kind`, an adapter-resolved `ref`, and settings
validated by the adapter descriptor. Use `add_tool_definition(...)` only when
the selected adapter accepts `tools.definitions` and publishes a
`tool_definition_schema`.
`tools.enabled` and `tools.blocked` select across the effective named tool
surface. Configure an allowlist, a blocklist, or both:
```python
from nemo_fabric import ToolsConfig
def with_tool_policy(base):
tool_config = base.model_copy(deep=True)
tool_config.tools = ToolsConfig(
enabled=["Read", "Edit", "Bash"],
blocked=["WebFetch"],
)
return tool_config
```
`tools.enabled=None` preserves the harness default. An empty list disables every
executable tool. A tool cannot appear in both lists. The selected adapter must
enforce the complete configured policy or planning fails. Hermes interprets
these adapter-native selectors as Hermes toolset names.
Config helpers edit the typed config before planning or starting a runtime.
They do not modify already-started runtimes. Use `block_tools(...)`,
`remove_mcp_server(name)`, and `remove_skill_path(path)` to edit a copied config.
Telemetry is enabled by adding entries to `telemetry.providers`; settings
specific to NeMo Relay live in the top-level `relay` block.
For evaluation or deployment variations, use ordinary Python functions and
copies of the typed config. Supply the complete final config to NeMo Fabric.
```python
def review_agent_config(base, *, github_mcp: bool, relay: bool):
config = base.model_copy(deep=True)
if github_mcp:
config.add_mcp_server(
"github",
transport="streamable-http",
url="${GITHUB_MCP_URL}",
exposure="harness_native",
)
if relay:
config.enable_relay(project="fabric-review", output_dir="./artifacts/relay")
return config
variant = review_agent_config(config, github_mcp=True, relay=True)
```
NeMo Relay observability is represented directly in the SDK config's top-level
`relay` block. Configure ATOF file and stream sinks as follows:
```python
from nemo_fabric import (
RelayAtofConfig,
RelayAtofFileSinkConfig,
RelayAtofStreamSinkConfig,
RelayObservabilityConfig,
)
relay_config = config.model_copy(deep=True)
relay_config.enable_relay(
output_dir="./artifacts/relay",
observability=RelayObservabilityConfig(
atof=RelayAtofConfig(
enabled=True,
sinks=[
RelayAtofFileSinkConfig(
output_directory="./artifacts/relay",
filename="events.atof.jsonl",
mode="overwrite",
),
RelayAtofStreamSinkConfig(
name="nemo-fabric-stream",
url="http://localhost:4319",
transport="ndjson",
),
],
),
),
)
```
To write OpenTelemetry spans to local files, add `file_sinks` to the `opentelemetry` section. File sinks need NeMo Relay 0.10 or later. You can use them instead of endpoints or alongside them. When `opentelemetry` is enabled, it needs at least one endpoint or file sink.
The following example writes OpenInference spans to files instead of sending them to a collector:
```python
relay_config.enable_relay(
output_dir="./artifacts/relay",
observability={
"opentelemetry": {
"enabled": True,
"file_sinks": [
{"type": "openinference", "output_directory": "./artifacts/relay"}
],
},
},
)
```
NeMo Fabric handles these files as follows:
* As with ATOF and ATIF files, each runtime writes to its own directory: `/`. If you omit `output_directory`, NeMo Fabric uses `artifacts/relay`.
* If you omit `filename`, Relay names each file with a timestamp when it starts. A later invocation can then write to a new file.
* File sinks append by default. If you set `filename`, that file keeps the spans from earlier invocations. To replace it each time Relay starts, set `"mode": "overwrite"`.
* `RunResult.artifacts` lists each file as an artifact of kind `otlp`. Its media type is `application/x-ndjson` for the default `json_lines` format and `application/x-protobuf` for the `proto` format.
Relay buffers spans and writes them in batches. It writes the final batch when it shuts down, so read the files after the runtime stops. If Relay crashes, or is forcibly stopped because it does not shut down in time, a file can be missing its most recent spans.
`RelayAtifConfig.model_name` is an optional ATIF-only override. When you omit
it, NeMo Fabric writes the selected `default` model, or the only configured
model, into the Relay plugin configuration. If `config.models` is empty, the
resolved configuration uses `"unknown"`, matching the adapter fallback.
Multiple model roles without `default` fail planning before telemetry
resolution.
The derived label preserves `ModelConfig.model` exactly, including provider
prefixes such as `openai/gpt-5-codex`. Adapter-specific wire-model translation
does not change this telemetry label. Set the override explicitly only when the
trajectory must use a different label.
Additional NeMo Relay plugin components can be supplied generically with
`RelayComponentConfig` when their component package is available in the runtime
environment.
The repository's
[code-review example](https://github.com/NVIDIA/NeMo-Fabric/tree/v0.6.0/examples/code_review_agent)
uses this pattern for complete Hermes Agent, Codex, Deep Agents,
environment, MCP, and telemetry variants.
If a config contains relative paths, pass a `base_dir` to `plan(...)`,
`doctor(...)`, `run(...)`, `prepare_service(...)`, `attach_service(...)`, or
`start_runtime(...)`. The base directory anchors skills, workspaces, artifacts,
and other relative paths to the caller's package or job layout.
### Configure Dynamic Relay Plugins
Dynamic Relay plugins are configured separately under `relay.plugins.dynamic`. Enable `telemetry.providers.relay` with `enable_relay()` so NeMo Fabric forwards this configuration to the adapter's Relay integration. The following example configures a dynamic plugin without static components:
```python
from nemo_fabric import RelayConfig
plugin_config = config.model_copy(deep=True)
plugin_config.relay = RelayConfig(
plugins={
"dynamic": [
{
"manifest": "/opt/switchyard/relay-plugin.toml",
"enabled": True,
}
]
}
)
plugin_config.enable_relay()
```
NeMo Fabric preserves the `relay.plugins` section alongside any configured `relay.observability`, `relay.components`, and `relay.policy`. Relay owns manifest validation, loading, and activation. Use absolute manifest paths accessible inside the agent environment. Plugin activation depends on the selected adapter's Relay integration and the installed Relay version. For Hermes installation requirements, refer to [Use the Hermes Agent Adapter](/nemo/fabric/integrations/harness-integrations/hermes-agent).
## API Inventory
Create `Fabric()` as the primary SDK entrypoint. It is a regular Python object,
not a lifecycle context manager, and may be reused to plan, diagnose, or start
multiple independent runtimes.
The following table summarizes the SDK entry points and runtime methods:
| API | Async | Use When | Notes |
| --------------------------------------------------------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Fabric.plan(config, base_dir=...)` | No | You need to inspect the selected adapter, capability mapping, and runtime capabilities before running. | Does not start a runtime. |
| `Fabric.doctor(config, *, base_dir=...)` | Yes | You need preflight diagnostics for adapter resolution, capability routing, declared requirements, and environment assumptions. | Checks can inspect local binaries, environment variables, and files without starting a runtime. |
| `Fabric.run(config, base_dir=..., input=...)` | Yes | You need one complete start, invoke, result, stop lifecycle. | `base_dir` is optional. Pass a `RunRequest` instead when the invocation needs IDs, context, or overrides. |
| `Fabric.prepare_service(config, base_dir=...)` | Yes | NeMo Fabric should configure, start, and supervise a shared adapter service. | Returns a NeMo Fabric-owned `Service`. Release stops the service after every connected runtime stops. |
| `Fabric.attach_service(config, reference, base_dir=...)` | Yes | A deployment system already owns the service. | Returns a caller-owned `Service`. Release detaches NeMo Fabric without stopping the external service. |
| `Fabric.start_runtime(config, ..., streaming=False, launch_collector=None, completion_wait_timeout=1.0, service=...)` | Yes | You need state across multiple ordered invocations, optionally on a shared service. | Returns a `Runtime`. Use `streaming=True` with NeMo Relay enabled only to provision Relay ATOF streaming. |
| `Runtime.invoke(...)` | Yes | You need one turn on an existing runtime. | A runtime permits one active invocation at a time. |
| `Runtime.invoke_openai_stream(...)` | No | You need adapter-native OpenAI Chat Completions chunks for one turn. | Requires descriptor `capabilities.streaming`. Returns an async `OpenAIInvokeStream`; await `stream.result()` for the separate terminal `RunResult`. |
| `Runtime.invoke_stream(...)` | No | You need live ATOF records generated by NeMo Relay for one turn. | Returns an async `InvokeStream`; await `stream.result()` for the terminal `RunResult`. |
| `Runtime.resource_snapshot()` | Yes | You need the memory, process count, or CPU time of a started runtime. | Does not wait for an active invocation. Refer to [Runtime Resource Snapshots](#runtime-resource-snapshots). |
| `Runtime.stop()` | Yes | You need to stop or detach from the runtime. | Called automatically when using `async with`. |
| `Service.release()` | Yes | You need to stop a prepared service or detach from an attached service. | Fails while any connected runtime remains active. Called automatically when using `async with`. |
## Shared Services
Use `prepare_service(...)` when NeMo Fabric should create and supervise a
long-lived service. Start independent runtimes against the returned service,
then stop every runtime before releasing it.
```python
import asyncio
from nemo_fabric import Fabric
async def main():
fabric = Fabric()
async with await fabric.prepare_service(config) as service:
async with await fabric.start_runtime(config, service=service) as first:
async with await fabric.start_runtime(config, service=service) as second:
first_result = await first.invoke(input="Inspect the repository")
second_result = await second.invoke(input="Review the latest patch")
asyncio.run(main())
```
Use `attach_service(...)` when another system owns deployment and supervision.
The adapter defines the required `ServiceReference.connection` fields. The
reference contains locators and secret references, not secret values. Releasing
an attachment only removes NeMo Fabric-owned clients and temporary state.
```python
import asyncio
from nemo_fabric import Fabric, ServiceReference
async def main():
reference = ServiceReference.from_mapping(
{
"adapter_id": "nvidia.fabric.openclaw",
"service_type": "openclaw_gateway",
"connection": {
"gateway_url": "https://openclaw.example.com",
"gateway_token_env": "OPENCLAW_GATEWAY_TOKEN",
},
}
)
fabric = Fabric()
async with await fabric.attach_service(config, reference) as service:
async with await fabric.start_runtime(config, service=service) as runtime:
result = await runtime.invoke(input="Review the latest patch")
asyncio.run(main())
```
`Service.handle` is safe to inspect but is valid only in the process that
created it. Cold-resume behavior will be defined separately.
## Single-Invocation Runs
Use `run(...)` when the application has one input and does not need to preserve
runtime state after the result is collected.
```python
from nemo_fabric import Fabric, RunRequest
request = RunRequest(
input="Review the workspace changes.",
request_id="request-123",
context={"source": "review-service"},
)
fabric = Fabric()
result = await fabric.run(
config,
base_dir="/workspace/review-agent",
request=request,
)
print(result.status)
print(result.output.response)
print(result.artifacts)
```
Use `input=...` for the common case. Use `request=RunRequest(...)` for structured
invocation metadata. Applications read files themselves and pass either the
loaded input or a validated request to NeMo Fabric.
NeMo Fabric generates runtime and invocation IDs for lifecycle correlation. An
application may include its own identifiers in opaque request metadata, but
NeMo Fabric does not interpret them as job, session, scheduling, or resume state.
## Multi-Turn Runtimes
Use `start_runtime(...)` when the selected harness should keep state across
turns. Every call creates a new logical NeMo Fabric runtime; callers reuse the
returned object rather than selecting it with a job or session ID. The runtime
stops when its async context exits.
```python
from nemo_fabric import Fabric
fabric = Fabric()
async with await fabric.start_runtime(
config,
base_dir="/workspace/review-agent",
) as runtime:
first = await runtime.invoke(input="Inspect the repository")
second = await runtime.invoke(input="Now review the latest patch")
print(first.status, second.status)
```
The adapter reuses its native state between calls. Codex maps the calls to turns
on one live thread, Deep Agents invokes one compiled graph and checkpointer,
Hermes Agent reuses one agent and session database, and Claude keeps one connected SDK
client. Harness-native identifiers remain adapter-internal.
## Runtime Resource Snapshots
`Runtime.resource_snapshot()` returns a timestamped `RuntimeResourceSnapshot`
for a started local runtime. It does not wait for an active invocation, so a
monitor can call it while `invoke()` runs:
```python
snapshot = await runtime.resource_snapshot()
print(snapshot.observed_at, snapshot.resident_memory_bytes)
print(snapshot.process_count, snapshot.cpu_time_seconds)
```
| Field | Meaning |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `runtime_id` | Runtime the snapshot describes. |
| `started_at` | When NeMo Fabric started the runtime's adapter host, as an RFC 3339 UTC timestamp. |
| `observed_at` | When NeMo Fabric read the values, as an RFC 3339 UTC timestamp. |
| `resident_memory_bytes` | Sum of the resident set sizes of the live processes. Pages shared between processes count once per process. |
| `process_count` | Live processes, including the adapter host. Exited processes that are not yet reaped are not counted. |
| `cpu_time_seconds` | CPU time used since the adapter host started, including exited descendants. It can drop when a descendant leaves the tree. |
The scope is fixed: the local adapter host process that NeMo Fabric started for
the runtime, plus its descendants. For a runtime started on a shared service,
the service process is not included. A descendant that starts a new session or
process group still counts while its parent chain leads to the host. A process
whose parent exited is reparented outside the tree and no longer counts. The
snapshot finds the tree from each process's list of children: Linux reads
`/proc//task//children` and `/proc//stat`, and macOS reads
`proc_listchildpids` and `proc_pid_rusage`. Both read only the runtime's own
processes. On a Linux kernel without those children lists, the snapshot reads
the parent ID of every process on the machine to find the tree. The snapshot
exposes only the fields above, never command lines, environment variables,
credentials, or processes outside the runtime's tree.
When NeMo Fabric cannot read the values, it raises an error instead of
returning zeros or leaving a process out:
* `ResourceObservationUnavailableError` when the runtime stopped, is stopping,
its adapter host process exited, a live process in its tree or the process
table cannot be read, or the process tree kept changing during the read. For
example, on macOS, the snapshot is unavailable while a descendant runs as
another user, including setuid programs such as `ps`, `top`, and `sudo`.
* `ResourceObservationUnsupportedError` when the runtime's adapter kind,
environment provider, or operating system has no resource reader. The `http`
and `native_plugin` adapter kinds and Windows raise this error.
On Linux with `/proc` mounted `hidepid=invisible` or `hidepid=ptraceable`, the
snapshot is unavailable while `/proc` hides a descendant, such as one that runs
as another user. On a kernel without children lists, every snapshot is
unavailable under those mount options, because hidden processes are missing
from the process table that the snapshot reads to find the tree.
By default, each `RunResult` from a local runtime carries `system_stats` with before/after observations and a monotonic duration for the adapter exchange. Both observations run under the invocation lock, so queued invocations cannot contaminate the baseline. Runtime startup, endpoint collection, artifact preparation, and teardown are excluded from `elapsed_seconds`. Optional periodic sampling runs concurrently and can affect execution; the elapsed measurement does not subtract that effect. The existing `resource_snapshot` field retains its independent final process-tree reading even when invocation stats or the process group is disabled. When the process group is selected, it reuses `system_stats.after.snapshot`, or is `None` if unavailable.
### Invocation System Stats
For a hands-on walkthrough with configuration examples and annotated results, see [Resource Usage and Stats](/nemo/fabric/sdk/resource-usage-and-stats).
**Identical configs can produce different amounts of agent work and encounter different system conditions. Stats should help separate those two.**
Omitting `system_stats` selects every supported metric group at the invocation endpoints. Explicit `None` disables invocation stats, including before/after collection and stats artifacts. The SDK preserves that explicit null across planning and task-runner serialization.
```python
from nemo_fabric import SystemStatsConfig, SystemStatsSamplingConfig
config.system_stats = None # Disable invocation stats.
config.system_stats = SystemStatsConfig() # All groups, endpoints only.
config.system_stats = SystemStatsConfig(
collect=["process", "cgroup.cpu"],
sampling=SystemStatsSamplingConfig(interval_ms=1000, max_samples=10_000),
)
config.system_stats.sampling = None # Endpoints only for subsequent runs.
```
| `collect` group | Measurements |
| ----------------- | ---------------------------------------------------------------------------- |
| `process` | Adapter-host tree CPU time, RSS, and process count. |
| `cgroup.cpu` | Visible CPU limits, usage, and throttling. |
| `cgroup.memory` | Visible memory limits, accounted memory, provider peak, swap, and OOM kills. |
| `cgroup.pressure` | CPU, memory, and I/O pressure counters. |
Omit `collect` for all supported groups. Empty lists, duplicates, and unknown names are rejected; use `system_stats=None` to disable stats. The invocation stats collector skips excluded groups in both endpoints and periodic samples. `collected_groups` records the selected groups, including those unavailable on the current platform. A `not_collected` status identifies excluded process observations and cgroup measurements; it differs from `unavailable` and `unsupported`, and never substitutes zero. Unselected process gauge reductions and an entirely excluded cgroup block are omitted. Identity, duration, scope, and sampling-quality metadata remain available. Metric selection does not change the separate `resource_snapshot` contract.
```python
result = await runtime.invoke(input="Fix the failing test")
stats = result.system_stats
if stats is not None:
print(stats.elapsed_seconds, stats.cpu_time_seconds_estimate)
if stats.after.status == "available":
print(stats.after.snapshot.resident_memory_bytes)
elif stats.after.status in {"unavailable", "unsupported"}:
print(stats.after.status, stats.after.reason)
else:
print(stats.after.status)
```
For example, a serialized `RunResult` contains this fragment. The IDs already associate the measurements with one invocation; no additional identity file is required.
```json
{
"runtime_id": "runtime-1",
"invocation_id": "invocation-2",
"request_id": "trial-42",
"system_stats": {
"elapsed_seconds": 8.0,
"before": {
"status": "available",
"snapshot": {
"runtime_id": "runtime-1",
"started_at": "2026-10-21T09:00:00.000Z",
"observed_at": "2026-10-21T09:01:00.000Z",
"resident_memory_bytes": 52428800,
"process_count": 1,
"cpu_time_seconds": 12.0
}
},
"after": {
"status": "available",
"snapshot": {
"runtime_id": "runtime-1",
"started_at": "2026-10-21T09:00:00.000Z",
"observed_at": "2026-10-21T09:01:08.000Z",
"resident_memory_bytes": 83886080,
"process_count": 1,
"cpu_time_seconds": 14.5
}
},
"cpu_time_seconds_estimate": 2.5
}
}
```
The CPU estimate is 2.5 seconds for this invocation, excluding the warm runtime's previous 12 seconds. It is omitted when either reading is missing, runtime identities differ, or the counter decreases. Process reaping and reparenting can distort even a positive difference; this is an estimate, not exact accounting. Background work in the same process tree also contributes; the difference does not isolate agent-only CPU. Endpoint memory and process counts are instantaneous gauges, not peaks, averages, or invocation allocations.
For Harbor, these measurements come from the adapter host inside the task runner, even when a separate host selects the task and capabilities. Harbor's existing result JSON carries the new field without a new stats file. For a model team comparing identical configs, pair `usage` with these measurements: more tokens or tool calls can explain more agent work; similar work with longer elapsed time warrants investigating execution conditions. Endpoint observations alone miss intermediate peaks and cannot explain every source of latency. Enable sampled stats and inspect native cgroup evidence below; remote inference latency requires serving-side evidence.
A failed read appears as `{"status": "unavailable", "reason": "..."}`; an unsupported provider or operating system uses `"unsupported"`. Collection failures preserve the adapter outcome, including a normalized failed result. Disabled collection, older results, and execution backends without collection have `system_stats=None`. Protocol failures and timeouts that raise without a `RunResult` still raise. When sampling is enabled, NeMo Fabric retains sample and summary artifacts where possible even if the adapter exchange fails. These measurements require neither NeMo Relay collection nor invocation recording.
### Sampled Peaks and Means
A process can allocate 400 MiB during execution and return to 80 MiB before the final reading. Endpoints miss that spike. Enable native sampling independently of Relay or invocation recording:
```python
from nemo_fabric import SystemStatsConfig, SystemStatsSamplingConfig
config.system_stats = SystemStatsConfig(
sampling=SystemStatsSamplingConfig(interval_ms=1000, max_samples=10_000),
)
config.runtime.artifacts = "artifacts"
result = await Fabric().run(config, input="Fix the failing test")
stats = result.system_stats
sampled = None if stats is None else stats.sampled
if sampled is not None:
rss = sampled.resident_memory_bytes
if rss is not None:
print(rss.sampled_peak, rss.time_weighted_mean, rss.covered_fraction)
if sampled.samples is not None:
print(sampled.samples.path)
```
The mean holds the preceding reading across adjacent valid observations and excludes missing readings or gaps larger than twice cadence. Retained JSONL offsets allow benchmark tooling to correlate samples with trajectory events and select its own analysis windows. Temporal overlap with a span does not prove exclusive attribution.
For readings **80 MiB → 400 MiB → 80 MiB** at 0, 1, and 2 seconds, the final value is 80 MiB, sampled peak is 400 MiB, range is 80–400 MiB, and hold-last-value mean is 240 MiB. A sampled maximum can miss brief spikes between observations. It is not a true high-water mark.
Sampling uses one process-wide scheduler, retains both endpoints, and bounds intermediate retention with `max_samples`. The result exposes retained/dropped sample counts, collection time, largest gap, truncation, per-gauge covered time, diagnostics, and artifact references. Late samples are discarded after invocation closes. A full series preserves the final endpoint rather than silently reporting an old value as final. Counter resets, changed targets, missing observations, and long gaps invalidate affected sampled CPU estimates. Sampling does not replace endpoint-derived CPU or cgroup deltas; collection quality and continuity are reported under `sampled`.
With an artifact root, the layout is:
```text
artifacts/.fabric///
adapter-invocation.json
system-stats.jsonl
system-stats-summary.json
stdout.txt
```
Every sample and saved summary carries the existing runtime, invocation, and request IDs. `RunResult.artifacts` references both stats files. Without an explicit artifact root, these files use the runtime's temporary `.fabric` directory. Optional stats persistence failures produce diagnostics and preserve the agent outcome. A known series write failure suppresses its artifact reference while retaining in-memory reductions and a `sample_write_failed` diagnostic. Samples and a summary are retained even if the adapter protocol raises without a normalized result; they do not invent a terminal agent status or a second lifecycle journal. Consumers should handle an interrupted last JSONL line explicitly.
### Native Cgroup Constraints
On Linux cgroup v2, `result.system_stats.cgroup` contains before/after observations for the **adapter host's cgroup**, plus CPU-use, throttled-time, and OOM-kill deltas. Snapshots expose visible CPU/memory limits, container-accounted current memory, swap, provider memory high-water, and CPU/memory/I/O pressure totals. Missing limit sources report `unsupported`. Visible numeric limits take precedence over `max`; `unlimited` requires an explicit `max` source with no numeric bound. Individual missing measurements remain explicit; unsupported platforms do not substitute host-wide readings.
This cgroup can contain the NeMo Fabric runner, the adapter tree, and other workloads. Its totals overlap process-tree totals and must not be added to them. Limits account for visible ancestors only; namespaces can hide additional restrictions. `memory_peak_bytes` covers cgroup lifetime or an external reset, not one invocation. The collector does not reset shared counters or write to cgroup control files.
Pair these observations with `usage` and reward: more tokens/tool calls can explain more agent work; similar work with throttled CPU and a lower visible limit suggests a constrained execution environment. These signals support investigation and do not establish causality. Remote inference-server GPU usage still requires serving-side telemetry.
The [native stats example](https://github.com/NVIDIA/NeMo-Fabric/tree/v0.6.0/examples/system_stats) produces a transient memory spike and retained artifacts without model credentials.
Snapshots observe resources only. They do not enforce limits, stop runtimes, or
queue requests. Use the operating system or execution environment, such as
cgroups or container limits, to enforce resource limits.
## Native OpenAI Streaming
Use `Runtime.invoke_openai_stream()` when the selected adapter provides native
OpenAI Chat Completions chunks. `Runtime.supports_openai_streaming` reflects the
selected descriptor's existing `capabilities.streaming` claim. If the claim is
false, `invoke_openai_stream()` raises `FabricCapabilityError` with code
`openai_streaming_unavailable`.
```python
from nemo_fabric import Fabric
async with await Fabric().start_runtime(config) as runtime:
if not runtime.supports_openai_streaming:
raise RuntimeError("the selected adapter does not support OpenAI streaming")
stream = runtime.invoke_openai_stream(input="Review the latest patch")
async for chunk in stream:
print(chunk)
result = await stream.result()
print(result.status, result.output)
```
`invoke_openai_stream(...)` is synchronous and starts exactly one adapter
invocation in the background. Its `OpenAIInvokeStream` yields only JSON mappings
whose `object` field is `chat.completion.chunk`. An empty stream is valid. The
terminal normalized `RunResult` is not a final chunk; obtain it separately with
`await stream.result()` and treat it as authoritative.
Only one turn can be active on a runtime. Fully consume the stream before
starting another turn. If iteration stops early, call `await stream.aclose()`;
it drains the invocation and discards unread chunks so the runtime can be reused.
It does not cancel the target invocation. Awaiting `stream.result()` also drains
and discards any unread chunks before it returns, so consume the iterator first
when the application needs every chunk.
The SDK owns an authenticated listener on the loopback interface and supplies
the adapter host with a single-invocation HTTP sink that uses chunked NDJSON
framing. It validates chunk shape, framing, authentication, and invocation
correlation. This transport is an implementation detail; consumers provide no
listener address, token, or protocol configuration.
Native OpenAI streaming does not require NeMo Relay or
`start_runtime(..., streaming=True)`. It is independent of Relay ATOF streaming:
`supports_openai_streaming` and `invoke_openai_stream()` describe adapter-native
OpenAI chunks, while `supports_streaming` and `invoke_stream()` describe Relay
ATOF. Enabling one does not enable or change the other.
## NeMo Relay Streaming
Enable NeMo Relay before starting a runtime to consume raw ATOF records while a
turn runs. `Runtime.supports_streaming` reports whether this NeMo Relay path is
available. It is separate from `RuntimeCapabilities.streaming`, which describes
adapter-native progressive output. This separation is intentional:
`Runtime.invoke_stream()` exposes only ATOF records generated by NeMo Relay; it
does not call the adapter's native OpenAI stream operation.
Install the matching collector for the default embedded streaming path:
```bash
pip install "nemo-fabric[streaming]"
```
This extra is not required when `launch_collector=False` selects an externally
managed collector.
The following example streams one invocation and collects its terminal result:
```python
import asyncio
from nemo_fabric import (
EnvironmentConfig,
Fabric,
FabricConfig,
HarnessConfig,
MetadataConfig,
ModelConfig,
)
async def main() -> None:
config = FabricConfig(
metadata=MetadataConfig(name="streaming-agent"),
harness=HarnessConfig(adapter_id="nvidia.fabric.codex"),
models={
"default": ModelConfig(provider="openai", model="openai/gpt-5.4")
},
environment=EnvironmentConfig(provider="local", workspace="."),
).enable_relay()
async with await Fabric().start_runtime(config, streaming=True) as runtime:
stream = runtime.invoke_stream(input="Review the latest patch")
async for record in stream:
print(record)
result = await stream.result()
print(result.status, result.output)
asyncio.run(main())
```
`invoke_stream(...)` is synchronous and starts the invocation in the background.
The returned `InvokeStream` is an async iterator of raw NeMo Relay ATOF
dictionaries. The terminal `RunResult` stays out of band and is available only
through `await stream.result()`. Treat that terminal result as authoritative:
replace any provisional rendering instead of appending it. Calling
`stream.result()` does not finalize unread stream records. Calling
`Runtime.stop()` after the result completes finalizes the stream and discards
those unread records.
Streaming has the following v0.1 constraints:
* Streaming requires two explicit settings: enable NeMo Relay in `FabricConfig`
and pass `streaming=True` to `start_runtime(...)`. The `streaming=True` flag
does not enable NeMo Relay by itself; without NeMo Relay telemetry,
`start_runtime(...)` raises `FabricConfigError`. By default, streaming starts
an embedded collector and configures Relay to publish NDJSON to it. Set
`launch_collector=False` to use an externally managed collector. In that
case, configure exactly one `nemo-fabric-stream` sink with an `http` or
`https` collector base URL and `transport="ndjson"`; Fabric sends Relay to
`/v1/atof`. Pi is the exception as Pi streaming supports only the
default embedded collector. Do not set `launch_collector=False` for Pi
because its extension does not attach the NeMo Fabric request IDs required
for external correlation. The embedded collector waits one second for a late
Pi `agent_settled` marker by default. Increase `completion_wait_timeout` when
Relay delivery can take longer.
* The default `streaming=False` leaves the existing NeMo Relay configuration
unchanged; it neither enables nor disables ATOF. It also avoids collector
startup and streaming control traffic for runtimes that use only ATIF,
OpenTelemetry, OpenInference, or application-configured ATOF sinks.
* One invocation can be active on a runtime. Fully consume the stream or call
`await stream.aclose()` before starting another turn.
* Breaking an `async for` loop does not finalize the stream. Call
`await stream.aclose()` explicitly. It waits for the invocation to finish and
discards unread records; it does not cancel the harness turn.
* The end of async iteration only means that no more ATOF records are available.
It does not indicate invocation success. Always await `stream.result()`.
Invocation exceptions raise from that call, while harness-reported failures
are represented by the returned result's status and error fields.
* The collector limits each request queue to 1,024 records and 16 MiB of
encoded record data. It rejects individual records larger than 1 MiB and
applies TCP backpressure for up to 30 seconds. If the queue remains full,
the collector drops the record and logs a warning.
NeMo Relay can drop records if a consumer stalls longer than its delivery
timeout.
* ATOF granularity depends on the NeMo Relay integration. Gateway harnesses expose
per-delta event structure, but current ATOF records keep token text in the
terminal scope. In-process harnesses expose scope-level progress.
* Reconstruct nested and parallel work with `uuid` and `parent_uuid`. Stream
order alone does not define the Deep Agents scope tree.
* For request-correlated adapters, the runtime registers its request ID with
the collector before invoking the agent. The collector routes the ATOF root
scope carrying that request ID and its descendants, based on UUID ancestry,
to that invocation's private queue. With `relay_streaming: true`, Remote
Agent services receive the ID as `metadata.nemo_fabric_request_id` in the
mapped request body and must carry it into Relay metadata. Records that
cannot be correlated to a registered request are discarded. Pi instead
serializes both streaming and plain invocations behind one embedded-collector
lease. Streaming capture begins at the first `turn_start` and closes at
`agent_settled`. If output is interrupted or late, the collector discards the
remainder through that same terminal marker before allowing the next
invocation to start.
* An external collector can be shared by multiple NeMo Fabric runtimes whose
adapters propagate request IDs. It must be reachable by both Relay
publishers and NeMo Fabric clients. Use HTTPS and its configured authentication
when it is deployed beyond a trusted local boundary.
Claude and Codex streaming use the NeMo Relay `nemo-relay` gateway CLI and
require a release in the `>=0.9,<0.12` range. Their adapter `harness` and
`full` extras install the supported CLI. Deep Agents uses its in-process NeMo
Relay integration with Relay 0.11. Hermes Agent keeps Relay 0.10 with the pinned Hermes source in a separate Python 3.14 environment.
Remote Agent services own their Relay installation and publish to the same
configured `nemo-fabric-stream` collector
base URL. The local adapter does not install or start Relay for the service.
## Application-Owned Parallelism
Applications create independent runtimes when they want parallel work. NeMo Fabric
does not own a queue, worker pool, semaphore, retry policy, timeout policy, or
numeric concurrency limit. Each `Runtime` accepts one invocation at a time so
its ordered harness state cannot be changed by two calls at once. If an
application overlaps calls on the same `Runtime`, the second call raises
`FabricStateError`. To perform work in parallel, start independent runtimes;
the application decides how many to run.
Async lifecycle calls run blocking native work outside the Python event loop,
so independent runtimes can make progress concurrently. This does not add a
NeMo Fabric concurrency limit or scheduler.
```python
import asyncio
async def review_one(prompt: str):
return await Fabric().run(config, input=prompt)
async def main() -> None:
results = await asyncio.gather(
review_one("Review patch A"),
review_one("Review patch B"),
)
print([result.status for result in results])
asyncio.run(main())
```
For example, each Harbor job starts an independent NeMo Fabric runtime. Harbor owns
job IDs and concurrency policy; NeMo Fabric does not use a job ID to select or
resume runtime state.
## Unified Run Results
Every invocation that reaches the adapter boundary returns a normalized
`RunResult`, even when the harness invocation itself failed. Inspect `status`,
`error`, `events`, and `artifacts` first, then process `output` when the status
is successful.
Important fields:
| Field | Meaning |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | Terminal invocation status such as success, failure, or cancellation. |
| `output` | Harness output normalized to the configured output schema. |
| `error` | Structured failure metadata when available. |
| `usage` | Optional `RunUsage` with invocation-local input, cached-input, output, and total token counts, reported cost in USD, and provider metadata. |
| `system_stats` | Optional `InvocationSystemStats`: monotonic adapter exchange duration, before/after observations with availability, and an optional CPU counter difference estimate. |
| `resource_snapshot` | Optional `RuntimeResourceSnapshot` of the runtime, read once after the adapter returned the result. |
| `artifacts` | Output files, logs, patches, native artifacts, and other materialized references. |
| `telemetry` | References to NeMo Relay or other telemetry streams produced by the run. |
| `events` | Ordered normalized lifecycle and invocation events. |
| `metadata` | Result-specific structured metadata. |
| `runtime_id`, `invocation_id`, `request_id` | IDs for correlation across runtimes, logs, telemetry, and artifacts. |
These are structured correlation fields, not interchangeable metadata:
`runtime_id` identifies the runtime lifecycle, `invocation_id` identifies one
invocation within that runtime, and `request_id` correlates the caller's
request. NeMo Fabric-generated values use type-specific prefixes such as `runtime-`,
`invocation-`, and `request-`; callers may provide their own `request_id`.
Consumers should store and log each field separately and otherwise treat its
value as opaque rather than parsing the identifier encoding.
Usage is absent when the target does not report it. Read the typed values only
when `usage` is available:
```python
if result.usage is not None:
print(result.usage.get("input_tokens"))
print(result.usage.get("cached_input_tokens"))
print(result.usage.get("input_tokens_include_cache"))
print(result.usage.get("output_tokens"))
print(result.usage.get("total_tokens"))
print(result.usage.get("cost_usd"))
```
Each counter is optional: missing values are unknown, not zero. `cached_input_tokens` reports cached prompt tokens. `input_tokens_include_cache` is `True` when those tokens are already included in `input_tokens`, `False` when they are excluded, and absent when the adapter cannot establish the semantics. Consumers must not add cached tokens to an inclusive input count. Adapters can report cache-write, reasoning, peak request, and per-model detail through the descriptor-declared `usage.extensions["nemo_fabric.detailed_token_usage.v1"]` object. NeMo Fabric preserves that object at `RunUsage.metadata["nemo_fabric.detailed_token_usage.v1"]`. Consumers treat detailed input and output independently: any detailed accounting field for a dimension makes that dimension authoritative, with no fallback to its top-level counters. The views can diverge, and an inclusive total remains unknown when the selected detailed counters do not provide enough information to calculate it. `cost_usd` contains reported cost only; keep estimates separately in provider metadata rather than presenting them as measured charges.
If NeMo Fabric cannot resolve config, start a runtime, or obtain a normalized result,
the SDK raises a `FabricError` subclass instead of returning a partial
`RunResult`.
## Events
Each `RunResult` includes the normalized events collected for that invocation.
Event kinds and detail may vary by adapter, but their lifecycle and correlation
fields use the common NeMo Fabric contract.
Events are useful for:
* rendering invocation history in application or service UIs;
* forwarding logs and status to evaluation harnesses;
* correlating runtime, invocation, adapter, and telemetry IDs;
* reporting structured failures alongside the terminal result.
## Feature Support Across Harness Adapters
The SDK presents one consistent shape across adapters, but adapters differ in
their runtime requirements, accepted configuration, and optional capabilities.
Use `plan(...)` and `doctor(...)` before relying on optional features:
```python
fabric = Fabric()
plan = fabric.plan(config)
report = await fabric.doctor(config)
print(plan.adapter.adapter_id)
print(report.status)
```
Use the plan to confirm adapter selection and capability routing. Use the doctor
report to catch unresolved adapter descriptors, unsupported normalized
capabilities, missing declared requirements, and environment problems before
starting a runtime.
Capability routes assign execution ownership; they do not describe network
routing. `harness_native` means the selected adapter executes the capability
through its harness. `fabric_managed` means NeMo Fabric executes it outside the
harness-native surface. `unsupported` means neither side can execute it.
Routes apply to tools, skills, and MCP servers. Scalar fields such as
`instructions.system` and `runtime.max_turns` are validated separately against
the adapter descriptor.
## Install And Runtime Responsibilities
In production, the consumer or execution environment is responsible for
installing NeMo Fabric, the selected harness, adapter dependencies, model access,
credentials, and any required native tools. NeMo Fabric validates and diagnoses the
runtime assumptions, but it does not silently install harnesses or credentials at
invocation time.
Development environments may use extras, virtual environments, or local source
checkouts to make iteration easy. Production environments should prefer explicit
images, preinstalled dependencies, or managed deployment packages.
Runtime compatibility checks should validate:
* NeMo Fabric SDK and native extension versions;
* selected adapter version;
* selected harness version or version range;
* required environment variables or secret references;
* optional capability support such as NeMo Relay, MCP, or tool exposure.
## Custom Fields And Adapter Settings
Use normalized NeMo Fabric fields for portable behavior: models, system
instructions, turn limit, runtime, environment, skills, MCP, telemetry, tools,
and artifacts.
Supply request context through `RunRequest.context` for each invocation. Request
context is not part of `FabricConfig`.
Use `harness.settings` only for stable, harness-native behavior declared by the
selected descriptor. Each bundled adapter declares a closed settings schema.
Executable paths, Relay command discovery, state directories, and similar
launch mechanics are runtime implementation details rather than public adapter
settings.
Use the optional `workflow` block to select a registered target by `target_id`
and provide its immutable construction settings. The Adapter Target Descriptor
selects the adapter, supplies the target-native entry point, and validates
`workflow.settings` during planning. Fabric then projects the entry point and
settings into `AgentConfig.workflow`; consumer configuration does not repeat
adapter-specific entry-point semantics. Validation does not apply schema
defaults.
Use `FabricConfig.metadata` for human-readable agent identity and caller-owned
annotations. NeMo Fabric preserves these values in the resolved configuration
but does not copy them into `RunResult.metadata`. Use `request_id` and
`RunRequest.context` for per-invocation correlation. `RunResult.metadata`
contains adapter-specific result details.
To group several invocations into one NeMo Relay session, set
`RunRequest.relay_session_root` to the same UUID string on each
invocation. Deep Agents and mini-SWE-agent, which own an in-process Relay Agent
scope, use it as the propagated Relay root. A UUID `request_id` stays the
parent; otherwise the session root is also the parent. ATIF exports one
trajectory per invocation under one `session_id`. An unusable value falls back
to the per-request root without an error, and the invocation still succeeds. A
context key named `session_id` does not group traces.
For example, invoke each turn with `RunRequest(input=message, relay_session_root=conversation_uuid)`. Core forwards the typed field as `AgentRunRequest.relay_session_root`. Context keys do not control Relay grouping. Omitting the field preserves per-request behavior. Use matching runtime and adapter-contract versions when sending this field.
Adapter settings are not portable by default. The exact descriptor selected
during planning owns their JSON Schema. Planning and doctor reject settings
that do not satisfy that schema. A descriptor without `settings_schema` accepts
an empty settings map and rejects a non-empty map. Validation does not apply
schema defaults or otherwise change the settings preserved in `RunPlan`.
Validation errors identify the adapter, descriptor source and path, and
offending settings path.
## Errors
All public SDK errors inherit from `FabricError`.
| Error | Meaning |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FabricConfigError` | Invalid config, request, or override. |
| `FabricCapabilityError` | Selected adapter does not support the requested operation. |
| `FabricRuntimeError` | Runtime startup, invocation, or shutdown failed before a normalized result could be returned. |
| `FabricStateError` | Invalid runtime state transition, such as invoking after stop or starting overlapping invocations. |
| `FabricNativeUnavailableError` | Native extension is not installed or importable. |
| `ResourceObservationUnavailableError` | `Runtime.resource_snapshot()` cannot read the runtime because it stopped, is stopping, its adapter host exited, a live process in its tree or the process table cannot be read, or the process tree kept changing during the read. Subclass of `FabricRuntimeError`. |
| `ResourceObservationUnsupportedError` | The runtime's adapter kind, environment provider, or operating system has no resource reader. Subclass of `FabricRuntimeError`. |
Consumers own job-level retries and rollout-level failure policy.
Core host-operation deadlines (`host_timeout`) become Python `TimeoutError`, then
`FabricRuntimeError` with `code == "timeout"` and the failed lifecycle stage.
If invocation succeeds in `Fabric.run()` but stop reaches this deadline,
it returns a failed `RunResult` with `error.stage == "stop"` and
`error.code == "timeout"`. Check the returned result as well as raised errors.
Adapters can report execution deadlines in normalized results with
`result.error.code == "timeout"`. Other adapter-specific timeout codes are not
automatically reclassified. Diagnostic text is not a timeout classifier.
Single-invocation runs attempt to stop the runtime before returning. A `Runtime`
used with `async with` also attempts cleanup after an invocation error; if cleanup
fails, that failure is attached to the original exception rather than replacing
it. NeMo Fabric records structured error metadata when possible and returns enough
detail for the consumer to decide what to do next.
## SDK Contract Boundaries
NeMo Fabric keeps a narrow execution contract. Applications own product behavior
around that contract.
### Schemas And Python Models
The SDK's Pydantic models are maintained against the Rust-generated public
schemas. The schemas describe the typed cross-language contracts. Use the
generated API reference for exact Python signatures.
| Contract | Source |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| SDK Pydantic models | `nemo_fabric.models`; refer to the generated [Models reference](/nemo/fabric/reference/api/python-library-reference/models) |
| Agent config | `schemas/sdk/agent.schema.json` |
| Adapter contract | `schemas/adapter-contract/`, including the descriptor, `AgentConfig`, execution types, and runtime context |
| Run plan | `schemas/sdk/run-plan.schema.json` |
| Request, result, and events | `schemas/sdk/run-request.schema.json`, `schemas/sdk/run-result.schema.json`, `schemas/sdk/fabric-event.schema.json` |
| Runtime and invocation handles | `schemas/sdk/runtime-handle.schema.json`, `schemas/sdk/invocation-handle.schema.json` |
| Artifacts and errors | `schemas/sdk/artifact-manifest.schema.json`, `schemas/sdk/error-info.schema.json` |
| Runtime resource snapshot | `schemas/sdk/runtime-resource-snapshot.schema.json` |
### Versioning
NeMo Fabric uses explicit contract versions where persisted or independently
maintained artifacts cross package boundaries:
* `schema_version` identifies the typed NeMo Fabric config contract transported
across SDK and language boundaries.
* `contract_version` identifies the contract implemented by
`*.fabric-adapter.json` and `*.fabric-target.json` records.
* Python and Rust package versions identify the installed SDK/core release.
NeMo Fabric versions top-level typed contracts and independently maintained adapter
metadata. It does not version each config subsection independently. For
example, MCP, skills, models, telemetry, and runtime fields evolve under the
enclosing `schema_version`.
NeMo Fabric validates adapter descriptor contract versions during planning. Package
semver identifies the installed implementation, but it is not the compatibility
contract for independently maintained adapters.
### Config Extensibility
The public schema has typed fields for stable NeMo Fabric concepts and controlled
extension points for adapter- or application-owned data. Use known fields for
portable concepts such as harness selection, models, runtime, skills, MCP,
telemetry, and artifacts. Use adapter-owned `harness.settings` for settings
validated by the selected descriptor and interpreted by its adapter. Use
metadata or preserved extension fields for data NeMo Fabric should carry but
not interpret.
Additive optional fields may be introduced within the existing document schema
version when old configs remain valid. Required fields, removed fields, or
semantic changes that alter how existing configs are interpreted require a new
enclosing document schema version or an explicit compatibility path.
Unknown data is not the same as supported behavior. An adapter must advertise
and implement a capability before NeMo Fabric treats it as runnable.
### Resilience
NeMo Fabric reports lifecycle failures; applications own recovery policy. If a
runtime process dies, a connection is permanently lost, or an adapter cannot
complete an invocation, NeMo Fabric marks the relevant runtime/invocation as
failed and returns structured error metadata when possible.
Transient I/O failures may be marked with retryable error metadata, but NeMo Fabric
does not perform job-level retries by default. Consumers decide whether to retry
the request, start a replacement runtime, fail the job, or escalate to a user.
### Capacity And Backpressure
If a harness reports capacity pressure, an adapter should surface it as a
structured error or event such as busy, rate limited, capacity exceeded, or
backpressure. The consumer decides whether to wait, retry, scale out, or fail.
## Next Steps
* Review the examples in the GitHub repo [`examples/`](https://github.com/NVIDIA/NeMo-Fabric/tree/v0.6.0/examples).
## Selected Software Evidence
`RunResult.execution` optionally contains selected software inventories and existing infrastructure software reports. Each inventory has an opaque result-local `context_ref`, `participants`, availability, coverage, and component `entries`. Component roles belong in `entries[].role`. Equal versions do not establish a shared installation. Version observations retain the full value, observer/method in `source`, `origin`, and original UTC `observed_at`; missing values carry a status and reason. An unavailable version can still be represented in a complete selected inventory.
`FabricConfig.provenance` defaults to `ProvenanceConfig(enabled=True)`. Set `provenance=ProvenanceConfig(enabled=False)` (JSON: `"provenance": {"enabled": false}`) before runtime initialization to skip collector work on both sides of dispatch. `provenance.reports` accepts a typed `nemo_fabric_evidence.evidence.ExecutionEvidence` snapshot from the caller or environment owner. Supply it in the existing configuration dispatch; the guest collects its own snapshot. No callback or additional network service is involved. Reports are bounded to 16 contexts, 128 total entries and 64 KiB of compact UTF-8 JSON. Over-budget or malformed optional reports become explicit incomplete evidence without changing the invocation outcome.
`nemo-fabric-runtime` and `nemo-fabric-adapter-contract` both use the version-matched `nemo-fabric-evidence` package for shared software evidence and validation. The shared package uses only the standard library and owns the bounded metadata primitives used by both layers. Package selection and snapshot caching remain in the SDK and adapters. The SDK does not depend on the southbound adapter contract. Existing adapter-contract evidence imports remain compatible re-exports; the `nemo_fabric` SDK facade is unchanged.
Old saved results may omit `execution`. A receiving boundary expecting guest/adapter evidence records missing reports explicitly. Compare participants, component kind/name, full versions, topology and coverage separately; exclude capture times, random context/invocation IDs and artifact roots. Unknown values do not establish equality. This API provides observations, not a dependency inventory, reproducibility guarantee or attestation. Inputs/configuration capture is a follow-up.
### Snapshot Lifetime And Collection Bounds
The Python SDK captures its selected distribution and executing interpreter when a `Fabric` instance first initializes an enabled runtime, service, or environment. It reuses that snapshot for that SDK instance. Construct a new `Fabric` after a known installation change or SDK process restart. Each adapter host captures its own selected packages after startup; replacing/restarting the host creates a fresh snapshot. The compiled core reports its executing version independently. Inventories stay separate when no shared resolution context is established; equal versions or interpreter paths are insufficient evidence for joining installations.
Python package selection is adapter-owned. Loaded-module attribution checks prevent installed metadata from identifying a shadowing source tree. Missing metadata or an unattributed editable checkout produces an explicit unavailable version. Collector work uses at most 32 selected package lookups and 4,096 candidate metadata files per target, with at most 64 KiB of metadata headers and 512 KiB of RECORD bytes per target, a 250 ms initialization deadline and at most one outstanding daemon worker per process. Unsupported metadata backends remain explicit unknowns. TypeScript uses asynchronous selected manifest reads through at most eight node_modules ancestor levels, a 64 KiB manifest read limit, 32 selected package targets and a 250 ms deadline. Shared host and contract metadata use the executing common host's resolution context; selected harness packages use the adapter's context. Report limits apply after composition too: entries are dropped whole with incomplete coverage rather than shortening version values. Lookup failures preserve invocation output, errors and status. The escape hatch skips the collectors before lookup work begins.
Claude initialization messages and Relay's existing startup version check can supply passive observations. Receiving and forwarding these reports preserves their source, origin and first capture time. Installed packages are included for telemetry only when the selected component enables that integration; package presence alone does not establish use.
### Dispatch And Shared Services
OpenShell forwards the caller snapshot in `FabricConfig.provenance.reports` through the existing task start command. The guest SDK and adapter initialize their own snapshots. The host retains its copy independently, including when an older intermediary drops `execution`; that case produces a `guest_sdk` coverage gap while preserving the invocation outcome. Context refs are rebased when reports are composed. The host reuses its own forwarded prefix only when it is an exact copy of its request, including original capture times; matching versions never merge installations.
Existing OpenShell gateway health responses supply provider versions and capture times. Older provider reports without capture times are observed once at host runtime initialization with source `host:provider_report_at_runtime_start`. Provider reports reference guest contexts. NeMo Fabric does not infer a container engine. Explicit container-owner reports can use `provenance.reports` with `infrastructure.container.engine_version`, which means the engine server version.
OpenClaw service startup reports retain their original source and capture time when multiple runtimes connect to the same service. A separate service inventory contains the reported OpenClaw version without inventing the remote interpreter or package inventory. Existing startup checks can refresh a changed version; a newly prepared or attached service creates a new snapshot. Software observations and their timestamps do not participate in runtime/service binding hashes. Runtime bindings retain the collection control chosen at initialization; service bindings allow a connecting runtime to choose its own collection control. A runtime can disable collection while connecting to a previously initialized service.
### Calling Application Reports
Calling applications and environment owners supply typed `ProvenanceConfig.reports` snapshots through the configuration already sent to NeMo Fabric. Capture them at application initialization and preserve their original `source`, `origin`, availability and `observed_at`. A guest never inspects its caller. Harbor implements this seam with `fabric_provenance`, whose shape matches the SDK control; `--ak 'fabric_provenance={"enabled":false}'` disables collection before host initialization and propagates to the guest.
The [software evidence example](https://github.com/NVIDIA/NeMo-Fabric/tree/v0.6.0/examples/provenance) provides six execution-path checks, separate-venv runs, saved RunResults and private comparison fixtures. Its local provider/service fixtures are labeled separately from live deployments. Safe configuration capture is the P0 follow-up and is outside this software-version contract.