NVIDIA NeMo Fabric Python SDK

The Python SDK is the application-facing interface for NeMo Fabric. Use it to configure an agent harness, inspect the resolved plan, run one request or a multi-turn runtime, and collect normalized results, events, artifacts, and telemetry references.

The SDK is config-first. Applications should construct a Pydantic FabricConfig from their own job, deployment, or evaluation config. Examples, CI jobs, and applications construct the typed config directly and can use ordinary Python functions to create variants.

Generated API reference pages remain the source of truth for exact signatures. This guide explains how the pieces are intended to fit together.

For installation and a package-backed example, start with the NeMo Fabric overview.

Start With One Run

This example uses an NVIDIA-hosted model. Export your NVIDIA API key before running it:

export NVIDIA_API_KEY=...

Construct a typed config, then pass it to Fabric.run(...). The SDK starts a runtime, invokes it once, collects the result, and stops the runtime.

import asyncio
from nemo_fabric import (
    Fabric,
    FabricConfig,
    HarnessConfig,
    InstructionConfig,
    InstructionsConfig,
    MetadataConfig,
    ModelConfig,
    RuntimeConfig,
)
config = FabricConfig(
    metadata=MetadataConfig(name="review-agent"),
    harness=HarnessConfig(adapter_id="nvidia.fabric.hermes"),
    instructions=InstructionsConfig(
        system=InstructionConfig(
            content="Review code for concrete correctness risks.",
            mode="replace",
        )
    ),
    runtime=RuntimeConfig(max_turns=8),
    models={
        "default": ModelConfig(
            provider="nvidia",
            model="nvidia/nemotron-3-nano-omni-30b-a3b-reasoning",
            api_key_env="NVIDIA_API_KEY",
        )
    },
)
async def main() -> None:
    fabric = Fabric()
    result = await fabric.run(
        config,
        input="Review the workspace changes.",
    )
    print(result.status)
    print(result.output.response)
asyncio.run(main())

Omit instructions.system to preserve the harness’s native system instruction. When present, InstructionConfig.mode defaults to replace. Use append only with an adapter whose descriptor advertises that mode; planning rejects other combinations before the runtime starts.

Use base_dir to resolve relative paths in an in-memory config. Before running in a new environment, call plan(...) to inspect adapter selection and doctor(...) to check runtime requirements.

The remaining examples reuse this config value.

Inspect Metadata On A Separate Host

Use inspect_adapter(config, descriptor) when the admission host cannot access the task’s installed adapter. Supply canonical metadata from the matching nemo-fabric-adapter-catalog release or an external adapter:

from nemo_fabric import inspect_adapter
from nemo_fabric_adapter_catalog import get_adapter_descriptor
profile = inspect_adapter(config, get_adapter_descriptor(config.harness.adapter_id))
print(profile.skills, profile.mcp, profile.atif)
# In the task environment, resolve its own executable descriptor.
result = await Fabric().run(
    config,
    input="Review the changes.",
    expected_descriptor_sha256=profile.descriptor_sha256,
)

The immutable AdapterCapabilityProfile reports declared skills and MCP support and ATIF availability for the selected telemetry configuration. Inspection uses the existing native planner without importing the harness SDK or reading task-local discovery paths. Unsupported requested configuration fails admission. Passing None as the descriptor permits basic execution with false capability claims, but rejects requested skills, MCP, or telemetry.

The optional expected_descriptor_sha256 argument on run() and start_runtime() verifies the normalized descriptor in the actual execution plan before startup. Catalog metadata never becomes an execution origin. A fingerprint confirms descriptor correspondence, not installed software versions or successful evidence production. ATIF requires a selected provider declaring that output; Relay additionally requires one explicitly enabled observability component with ATIF enabled. Unconfirmed ATIF requests, including ambiguous duplicate components, fail admission. For a registered workflow, also supply its canonical target metadata: inspect_adapter(config, descriptor, target_descriptor=get_target_descriptor(config.workflow.target_id)). Import get_target_descriptor from nemo_fabric_adapter_catalog. The fingerprint then covers both normalized descriptors, including the target factory reference and settings schema. Missing or mismatched target metadata fails admission. Task execution still resolves its own installed target and verifies both descriptors before starting the factory. Standalone fingerprints are unchanged. Attached services require separate route-specific admission.

Execution Model

NeMo Fabric separates configuration, planning, runtime lifecycle, and individual invocations. It does not expose a separate portable session layer.

Most application code works with five objects:

Fabric is a lightweight, reusable SDK facade. It resolves configuration, creates plans and runtimes, and provides a single-invocation convenience API, but it does not represent a started execution and does not require cleanup. A Runtime owns stateful execution and shutdown, so it is the object used as an async context manager.

RuntimeHandle and InvocationHandle carry lifecycle identity across the native boundary. Most Python callers use their runtime_id and invocation_id through Runtime and RunResult rather than manipulating the handles directly.

A runtime is a logical execution boundary, not necessarily an operating-system process. An adapter may use an in-process SDK, a process, or shared service infrastructure while preserving isolated state for each NeMo Fabric runtime.

Runtime hosting is selected by the adapter kind, not by a public FabricConfig setting. Every local Process or Python adapter implements the persistent local-host wire protocol. NeMo Fabric starts one host during start_runtime(...), reuses its adapter-owned native resources across ordered invocations, and attempts to release them during stop(). The lifecycle start operation carries the resolved configuration and capability plan. Each subsequent AdapterInvocation carries only runtime_context and request.

Third-party local adapters must implement the persistent local-host contract. Adapters that advertise service support can also prepare a NeMo Fabric-owned service or attach to a caller-owned service. Multiple runtimes can connect to one service while retaining distinct runtime IDs and harness sessions. Service handles are process-local in this release and must not be persisted for cold resume. A crashed persistent host is terminal for its runtime or service. The same applies when the host exceeds the protocol response timeout. NeMo Fabric does not silently respawn the host or replay a request.

Harness-native threads, sessions, and conversations remain adapter-owned state associated with the NeMo Fabric runtime. They are not additional NeMo Fabric lifecycle objects.

NeMo Fabric provides the runtime contract. Applications own scheduling, queues, retries, worker scaling, and the number of runtimes to run.

Explicit Environment Lifecycle

prepare_environment(config) returns an immutable EnvironmentHandle without starting an agent. Pass it to start_runtime_in(config, environment) to start a runtime, then invoke and stop that runtime normally. Call release_environment(environment) separately after collecting results. Local release preserves caller-owned files and supports multiple runtimes.

attach_environment(config, EnvironmentReference.from_mapping(...)) asks the selected provider to verify an existing resource. Providers that do not support attachment return an unsupported-provider error. Attachment retains caller ownership; release detaches without deleting that resource. Provider availability and runtime reuse limits depend on the selected environment implementation. Non-local workspace and artifact paths refer to the task environment and are not resolved against the host’s base_dir. The OpenShell provider starts the agent through the task SDK command bridge. One sandbox can host multiple independent runtime IDs; invoke and stop target a specific ID, and stopping one does not stop the others. Streaming and shared-service hosting remain local SDK capabilities.

The optional OpenShell provider uses the task command bridge and returns the task-generated SDK runtime ID and normalized results. Declared files are collected under the host’s runtime.artifacts/<runtime_id>/<invocation_id>/ directory. The task artifact root is separate. OpenShell invocation is buffered; streaming and shared-service hosting remain local SDK capabilities. Commands with uncertain outcomes are never replayed; an uncertain start blocks further starts in that sandbox.

Runtime Health

Use Fabric.check(environment=..., request=...) to check runtime health. Refer to the health-check overview for scopes, outcomes, and deadlines, and the adapter support table for supported levels and native evidence.

The runtime ID selects the exact session inside the supplied environment:

from nemo_fabric import CheckRequest, Fabric
fabric = Fabric()
environment = await fabric.prepare_environment(config)
runtime = await fabric.start_runtime_in(config, environment)
try:
    report = await fabric.check(
        environment=environment,
        request=CheckRequest(
            scope="runtime", runtime_id=runtime.runtime_id, level="ready"
        ),
    )
    print(report.outcome)
finally:
    await runtime.stop()
    await fabric.release_environment(environment)

The default level is live, and the default timeout is three seconds. A request for ready evaluates live, active, and ready in order, stopping at the first nonpassing result. One deadline covers routing, every probe, and response decoding. Checks can run during an invocation; a busy session can pass all three levels. A timeout returns unknown/probe_timeout. Stop or environment release during a probe returns unknown/observation_changed.

CheckReport is an immutable mapping with outcomes passed, failed, unknown, and unsupported. Reports contain fixed evidence-source IDs, UTC timestamps, and elapsed milliseconds. An unavailable runtime returns unknown/runtime_unavailable. Adapters without health support return unsupported/unsupported_adapter.

Cline live checks verify native session residency, identity, and closure. active checks also inspect request-interface state. ready checks additionally call ClineCore.get() to read a fresh native session snapshot. Cline checks can run during the first invocation and do not require model access. Refer to the Cline health contract.

Codex checks its owned app-server process and a fresh loaded-thread response; native idle or active status is the readiness equivalent. Pi checks native session liveness, initialization and request-interface guards, and required model, tool, and MCP connection state. OpenClaw checks Gateway liveness, completed startup, selected-agent registration, the enabled Chat Completions endpoint, and native readiness including configured channels. Deep Agents checks its owned checkpoint connection, compiled message interface, and native state for the selected LangGraph thread. Hermes checks its owned native session, initialized invocation interface, and a fresh get_activity_summary() diagnostic response. Runtime activity covers NeMo Fabric invocations. Refer to the OpenClaw runtime health guide.

A local CheckRequest(scope="server") returns unsupported/unsupported_scope. For OpenShell environments, Fabric.check() maps to fabric-agent-ctl check in the sandbox. Server requests stop at the runtime server; runtime requests select the owned agent and use the local SDK health path inside the sandbox. Health uses an independent provider worker during remote invocations.

The diagram shows the local runtime adapter path. OpenShell adds the provider and command bridge before this path; server checks stop at the runtime server.

The lifecycle dispatcher awaits each start, invocation, or stop operation. The health listener checks the same adapter runtime through a separate connection, so checks can run while an invocation is pending. The host publishes the port and a per-process shared secret in a private health.endpoint.json file under FABRIC_HOME; the Rust core reads that file and authenticates health requests and responses. Each health request carries the remaining timeout budget, which the host applies to the probe.

Gateway Service Health

Check a prepared or attached OpenClaw Gateway with Fabric.check(service=..., request=...). Service checks use the selected process-local ServiceHandle:

from nemo_fabric import CheckRequest, Fabric
fabric = Fabric()
async with await fabric.prepare_service(config) as service:
    report = await fabric.check(
        service=service.handle,
        request=CheckRequest(
            scope="service", service_id=service.service_id, level="ready"
        ),
    )
    print(report.outcome)

For OpenClaw, live validates /healthz, active validates an authenticated /v1/models response, and ready follows native /readyz, including configured channel health. A failed channel can fail Gateway readiness while live and active pass. Refer to the OpenClaw Gateway health guide.

Service checks evaluate earlier levels in order under the same three-second default deadline. They can run during invocations. Reports have scope="service", the selected service_id, and null runtime evidence. Release during a check invalidates evidence with unknown/observation_changed; a released or unavailable binding returns unknown/service_unavailable. Service hosting and checks remain local SDK capabilities; the sandbox command bridge continues to support server and runtime checks.

Sandbox Command Bridge

fabric-agent-ctl check checks server responsiveness with a three-second default deadline. Use --timeout-seconds SECONDS to set another positive, finite budget. Server checks work before a runtime exists and during an invocation. Completed checks return a fabric.check/v1alpha1 report in result, with changed: false; passed health exits 0, while failed, unknown, or unsupported health exits 1. Use fabric-agent-ctl check --agent researcher --runtime-id R1 --ready to check one runtime and return a runtime-scoped report. Runtime checks require both target flags and default to --live; --active and --ready are mutually exclusive alternatives. OpenShell SDK requests map to these named commands. Each layer forwards the remaining deadline, and failed health evidence stays in the typed SDK report.

fabric.check/v1alpha1 identifies the shared report schema returned by Fabric.check() and fabric-agent-ctl check. The executable ships with nemo-fabric-runtime and follows the SDK package release.

fabric-agent-ctl serve starts one runtime server per sandbox. The start, invoke, and stop commands call the existing Fabric.start_runtime(), Runtime.invoke(), and Runtime.stop() methods. Start accepts FabricConfig and returns the SDK RuntimeHandle mapping. Invoke accepts a full RunRequest through --request or any JSON SDK input through --input, and returns the SDK RunResult mapping. Stop returns JSON null. These payloads occupy the response envelope’s result field; command diagnostics remain separate. Targeted commands require the returned runtime ID and exact agent name. The caller waits for the private socket before start. Caller-side preflight uses Fabric.plan() and Fabric.doctor(); startup validates configuration against installed descriptors.

One sandbox can own multiple runtimes, each addressed by its returned runtime ID. Invoke and stop operations are independent across IDs; stopping one runtime leaves the others usable, and another runtime can be started after a stop. An uncertain start blocks further starts because the server cannot safely determine whether an unaddressable runtime exists. Ordinary SDK stop does not confirm descendant cleanup, so the runtime server retains fabric.unclean after runtime use and reports cleanup_unconfirmed if previous sandbox process cleanup cannot be confirmed. Refer to the runtime server command guide.

Configure Agents In Code

Normalized Configuration Compatibility

NeMo Fabric-owned fields such as workspace, environment variables, and the invocation deadline apply to every local adapter. Adapter-translated fields are validated against the selected adapter descriptor during planning.

Core means NeMo Fabric owns the behavior and applies it uniformly before or around adapter execution. Yes means the adapter translates the normalized field into its harness. No means an explicitly configured value fails planning instead of being ignored. The following table groups provider-specific Relay subfields:

Model selection is deterministic: the default role wins; otherwise a single named role is selected. More than one role without default fails planning. All bundled adapters except Hermes publish a descriptor-owned model_schema for every configured model role. The Claude and Codex native providers (anthropic and openai, respectively) keep the existing authentication path. Other Claude and Codex providers remain valid only with an explicit base_url and api_key_env. Deep Agents keeps dynamic LangChain provider selection. Each published schema rejects undeclared ModelConfig.settings during planning and reports each issue through doctor(...) before adapter startup. Existing configurations that supplied top_p and max_tokens as flattened model extensions retain the same wire shape when loaded as normalized fields. Legacy adapter descriptors that accept either name through extension_schemas.model continue receiving it in AgentModelConfig.extensions; adapters that advertise the normalized capability receive the typed field instead.

If a normalized field has no complete mapping, plan(...) and runtime start fail with a configuration compatibility error naming the adapter and field. doctor(...) retains the incompatibility and returns a failed diagnostic check so callers can inspect all preflight findings. NeMo Fabric does not silently drop the field. runtime.max_turns is optional, so a config that omits it remains portable across adapters with different native turn-limit support.

Build the complete nested FabricConfig directly, or start with a base config and use helpers to add capabilities. For example, extend the config from the first example with skills, MCP, and telemetry:

capability_config = config.model_copy(deep=True)
capability_config.add_skill_path("./skills/code-review")
capability_config.add_mcp_server(
    "github",
    transport="stdio",
    url="github-mcp",
    args=["stdio"],
    env={"GITHUB_TOKEN": "${GITHUB_TOKEN}"},
    exposure="harness_native",
)
capability_config.enable_relay(
    project="fabric-review",
    output_dir="./artifacts/relay",
)

Set Environment Values Or Forward Them By Name

Use environment.env to set literal values. Use environment.env_names to forward variables that are already available to the NeMo Fabric process.

For example, set GITHUB_TOKEN in the shell that launches your application:

export GITHUB_TOKEN="example-token"

Then extend the configuration with a literal log level and the token’s name:

from nemo_fabric import EnvironmentConfig
config.environment = EnvironmentConfig(
    env={"LOG_LEVEL": "debug"},
    env_names=["GITHUB_TOKEN"],
)

NeMo Fabric looks up GITHUB_TOKEN in its own process environment and passes the actual value, example-token, to the adapter in RuntimeContext.environment.env. The adapter receives the value, not a name that it must look up. The configuration and run plan contain only the name; NeMo Fabric redacts environment values in the saved adapter invocation. Execution fails if a listed variable is unset. A name cannot appear in both env and env_names.

environment.env does not expand $GITHUB_TOKEN or ${GITHUB_TOKEN}. For example, env={"GITHUB_TOKEN": "$GITHUB_TOKEN"} passes the literal text $GITHUB_TOKEN. Use env_names=["GITHUB_TOKEN"] to request lookup. If your application instead reads the token and places its value in env, that value becomes part of the saved configuration and run plan.

A .env file stores names and values, such as GITHUB_TOKEN=example-token. Neither env nor env_names loads that file. Your launcher or application must load it into the process environment before NeMo Fabric starts the adapter. The same forwarding mechanism works with variables supplied by a shell, CI secrets, or a container.

With Harbor, the NeMo Fabric process runs inside the task environment. env_names reads that process’s variables; listing a name does not copy its value from the Harbor host into the task. Harbor owns delivery through its managed execution environment. The existing Harbor bridge uses a separate environment_env_names payload field and resolves those names into environment.env inside the task runner. That bridge behavior is unchanged by env_names. Refer to Harbor Environment Forwarding for the two-stage flow.

NeMo Fabric reads named values at adapter start and checks them before each invocation. Persistent harnesses and shell backends can retain their startup environment, so NeMo Fabric rejects an invocation if a named value has changed instead of sending a context that disagrees with that environment. The error identifies the variable without exposing its value. Restart the runtime and any associated service after changing a forwarded value. Connecting a new runtime to a service also fails if the named values differ from the service’s startup values.

Configure MCP Authentication And Headers

MCP authentication and custom-header support vary by adapter:

Claude and Deep Agents return a configuration error when authentication is set. The bundled adapters do not support service_account MCP authentication.

Configure OAuth 2.0 with McpAuthenticationConfig. The adapter and MCP server complete the authorization flow before the first agent turn. The flow can require interactive browser authorization.

from nemo_fabric import McpAuthenticationConfig
capability_config.add_mcp_server(
    "github",
    transport="streamable-http",
    url="https://mcp.example.com/mcp",
    authentication=McpAuthenticationConfig(
        type="oauth2",
        scopes=["repo:read"],
    ),
    exposure="harness_native",
)

Use custom_headers to send HTTP headers with an sse or streamable-http server. Header names and values must be valid HTTP header text. To provide a sensitive value, set it in the process environment and use a ${NAME} reference in the header value:

import getpass
import os
os.environ["MCP_ACCESS_TOKEN"] = getpass.getpass("MCP access token: ")
capability_config.add_mcp_server(
    "private-api",
    transport="streamable-http",
    url="https://mcp.example.com/mcp",
    authentication=None,
    custom_headers={
        "Authorization": "Bearer ${MCP_ACCESS_TOKEN}",
        "X-Tenant": "example-team",
    },
)

Set referenced environment variables before calling run(...) or start_runtime(...). Do not use McpServerConfig.env for HTTP header variables; env applies only to stdio MCP server processes.

MCP tool policy is scoped to one server and uses the tool names that server advertises. allowed_tools=None exposes every discovered tool, while an empty allowlist exposes none. blocked_tools is then removed from the allowed or discovered set, and the same tool cannot appear in both lists:

capability_config.add_mcp_server(
    "github",
    transport="streamable-http",
    url="${GITHUB_MCP_URL}",
    exposure="harness_native",
    allowed_tools=["issues.read", "pull_requests.read"],
    blocked_tools=["issues.delete"],
)

An adapter must declare both mcp and mcp.tool_filters support before planning accepts these fields. The bundled adapters currently support unfiltered MCP servers only. Per-server MCP policy is distinct from the root ToolsConfig, which controls adapter-native tools across the harness. An MCP server requiring authentication also requires mcp.auth.oauth2 or mcp.auth.service_account, matching its configured authentication type.

tools.definitions names tools and tool groups that the adapter constructs. Each definition has a portable kind, an adapter-resolved ref, and settings validated by the adapter descriptor. Use add_tool_definition(...) only when the selected adapter accepts tools.definitions and publishes a tool_definition_schema.

tools.enabled and tools.blocked select across the effective named tool surface. Configure an allowlist, a blocklist, or both:

from nemo_fabric import ToolsConfig
def with_tool_policy(base):
    tool_config = base.model_copy(deep=True)
    tool_config.tools = ToolsConfig(
        enabled=["Read", "Edit", "Bash"],
        blocked=["WebFetch"],
    )
    return tool_config

tools.enabled=None preserves the harness default. An empty list disables every executable tool. A tool cannot appear in both lists. The selected adapter must enforce the complete configured policy or planning fails. Hermes interprets these adapter-native selectors as Hermes toolset names.

Config helpers edit the typed config before planning or starting a runtime. They do not modify already-started runtimes. Use block_tools(...), remove_mcp_server(name), and remove_skill_path(path) to edit a copied config. Telemetry is enabled by adding entries to telemetry.providers; settings specific to NeMo Relay live in the top-level relay block.

For evaluation or deployment variations, use ordinary Python functions and copies of the typed config. Supply the complete final config to NeMo Fabric.

def review_agent_config(base, *, github_mcp: bool, relay: bool):
    config = base.model_copy(deep=True)
    if github_mcp:
        config.add_mcp_server(
            "github",
            transport="streamable-http",
            url="${GITHUB_MCP_URL}",
            exposure="harness_native",
        )
    if relay:
        config.enable_relay(project="fabric-review", output_dir="./artifacts/relay")
    return config
variant = review_agent_config(config, github_mcp=True, relay=True)

NeMo Relay observability is represented directly in the SDK config’s top-level relay block. Configure ATOF file and stream sinks as follows:

from nemo_fabric import (
    RelayAtofConfig,
    RelayAtofFileSinkConfig,
    RelayAtofStreamSinkConfig,
    RelayObservabilityConfig,
)
relay_config = config.model_copy(deep=True)
relay_config.enable_relay(
    output_dir="./artifacts/relay",
    observability=RelayObservabilityConfig(
        atof=RelayAtofConfig(
            enabled=True,
            sinks=[
                RelayAtofFileSinkConfig(
                    output_directory="./artifacts/relay",
                    filename="events.atof.jsonl",
                    mode="overwrite",
                ),
                RelayAtofStreamSinkConfig(
                    name="nemo-fabric-stream",
                    url="http://localhost:4319",
                    transport="ndjson",
                ),
            ],
        ),
    ),
)

To write OpenTelemetry spans to local files, add file_sinks to the opentelemetry section. File sinks need NeMo Relay 0.10 or later. You can use them instead of endpoints or alongside them. When opentelemetry is enabled, it needs at least one endpoint or file sink.

The following example writes OpenInference spans to files instead of sending them to a collector:

relay_config.enable_relay(
    output_dir="./artifacts/relay",
    observability={
        "opentelemetry": {
            "enabled": True,
            "file_sinks": [
                {"type": "openinference", "output_directory": "./artifacts/relay"}
            ],
        },
    },
)

NeMo Fabric handles these files as follows:

• As with ATOF and ATIF files, each runtime writes to its own directory: <output_directory>/<runtime_id>. If you omit output_directory, NeMo Fabric uses artifacts/relay.
• If you omit filename, Relay names each file with a timestamp when it starts. A later invocation can then write to a new file.
• File sinks append by default. If you set filename, that file keeps the spans from earlier invocations. To replace it each time Relay starts, set "mode": "overwrite".
• RunResult.artifacts lists each file as an artifact of kind otlp. Its media type is application/x-ndjson for the default json_lines format and application/x-protobuf for the proto format.

Relay buffers spans and writes them in batches. It writes the final batch when it shuts down, so read the files after the runtime stops. If Relay crashes, or is forcibly stopped because it does not shut down in time, a file can be missing its most recent spans.

RelayAtifConfig.model_name is an optional ATIF-only override. When you omit it, NeMo Fabric writes the selected default model, or the only configured model, into the Relay plugin configuration. If config.models is empty, the resolved configuration uses "unknown", matching the adapter fallback. Multiple model roles without default fail planning before telemetry resolution. The derived label preserves ModelConfig.model exactly, including provider prefixes such as openai/gpt-5-codex. Adapter-specific wire-model translation does not change this telemetry label. Set the override explicitly only when the trajectory must use a different label.

Additional NeMo Relay plugin components can be supplied generically with RelayComponentConfig when their component package is available in the runtime environment.

The repository’s code-review example uses this pattern for complete Hermes Agent, Codex, Deep Agents, environment, MCP, and telemetry variants.

If a config contains relative paths, pass a base_dir to plan(...), doctor(...), run(...), prepare_service(...), attach_service(...), or start_runtime(...). The base directory anchors skills, workspaces, artifacts, and other relative paths to the caller’s package or job layout.

Configure Dynamic Relay Plugins

Dynamic Relay plugins are configured separately under relay.plugins.dynamic. Enable telemetry.providers.relay with enable_relay() so NeMo Fabric forwards this configuration to the adapter’s Relay integration. The following example configures a dynamic plugin without static components:

from nemo_fabric import RelayConfig
plugin_config = config.model_copy(deep=True)
plugin_config.relay = RelayConfig(
    plugins={
        "dynamic": [
            {
                "manifest": "/opt/switchyard/relay-plugin.toml",
                "enabled": True,
            }
        ]
    }
)
plugin_config.enable_relay()

NeMo Fabric preserves the relay.plugins section alongside any configured relay.observability, relay.components, and relay.policy. Relay owns manifest validation, loading, and activation. Use absolute manifest paths accessible inside the agent environment. Plugin activation depends on the selected adapter’s Relay integration and the installed Relay version. For Hermes installation requirements, refer to Use the Hermes Agent Adapter.

API Inventory

Create Fabric() as the primary SDK entrypoint. It is a regular Python object, not a lifecycle context manager, and may be reused to plan, diagnose, or start multiple independent runtimes.

The following table summarizes the SDK entry points and runtime methods:

Shared Services

Use prepare_service(...) when NeMo Fabric should create and supervise a long-lived service. Start independent runtimes against the returned service, then stop every runtime before releasing it.

import asyncio
from nemo_fabric import Fabric
async def main():
    fabric = Fabric()
    async with await fabric.prepare_service(config) as service:
        async with await fabric.start_runtime(config, service=service) as first:
            async with await fabric.start_runtime(config, service=service) as second:
                first_result = await first.invoke(input="Inspect the repository")
                second_result = await second.invoke(input="Review the latest patch")
asyncio.run(main())

Use attach_service(...) when another system owns deployment and supervision. The adapter defines the required ServiceReference.connection fields. The reference contains locators and secret references, not secret values. Releasing an attachment only removes NeMo Fabric-owned clients and temporary state.

import asyncio
from nemo_fabric import Fabric, ServiceReference
async def main():
    reference = ServiceReference.from_mapping(
        {
            "adapter_id": "nvidia.fabric.openclaw",
            "service_type": "openclaw_gateway",
            "connection": {
                "gateway_url": "https://openclaw.example.com",
                "gateway_token_env": "OPENCLAW_GATEWAY_TOKEN",
            },
        }
    )
    fabric = Fabric()
    async with await fabric.attach_service(config, reference) as service:
        async with await fabric.start_runtime(config, service=service) as runtime:
            result = await runtime.invoke(input="Review the latest patch")
asyncio.run(main())

Service.handle is safe to inspect but is valid only in the process that created it. Cold-resume behavior will be defined separately.

Single-Invocation Runs

Use run(...) when the application has one input and does not need to preserve runtime state after the result is collected.

from nemo_fabric import Fabric, RunRequest
request = RunRequest(
    input="Review the workspace changes.",
    request_id="request-123",
    context={"source": "review-service"},
)
fabric = Fabric()
result = await fabric.run(
    config,
    base_dir="/workspace/review-agent",
    request=request,
)
print(result.status)
print(result.output.response)
print(result.artifacts)

Use input=... for the common case. Use request=RunRequest(...) for structured invocation metadata. Applications read files themselves and pass either the loaded input or a validated request to NeMo Fabric.

NeMo Fabric generates runtime and invocation IDs for lifecycle correlation. An application may include its own identifiers in opaque request metadata, but NeMo Fabric does not interpret them as job, session, scheduling, or resume state.

Multi-Turn Runtimes

Use start_runtime(...) when the selected harness should keep state across turns. Every call creates a new logical NeMo Fabric runtime; callers reuse the returned object rather than selecting it with a job or session ID. The runtime stops when its async context exits.

from nemo_fabric import Fabric
fabric = Fabric()
async with await fabric.start_runtime(
    config,
    base_dir="/workspace/review-agent",
) as runtime:
    first = await runtime.invoke(input="Inspect the repository")
    second = await runtime.invoke(input="Now review the latest patch")
print(first.status, second.status)

The adapter reuses its native state between calls. Codex maps the calls to turns on one live thread, Deep Agents invokes one compiled graph and checkpointer, Hermes Agent reuses one agent and session database, and Claude keeps one connected SDK client. Harness-native identifiers remain adapter-internal.

Runtime Resource Snapshots

Runtime.resource_snapshot() returns a timestamped RuntimeResourceSnapshot for a started local runtime. It does not wait for an active invocation, so a monitor can call it while invoke() runs:

snapshot = await runtime.resource_snapshot()
print(snapshot.observed_at, snapshot.resident_memory_bytes)
print(snapshot.process_count, snapshot.cpu_time_seconds)

The scope is fixed: the local adapter host process that NeMo Fabric started for the runtime, plus its descendants. For a runtime started on a shared service, the service process is not included. A descendant that starts a new session or process group still counts while its parent chain leads to the host. A process whose parent exited is reparented outside the tree and no longer counts. The snapshot finds the tree from each process’s list of children: Linux reads /proc/<pid>/task/<tid>/children and /proc/<pid>/stat, and macOS reads proc_listchildpids and proc_pid_rusage. Both read only the runtime’s own processes. On a Linux kernel without those children lists, the snapshot reads the parent ID of every process on the machine to find the tree. The snapshot exposes only the fields above, never command lines, environment variables, credentials, or processes outside the runtime’s tree.

When NeMo Fabric cannot read the values, it raises an error instead of returning zeros or leaving a process out:

• ResourceObservationUnavailableError when the runtime stopped, is stopping, its adapter host process exited, a live process in its tree or the process table cannot be read, or the process tree kept changing during the read. For example, on macOS, the snapshot is unavailable while a descendant runs as another user, including setuid programs such as ps, top, and sudo.
• ResourceObservationUnsupportedError when the runtime’s adapter kind, environment provider, or operating system has no resource reader. The http and native_plugin adapter kinds and Windows raise this error.

On Linux with /proc mounted hidepid=invisible or hidepid=ptraceable, the snapshot is unavailable while /proc hides a descendant, such as one that runs as another user. On a kernel without children lists, every snapshot is unavailable under those mount options, because hidden processes are missing from the process table that the snapshot reads to find the tree.

By default, each RunResult from a local runtime carries system_stats with before/after observations and a monotonic duration for the adapter exchange. Both observations run under the invocation lock, so queued invocations cannot contaminate the baseline. Runtime startup, endpoint collection, artifact preparation, and teardown are excluded from elapsed_seconds. Optional periodic sampling runs concurrently and can affect execution; the elapsed measurement does not subtract that effect. The existing resource_snapshot field retains its independent final process-tree reading even when invocation stats or the process group is disabled. When the process group is selected, it reuses system_stats.after.snapshot, or is None if unavailable.

Invocation System Stats

For a hands-on walkthrough with configuration examples and annotated results, see Resource Usage and Stats.

Identical configs can produce different amounts of agent work and encounter different system conditions. Stats should help separate those two.

Omitting system_stats selects every supported metric group at the invocation endpoints. Explicit None disables invocation stats, including before/after collection and stats artifacts. The SDK preserves that explicit null across planning and task-runner serialization.

from nemo_fabric import SystemStatsConfig, SystemStatsSamplingConfig
config.system_stats = None  # Disable invocation stats.
config.system_stats = SystemStatsConfig()  # All groups, endpoints only.
config.system_stats = SystemStatsConfig(
    collect=["process", "cgroup.cpu"],
    sampling=SystemStatsSamplingConfig(interval_ms=1000, max_samples=10_000),
)
config.system_stats.sampling = None  # Endpoints only for subsequent runs.

Omit collect for all supported groups. Empty lists, duplicates, and unknown names are rejected; use system_stats=None to disable stats. The invocation stats collector skips excluded groups in both endpoints and periodic samples. collected_groups records the selected groups, including those unavailable on the current platform. A not_collected status identifies excluded process observations and cgroup measurements; it differs from unavailable and unsupported, and never substitutes zero. Unselected process gauge reductions and an entirely excluded cgroup block are omitted. Identity, duration, scope, and sampling-quality metadata remain available. Metric selection does not change the separate resource_snapshot contract.

result = await runtime.invoke(input="Fix the failing test")
stats = result.system_stats
if stats is not None:
    print(stats.elapsed_seconds, stats.cpu_time_seconds_estimate)
    if stats.after.status == "available":
        print(stats.after.snapshot.resident_memory_bytes)
    elif stats.after.status in {"unavailable", "unsupported"}:
        print(stats.after.status, stats.after.reason)
    else:
        print(stats.after.status)

For example, a serialized RunResult contains this fragment. The IDs already associate the measurements with one invocation; no additional identity file is required.

{
  "runtime_id": "runtime-1",
  "invocation_id": "invocation-2",
  "request_id": "trial-42",
  "system_stats": {
    "elapsed_seconds": 8.0,
    "before": {
      "status": "available",
      "snapshot": {
        "runtime_id": "runtime-1",
        "started_at": "2026-10-21T09:00:00.000Z",
        "observed_at": "2026-10-21T09:01:00.000Z",
        "resident_memory_bytes": 52428800,
        "process_count": 1,
        "cpu_time_seconds": 12.0
      }
    },
    "after": {
      "status": "available",
      "snapshot": {
        "runtime_id": "runtime-1",
        "started_at": "2026-10-21T09:00:00.000Z",
        "observed_at": "2026-10-21T09:01:08.000Z",
        "resident_memory_bytes": 83886080,
        "process_count": 1,
        "cpu_time_seconds": 14.5
      }
    },
    "cpu_time_seconds_estimate": 2.5
  }
}

The CPU estimate is 2.5 seconds for this invocation, excluding the warm runtime’s previous 12 seconds. It is omitted when either reading is missing, runtime identities differ, or the counter decreases. Process reaping and reparenting can distort even a positive difference; this is an estimate, not exact accounting. Background work in the same process tree also contributes; the difference does not isolate agent-only CPU. Endpoint memory and process counts are instantaneous gauges, not peaks, averages, or invocation allocations.

For Harbor, these measurements come from the adapter host inside the task runner, even when a separate host selects the task and capabilities. Harbor’s existing result JSON carries the new field without a new stats file. For a model team comparing identical configs, pair usage with these measurements: more tokens or tool calls can explain more agent work; similar work with longer elapsed time warrants investigating execution conditions. Endpoint observations alone miss intermediate peaks and cannot explain every source of latency. Enable sampled stats and inspect native cgroup evidence below; remote inference latency requires serving-side evidence.

A failed read appears as {"status": "unavailable", "reason": "..."}; an unsupported provider or operating system uses "unsupported". Collection failures preserve the adapter outcome, including a normalized failed result. Disabled collection, older results, and execution backends without collection have system_stats=None. Protocol failures and timeouts that raise without a RunResult still raise. When sampling is enabled, NeMo Fabric retains sample and summary artifacts where possible even if the adapter exchange fails. These measurements require neither NeMo Relay collection nor invocation recording.

Sampled Peaks and Means

A process can allocate 400 MiB during execution and return to 80 MiB before the final reading. Endpoints miss that spike. Enable native sampling independently of Relay or invocation recording:

from nemo_fabric import SystemStatsConfig, SystemStatsSamplingConfig
config.system_stats = SystemStatsConfig(
    sampling=SystemStatsSamplingConfig(interval_ms=1000, max_samples=10_000),
)
config.runtime.artifacts = "artifacts"
result = await Fabric().run(config, input="Fix the failing test")
stats = result.system_stats
sampled = None if stats is None else stats.sampled
if sampled is not None:
    rss = sampled.resident_memory_bytes
    if rss is not None:
        print(rss.sampled_peak, rss.time_weighted_mean, rss.covered_fraction)
    if sampled.samples is not None:
        print(sampled.samples.path)

The mean holds the preceding reading across adjacent valid observations and excludes missing readings or gaps larger than twice cadence. Retained JSONL offsets allow benchmark tooling to correlate samples with trajectory events and select its own analysis windows. Temporal overlap with a span does not prove exclusive attribution.

For readings 80 MiB → 400 MiB → 80 MiB at 0, 1, and 2 seconds, the final value is 80 MiB, sampled peak is 400 MiB, range is 80–400 MiB, and hold-last-value mean is 240 MiB. A sampled maximum can miss brief spikes between observations. It is not a true high-water mark.

Sampling uses one process-wide scheduler, retains both endpoints, and bounds intermediate retention with max_samples. The result exposes retained/dropped sample counts, collection time, largest gap, truncation, per-gauge covered time, diagnostics, and artifact references. Late samples are discarded after invocation closes. A full series preserves the final endpoint rather than silently reporting an old value as final. Counter resets, changed targets, missing observations, and long gaps invalidate affected sampled CPU estimates. Sampling does not replace endpoint-derived CPU or cgroup deltas; collection quality and continuity are reported under sampled.

With an artifact root, the layout is:

artifacts/.fabric/<runtime_id>/<invocation_id>/
  adapter-invocation.json
  system-stats.jsonl
  system-stats-summary.json
  stdout.txt

Every sample and saved summary carries the existing runtime, invocation, and request IDs. RunResult.artifacts references both stats files. Without an explicit artifact root, these files use the runtime’s temporary .fabric directory. Optional stats persistence failures produce diagnostics and preserve the agent outcome. A known series write failure suppresses its artifact reference while retaining in-memory reductions and a sample_write_failed diagnostic. Samples and a summary are retained even if the adapter protocol raises without a normalized result; they do not invent a terminal agent status or a second lifecycle journal. Consumers should handle an interrupted last JSONL line explicitly.

Native Cgroup Constraints

On Linux cgroup v2, result.system_stats.cgroup contains before/after observations for the adapter host’s cgroup, plus CPU-use, throttled-time, and OOM-kill deltas. Snapshots expose visible CPU/memory limits, container-accounted current memory, swap, provider memory high-water, and CPU/memory/I/O pressure totals. Missing limit sources report unsupported. Visible numeric limits take precedence over max; unlimited requires an explicit max source with no numeric bound. Individual missing measurements remain explicit; unsupported platforms do not substitute host-wide readings.

This cgroup can contain the NeMo Fabric runner, the adapter tree, and other workloads. Its totals overlap process-tree totals and must not be added to them. Limits account for visible ancestors only; namespaces can hide additional restrictions. memory_peak_bytes covers cgroup lifetime or an external reset, not one invocation. The collector does not reset shared counters or write to cgroup control files.

Pair these observations with usage and reward: more tokens/tool calls can explain more agent work; similar work with throttled CPU and a lower visible limit suggests a constrained execution environment. These signals support investigation and do not establish causality. Remote inference-server GPU usage still requires serving-side telemetry.

The native stats example produces a transient memory spike and retained artifacts without model credentials.

Snapshots observe resources only. They do not enforce limits, stop runtimes, or queue requests. Use the operating system or execution environment, such as cgroups or container limits, to enforce resource limits.

Native OpenAI Streaming

Use Runtime.invoke_openai_stream() when the selected adapter provides native OpenAI Chat Completions chunks. Runtime.supports_openai_streaming reflects the selected descriptor’s existing capabilities.streaming claim. If the claim is false, invoke_openai_stream() raises FabricCapabilityError with code openai_streaming_unavailable.

from nemo_fabric import Fabric
async with await Fabric().start_runtime(config) as runtime:
    if not runtime.supports_openai_streaming:
        raise RuntimeError("the selected adapter does not support OpenAI streaming")
    stream = runtime.invoke_openai_stream(input="Review the latest patch")
    async for chunk in stream:
        print(chunk)
    result = await stream.result()
    print(result.status, result.output)

invoke_openai_stream(...) is synchronous and starts exactly one adapter invocation in the background. Its OpenAIInvokeStream yields only JSON mappings whose object field is chat.completion.chunk. An empty stream is valid. The terminal normalized RunResult is not a final chunk; obtain it separately with await stream.result() and treat it as authoritative.

Only one turn can be active on a runtime. Fully consume the stream before starting another turn. If iteration stops early, call await stream.aclose(); it drains the invocation and discards unread chunks so the runtime can be reused. It does not cancel the target invocation. Awaiting stream.result() also drains and discards any unread chunks before it returns, so consume the iterator first when the application needs every chunk.

The SDK owns an authenticated listener on the loopback interface and supplies the adapter host with a single-invocation HTTP sink that uses chunked NDJSON framing. It validates chunk shape, framing, authentication, and invocation correlation. This transport is an implementation detail; consumers provide no listener address, token, or protocol configuration.

Native OpenAI streaming does not require NeMo Relay or start_runtime(..., streaming=True). It is independent of Relay ATOF streaming: supports_openai_streaming and invoke_openai_stream() describe adapter-native OpenAI chunks, while supports_streaming and invoke_stream() describe Relay ATOF. Enabling one does not enable or change the other.

NeMo Relay Streaming

Enable NeMo Relay before starting a runtime to consume raw ATOF records while a turn runs. Runtime.supports_streaming reports whether this NeMo Relay path is available. It is separate from RuntimeCapabilities.streaming, which describes adapter-native progressive output. This separation is intentional: Runtime.invoke_stream() exposes only ATOF records generated by NeMo Relay; it does not call the adapter’s native OpenAI stream operation.

Install the matching collector for the default embedded streaming path:

pip install "nemo-fabric[streaming]"

This extra is not required when launch_collector=False selects an externally managed collector.

The following example streams one invocation and collects its terminal result:

import asyncio
from nemo_fabric import (
    EnvironmentConfig,
    Fabric,
    FabricConfig,
    HarnessConfig,
    MetadataConfig,
    ModelConfig,
)
async def main() -> None:
    config = FabricConfig(
        metadata=MetadataConfig(name="streaming-agent"),
        harness=HarnessConfig(adapter_id="nvidia.fabric.codex"),
        models={
            "default": ModelConfig(provider="openai", model="openai/gpt-5.4")
        },
        environment=EnvironmentConfig(provider="local", workspace="."),
    ).enable_relay()
    async with await Fabric().start_runtime(config, streaming=True) as runtime:
        stream = runtime.invoke_stream(input="Review the latest patch")
        async for record in stream:
            print(record)
        result = await stream.result()
        print(result.status, result.output)
asyncio.run(main())

invoke_stream(...) is synchronous and starts the invocation in the background. The returned InvokeStream is an async iterator of raw NeMo Relay ATOF dictionaries. The terminal RunResult stays out of band and is available only through await stream.result(). Treat that terminal result as authoritative: replace any provisional rendering instead of appending it. Calling stream.result() does not finalize unread stream records. Calling Runtime.stop() after the result completes finalizes the stream and discards those unread records.

Streaming has the following v0.1 constraints:

• Streaming requires two explicit settings: enable NeMo Relay in FabricConfig and pass streaming=True to start_runtime(...). The streaming=True flag does not enable NeMo Relay by itself; without NeMo Relay telemetry, start_runtime(...) raises FabricConfigError. By default, streaming starts an embedded collector and configures Relay to publish NDJSON to it. Set launch_collector=False to use an externally managed collector. In that case, configure exactly one nemo-fabric-stream sink with an http or https collector base URL and transport="ndjson"; Fabric sends Relay to <base-url>/v1/atof. Pi is the exception as Pi streaming supports only the default embedded collector. Do not set launch_collector=False for Pi because its extension does not attach the NeMo Fabric request IDs required for external correlation. The embedded collector waits one second for a late Pi agent_settled marker by default. Increase completion_wait_timeout when Relay delivery can take longer.
• The default streaming=False leaves the existing NeMo Relay configuration unchanged; it neither enables nor disables ATOF. It also avoids collector startup and streaming control traffic for runtimes that use only ATIF, OpenTelemetry, OpenInference, or application-configured ATOF sinks.
• One invocation can be active on a runtime. Fully consume the stream or call await stream.aclose() before starting another turn.
• Breaking an async for loop does not finalize the stream. Call await stream.aclose() explicitly. It waits for the invocation to finish and discards unread records; it does not cancel the harness turn.
• The end of async iteration only means that no more ATOF records are available. It does not indicate invocation success. Always await stream.result(). Invocation exceptions raise from that call, while harness-reported failures are represented by the returned result’s status and error fields.
• The collector limits each request queue to 1,024 records and 16 MiB of encoded record data. It rejects individual records larger than 1 MiB and applies TCP backpressure for up to 30 seconds. If the queue remains full, the collector drops the record and logs a warning. NeMo Relay can drop records if a consumer stalls longer than its delivery timeout.
• ATOF granularity depends on the NeMo Relay integration. Gateway harnesses expose per-delta event structure, but current ATOF records keep token text in the terminal scope. In-process harnesses expose scope-level progress.
• Reconstruct nested and parallel work with uuid and parent_uuid. Stream order alone does not define the Deep Agents scope tree.
• For request-correlated adapters, the runtime registers its request ID with the collector before invoking the agent. The collector routes the ATOF root scope carrying that request ID and its descendants, based on UUID ancestry, to that invocation’s private queue. With relay_streaming: true, Remote Agent services receive the ID as metadata.nemo_fabric_request_id in the mapped request body and must carry it into Relay metadata. Records that cannot be correlated to a registered request are discarded. Pi instead serializes both streaming and plain invocations behind one embedded-collector lease. Streaming capture begins at the first turn_start and closes at agent_settled. If output is interrupted or late, the collector discards the remainder through that same terminal marker before allowing the next invocation to start.
• An external collector can be shared by multiple NeMo Fabric runtimes whose adapters propagate request IDs. It must be reachable by both Relay publishers and NeMo Fabric clients. Use HTTPS and its configured authentication when it is deployed beyond a trusted local boundary.

Claude and Codex streaming use the NeMo Relay nemo-relay gateway CLI and require a release in the >=0.9,<0.12 range. Their adapter harness and full extras install the supported CLI. Deep Agents uses its in-process NeMo Relay integration with Relay 0.11. Hermes Agent keeps Relay 0.10 with the pinned Hermes source in a separate Python 3.14 environment. Remote Agent services own their Relay installation and publish to the same configured nemo-fabric-stream collector base URL. The local adapter does not install or start Relay for the service.

Application-Owned Parallelism

Applications create independent runtimes when they want parallel work. NeMo Fabric does not own a queue, worker pool, semaphore, retry policy, timeout policy, or numeric concurrency limit. Each Runtime accepts one invocation at a time so its ordered harness state cannot be changed by two calls at once. If an application overlaps calls on the same Runtime, the second call raises FabricStateError. To perform work in parallel, start independent runtimes; the application decides how many to run.

Async lifecycle calls run blocking native work outside the Python event loop, so independent runtimes can make progress concurrently. This does not add a NeMo Fabric concurrency limit or scheduler.

import asyncio
async def review_one(prompt: str):
    return await Fabric().run(config, input=prompt)
async def main() -> None:
    results = await asyncio.gather(
        review_one("Review patch A"),
        review_one("Review patch B"),
    )
    print([result.status for result in results])
asyncio.run(main())

For example, each Harbor job starts an independent NeMo Fabric runtime. Harbor owns job IDs and concurrency policy; NeMo Fabric does not use a job ID to select or resume runtime state.

Unified Run Results

Every invocation that reaches the adapter boundary returns a normalized RunResult, even when the harness invocation itself failed. Inspect status, error, events, and artifacts first, then process output when the status is successful.

Important fields:

These are structured correlation fields, not interchangeable metadata: runtime_id identifies the runtime lifecycle, invocation_id identifies one invocation within that runtime, and request_id correlates the caller’s request. NeMo Fabric-generated values use type-specific prefixes such as runtime-, invocation-, and request-; callers may provide their own request_id. Consumers should store and log each field separately and otherwise treat its value as opaque rather than parsing the identifier encoding.

Usage is absent when the target does not report it. Read the typed values only when usage is available:

if result.usage is not None:
    print(result.usage.get("input_tokens"))
    print(result.usage.get("cached_input_tokens"))
    print(result.usage.get("input_tokens_include_cache"))
    print(result.usage.get("output_tokens"))
    print(result.usage.get("total_tokens"))
    print(result.usage.get("cost_usd"))

Each counter is optional: missing values are unknown, not zero. cached_input_tokens reports cached prompt tokens. input_tokens_include_cache is True when those tokens are already included in input_tokens, False when they are excluded, and absent when the adapter cannot establish the semantics. Consumers must not add cached tokens to an inclusive input count. Adapters can report cache-write, reasoning, peak request, and per-model detail through the descriptor-declared usage.extensions["nemo_fabric.detailed_token_usage.v1"] object. NeMo Fabric preserves that object at RunUsage.metadata["nemo_fabric.detailed_token_usage.v1"]. Consumers treat detailed input and output independently: any detailed accounting field for a dimension makes that dimension authoritative, with no fallback to its top-level counters. The views can diverge, and an inclusive total remains unknown when the selected detailed counters do not provide enough information to calculate it. cost_usd contains reported cost only; keep estimates separately in provider metadata rather than presenting them as measured charges.

If NeMo Fabric cannot resolve config, start a runtime, or obtain a normalized result, the SDK raises a FabricError subclass instead of returning a partial RunResult.

Events

Each RunResult includes the normalized events collected for that invocation. Event kinds and detail may vary by adapter, but their lifecycle and correlation fields use the common NeMo Fabric contract.

Events are useful for:

• rendering invocation history in application or service UIs;
• forwarding logs and status to evaluation harnesses;
• correlating runtime, invocation, adapter, and telemetry IDs;
• reporting structured failures alongside the terminal result.

Feature Support Across Harness Adapters

The SDK presents one consistent shape across adapters, but adapters differ in their runtime requirements, accepted configuration, and optional capabilities.

Use plan(...) and doctor(...) before relying on optional features:

fabric = Fabric()
plan = fabric.plan(config)
report = await fabric.doctor(config)
print(plan.adapter.adapter_id)
print(report.status)

Use the plan to confirm adapter selection and capability routing. Use the doctor report to catch unresolved adapter descriptors, unsupported normalized capabilities, missing declared requirements, and environment problems before starting a runtime.

Capability routes assign execution ownership; they do not describe network routing. harness_native means the selected adapter executes the capability through its harness. fabric_managed means NeMo Fabric executes it outside the harness-native surface. unsupported means neither side can execute it. Routes apply to tools, skills, and MCP servers. Scalar fields such as instructions.system and runtime.max_turns are validated separately against the adapter descriptor.

Install And Runtime Responsibilities

In production, the consumer or execution environment is responsible for installing NeMo Fabric, the selected harness, adapter dependencies, model access, credentials, and any required native tools. NeMo Fabric validates and diagnoses the runtime assumptions, but it does not silently install harnesses or credentials at invocation time.

Development environments may use extras, virtual environments, or local source checkouts to make iteration easy. Production environments should prefer explicit images, preinstalled dependencies, or managed deployment packages.

Runtime compatibility checks should validate:

• NeMo Fabric SDK and native extension versions;
• selected adapter version;
• selected harness version or version range;
• required environment variables or secret references;
• optional capability support such as NeMo Relay, MCP, or tool exposure.

Custom Fields And Adapter Settings

Use normalized NeMo Fabric fields for portable behavior: models, system instructions, turn limit, runtime, environment, skills, MCP, telemetry, tools, and artifacts.

Supply request context through RunRequest.context for each invocation. Request context is not part of FabricConfig.

Use harness.settings only for stable, harness-native behavior declared by the selected descriptor. Each bundled adapter declares a closed settings schema. Executable paths, Relay command discovery, state directories, and similar launch mechanics are runtime implementation details rather than public adapter settings.

Use the optional workflow block to select a registered target by target_id and provide its immutable construction settings. The Adapter Target Descriptor selects the adapter, supplies the target-native entry point, and validates workflow.settings during planning. Fabric then projects the entry point and settings into AgentConfig.workflow; consumer configuration does not repeat adapter-specific entry-point semantics. Validation does not apply schema defaults.

Use FabricConfig.metadata for human-readable agent identity and caller-owned annotations. NeMo Fabric preserves these values in the resolved configuration but does not copy them into RunResult.metadata. Use request_id and RunRequest.context for per-invocation correlation. RunResult.metadata contains adapter-specific result details.

To group several invocations into one NeMo Relay session, set RunRequest.relay_session_root to the same UUID string on each invocation. Deep Agents and mini-SWE-agent, which own an in-process Relay Agent scope, use it as the propagated Relay root. A UUID request_id stays the parent; otherwise the session root is also the parent. ATIF exports one trajectory per invocation under one session_id. An unusable value falls back to the per-request root without an error, and the invocation still succeeds. A context key named session_id does not group traces.

For example, invoke each turn with RunRequest(input=message, relay_session_root=conversation_uuid). Core forwards the typed field as AgentRunRequest.relay_session_root. Context keys do not control Relay grouping. Omitting the field preserves per-request behavior. Use matching runtime and adapter-contract versions when sending this field.

Adapter settings are not portable by default. The exact descriptor selected during planning owns their JSON Schema. Planning and doctor reject settings that do not satisfy that schema. A descriptor without settings_schema accepts an empty settings map and rejects a non-empty map. Validation does not apply schema defaults or otherwise change the settings preserved in RunPlan. Validation errors identify the adapter, descriptor source and path, and offending settings path.

Errors

All public SDK errors inherit from FabricError.

Consumers own job-level retries and rollout-level failure policy. Core host-operation deadlines (host_timeout) become Python TimeoutError, then FabricRuntimeError with code == "timeout" and the failed lifecycle stage. If invocation succeeds in Fabric.run() but stop reaches this deadline, it returns a failed RunResult with error.stage == "stop" and error.code == "timeout". Check the returned result as well as raised errors. Adapters can report execution deadlines in normalized results with result.error.code == "timeout". Other adapter-specific timeout codes are not automatically reclassified. Diagnostic text is not a timeout classifier. Single-invocation runs attempt to stop the runtime before returning. A Runtime used with async with also attempts cleanup after an invocation error; if cleanup fails, that failure is attached to the original exception rather than replacing it. NeMo Fabric records structured error metadata when possible and returns enough detail for the consumer to decide what to do next.

SDK Contract Boundaries

NeMo Fabric keeps a narrow execution contract. Applications own product behavior around that contract.

Schemas And Python Models

The SDK’s Pydantic models are maintained against the Rust-generated public schemas. The schemas describe the typed cross-language contracts. Use the generated API reference for exact Python signatures.

Versioning

NeMo Fabric uses explicit contract versions where persisted or independently maintained artifacts cross package boundaries:

• schema_version identifies the typed NeMo Fabric config contract transported across SDK and language boundaries.
• contract_version identifies the contract implemented by *.fabric-adapter.json and *.fabric-target.json records.
• Python and Rust package versions identify the installed SDK/core release.

NeMo Fabric versions top-level typed contracts and independently maintained adapter metadata. It does not version each config subsection independently. For example, MCP, skills, models, telemetry, and runtime fields evolve under the enclosing schema_version.

NeMo Fabric validates adapter descriptor contract versions during planning. Package semver identifies the installed implementation, but it is not the compatibility contract for independently maintained adapters.

Config Extensibility

The public schema has typed fields for stable NeMo Fabric concepts and controlled extension points for adapter- or application-owned data. Use known fields for portable concepts such as harness selection, models, runtime, skills, MCP, telemetry, and artifacts. Use adapter-owned harness.settings for settings validated by the selected descriptor and interpreted by its adapter. Use metadata or preserved extension fields for data NeMo Fabric should carry but not interpret.

Additive optional fields may be introduced within the existing document schema version when old configs remain valid. Required fields, removed fields, or semantic changes that alter how existing configs are interpreted require a new enclosing document schema version or an explicit compatibility path.

Unknown data is not the same as supported behavior. An adapter must advertise and implement a capability before NeMo Fabric treats it as runnable.

Resilience

NeMo Fabric reports lifecycle failures; applications own recovery policy. If a runtime process dies, a connection is permanently lost, or an adapter cannot complete an invocation, NeMo Fabric marks the relevant runtime/invocation as failed and returns structured error metadata when possible.

Transient I/O failures may be marked with retryable error metadata, but NeMo Fabric does not perform job-level retries by default. Consumers decide whether to retry the request, start a replacement runtime, fail the job, or escalate to a user.

Capacity And Backpressure

If a harness reports capacity pressure, an adapter should surface it as a structured error or event such as busy, rate limited, capacity exceeded, or backpressure. The consumer decides whether to wait, retry, scale out, or fail.

Next Steps

• Review the examples in the GitHub repo examples/.

Selected Software Evidence

RunResult.execution optionally contains selected software inventories and existing infrastructure software reports. Each inventory has an opaque result-local context_ref, participants, availability, coverage, and component entries. Component roles belong in entries[].role. Equal versions do not establish a shared installation. Version observations retain the full value, observer/method in source, origin, and original UTC observed_at; missing values carry a status and reason. An unavailable version can still be represented in a complete selected inventory.

FabricConfig.provenance defaults to ProvenanceConfig(enabled=True). Set provenance=ProvenanceConfig(enabled=False) (JSON: "provenance": {"enabled": false}) before runtime initialization to skip collector work on both sides of dispatch. provenance.reports accepts a typed nemo_fabric_evidence.evidence.ExecutionEvidence snapshot from the caller or environment owner. Supply it in the existing configuration dispatch; the guest collects its own snapshot. No callback or additional network service is involved. Reports are bounded to 16 contexts, 128 total entries and 64 KiB of compact UTF-8 JSON. Over-budget or malformed optional reports become explicit incomplete evidence without changing the invocation outcome.

nemo-fabric-runtime and nemo-fabric-adapter-contract both use the version-matched nemo-fabric-evidence package for shared software evidence and validation. The shared package uses only the standard library and owns the bounded metadata primitives used by both layers. Package selection and snapshot caching remain in the SDK and adapters. The SDK does not depend on the southbound adapter contract. Existing adapter-contract evidence imports remain compatible re-exports; the nemo_fabric SDK facade is unchanged.

Old saved results may omit execution. A receiving boundary expecting guest/adapter evidence records missing reports explicitly. Compare participants, component kind/name, full versions, topology and coverage separately; exclude capture times, random context/invocation IDs and artifact roots. Unknown values do not establish equality. This API provides observations, not a dependency inventory, reproducibility guarantee or attestation. Inputs/configuration capture is a follow-up.

Snapshot Lifetime And Collection Bounds

The Python SDK captures its selected distribution and executing interpreter when a Fabric instance first initializes an enabled runtime, service, or environment. It reuses that snapshot for that SDK instance. Construct a new Fabric after a known installation change or SDK process restart. Each adapter host captures its own selected packages after startup; replacing/restarting the host creates a fresh snapshot. The compiled core reports its executing version independently. Inventories stay separate when no shared resolution context is established; equal versions or interpreter paths are insufficient evidence for joining installations.

Python package selection is adapter-owned. Loaded-module attribution checks prevent installed metadata from identifying a shadowing source tree. Missing metadata or an unattributed editable checkout produces an explicit unavailable version. Collector work uses at most 32 selected package lookups and 4,096 candidate metadata files per target, with at most 64 KiB of metadata headers and 512 KiB of RECORD bytes per target, a 250 ms initialization deadline and at most one outstanding daemon worker per process. Unsupported metadata backends remain explicit unknowns. TypeScript uses asynchronous selected manifest reads through at most eight node_modules ancestor levels, a 64 KiB manifest read limit, 32 selected package targets and a 250 ms deadline. Shared host and contract metadata use the executing common host’s resolution context; selected harness packages use the adapter’s context. Report limits apply after composition too: entries are dropped whole with incomplete coverage rather than shortening version values. Lookup failures preserve invocation output, errors and status. The escape hatch skips the collectors before lookup work begins.

Claude initialization messages and Relay’s existing startup version check can supply passive observations. Receiving and forwarding these reports preserves their source, origin and first capture time. Installed packages are included for telemetry only when the selected component enables that integration; package presence alone does not establish use.

Dispatch And Shared Services

OpenShell forwards the caller snapshot in FabricConfig.provenance.reports through the existing task start command. The guest SDK and adapter initialize their own snapshots. The host retains its copy independently, including when an older intermediary drops execution; that case produces a guest_sdk coverage gap while preserving the invocation outcome. Context refs are rebased when reports are composed. The host reuses its own forwarded prefix only when it is an exact copy of its request, including original capture times; matching versions never merge installations.

Existing OpenShell gateway health responses supply provider versions and capture times. Older provider reports without capture times are observed once at host runtime initialization with source host:provider_report_at_runtime_start. Provider reports reference guest contexts. NeMo Fabric does not infer a container engine. Explicit container-owner reports can use provenance.reports with infrastructure.container.engine_version, which means the engine server version.

OpenClaw service startup reports retain their original source and capture time when multiple runtimes connect to the same service. A separate service inventory contains the reported OpenClaw version without inventing the remote interpreter or package inventory. Existing startup checks can refresh a changed version; a newly prepared or attached service creates a new snapshot. Software observations and their timestamps do not participate in runtime/service binding hashes. Runtime bindings retain the collection control chosen at initialization; service bindings allow a connecting runtime to choose its own collection control. A runtime can disable collection while connecting to a previously initialized service.

Calling Application Reports

Calling applications and environment owners supply typed ProvenanceConfig.reports snapshots through the configuration already sent to NeMo Fabric. Capture them at application initialization and preserve their original source, origin, availability and observed_at. A guest never inspects its caller. Harbor implements this seam with fabric_provenance, whose shape matches the SDK control; --ak 'fabric_provenance={"enabled":false}' disables collection before host initialization and propagates to the guest.

The software evidence example provides six execution-path checks, separate-venv runs, saved RunResults and private comparison fixtures. Its local provider/service fixtures are labeled separately from live deployments. Safe configuration capture is the P0 follow-up and is outside this software-version contract.