> This page is for NeMo Fabric, version v0.7.0.
> For other versions, use one of these documentation indexes:
> - Latest (v0.7.0) (default): https://docs.nvidia.com/nemo/fabric/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/fabric/dev/llms.txt
> - v0.7.0: https://docs.nvidia.com/nemo/fabric/v0.7.0/llms.txt
> - v0.6.1: https://docs.nvidia.com/nemo/fabric/v0.6.1/llms.txt
> - v0.6.0: https://docs.nvidia.com/nemo/fabric/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/nemo/fabric/v0.5.0/llms.txt
> - v0.4.0: https://docs.nvidia.com/nemo/fabric/v0.4.0/llms.txt
> - v0.3.0: https://docs.nvidia.com/nemo/fabric/v0.3.0/llms.txt
> - v0.2.0: https://docs.nvidia.com/nemo/fabric/v0.2.0/llms.txt
> - v0.1.1: https://docs.nvidia.com/nemo/fabric/v0.1.1/llms.txt
> - v0.1.0: https://docs.nvidia.com/nemo/fabric/v0.1.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/fabric/llms.txt.

# Requests and Runtimes

> Run requests, retain and resume state, and inspect recorded runtime status.

For examples that reuse `config`, start with [Start With One Run](/nemo/fabric/sdk/python-sdk#start-with-one-run).

## Single-Invocation Runs

Use `run(...)` when the application has one input and does not need to preserve
runtime state after the result is collected.

```python
from nemo_fabric import Fabric, RunRequest

request = RunRequest(
    input="Review the workspace changes.",
    request_id="request-123",
    context={"source": "review-service"},
)

fabric = Fabric()
result = await fabric.run(
    config,
    base_dir="/workspace/review-agent",
    request=request,
)

print(result.status)
print(result.output.response)
print(result.artifacts)
```

Use `input=...` for the common case. Use `request=RunRequest(...)` for structured
invocation metadata. Applications read files themselves and pass either the
loaded input or a validated request to NeMo Fabric.

NeMo Fabric generates runtime and invocation IDs for lifecycle correlation. An
application may include its own identifiers in opaque request metadata, but
NeMo Fabric does not interpret them as job, session, scheduling, or resume state.

## Multi-Turn Runtimes

Use `start_runtime(...)` when the selected harness should keep state across
turns. Every call creates a new logical NeMo Fabric runtime; callers reuse the
returned object rather than selecting it with a job or session ID. The runtime
stops when its async context exits.

```python
from nemo_fabric import Fabric

fabric = Fabric()
async with await fabric.start_runtime(
    config,
    base_dir="/workspace/review-agent",
) as runtime:
    first = await runtime.invoke(input="Inspect the repository")
    second = await runtime.invoke(input="Now review the latest patch")

print(first.status, second.status)
```

The adapter reuses its native state between calls. Codex maps the calls to turns
on one live thread, Deep Agents invokes one compiled graph and checkpointer,
Hermes Agent reuses one agent and session database, and Claude keeps one connected SDK
client. Durable Hermes conversations expose their native identifier inside an
opaque `SessionRef`.  NeMo Fabric does not interpret that identifier.

Buffered runtime shutdown can interrupt an active invocation on supported Unix and Windows hosts. Windows shutdown requires an owned process job to confirm descendant cleanup. `Runtime.stop()` waits for the interrupted invocation to settle before reporting stopped. Cleanup errors retain ownership for retry; the runtime server blocks replacement for the same agent until cleanup succeeds. Streaming shutdown continues to use the stream lifecycle. An invocation interrupted by a requested stop reports `interrupted`; genuine host crashes still report `host_crashed`. Cleanup confirms processes that remain in Fabric’s owned host tree. The sandbox supervisor must terminate processes that detached before Fabric captured that tree.

If `Fabric.run()`, `Fabric.start_runtime()`, `Fabric.prepare_service()`, or `Fabric.attach_service()` cannot confirm cleanup after a failed lifecycle operation, the raised error provides `cleanup_runtime`. Keep that object and retry `await error.cleanup_runtime.stop()` in the same process. Cancellation can carry the same recovery object. The recovery handle is excluded from error text and structured details. When an exited parent leaves undiscoverable children, cleanup remains unconfirmed and the owning process or sandbox must recover.

Retained hosts from a timed-out invocation or failed start accept cleanup only. New and queued invocations return `host_unavailable` after acquiring the host lock. Failed connected starts keep their service membership until cleanup succeeds, so the service cannot be released while that runtime remains owned.

Runtime health checks report pending shutdown as `runtime_busy`, with state `unknown` and activity `stopping`. After failed cleanup, checks report `cleanup_unconfirmed`, with state `failed` and activity `unknown`. These reports describe recorded lifecycle state and do not probe the harness during cleanup.

### Keep Agent State Across Runtimes

By default, each runtime keeps its native state in its own folder, so a new
runtime starts empty. Set `"keep_state": true` in the config's `runtime`
mapping to give every runtime of the agent one folder instead:
`<artifact root>/.fabric/state/<adapter id>/<agent name>`. The config needs an
artifact directory, and `metadata.name` and the adapter ID must each be usable
as one folder name on Linux, macOS, and Windows: no `/`, `\`, `<>:"|?*`, or
control characters, no trailing space or `.`, and not a Windows device name such
as `CON` or `NUL`. On Linux and macOS, only the folder's owner can read it.
Only one runtime can use the folder at a time: `Fabric.start_runtime(...)` fails while another runtime of the agent runs. A service and one connected runtime share the folder. The service holds the lock until `Service.release()`, even with no runtime connected.

The operating system releases the lock after all copies of the lock have closed:

* **Linux and macOS:** Each adapter host inherits a lock descriptor, including a runtime connected to a service. Stopping a standalone runtime normally releases the lock while the application continues running. Killing only the adapter host leaves the application's lock descriptor open.
* **Windows:** Adapter hosts do not inherit the lock. Only the NeMo Fabric process holds the lock, so the folder becomes available if that process exits while an adapter host is still running.
* **Detached processes:** A native process can continue accessing the state folder after all lock descriptors close. A replacement runtime can then acquire the lock while the detached process still writes to the folder.

Planning fails without an artifact directory or for an adapter that doesn't declare `runtime.keep_state`. The Hermes Agent and OpenClaw adapters declare `runtime.keep_state`.
See the [OpenClaw cold resume guide](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.7.0/adapters/python/openclaw/README.md#cold-resume)
for native Gateway and service behavior.

Enabling `runtime.keep_state` preserves earlier Hermes sessions in the database. Omitting `session_ref` starts a fresh Hermes SDK conversation.

### Resume a Hermes Conversation

Hermes `SessionRef` resume requires `harness.settings.sdk.enabled` to be `true`, which is the default. `RunPlan.capabilities.session_resume` reports the adapter's declared capability. Native-only runtimes can retain state with `runtime.keep_state` and start without a session reference. Hermes rejects explicit SDK session selection in native-only mode with `session_resume_unsupported` before native startup.

Retention preserves state. Omitting `session_ref` starts a fresh conversation.
Resume requires the exact `SessionRef` and access to the original retained
storage. Hermes accepts `session_ref` through `Fabric.run()`, `start_runtime()`,
and `start_runtime_in()`:

```python
from nemo_fabric import Fabric, SessionRef

fabric = Fabric()
# config.runtime.keep_state must be True and an artifact root must be supplied.
async with await fabric.start_runtime(config) as runtime:
    reference_data = runtime.session_ref.to_mapping()
    await runtime.invoke(input="Inspect the repository")

# Persist reference_data and the retained artifact root in caller-owned storage.
reference = SessionRef.from_mapping(reference_data)
async with await fabric.start_runtime(config, session_ref=reference) as resumed:
    if not resumed.recovery_warnings:
        await resumed.invoke(input="Continue the review")
```

The reference contains `schema_version` (`fabric.session_ref/v1alpha1`),
`adapter_id`, and `session_id`. A resumed runtime has a new runtime ID and the
same session reference. Results confirm the reference in `RunResult.session_ref`.
An agent name is a label: correcting the name does not change the selected
session. NeMo Fabric resolves the reference only within the supplied artifact root,
using private relative storage bindings under `.fabric/state/<adapter id>/.sessions`.
Move or retain the complete state tree, including these bindings. NeMo Fabric does
not maintain a consumer identity registry. The caller controls storage access.
NeMo Fabric supplies the current configuration and instructions when the runtime
starts. Hermes can restore its saved system prompt and pinned tools according
to native resume rules, so new instructions do not necessarily replace the
conversation's saved prompt.

Missing sessions, mismatched adapters, unsupported capabilities, incompatible
state, or an unconfirmed native identity fail explicitly. NeMo Fabric never falls
back to a fresh conversation after an explicit resume failure. Structured SDK
errors expose codes such as `session_not_found`, `session_adapter_mismatch`,
`session_resume_unsupported`, and `state_in_use`.

If a previous turn might be incomplete, Hermes can restore the native history
and return `Runtime.recovery_warnings` with `session_recovery_required`. NeMo Fabric
does not automatically submit another invocation or supply fabricated tool results.
The restored history retains unfinished requests, and the model may act on those
requests after acknowledgment. Acknowledgment allows continuation with that history.
Inspect the native history and any external effects before sending a new request with
`RunRequest(input=..., acknowledge_recovery=True)`. Ordinary NeMo Fabric invocations
remain blocked until acknowledgment. A durable completed response does not
become uncertain merely because the caller lost the reply. A recorded tool
failure followed by a completed response also does not require recovery.

Writer ownership remains held while cleanup is unconfirmed. A busy managed
stop returns `runtime_busy`. Retry stop after the invocation ends. A failed or
timed out stop does not establish that replacement is safe. Keep the runtime
handle and retry cleanup. After a local startup fails without returning a
handle, the next start for the same agent storage retries cleanup before
starting a replacement. Connected starts and service release also retry known
failed connection cleanup. A local one-shot run whose shutdown could not be
confirmed also retains cleanup ownership. The next start for the same storage
retries that cleanup before starting a replacement. Ownership remains held when
cleanup fails.

These local cleanup checks apply to every adapter, including runtimes with
`runtime.keep_state: false`. If the operating system prevents NeMo Fabric from
confirming a surviving child's identity, further starts for the same cleanup
scope remain blocked with `cleanup_unconfirmed`. Retrying while that child
remains unidentifiable cannot confirm cleanup. Use operating system controls to
confirm that the old adapter and native child processes have stopped, then retry
cleanup. If the application cannot recover its cleanup handle, restart the
application that embeds NeMo Fabric after confirming those processes have stopped.

Hermes native interface children retain the state
lock when their adapter host exits, so a surviving writer prevents replacement.
These controls do not change how the native Hermes API or dashboard chooses
conversations or accepts work.

## Recorded Runtime Status

Use `Fabric.runtime_status()` to read the recorded lifecycle state and rediscover a runtime handle without starting, invoking, stopping, or probing the harness:

```python
report = await fabric.runtime_status(
    runtime_id=runtime.runtime_id,
    agent_name=config.metadata.name,
)
print(report.status)  # "active", "stopped", or "failed"
```

A direct local lookup selects a runtime started by the same `Fabric` instance. Each client holds at most 128 records across active, failed, stopped, and pending runtimes. New starts evict the oldest successfully stopped records first. When active, failed, and pending runtimes fill the limit, startup raises `FabricStateError` with `code="runtime_limit_reached"` before creating runtime resources. The client retains active and failed runtime objects until successful stop, even when the caller drops the original Python reference. Callers remain responsible for stopping runtimes. An optional local `EnvironmentHandle` restricts the lookup to that environment. Retained stopped records have `handle=None`; evicted records are unavailable.

For OpenShell, pass a verified environment handle and the runtime identifier returned by start. The SDK sends `fabric-agent-ctl status` to the original owning server, so a later client process can perform the same read:

```python
report = await Fabric().runtime_status(
    environment=environment,
    runtime_id=runtime_id,
    agent_name="researcher",
)
```

`RuntimeStatusReport` contains `runtime_id`, `agent_name`, `status`, `handle`, and `server_state`. The [runtime status schema](https://github.com/NVIDIA/NeMo-Fabric/blob/v0.7.0/schemas/sdk/runtime-status-report.schema.json) defines the JSON shape for SDK and command consumers. The server state is `running`, `stopping`, `stopped`, or `unknown`; direct local reads use `None`. Missing runtimes, wrong agents, and mismatched local environment identifiers raise `FabricStateError`. The SDK validates remote report identity and uses fixed diagnostics for transport failures.

The returned handle is a discovery snapshot. The snapshot retains runtime identity, the opaque binding, adapter and service identifiers, and environment identity and paths. The snapshot excludes environment variable values, provider connection metadata, provider metadata, and unknown handle extensions. Use the original `Runtime` for invocation and cleanup. OpenShell reports the handle owned inside the sandbox, rather than a new host-side runtime binding. Stopped runtimes have no current handle. Records are held by the owning process and do not survive server restart.

Recorded `active` state does not prove the harness is currently healthy. Use `Fabric.check()` for a fresh health observation. This status operation adds no native-interface endpoints or credential locations.

## Application-Owned Parallelism

Applications create independent runtimes when they want parallel work. NeMo Fabric
does not own a queue, worker pool, semaphore, retry policy, timeout policy, or
numeric concurrency limit. Each `Runtime` accepts one invocation at a time so
its ordered harness state cannot be changed by two calls at once. If an
application overlaps calls on the same `Runtime`, the second call raises
`FabricStateError`. To perform work in parallel, start independent runtimes;
the application decides how many to run.

Async lifecycle calls run blocking native work outside the Python event loop,
so independent runtimes can make progress concurrently. This does not add a
NeMo Fabric concurrency limit or scheduler.

```python
import asyncio

async def review_one(prompt: str):
    return await Fabric().run(config, input=prompt)

async def main() -> None:
    results = await asyncio.gather(
        review_one("Review patch A"),
        review_one("Review patch B"),
    )
    print([result.status for result in results])

asyncio.run(main())
```

For example, each Harbor job starts an independent NeMo Fabric runtime. Harbor owns
job IDs and concurrency policy; NeMo Fabric does not use a job ID to select or
resume runtime state.