Skip to navigation

Requests and Runtimes

View as MarkdownOpen in Claude

For examples that reuse config, start with Start With One Run.

Single-Invocation Runs

Use run(...) when the application has one input and does not need to preserve runtime state after the result is collected.

from nemo_fabric import Fabric, RunRequest
request = RunRequest(
input="Review the workspace changes.",
request_id="request-123",
context={"source": "review-service"},
)
fabric = Fabric()
result = await fabric.run(
config,
base_dir="/workspace/review-agent",
request=request,
)
print(result.status)
print(result.output.response)
print(result.artifacts)

Use input=... for the common case. Use request=RunRequest(...) for structured invocation metadata. Applications read files themselves and pass either the loaded input or a validated request to NeMo Fabric.

NeMo Fabric generates runtime and invocation IDs for lifecycle correlation. An application may include its own identifiers in opaque request metadata, but NeMo Fabric does not interpret them as job, session, scheduling, or resume state.

Multi-Turn Runtimes

Use start_runtime(...) when the selected harness should keep state across turns. Every call creates a new logical NeMo Fabric runtime; callers reuse the returned object rather than selecting it with a job or session ID. The runtime stops when its async context exits.

from nemo_fabric import Fabric
fabric = Fabric()
async with await fabric.start_runtime(
config,
base_dir="/workspace/review-agent",
) as runtime:
first = await runtime.invoke(input="Inspect the repository")
second = await runtime.invoke(input="Now review the latest patch")
print(first.status, second.status)

The adapter reuses its native state between calls. Codex maps the calls to turns on one live thread, Deep Agents invokes one compiled graph and checkpointer, Hermes Agent reuses one agent and session database, and Claude keeps one connected SDK client. Durable Hermes conversations expose their native identifier inside an opaque SessionRef. NeMo Fabric does not interpret that identifier.

Buffered runtime shutdown can interrupt an active invocation on supported Unix and Windows hosts. Windows shutdown requires an owned process job to confirm descendant cleanup. Runtime.stop() waits for the interrupted invocation to settle before reporting stopped. Cleanup errors retain ownership for retry; the runtime server blocks replacement for the same agent until cleanup succeeds. Streaming shutdown continues to use the stream lifecycle. An invocation interrupted by a requested stop reports interrupted; genuine host crashes still report host_crashed. Cleanup confirms processes that remain in Fabric’s owned host tree. The sandbox supervisor must terminate processes that detached before Fabric captured that tree.

If Fabric.run(), Fabric.start_runtime(), Fabric.prepare_service(), or Fabric.attach_service() cannot confirm cleanup after a failed lifecycle operation, the raised error provides cleanup_runtime. Keep that object and retry await error.cleanup_runtime.stop() in the same process. Cancellation can carry the same recovery object. The recovery handle is excluded from error text and structured details. When an exited parent leaves undiscoverable children, cleanup remains unconfirmed and the owning process or sandbox must recover.

Retained hosts from a timed-out invocation or failed start accept cleanup only. New and queued invocations return host_unavailable after acquiring the host lock. Failed connected starts keep their service membership until cleanup succeeds, so the service cannot be released while that runtime remains owned.

Runtime health checks report pending shutdown as runtime_busy, with state unknown and activity stopping. After failed cleanup, checks report cleanup_unconfirmed, with state failed and activity unknown. These reports describe recorded lifecycle state and do not probe the harness during cleanup.

Keep Agent State Across Runtimes

By default, each runtime keeps its native state in its own folder, so a new runtime starts empty. Set "keep_state": true in the config’s runtime mapping to give every runtime of the agent one folder instead: <artifact root>/.fabric/state/<adapter id>/<agent name>. The config needs an artifact directory, and metadata.name and the adapter ID must each be usable as one folder name on Linux, macOS, and Windows: no /, \, <>:"|?*, or control characters, no trailing space or ., and not a Windows device name such as CON or NUL. On Linux and macOS, only the folder’s owner can read it. Only one runtime can use the folder at a time: Fabric.start_runtime(...) fails while another runtime of the agent runs. A service and one connected runtime share the folder. The service holds the lock until Service.release(), even with no runtime connected.

The operating system releases the lock after all copies of the lock have closed:

  • Linux and macOS: Each adapter host inherits a lock descriptor, including a runtime connected to a service. Stopping a standalone runtime normally releases the lock while the application continues running. Killing only the adapter host leaves the application’s lock descriptor open.
  • Windows: Adapter hosts do not inherit the lock. Only the NeMo Fabric process holds the lock, so the folder becomes available if that process exits while an adapter host is still running.
  • Detached processes: A native process can continue accessing the state folder after all lock descriptors close. A replacement runtime can then acquire the lock while the detached process still writes to the folder.

Planning fails without an artifact directory or for an adapter that doesn’t declare runtime.keep_state. The Hermes Agent and OpenClaw adapters declare runtime.keep_state. See the OpenClaw cold resume guide for native Gateway and service behavior.

Enabling runtime.keep_state preserves earlier Hermes sessions in the database. Omitting session_ref starts a fresh Hermes SDK conversation.

Resume a Hermes Conversation

Hermes SessionRef resume requires harness.settings.sdk.enabled to be true, which is the default. RunPlan.capabilities.session_resume reports the adapter’s declared capability. Native-only runtimes can retain state with runtime.keep_state and start without a session reference. Hermes rejects explicit SDK session selection in native-only mode with session_resume_unsupported before native startup.

Retention preserves state. Omitting session_ref starts a fresh conversation. Resume requires the exact SessionRef and access to the original retained storage. Hermes accepts session_ref through Fabric.run(), start_runtime(), and start_runtime_in():

from nemo_fabric import Fabric, SessionRef
fabric = Fabric()
# config.runtime.keep_state must be True and an artifact root must be supplied.
async with await fabric.start_runtime(config) as runtime:
reference_data = runtime.session_ref.to_mapping()
await runtime.invoke(input="Inspect the repository")
# Persist reference_data and the retained artifact root in caller-owned storage.
reference = SessionRef.from_mapping(reference_data)
async with await fabric.start_runtime(config, session_ref=reference) as resumed:
if not resumed.recovery_warnings:
await resumed.invoke(input="Continue the review")

The reference contains schema_version (fabric.session_ref/v1alpha1), adapter_id, and session_id. A resumed runtime has a new runtime ID and the same session reference. Results confirm the reference in RunResult.session_ref. An agent name is a label: correcting the name does not change the selected session. NeMo Fabric resolves the reference only within the supplied artifact root, using private relative storage bindings under .fabric/state/<adapter id>/.sessions. Move or retain the complete state tree, including these bindings. NeMo Fabric does not maintain a consumer identity registry. The caller controls storage access. NeMo Fabric supplies the current configuration and instructions when the runtime starts. Hermes can restore its saved system prompt and pinned tools according to native resume rules, so new instructions do not necessarily replace the conversation’s saved prompt.

Missing sessions, mismatched adapters, unsupported capabilities, incompatible state, or an unconfirmed native identity fail explicitly. NeMo Fabric never falls back to a fresh conversation after an explicit resume failure. Structured SDK errors expose codes such as session_not_found, session_adapter_mismatch, session_resume_unsupported, and state_in_use.

If a previous turn might be incomplete, Hermes can restore the native history and return Runtime.recovery_warnings with session_recovery_required. NeMo Fabric does not automatically submit another invocation or supply fabricated tool results. The restored history retains unfinished requests, and the model may act on those requests after acknowledgment. Acknowledgment allows continuation with that history. Inspect the native history and any external effects before sending a new request with RunRequest(input=..., acknowledge_recovery=True). Ordinary NeMo Fabric invocations remain blocked until acknowledgment. A durable completed response does not become uncertain merely because the caller lost the reply. A recorded tool failure followed by a completed response also does not require recovery.

Writer ownership remains held while cleanup is unconfirmed. A busy managed stop returns runtime_busy. Retry stop after the invocation ends. A failed or timed out stop does not establish that replacement is safe. Keep the runtime handle and retry cleanup. After a local startup fails without returning a handle, the next start for the same agent storage retries cleanup before starting a replacement. Connected starts and service release also retry known failed connection cleanup. A local one-shot run whose shutdown could not be confirmed also retains cleanup ownership. The next start for the same storage retries that cleanup before starting a replacement. Ownership remains held when cleanup fails.

These local cleanup checks apply to every adapter, including runtimes with runtime.keep_state: false. If the operating system prevents NeMo Fabric from confirming a surviving child’s identity, further starts for the same cleanup scope remain blocked with cleanup_unconfirmed. Retrying while that child remains unidentifiable cannot confirm cleanup. Use operating system controls to confirm that the old adapter and native child processes have stopped, then retry cleanup. If the application cannot recover its cleanup handle, restart the application that embeds NeMo Fabric after confirming those processes have stopped.

Hermes native interface children retain the state lock when their adapter host exits, so a surviving writer prevents replacement. These controls do not change how the native Hermes API or dashboard chooses conversations or accepts work.

Recorded Runtime Status

Use Fabric.runtime_status() to read the recorded lifecycle state and rediscover a runtime handle without starting, invoking, stopping, or probing the harness:

report = await fabric.runtime_status(
runtime_id=runtime.runtime_id,
agent_name=config.metadata.name,
)
print(report.status) # "active", "stopped", or "failed"

A direct local lookup selects a runtime started by the same Fabric instance. Each client holds at most 128 records across active, failed, stopped, and pending runtimes. New starts evict the oldest successfully stopped records first. When active, failed, and pending runtimes fill the limit, startup raises FabricStateError with code="runtime_limit_reached" before creating runtime resources. The client retains active and failed runtime objects until successful stop, even when the caller drops the original Python reference. Callers remain responsible for stopping runtimes. An optional local EnvironmentHandle restricts the lookup to that environment. Retained stopped records have handle=None; evicted records are unavailable.

For OpenShell, pass a verified environment handle and the runtime identifier returned by start. The SDK sends fabric-agent-ctl status to the original owning server, so a later client process can perform the same read:

report = await Fabric().runtime_status(
environment=environment,
runtime_id=runtime_id,
agent_name="researcher",
)

RuntimeStatusReport contains runtime_id, agent_name, status, handle, and server_state. The runtime status schema defines the JSON shape for SDK and command consumers. The server state is running, stopping, stopped, or unknown; direct local reads use None. Missing runtimes, wrong agents, and mismatched local environment identifiers raise FabricStateError. The SDK validates remote report identity and uses fixed diagnostics for transport failures.

The returned handle is a discovery snapshot. The snapshot retains runtime identity, the opaque binding, adapter and service identifiers, and environment identity and paths. The snapshot excludes environment variable values, provider connection metadata, provider metadata, and unknown handle extensions. Use the original Runtime for invocation and cleanup. OpenShell reports the handle owned inside the sandbox, rather than a new host-side runtime binding. Stopped runtimes have no current handle. Records are held by the owning process and do not survive server restart.

Recorded active state does not prove the harness is currently healthy. Use Fabric.check() for a fresh health observation. This status operation adds no native-interface endpoints or credential locations.

Application-Owned Parallelism

Applications create independent runtimes when they want parallel work. NeMo Fabric does not own a queue, worker pool, semaphore, retry policy, timeout policy, or numeric concurrency limit. Each Runtime accepts one invocation at a time so its ordered harness state cannot be changed by two calls at once. If an application overlaps calls on the same Runtime, the second call raises FabricStateError. To perform work in parallel, start independent runtimes; the application decides how many to run.

Async lifecycle calls run blocking native work outside the Python event loop, so independent runtimes can make progress concurrently. This does not add a NeMo Fabric concurrency limit or scheduler.

import asyncio
async def review_one(prompt: str):
return await Fabric().run(config, input=prompt)
async def main() -> None:
results = await asyncio.gather(
review_one("Review patch A"),
review_one("Review patch B"),
)
print([result.status for result in results])
asyncio.run(main())

For example, each Harbor job starts an independent NeMo Fabric runtime. Harbor owns job IDs and concurrency policy; NeMo Fabric does not use a job ID to select or resume runtime state.