Health Checks
Use Fabric.check() to check the health of a runtime server, an agent runtime, or an OpenClaw Gateway service. NVIDIA NeMo Fabric returns an immutable CheckReport with the selected target, outcome, individual observations, timestamps, and elapsed time. The Python SDK guide shows runtime and service examples.
Choose the Scope
The scope selects the component that must answer the check:
Server checks do not take a health level. Runtime and service checks default to live. The sandbox command bridge supports server and runtime targets; service checks require the local SDK service handle.
To select a runtime through the command bridge, provide both the agent name and runtime ID:
Refer to the runtime server command guide for command syntax.
Read the Levels and Outcomes
The levels describe progressively stronger observations of the selected target:
Checks are cumulative: ready evaluates live, then active, then ready. Evaluation stops at the first result that does not pass. The report’s checks list shows which observations ran. Different adapters use different native evidence; the same level does not promise identical behavior across harnesses.
The report uses one of these outcomes:
Runtime activity is separate from health. A runtime can report busy and pass every supported level while an invocation waits for a model response. It can also pass with activity: unknown when the adapter has positive native health evidence but cannot observe whether work is in progress. A passing runtime check reports state: active; activity may be idle, busy, or unknown. An unknown activity value alone does not establish health, and stopping or stopped activity cannot pass.
This rule applies to every adapter. It does not change which native evidence each adapter must collect, or turn an outcome: unknown into a pass. Consumers waiting for an idle runtime must require activity: idle; a passing health result with unknown activity does not establish that an agent will accept another prompt.
Native Evidence by Adapter
The table covers all runtime adapters and the OpenClaw Gateway service. Each row lists native evidence and unsupported levels, with links to qualification tests or capability descriptors. Support depends on the installed adapter version and its descriptor. Check the installed adapter’s advertised health levels before selecting a level.
mini-SWE-agent, NOOA InteractiveAgent, NOOA BenchAgent, and Remote Agent advertise no runtime health levels. A health check for a running runtime using one of these adapters returns unsupported/unsupported_adapter. This outcome describes check support rather than an agent failure. Server checks remain available in OpenShell environments.
Pi readiness checks configured model, tool, and MCP state. Those observations do not perform a model request or execute a tool. OpenClaw readiness follows the Gateway’s channel policies and can use the Gateway’s cached observations. The linked OpenClaw endpoint tests use controlled Gateway HTTP responses.
On Linux and macOS, confirmed exit of an adapter host owned by NeMo Fabric returns failed/agent_exited at live for any adapter that advertises live. Checks evaluate live before higher levels, so a confirmed host exit fails a ready check even when the adapter does not support ready. The managed runtime can no longer serve requests, even if a separate native child process or attached service survives. This observation does not establish descendant cleanup. An unreadable process identity or a failed health connection alone remains unknown.
Adapter checks also preserve confirmed native-process exit and closed-session evidence when a native health request fails. A changed runtime or NeMo Fabric session invalidates the observation and returns unknown/observation_changed.
Kilo checks the NeMo Fabric session and runtime generation at every level. The active probe also returns unknown/observation_changed when the native session identifier differs from the selected session. Kilo’s ready probe checks server health without comparing the native session identifier; a full ready check still runs active first.
Passing a health check does not establish successful inference, valid model credentials, provider availability, or successful tool execution unless the particular probe verifies that property. For example, Claude’s readiness response does not require connected MCP servers, and Qwen’s command response does not test model access. The linked qualification tests hold a local model response to verify that health checks can complete during invocation; those tests provide separate invocation evidence rather than changing what the health result means.
Deadlines and Concurrent Invocations
One positive, finite timeout covers routing, worker queues, transport, all evaluated levels, and response decoding. The default is three seconds. A deadline is not restarted for each level. An expired check returns unknown/probe_timeout; the health timeout leaves the invocation running.
Some native SDK requests cannot be cancelled safely. A caller timeout can therefore leave a native health request pending. Adapters with this behavior bound their pending native work. A later check either waits within its own remaining deadline before obtaining a fresh response, or returns unknown/probe_timeout while the earlier request remains pending. A delayed response is not reused as fresh evidence for a later check.
Checks observe the selected native runtime through a health connection that is independent of the invocation request. A successful health response while an invocation is pending does not prove that the invocation will eventually succeed. Stop, release, and session replacement invalidate observations collected for an earlier owner.
Maintain the Evidence Table
When an adapter changes its advertised levels or native probe, update this table, the adapter guide, and its qualification tests together. Keep native response details in the adapter documentation and preserve the shared report schema across harnesses.
Runtime Health
For examples that reuse config, start with Start With One Run.
Use Fabric.check(environment=..., request=...) to check runtime health. Refer to the health-check overview for scopes, outcomes, and deadlines, and the adapter support table for supported levels and native evidence.
The runtime ID selects the exact session inside the supplied environment:
The default level is live, and the default timeout is three seconds. A request
for ready evaluates live, active, and ready in order, stopping at the
first nonpassing result. One deadline covers routing, every probe, and response
decoding. Checks can run during an invocation; a busy session can pass all
three levels. A timeout returns unknown/probe_timeout. Stop or environment release during a probe returns
unknown/observation_changed.
CheckReport is an immutable mapping with outcomes passed, failed, unknown,
and unsupported. Reports contain fixed evidence-source IDs, UTC timestamps,
and elapsed milliseconds. An unavailable runtime returns
unknown/runtime_unavailable. Adapters without health support return
unsupported/unsupported_adapter.
Cline live checks verify native session residency, identity, and closure. active checks also inspect request-interface state. ready checks additionally call ClineCore.get() to read a fresh native session snapshot. Cline checks can run during the first invocation and do not require model access. Refer to the Cline health contract.
Codex checks its owned app-server process and a fresh loaded-thread response; native idle or active status is the readiness equivalent. Pi checks native session liveness, initialization and request-interface guards, and required model, tool, and MCP connection state. OpenClaw checks Gateway liveness, completed startup, selected-agent registration, the enabled Chat Completions endpoint, and native readiness including configured channels. Deep Agents checks its owned checkpoint connection, compiled message interface, and native state for the selected LangGraph thread. Hermes checks its owned native session, initialized invocation interface, and a fresh get_activity_summary() diagnostic response. Runtime activity covers NeMo Fabric invocations. Refer to the OpenClaw runtime health guide.
A local CheckRequest(scope="server") returns unsupported/unsupported_scope. For OpenShell environments,
Fabric.check() maps to fabric-agent-ctl check in the sandbox. Server requests
stop at the runtime server; runtime requests select the owned agent and use the
local SDK health path inside the sandbox. Health uses an independent provider
worker during remote invocations.
The diagram shows the local runtime adapter path. OpenShell adds the provider and command bridge before this path; server checks stop at the runtime server.
The lifecycle dispatcher awaits each start, invocation, or stop operation. The
health listener checks the same adapter runtime through a separate connection,
so checks can run while an invocation is pending. The host publishes the port
and a per-process shared secret in a private health.endpoint.json file under
FABRIC_HOME; the Rust core reads that file and authenticates health requests
and responses. Each health request carries the remaining timeout budget, which the host applies
to the probe.
Gateway Service Health
Check a prepared or attached OpenClaw Gateway with Fabric.check(service=..., request=...). Service checks use the selected process-local ServiceHandle:
Use the OpenClaw service_config from Shared Services, including its installation and credential prerequisites.
For OpenClaw, live validates /healthz, active validates an authenticated /v1/models response, and ready follows native /readyz, including configured channel health. A failed channel can fail Gateway readiness while live and active pass. Refer to the OpenClaw Gateway health guide.
Service checks evaluate earlier levels in order under the same three-second default deadline. They can run during invocations. Reports have scope="service", the selected service_id, and null runtime evidence. Release during a check invalidates evidence with unknown/observation_changed; a released or unavailable binding returns unknown/service_unavailable. Service hosting and checks remain local SDK capabilities; the sandbox command bridge continues to support server and runtime checks.
Sandbox Command Bridge
fabric-agent-ctl check checks server responsiveness with a three-second default deadline. Use --timeout-seconds SECONDS to set another positive, finite budget. Server checks work before a runtime exists and during an invocation. Completed checks return a fabric.check/v1alpha1 report in result, with changed: false; passed health exits 0, while failed, unknown, or unsupported health exits 1. Use fabric-agent-ctl check --agent researcher --runtime-id R1 --ready to check one runtime and return a runtime-scoped report. Runtime checks require both target flags and default to --live; --active and --ready are mutually exclusive alternatives. OpenShell SDK requests map to these named commands. Each layer forwards the remaining deadline, and failed health evidence stays in the typed SDK report.
fabric.check/v1alpha1 identifies the shared report schema returned by
Fabric.check() and fabric-agent-ctl check. The executable ships with
nemo-fabric-runtime and follows the SDK package release. Health readers validate reason codes strictly. Use the same nemo-fabric-runtime release for the caller, sandbox server, and fabric-agent-ctl; mixed-release health reports are not supported. The schema identifier does not negotiate SDK releases.
fabric-agent-ctl serve starts one runtime server per sandbox. The start, invoke, stop, and status commands call Fabric.start_runtime(), Runtime.invoke(), Runtime.stop(), and Fabric.runtime_status(). Start accepts FabricConfig and returns the SDK RuntimeHandle mapping. Invoke accepts a full RunRequest through --request or any JSON SDK input through --input, and returns the SDK RunResult mapping. Stop returns JSON null. These payloads occupy the response envelope’s result field; command diagnostics remain separate. Targeted commands require the returned runtime ID and exact agent name. The caller waits for the private socket before start. Caller-side preflight uses Fabric.plan() and Fabric.doctor(); startup validates configuration against installed descriptors.
One sandbox can own multiple runtimes, each addressed by its returned runtime ID. Invoke and stop operations are independent across IDs; stopping one runtime leaves the others usable, and another runtime can be started after a stop. An uncertain start blocks further starts because the server cannot safely determine whether an unaddressable runtime exists. Buffered SDK stop confirms owned process cleanup before reporting success and retains ownership when cleanup is unconfirmed. The runtime server also retains fabric.unclean after runtime use or an uncertain start until previous sandbox process cleanup can be confirmed. Refer to the runtime server command guide.