Resource Usage and Stats
Resource stats help answer two questions: what resource limits apply to an agent invocation, and how does resource usage change while it runs? NVIDIA NeMo Fabric records duration, CPU, memory, and execution conditions to help answer them.
Before/after readings show the starting and ending state. Optional sampling reveals temporary demand between them, such as memory used by a tool that exits before the agent returns.
What Stats Measure
An invocation is one request sent to an agent through the runtime. Its stats describe two resource boundaries:
- Adapter process tree: the adapter host and its child processes. This includes the agent’s local work and tools that execute within that tree.
- Linux cgroup: the resource group containing the adapter host, including its descendant groups. It reports resource limits and usage, and can include other workloads sharing that group.
A resource limit is a ceiling, not a reservation or a measure of free capacity. Shared cgroup usage cannot be attributed exclusively to one invocation.
CPU and memory used by remote model and tool servers are outside this scope. Invocation duration includes time spent waiting for them. Measuring their resource usage requires collection where those services run.
Configure and Run
This example uses the maintained code-review agent with the Deep Agents harness and an OpenAI-compatible model endpoint. The same system_stats settings apply to other NeMo Fabric workflows.
Choose Collection Settings
FabricConfig.system_stats has three controls:
For the first run, keep all metric groups and enable sampling every second with the example flags below.
Prepare the Environment
Run from the NeMo Fabric repository root on a checkout containing this example’s --stats option. Install the locked Deep Agents dependencies and build the native runtime from the same source. The native build requires Rust and the platform’s compiler toolchain; see the installation guide.
ADAPTER_PYTHON ensures the adapter host uses the environment containing Deep Agents. Rebuilding the native runtime avoids mixing a current Python SDK with an older extension that does not recognize system_stats.
Configure your endpoint and the model ID it serves. The captured run used nvidia/nemotron-3-ultra-550b-a55b at a privately hosted endpoint. Replace the hostname below with your own reachable server. Set STATS_MODEL_API_KEY to its credential; for an endpoint without authentication, use a nonempty placeholder such as unused.
Enable Sampling and Run
Run the code-review agent with one-second sampling:
The command writes a normalized JSON result to $RUN_DIR/run-result.json. Do not add --show-output when redirecting to a JSON file: that option appends human-readable text. The run’s sampled readings and summary are separate artifacts under examples/code_review_agent/artifacts/deepagents/; their exact paths are in the result.
Add --plan to inspect the configuration without invoking the model. Look for config.system_stats: it should select all four metric groups, a 1000 ms sampling interval, and a 1000-sample bound. These settings belong to NeMo Fabric and are not forwarded as adapter configuration.
Compare Collection Modes
Repeat the command with a new RUN_DIR and change only these flags:
Sampling interval and retention flags require --stats sampled. Intervals range from 100 to 60000 ms; retention ranges from 2 to 100000 readings, including before/after readings. Without overrides, sampled mode uses 1000 ms and 10000 readings. Agent behavior may change between runs; differences do not by themselves measure collector overhead.
Select Metric Groups
The four metric groups are:
The example CLI selects all groups. To select a subset, configure the Python SDK directly:
Selection applies to both before/after and periodic reads. collected_groups in the result records the selection. Excluded observations or measurements use not_collected; unselected process reductions and an entirely excluded cgroup block are omitted.
Empty, duplicate, and unknown group selections are rejected. Use system_stats=None to disable collection, or system_stats.sampling=None to disable only periodic sampling. The separate final resource_snapshot remains independent of these controls.
Read the Results
For the code-review example, inspect $RUN_DIR/run-result.json with jq or open it directly in your editor:
A Captured Code-Review Run
The following values came from a Linux run of the command above using NeMo Fabric source 67acf1cff0d598fc3e6a32370cd63fc088e86958 plus the example’s stats CLI changes, Deep Agents 0.7.9, and the model named above. Model work and resource values vary between runs; these are observations, not expected outputs.
Memory values use MiB (bytes divided by 1048576). The final RSS is not memory leaked or left behind by the invocation: most of it was present before the invocation, and the final observation occurs before runtime teardown.
The shared cgroup’s 8.480 CPU-seconds cannot be attributed exclusively to this run. The configured memory limit was reported as unlimited within the visible hierarchy; this does not establish unlimited physical memory or available headroom.
Read collection quality alongside usage. A sampled maximum can miss a spike between readings even with 100% reported coverage. When sources are missing, availability states carry the reason rather than substituting zero. Disabled collection yields no system_stats; failures before a normalized result may leave only partial artifacts.
For stats collection with SWE-bench tasks, see the Harbor walkthrough.
Representative Metrics
Use these fields to interpret your own run. Paths are relative to result.system_stats.
Duration and CPU
CPU-seconds measure processor work, while elapsed seconds measure time passing. In the example, 0.14 CPU-seconds over 3.919 elapsed seconds describes the local adapter’s work; it does not measure remote model compute. The CPU estimate is absent if a reading is missing, identities differ, or the counter decreases. Processes leaving the tree can distort the estimate even when it is positive.
Memory
RSS sums the resident memory of the adapter host and its descendants; shared pages can be counted more than once. Before/after values are instantaneous observations. Sampling can reveal temporary demand that disappears before the final reading, but its peak can still miss shorter spikes. The mean holds each reading until the next valid reading and excludes intervals with missing readings or gaps longer than twice the requested cadence. Check coverage alongside the mean.
Resources and Execution Conditions
The limits below come from cgroup.after.snapshot when that observation is available. The CPU usage and throttling fields are invocation deltas under cgroup.
A limit is a ceiling, not a reservation or a measurement of free capacity. An unlimited memory limit means no numeric bound was found among the visible limit sources; it does not mean physical memory is unlimited. These counters cover the cgroup and its descendants, potentially including parallel workloads.
Throttled time is cumulative accounting and can exceed invocation wall time. It is not additional elapsed delay or proof that another task caused contention.
Collection Quality
Coverage of 1 means the full duration qualified for the mean under the collector’s gap rules, not that every memory spike was observed. Dropped samples disclose incomplete sampling; zero drops does not guarantee every measurement was available. Read availability states and diagnostics too. not_collected means a metric group was excluded, unsupported means the source is not supported, and unavailable means evidence could not be obtained. None of these states represents zero usage.
Complete Field Reference
See the RunResult JSON Schema for the complete field list, types, and required or optional structure. Start at system_stats and follow its definitions for invocation stats, cgroup observations, sampled summaries, and gauges. For configuration fields, see the FabricConfig JSON Schema. Use the schemas from the same release or commit as your runtime when checking saved results.
Collection Details
The Invocation Window
For a local adapter host, collection follows this sequence:
The lock serializes the adapter host’s request/response protocol and remains active when stats are disabled. Collection uses that existing lock to prevent another invocation on the same host from running between the before/after readings. Time waiting for the lock is excluded from elapsed_seconds, along with startup, before/after reads, artifact preparation, and teardown. Other processes in a shared cgroup can still run during this window.
CPU estimates subtract cumulative before/after counters. Memory and process counts are instantaneous readings. On a warm runtime, a new before reading establishes a baseline for each invocation; it does not reset the process or cgroup counters.
Periodic Sampling and Retention
When sampling is enabled, one background sampler thread per collector process serves its active invocations. It reads the selected groups at the requested cadence while the adapter executes. Cadence is a target: collection and scheduling take time, and sampling can affect execution. Late readings are discarded after an invocation’s sampling window closes.
The retention bound includes both before/after readings. Once the intermediate-sample budget is full, further periodic readings are skipped and counted as dropped; the final reading is preserved. Sampling produces a JSONL trace and a summary artifact when persistence succeeds. Before/after-only collection adds neither file.
Optional collection failures preserve the adapter outcome and appear as missing evidence or diagnostics. A protocol failure can still raise without a normalized result; available partial artifacts do not imply a completed invocation.
Local and Remote Execution
Process-tree measurements cover the adapter host and its descendants. In the Deep Agents example, the Fabric SDK starts a separate Python adapter host, and Deep Agents executes inside that host. Its memory and CPU usage, tools running in that process, and child processes launched for local tools fall within this scope. The calling application’s resources are excluded.
Remote model inference and remote MCP tool execution fall outside this process tree. The adapter’s local work to prepare requests and process responses is included, and invocation duration includes time spent waiting for those services. Their server-side CPU, memory, and GPU usage are not measured. A local MCP server is included only if it belongs to the observed process tree; a separately running local service is outside that scope too.
The same distinction applies to agents reached through ACP or gateways: local adapter measurements describe the client-side work, not the remote agent’s resource usage. Measuring remote execution requires collection where it runs, with an explicit execution scope and correlation to the invocation.
Linux cgroup measurements describe a different boundary: the adapter host’s cgroup and its descendants. That boundary can include other workloads. Do not attribute shared cgroup usage exclusively to this invocation or add it to overlapping process-tree usage.
Runtime Resource Snapshots
For examples that reuse config, start with Start With One Run.
For runtime in the snapshot example, use a started runtime from Multi-Turn Runtimes.
Runtime.resource_snapshot() returns a timestamped RuntimeResourceSnapshot
for a started local runtime. It does not wait for an active invocation, so a
monitor can call it while invoke() runs:
The scope is fixed: the local adapter host process that NeMo Fabric started for
the runtime, plus its descendants. For a runtime started on a shared service,
the service process is not included. A descendant that starts a new session or
process group still counts while its parent chain leads to the host. A process
whose parent exited is reparented outside the tree and no longer counts. The
snapshot finds the tree from each process’s list of children: Linux reads
/proc/<pid>/task/<tid>/children and /proc/<pid>/stat, and macOS reads
proc_listchildpids and proc_pid_rusage. Both read only the runtime’s own
processes. On a Linux kernel without those children lists, the snapshot reads
the parent ID of every process on the machine to find the tree. The snapshot
exposes only the fields above, never command lines, environment variables,
credentials, or processes outside the runtime’s tree.
When NeMo Fabric cannot read the values, it raises an error instead of returning zeros or leaving a process out:
ResourceObservationUnavailableErrorwhen the runtime stopped, is stopping, its adapter host process exited, a live process in its tree or the process table cannot be read, or the process tree kept changing during the read. For example, on macOS, the snapshot is unavailable while a descendant runs as another user, including setuid programs such asps,top, andsudo.ResourceObservationUnsupportedErrorwhen the runtime’s adapter kind, environment provider, or operating system has no resource reader. Thehttpandnative_pluginadapter kinds and Windows raise this error.
On Linux with /proc mounted hidepid=invisible or hidepid=ptraceable, the
snapshot is unavailable while /proc hides a descendant, such as one that runs
as another user. On a kernel without children lists, every snapshot is
unavailable under those mount options, because hidden processes are missing
from the process table that the snapshot reads to find the tree.
By default, each RunResult from a local runtime carries system_stats with before/after observations and a monotonic duration for the adapter exchange. Both observations run under the invocation lock, so queued invocations cannot contaminate the baseline. Runtime startup, endpoint collection, artifact preparation, and teardown are excluded from elapsed_seconds. Optional periodic sampling runs concurrently and can affect execution; the elapsed measurement does not subtract that effect. The existing resource_snapshot field retains its independent final process-tree reading even when invocation stats or the process group is disabled. When the process group is selected, it reuses system_stats.after.snapshot, or is None if unavailable.
Invocation System Stats
For a hands-on walkthrough with configuration examples and annotated results, see Resource Usage and Stats.
Identical configs can produce different amounts of agent work and encounter different system conditions. Stats should help separate those two.
Omitting system_stats selects every supported metric group at the invocation endpoints. Explicit None disables invocation stats, including before/after collection and stats artifacts. The SDK preserves that explicit null across planning and task-runner serialization.
Omit collect for all supported groups. Empty lists, duplicates, and unknown names are rejected; use system_stats=None to disable stats. The invocation stats collector skips excluded groups in both endpoints and periodic samples. collected_groups records the selected groups, including those unavailable on the current platform. A not_collected status identifies excluded process observations and cgroup measurements; it differs from unavailable and unsupported, and never substitutes zero. Unselected process gauge reductions and an entirely excluded cgroup block are omitted. Identity, duration, scope, and sampling-quality metadata remain available. Metric selection does not change the separate resource_snapshot contract.
For example, a serialized RunResult contains this fragment. The IDs already associate the measurements with one invocation; no additional identity file is required.
The CPU estimate is 2.5 seconds for this invocation, excluding the warm runtime’s previous 12 seconds. It is omitted when either reading is missing, runtime identities differ, or the counter decreases. Process reaping and reparenting can distort even a positive difference; this is an estimate, not exact accounting. Background work in the same process tree also contributes; the difference does not isolate agent-only CPU. Endpoint memory and process counts are instantaneous gauges, not peaks, averages, or invocation allocations.
For Harbor, these measurements come from the adapter host inside the task runner, even when a separate host selects the task and capabilities. Harbor’s existing result JSON carries the new field without a new stats file. For a model team comparing identical configs, pair usage with these measurements: more tokens or tool calls can explain more agent work; similar work with longer elapsed time warrants investigating execution conditions. Endpoint observations alone miss intermediate peaks and cannot explain every source of latency. Enable sampled stats and inspect native cgroup evidence below; remote inference latency requires serving-side evidence.
A failed read appears as {"status": "unavailable", "reason": "..."}; an unsupported provider or operating system uses "unsupported". Collection failures preserve the adapter outcome, including a normalized failed result. Disabled collection, older results, and execution backends without collection have system_stats=None. Protocol failures and timeouts that raise without a RunResult still raise. When sampling is enabled, NeMo Fabric retains sample and summary artifacts where possible even if the adapter exchange fails. These measurements require neither NeMo Relay collection nor invocation recording.
Sampled Peaks and Means
A process can allocate 400 MiB during execution and return to 80 MiB before the final reading. Endpoints miss that spike. Enable native sampling independently of Relay or invocation recording:
The mean holds the preceding reading across adjacent valid observations and excludes missing readings or gaps larger than twice cadence. Retained JSONL offsets allow benchmark tooling to correlate samples with trajectory events and select its own analysis windows. Temporal overlap with a span does not prove exclusive attribution.
For readings 80 MiB → 400 MiB → 80 MiB at 0, 1, and 2 seconds, the final value is 80 MiB, sampled peak is 400 MiB, range is 80–400 MiB, and hold-last-value mean is 240 MiB. A sampled maximum can miss brief spikes between observations. It is not a true high-water mark.
Sampling uses one process-wide scheduler, retains both endpoints, and bounds intermediate retention with max_samples. The result exposes retained/dropped sample counts, collection time, largest gap, truncation, per-gauge covered time, diagnostics, and artifact references. Late samples are discarded after invocation closes. A full series preserves the final endpoint rather than silently reporting an old value as final. Counter resets, changed targets, missing observations, and long gaps invalidate affected sampled CPU estimates. Sampling does not replace endpoint-derived CPU or cgroup deltas; collection quality and continuity are reported under sampled.
With an artifact root, the layout is:
Every sample and saved summary carries the existing runtime, invocation, and request IDs. RunResult.artifacts references both stats files. Without an explicit artifact root, these files use the runtime’s temporary .fabric directory. Optional stats persistence failures produce diagnostics and preserve the agent outcome. A known series write failure suppresses its artifact reference while retaining in-memory reductions and a sample_write_failed diagnostic. Samples and a summary are retained even if the adapter protocol raises without a normalized result; they do not invent a terminal agent status or a second lifecycle journal. Consumers should handle an interrupted last JSONL line explicitly.
Native Cgroup Constraints
On Linux cgroup v2, result.system_stats.cgroup contains before/after observations for the adapter host’s cgroup, plus CPU-use, throttled-time, and OOM-kill deltas. Snapshots expose visible CPU/memory limits, container-accounted current memory, swap, provider memory high-water, and CPU/memory/I/O pressure totals. Missing limit sources report unsupported. Visible numeric limits take precedence over max; unlimited requires an explicit max source with no numeric bound. Individual missing measurements remain explicit; unsupported platforms do not substitute host-wide readings.
This cgroup can contain the NeMo Fabric runner, the adapter tree, and other workloads. Its totals overlap process-tree totals and must not be added to them. Limits account for visible ancestors only; namespaces can hide additional restrictions. memory_peak_bytes covers cgroup lifetime or an external reset, not one invocation. The collector does not reset shared counters or write to cgroup control files.
Pair these observations with usage and reward: more tokens/tool calls can explain more agent work; similar work with throttled CPU and a lower visible limit suggests a constrained execution environment. These signals support investigation and do not establish causality. Remote inference-server GPU usage still requires serving-side telemetry.
The native stats example produces a transient memory spike and retained artifacts without model credentials.
Snapshots observe resources only. They do not enforce limits, stop runtimes, or queue requests. Use the operating system or execution environment, such as cgroups or container limits, to enforce resource limits.