Skip to navigation

Resource Metrics

View as MarkdownOpen in Claude

Use this reference for snapshot fields, units, collection contracts, and platform support. For activation, polling, and runnable collection examples, refer to Resource Metrics.

Snapshot Shape

Each snapshot has one UTC timestamp, an operating_system, a measurement_scope, and optional process_sampling metadata. Rust and Python use snake_case field names. Node.js uses camelCase property names. The values and meanings are the same.

The snapshot always has cpu, memory, process, disk, gpu, and network category fields. A disabled category is null. An enabled category is an object with a stable set of fields. A value that could not be collected is null, never zero. Each available measurement has only a value and a unit. This example has CPU enabled and every other category disabled:

Illustrative snapshot
{
"timestamp": "2026-09-29T14:30:00Z",
"operating_system": "linux",
"measurement_scope": "process_tree",
"process_sampling": {
"visible_processes": 1,
"sampled_processes": 1,
"field_sampled_processes": {
"cpu.user_time": 1,
"cpu.system_time": 1,
"cpu.total_time": 1
}
},
"cpu": {
"user_time": { "value": 120, "unit": "milliseconds" },
"system_time": { "value": 30, "unit": "milliseconds" },
"total_time": { "value": 150, "unit": "milliseconds" },
"consumption_rate": null,
"throttled_time": null,
"effective_limit": null,
"some_pressure_stall_time": null,
"full_pressure_stall_time": null,
"limit_events": []
},
"memory": null,
"process": null,
"disk": null,
"gpu": null,
"network": null
}

Every enabled category contains all of its fields. timestamp is the time collection finished. operating_system is linux, macos, or windows. measurement_scope gives the scope that Relay collected: global, application_process, or process_tree.

Global scope covers resources the operating system exposes to Relay. A process-tree snapshot includes its root and active children. The scope selects which process readings to include. System memory, filesystem capacity, device-wide GPU readings, and network readings describe their wider resources in every mode. network.measurement_scope is always global.

Some values are running totals. CPU time and disk I/O fields add the lifetime counters of processes visible at collection time. These sums can fall when a process exits. Relay can miss a process that starts and exits between polls.

If a process query shows that a child exited after Relay listed the process tree, Relay skips that child and reports metrics for the remaining live tree. The exited child does not make the process totals unavailable. Relay also excludes that child from the process IDs used for environment and GPU readings. This behavior is the same for polling and on-demand collection in every language.

A process total is still unavailable if Relay cannot read a remaining process, access is denied, a process ID has been reused, or any sampled process lacks that counter. If the root exits or its identity changes, Relay returns an unavailable snapshot. In global scope, each total adds only the processes that supply that counter. A global total is null if no process supplies the counter or the sum exceeds its numeric range.

Use process_sampling to check coverage before treating a global total as complete:

  • visible_processes counts processes listed when collection began, including children that exit before sampling. In process-tree mode, it can therefore be larger than sampled_processes even when the remaining tree is fully sampled.
  • sampled_processes counts processes whose base query succeeded.
  • field_sampled_processes counts readable values for each process-summed field. Keys use canonical paths, such as cpu.total_time, in every language. A zero count means no process supplied a usable value. Disk throughput keys count processes with a usable change over the sampling interval; they are zero on the first reading. A missing key means Relay did not attempt a process sum or disk-rate calculation for that field. Compare a field’s count with visible_processes to check its coverage of the listed processes.

The process_sampling metadata is null when Relay did not sample process counters or could not list processes. Polling marks include three coverage gauges, each measured in processes:

  • nemo.relay.resource.process_sampling.visible_count
  • nemo.relay.resource.process_sampling.sampled_count
  • nemo.relay.resource.process_sampling.field_sampled_count, with the canonical field path in the nemo_relay.resource.field attribute

If Relay cannot list processes, independent readings such as system memory, filesystem capacity, and device-wide GPU data can still be available.

Arrays have separate empty and unavailable meanings:

  • limit_events is an empty array when no supported limit events are present.
  • disk.filesystems is an empty array when no paths are configured.
  • network.interfaces is empty when no selected interface is visible.
  • A GPU subgroup is null when disabled or collection cannot start. It can be an empty array when collection finds no supported or matching records.
  • A configured filesystem path stays in filesystems when its capacity query fails; that entry’s capacity measurements are null.

CPU Category

CPU time uses milliseconds by default. For a process tree or global scope, the time fields add lifetime values from processes visible during collection. These sums can fall when a process exits.

For application-process and process-tree modes, CPU rate uses the change in each process’s CPU time between samples. A newly seen process contributes its current lifetime CPU time. Relay can miss work from processes that exit before the next sample. In global mode, CPU rate comes from system-wide CPU use reported by the operating system. Relay measures the time between samples with a clock that only moves forward, so changes to the system clock do not affect the calculated rate.

The CPU category contains these fields:

FieldMeaning
user_timeTime spent running application code across the measured process scope.
system_timeTime spent in operating-system code for those processes.
total_timeUser time plus system time across the measured process scope.
consumption_rateAverage CPU use, in logical processors by default.
throttled_timeTime the operating system stopped the environment from using CPU.
effective_limitCPU capacity available, in logical processors by default.
some_pressure_stall_timeTime when CPU pressure stalled some work.
full_pressure_stall_timeTime when CPU pressure stalled all work.
limit_eventsSupported cumulative CPU limit-event counts.

consumption_rate is not a percentage. With the default logical_processors unit, a value of 1.0 means the selected scope used about one logical processor over the sample interval. One logical processor is one CPU hardware thread working at full capacity. The same use is 1,000 millicores.

Pressure stall time measures how long work waited for CPU or memory resources. Pressure, throttling, limits, and event counts need operating-system support, so these values can be unavailable.

Very short polling intervals can also leave the global CPU rate null until the operating system provides another usable reading.

Memory Category

Memory values, including GPU memory, use KiB by default (1 KiB is 1,024 bytes). Collection converts byte and page values to KiB and drops any remainder before applying the selected output unit.

The memory category contains these fields:

FieldMeaning
system_usedSystem memory currently in use, in every mode.
system_totalTotal system memory, in every mode.
system_availableSystem memory available for use, in every mode. On macOS, this is active, inactive, free, and speculative memory, using the host page size, without subtracting compressor pages.
residentMemory used by the process or tree that is currently in physical RAM.
privatePrivate memory that is not shared with other processes.
physical_footprintMemory footprint as reported by the operating system.
virtual_memoryVirtual address space reported for the process.
peak_residentPeak memory for one process; unavailable for process trees.
limitMemory limit for the measured environment, when one exists.
environment_accountedMemory charged to the measured environment.
some_pressure_stall_timeTime some work stalled from memory pressure.
full_pressure_stall_timeTime all work stalled from memory pressure.
out_of_memory_event_countCumulative out-of-memory (OOM) events, when reported.
limit_eventsSupported cumulative memory limit-event counts.

The meaning of resident, private, footprint, and environment-accounted memory depends on the operating system. Refer to the compatibility tables. The three system_ fields describe system memory visible to Relay in every mode. They do not describe the selected process or tree. In global mode, per-process memory fields are null.

On macOS, system_available follows Apple’s available non-compressed memory definition. It includes active memory, so it is broader than the vm_stat free plus inactive pool. system_used and system_available can overlap; their sum does not necessarily equal system_total.

Process Category

Counts add across the measured process scope unless the description says otherwise. Global counts cover processes visible to the operating system. A process-tree count includes its root and active descendants.

The process category contains these fields:

FieldMeaning
active_countNumber of process IDs found when collection began.
descendant_countActive descendants, excluding the root process.
thread_countThreads across the measured scope.
lifetime_creation_countCumulative creations inside a Windows Job Object.
open_file_descriptor_countOpen files or descriptors reported by the OS.
windows_handle_countWindows kernel handles across the measured scope.
limit_eventsSupported cumulative process-limit event counts.

open_file_descriptor_count is not a count of every Windows kernel handle. Windows reports those separately as windows_handle_count.

Each limit_events entry has a resource (cpu, memory, or processes), an event name, and an optional count measurement in events. Current collectors report CPU throttled, memory high, maximum, or out_of_memory, and process maximum events where supported.

Disk Category

Disk transfer and filesystem capacity use bytes by default. Process I/O totals add lifetime counters and can fall when a process exits. Throughput uses the change between two samples and defaults to bytes per second. A newly seen process contributes its current lifetime counter to that interval. Relay can miss transfers from a process that exits before the next sample. Disk rates in global mode skip processes whose current or previous I/O counter is unavailable. Check process_sampling.field_sampled_processes for disk.read_throughput and disk.write_throughput to see how many processes supplied usable interval values. A rate is unavailable if no process supplies a usable change, a readable counter decreases, or the sum overflows. Rates for an application process or process tree require every sampled process to supply the required current and previous counters.

The operating systems count I/O differently. Linux byte fields count storage traffic, while its operation fields count read and write system calls. macOS reports disk-transfer bytes but not operation counts. Windows reports transfers and operations handled by its I/O manager. Those values can include activity beyond physical disk traffic. Compare samples from the same operating system. The shared unit does not mean the counters measure the same activity.

The disk category contains these fields:

FieldMeaning
read_dataData read by processes in the measured scope.
write_dataData written by processes in the measured scope.
read_throughputAverage read transfer rate since the previous sample.
write_throughputAverage write transfer rate since the previous sample.
read_operationsRead operations reported by the operating system.
write_operationsWrite operations reported by the operating system.
filesystemsCapacity values for each configured path.

Each filesystems entry contains the configured path and three optional measurements:

  • total_capacity is the filesystem’s total capacity.
  • available_capacity is the space available to the calling user.
  • free_capacity is total free space, which can include space reserved for an administrator.

For global scope, process I/O covers all processes visible to the operating system. For process_tree scope, it covers that tree. Set filesystem_paths = [] when you only want process I/O. Path capacity is measured for the filesystem containing each path, so two paths on the same filesystem can produce repeated capacity values. Throughput is null on the first sample, after a counter reset, or when Relay cannot read every needed process counter. Polling and on-demand collection keep separate earlier samples.

Network Category

Network data always describes selected system interfaces. It does not measure the selected process or its children. The network.measurement_scope field is global even when the snapshot’s measurement_scope is process_tree.

The category includes system, an aggregate of selected interfaces, and interfaces, one record per selected interface. Each record has the following measurements:

FieldMeaningDefault Unit
received_dataData received by the interface.bytes
transmitted_dataData sent by the interface.bytes
receive_throughputAverage receive rate since the previous sample.bytes_per_second
transmit_throughputAverage send rate since the previous sample.bytes_per_second
received_packetsReceived packet count.packets
transmitted_packetsSent packet count.packets
receive_errorsReceive error count.errors
transmit_errorsSend error count.errors

The system record has the same measurements. Each interface record also has its name. Set network.interfaces to names from a current snapshot, or leave it empty to include all visible interfaces. An unknown name produces no interface record. If no interfaces match, the system measurements are null. The first sample has no throughput reading. A counter reset makes that interval’s throughput null. If a new interface appears, the system throughput is null until that interface has two readings.

System values add selected interface counters. Traffic can cross both a virtual interface and a physical interface, so the aggregate can count it twice. Interface data totals and process disk totals come from different sources and should not be compared as if they measured the same traffic. On Windows, the network backend omits some disconnected and software interfaces. The aggregate covers the interfaces listed in the snapshot.

GPU Category

The gpu object has two optional groups: device_metrics and process_metrics. Each record can contain:

FieldMeaning
vendornvidia, amd, intel, apple, or other.
device_identifierID supplied by the GPU backend.
device_indexBackend-local index, when supplied.
memory_usedGPU memory in use, in KiB by default.
compute_utilizationGPU activity reported by the driver, from 0% to 100% by default.
process_idThe owning process ID; present only in process records.

Device records describe the whole GPU. Process records describe GPU use by a process in the measured scope. A record can have one available measurement and one null measurement if the driver cannot provide both. An empty devices list selects all supported devices that Relay detects. A selector can match a reported identifier or an index written as a string.

GPU compute use measures recent activity. NVML reports process activity since the previous query in the same polling or on-demand series. Linux DRM uses changes in GPU cycle counters or engine busy time. Apple uses changes in GPU time reported by its AGX driver. The first sample has no earlier reading, so process compute use is null. Process memory can still be available. Changing the collection target clears the earlier reading. GPU process records also depend on driver access and can be incomplete in global scope.

Configure Units

The typed APIs group units by what they measure. Each measurement field uses its matching unit type, so a duration cannot use a capacity or bandwidth unit. Capacity and transferred data have separate types even though they allow the same byte units. Rust uses enums. Python and Node.js use string enums. JSON measurements still contain only a value and a unit, such as { "value": 1, "unit": "seconds" }. An unavailable measurement is null.

The following table maps the unit types to their measurements:

Unit TypeMeasurements
DurationUnitCPU times, throttled time, and CPU or memory pressure time
CapacityUnitMemory, GPU memory, and filesystem capacity
DataUnitDisk and network data totals
BandwidthUnitDisk and network transfer rates
CpuUnitCPU consumption and effective CPU capacity
UtilizationUnitGPU compute use
CountUnitProcesses, threads, descriptors, handles, events, operations, packets, and errors

Each decimal storage unit is 1,000 times larger than the previous unit. Each binary unit is 1,024 times larger than the previous unit. A byte has 8 bits. One logical processor equals 1,000 millicores. A value of 100% equals a fraction of 1.

Exact whole-number conversions stay integers. Conversions that produce a fraction use a decimal number. Rust, Python, and Node.js expose the same value and unit; Node.js uses bigint for integers above its safe integer range. If the operating system rounded a reading, changing its unit cannot restore the lost detail. Measurements that Relay cannot collect or convert remain null.

Metric marks use a decimal number for a unit that can produce fractions, even when one reading happens to be whole. This keeps the mark’s number type the same across polls. Snapshots keep exact integers when the conversion allows them. The metric event format cannot carry an integer above 9,223,372,036,854,775,807. A snapshot can still contain that value, but its mark omits the measurement.

For allowed units, defaults, and configuration examples, refer to Configure Units.

Platform and Collection-Mode Compatibility

Choose a collection mode below to compare Linux, macOS, and Windows. These tables apply when the category is enabled. They cover both polling and on-demand collect(). Polling emits marks. collect() returns a snapshot without emitting a mark.

The tables use these terms:

  • Yes: Relay can collect the field, though a reading can still fail.
  • If available: Relay needs the named system interface, driver, or process group.
  • No: The field could apply, but Relay does not collect it on that system.
  • N/A: The field does not apply to that mode or operating system.

Both No and N/A give a null field or an empty event list. A field marked Yes can also be null if a process exits, access is denied, or a counter cannot be read.

A Linux control group (cgroup) sets limits for a group of processes. An exclusive cgroup contains only the process or tree that Relay measures. A Windows Job Object also groups processes so Relay can read group limits.

Linux CPU and memory limits include restrictions from parent cgroups up to the visible cgroup mount. These limits are upper bounds. Other processes that share a parent cgroup can reduce the resources available to the measured process or tree. Relay cannot read restrictions above the visible mount.

This is the CLI default. Process totals cover processes visible to Relay. Relay might see fewer processes than the machine runs. Check process_sampling for the number of visible and sampled processes.

The following table shows what global collection can measure:

Category and FieldsLinuxmacOSWindows
CPU: user, system, and total timeYesYesYes
CPU: consumption rateYesYesYes
CPU: pressure stall timesIf /proc/pressure/cpu is readableNoNo
Memory: system used, total, availableYesYesYes
Memory: pressure stall timesIf /proc/pressure/memory is readableNoNo
Process: active count and thread countYesYesYes
Process: open file descriptorsYesYesN/A
Process: Windows handlesN/AN/AYes
Disk: process read and write data and throughputYesYesYes
Disk: process read and write operationsYesNoYes
Disk: filesystem total, available, free capacityYesYesYes
GPU: device memory and compute useIf supported backend is availableIf driver diagnostics existIf NVML is available
GPU: process memory and compute useIf supported backend is availableApple compute onlyIf NVML is available
Network: system/interface data, packets, errors, throughputYesYesYes

Global CPU and disk totals add readable values for each counter. CPU use and system memory use system-wide readings. Linux pressure values come from the system pressure interface. GPU process records include only visible processes that the driver can identify. Global mode has no owned process group, so group limits and descendant counts do not apply. It reports system memory instead of per-process memory.

GPU Vendor Support in Every Collection Mode

Device records describe the whole GPU, even in application-process or process-tree mode. Process records cover visible processes in global mode, the selected process ID in application-process mode, or the selected process and its children in process-tree mode. A driver can provide memory without compute use, or compute use without memory. Choose a vendor to compare operating systems. These tables apply to every collection mode.

This table shows NVIDIA GPU measurements by operating system:

MeasurementLinuxmacOSWindows
Device memoryNVMLNoNVML
Device compute useNVMLNoNVML
Process memoryNVMLNoNVML
Process compute useNVML, if supportedNoNVML, if supported

A backend is the driver interface that provides GPU readings. Each backend has different requirements:

  • NVIDIA: NVML needs a working NVIDIA driver and library. Process compute use also needs the optional process-use API and an earlier reading in the same polling or on-demand series.
  • Linux AMD and Intel: Process readings need readable DRM fdinfo files and driver counters. Device readings come from Linux system files (sysfs). AMD device readings use mem_info_vram_used for memory and gpu_busy_percent for compute use. Intel device memory needs mem_info_vram_used. Relay does not collect Intel device compute use on Linux.
  • macOS: Device readings use driver PerformanceStatistics values when available. Apple process compute use comes from changes in AGX GPU time. These diagnostic values can change between macOS versions. Relay needs the ioreg utility and permission to read the device registry. Apple process data also needs exactly one detected Apple GPU that matches the selector.
  • Windows AMD and Intel: Relay has no collector for these GPUs.

Relay cannot collect Apple, AMD, or Intel process GPU memory on macOS. Apple process compute use needs an earlier reading. Relay caps it at 100% when parallel GPU work overlaps. A missing backend or field gives no record or a null measurement.

GPU memory has different meanings across backends. Linux AMD and Intel drivers report used video memory. NVIDIA NVML reports used device memory. macOS drivers can report video or system memory. Relay converts these readings to KiB before applying your output unit, but they do not measure the same kind of memory.

Compute percentages also cover different periods of activity. AMD reports whole-GPU busy time, NVIDIA reports GPU use, and macOS reports device use. Linux AMD and Intel drivers and macOS omit a device record when neither memory nor compute use is available. NVIDIA NVML can return a device record with both measurements null.

Limits That Apply to Every Mode

These conditions apply in every collection mode:

  • Filesystem capacity needs at least one configured filesystem_paths entry and a successful query for that path. An empty list produces an empty array.
  • Process I/O needs disk.process_io = true. macOS currently reports bytes, but not read or write operation counts.
  • Device-wide GPU records need gpu.device_metrics = true. Process GPU records need gpu.process_metrics = true. Both groups also need driver access.
  • GPU process utilization and CPU consumption rate need two readings in the same polling or on-demand series. Disk and network throughput also need two readings. Their first values are null.

By default, CPU and pressure times use milliseconds, memory uses KiB, disk transfers and capacity use bytes, network data uses bytes, and GPU use is a percentage. Operating systems can define similar counters differently, so compare them with care.

On platforms other than Linux, macOS, and Windows, the plugin cannot create a collection target and activation fails.

Collect One Snapshot

On-demand collection requires an active resource_metrics component, but it does not require polling. Await the collection call to get a snapshot. The call does not emit a polling mark. The first sample in a sampling series has no consumption_rate, because that value needs two samples to calculate a change over time.

After a Unix fork, the child cannot use the parent’s active plugin or its collection target. Start a new executable in the child and activate the plugin there before calling collect().

Collection runs OS and driver queries on a worker so they do not block your async runtime. Rust requires a Tokio runtime, and Python requires an active async event loop. Cancelling the wait does not stop a query already running. That query can finish and update the baseline used to calculate rates.

Python returns the timestamp as a timezone-aware datetime. Rust returns a UTC DateTime. Node.js returns a Promise. Await it to get a snapshot with an ISO string timestamp. Node.js uses bigint for integer measurements and process-sampling counts above JavaScript’s safe-integer range.

For the complete activation, collection, and cleanup procedure, refer to Collect One Snapshot.