Local Resource Monitor (Local)

Host-side per-process VRAM / PCIe / CPU exporter for engine-startup profiling — metrics and CLI flags.

View as Markdown

This tool is local-only. dynamo_local_resource_monitor.py runs on the host machine (not inside a container) and attributes GPU/CPU/IO usage to individual Dynamo processes by PID. It is not part of a Kubernetes deployment. For the setup walkthrough and design rationale, see the Local Resource Monitor guide.

The monitor samples per-process resource usage (VRAM, GPU utilization, PCIe bandwidth, CPU, disk I/O, network I/O) for Dynamo inference processes, labeled by GPU, PID, and process name. It always exposes Prometheus metrics at /metrics; when the dashboard dependencies are installed, the same endpoint also serves a WebSocket dashboard at /.

Run it on the host:

$pip install psutil nvidia-ml-py prometheus-client
$python3 dev/observability/dynamo_local_resource_monitor.py --host 0.0.0.0 --port 8051

Metrics

All metrics use the dynamo_ prefix. Unlike the runtime’s dynamo_component_* families, these are host-side gauges labeled by physical GPU and OS process.

GPU metrics

dynamo_gpu_memory_used_gib
gauge

GPU memory used by a process, in GiB. Labeled by gpu, pid, and process_name.

dynamo_gpu_memory_total_gib
gauge

Total GPU memory, in GiB. Labeled by gpu.

dynamo_gpu_utilization_percent
gauge

GPU utilization, as a percentage. Labeled by gpu.

dynamo_gpu_temperature_celsius
gauge

GPU temperature, in degrees Celsius. Labeled by gpu.

dynamo_gpu_pcie_tx_gbps
gauge

GPU PCIe transmit throughput, in GB/s. Labeled by gpu.

dynamo_gpu_pcie_rx_gbps
gauge

GPU PCIe receive throughput, in GB/s. Labeled by gpu.

CPU metrics

dynamo_cpu_utilization_percent
gauge

Overall CPU utilization, as a percentage.

dynamo_cpu_process_percent
gauge

CPU usage by process-name group, as a percentage. Labeled by process_name.

Network metrics

dynamo_network_sent_mbps
gauge

Network bytes sent, in MB/s.

dynamo_network_recv_mbps
gauge

Network bytes received, in MB/s.

Disk metrics

dynamo_disk_read_mbps
gauge

Disk read throughput, in MB/s.

dynamo_disk_write_mbps
gauge

Disk write throughput, in MB/s.

CLI flags

--port
integerDefaults to 8051

Port for the /metrics exporter (and the dashboard at / when dependencies are installed).

--host
stringDefaults to 0.0.0.0

Host address to bind.

--main-interval
integer (ms)Defaults to 200

Main collection interval in milliseconds, covering CPU, GPU memory/utilization/temperature, and network (5 samples/s).

--disk-interval
integer (ms)Defaults to 1000

Disk I/O collection interval in milliseconds (1 sample/s).

--window
integer (s)Defaults to 900

Dashboard rolling window in seconds (default 900 = 15 minutes).

--top-n
integerDefaults to 12

Dashboard process-series limit per chart.