Telemetry Guide

Overview

NIXL’s telemetry system collects performance metrics and transfer events for monitoring and debugging. Telemetry is disabled by default and must be enabled via environment variables or agent configuration.

The system supports two consumption patterns:

• Shared memory cyclic buffer — Events are written to a memory-mapped file and read by a separate telemetry reader process. This is the default exporter when NIXL_TELEMETRY_DIR is set.
• Prometheus exporter — Events are aggregated and exposed as Prometheus-compatible metrics on an HTTP endpoint.

Only one telemetry exporter plug-in can be loaded per NIXL agent instance.

Architecture

The telemetry system consists of three layers:

1. Telemetry Collection — Built into the core library, intercepts agent operations (memory registration, transfers, metadata exchange) and generates events with microsecond-precision timestamps.

2. Event Buffer — A cyclic (ring) buffer stores events. Buffer size is configurable via NIXL_TELEMETRY_BUFFER_SIZE (default: 4096 events). When the buffer is full, the oldest events are overwritten silently — the current design allows telemetry loss under high event rates.

3. Exporter Plugins — Flush events from the buffer to an external consumer at configurable intervals. The built-in shared memory buffer exporter uses a memory-mapped file that can be read by external telemetry reader applications. The Prometheus exporter aggregates events into metrics exposed on an HTTP endpoint.

Event Structure

Each telemetry event contains four fields:

Event Types

The telemetry system defines eight event categories:

Metrics

The following table lists all built-in telemetry metrics and events generated by NIXL:

The shared memory buffer exporter stores raw per-event data without aggregation. Each event is recorded individually, preserving the full time-series for offline analysis.

Enabling Telemetry

Telemetry is controlled by environment variables set before agent initialization:

NIXL_TELEMETRY_ENABLE accepts y, yes, on, true, enable, or 1 (case-insensitive). Any other value or absence disables telemetry.

Behavior rules:

• If telemetry is disabled via the environment but enabled programmatically (capture_telemetry in agent config), telemetry is captured internally but not exported. This may incur a performance penalty.
• If telemetry is enabled but no exporter is set and NIXL_TELEMETRY_DIR is unset, no telemetry file is generated and NIXL_TELEMETRY_RUN_INTERVAL is unused.
• If telemetry is enabled and NIXL_TELEMETRY_DIR is set, the shared memory buffer exporter writes events to a file in that directory.

# Enable telemetry with shared memory buffer exporter
export NIXL_TELEMETRY_ENABLE=true
export NIXL_TELEMETRY_DIR=/tmp/nixl_telemetry
export NIXL_TELEMETRY_BUFFER_SIZE=8192
# Run your NIXL application
./my_nixl_app

Telemetry API

getXferTelemetry (C++)

Retrieve per-transfer telemetry for a completed transfer request. Requires captureTelemetry = true in nixlAgentConfig.

The returned nixl_xfer_telem_t structure contains:

• startTime — Timestamp when the transfer was initiated
• postDuration — Time from start to posting to the backend
• xferDuration — Total transfer time from start to completion
• totalBytes — Total bytes transferred
• descCount — Number of descriptors in the transfer

nixl_xfer_telem_t telem;
if (agent.getXferTelemetry(req, telem) == NIXL_SUCCESS) {
    std::cout << "Transfer took " << telem.xferDuration.count() << " us" << std::endl;
    std::cout << "Total bytes: " << telem.totalBytes << std::endl;
}

get_xfer_telemetry (Python)

Python equivalent of the C++ method. Requires capture_telemetry=True in nixl_agent_config.

telem = agent.get_xfer_telemetry(xfer_handle)
print(f"Transfer duration: {telem.xferDuration} us")
print(f"Total bytes: {telem.totalBytes}")

The returned object has the same fields as the C++ version: startTime, postDuration, xferDuration, totalBytes, and descCount.

addTelemetryEvent / getTelemetryEvents (Backend API)

These methods are available on the backend engine base class (nixlBackendEngine) and are used by backend plug-in authors to emit custom telemetry events:

• addTelemetryEvent(event_name, value) — Add a custom event to the telemetry buffer
• getTelemetryEvents() — Retrieve recorded events from a backend

These are internal to the backend plug-in interface.

Telemetry Reader

NIXL provides telemetry reader utilities in both C++ and Python for consuming events from the shared memory cyclic buffer. Below are short snippets showing the core reading loop.

import time
from nixl_telemetry_reader import SharedRingBuffer
# Open the shared memory telemetry buffer
buffer = SharedRingBuffer("/tmp/nixl_telemetry/agent_name", version=1)
print(f"Buffer capacity: {buffer.get_capacity()} events")
# Read events in a loop
while True:
    event = buffer.pop()
    if event:
        name = event.event_name.decode("utf-8").rstrip("\x00")
        print(f"[{event.timestamp_us}] {name}: {event.value}")
    else:
        time.sleep(0.5)  # No events available, wait briefly

Running the readers:

# C++ telemetry reader
./builddir/examples/cpp/telemetry_reader /tmp/nixl_telemetry/agent_name
# Python telemetry reader
python3 examples/python/telemetry_reader.py --telemetry_path /tmp/nixl_telemetry/agent_name

Example output:

=== NIXL Telemetry Event ===
Timestamp: 2025-01-15 14:30:25.123456
Category: TRANSFER
Event: agent_tx_bytes
Value: 1048576
===========================
=== NIXL Telemetry Event ===
Timestamp: 2025-01-15 14:30:25.124567
Category: MEMORY
Event: agent_memory_registered
Value: 4096
===========================

Prometheus Integration

NIXL includes an experimental Prometheus-compatible telemetry exporter that aggregates events into metrics and exposes them on an HTTP endpoint.

Setup

To enable Prometheus export, set the following environment variables:

export NIXL_TELEMETRY_ENABLE=true
export NIXL_TELEMETRY_EXPORTER=prometheus
export NIXL_TELEMETRY_PROMETHEUS_PORT=9090     # Default port
export NIXL_TELEMETRY_PROMETHEUS_LOCAL=false    # Bind to all interfaces

Scraping Metrics

Once the exporter is running, configure your Prometheus instance to scrape the NIXL metrics endpoint:

# prometheus.yml
scrape_configs:
  - job_name: 'nixl'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 5s

You can verify the endpoint is serving metrics:

curl http://localhost:9090/metrics

Custom Telemetry Plug-ins

NIXL supports custom telemetry exporter plug-ins that implement the telemetry export interface. A custom plug-in consumes events from the cyclic buffer and routes them to any monitoring backend — CSV files, dashboards, or cloud monitoring services.