Pipeline on ZAPP

Containers

Mon, 01 Jan 0001 00:00:00 +0000

ZAPP containers can embed multiple data formats in one container. Key features include:

Minimal overhead: 4-byte container header and 10-byte header for each payload
Variable-length multiple payloads with no inherent container size limit
Extensible via IANA ID + TypeID mechanism

gRPC pipeline component

Mon, 01 Jan 0001 00:00:00 +0000

Use grpcServer when a listener should expose telemetry directly to consumers. The component is terminal: it accepts pipeline input and serves the ZAPP gRPC service, but it does not produce downstream pipeline targets.

components:
 grpc:
 type: grpcServer
 address: :9998

Important fields:

type must be grpcServer
address is the TCP listen address for the gRPC server

Accepted input types:

*timeseries.Model
[]*zapp.Payload

Use grpcServer for direct listener-to-consumer deployments. Use the ZAPP Aggregator consumer API when you need one endpoint that combines telemetry from multiple listeners.

NVIDIA ZAPP documentation

Mon, 01 Jan 0001 00:00:00 +0000

Important

NVIDIA ZAPP Aggregation Protocol Pipeline is currently offered as a Developer Preview. The software may have limitations, may change significantly, and includes functionality that continues to be refined through testing and feedback. It should not be used in production environments except as part of an authorized collaboration.

Use this documentation to deploy and operate ZAPP telemetry infrastructure on Kubernetes.

These pages cover ZAPP concepts, components, pipeline configuration, plugins, deployment, and example listener configurations.

Redfish listener architecture

Mon, 01 Jan 0001 00:00:00 +0000

The Redfish listener example combines Redfish sensor discovery, a ZAPP Listener pipeline, and ZAPP Aggregator forwarding.

The Redfish listener discovers or collects telemetry from Redfish-enabled baseboard management controllers (BMCs) and forwards telemetry to the ZAPP Aggregator.

Main components

BMC Redfish endpoints: BMC Redfish endpoints provide sensor metadata and telemetry from managed infrastructure.
Sensor pipeline: The envPlugins.sensorPipeline configuration runs before pipeline components start. It loads or discovers sensors and populates the listener sensor manifest.
Redfish listener pipeline: The listener pipeline runs inside the zapp container. For Redfish deployments, the pipeline includes Redfish configuration and forwards telemetry to an aggregatorStreamer component.
ZAPP Aggregator: The ZAPP Aggregator receives telemetry from the listener and exposes metrics, manifest data, and subscriber gRPC access.

Data flow

The listener reads the pipeline file and expands environment and secret references.
The sensor pipeline reads envPlugins.sensorPipeline.
ZAPP discovers or loads Redfish sensors into the manifest.
Pipeline components start.
The listener forwards telemetry to the ZAPP Aggregator.
Operators inspect /metrics, /manifest, or /manifest.json to validate deployment health.

The Helm chart mounts the pipeline at /config/pipeline.yaml, mounts Redfish credential files as Kubernetes Secrets, and passes the ZAPP Aggregator address to the listener with --aggregator.

Redfish listener example

Mon, 01 Jan 0001 00:00:00 +0000

Use this example to understand, configure, monitor, and troubleshoot ZAPP telemetry collection for Redfish-enabled infrastructure.

Start with Deploy ZAPP Listeners if you already know the topology and need Helm values.

Time-series telemetry

Mon, 01 Jan 0001 00:00:00 +0000

ZAPP represents decoded sensor telemetry as time-series models. Operators usually interact with this model indirectly through pipeline components, the ZAPP Aggregator, and gRPC subscriptions.

A time-series model contains:

readings, which group telemetry by sensor ID
Instance readings, which represent individual samples
Window readings, which represent aggregated samples over a time interval
snapshots, which represent the latest known values for multiple sensors at one timestamp

Pipeline components use time-series models when telemetry has been decoded and associated with the sensor manifest. For example, payloadProcessor converts NVIDIA time-series payloads into *timeseries.Model values, and redfishCollector emits *timeseries.Model values from Redfish polling.

ZAPP Listener

Mon, 01 Jan 0001 00:00:00 +0000

A ZAPP Listener runs a telemetry pipeline from a YAML configuration file. A listener can receive ZAPP payloads, poll Redfish endpoints, process payload batches, forward telemetry to the ZAPP Aggregator, or expose telemetry directly to consumers.

A listener pipeline can poll BMC Redfish endpoints and expose the resulting time-series telemetry directly through a grpcServer component.

ZAPP overview

Mon, 01 Jan 0001 00:00:00 +0000

ZAPP collects telemetry from infrastructure systems and forwards it through a configurable pipeline. Many deployments include one or more ZAPP Listeners and one ZAPP Aggregator.

ZAPP Listeners collect or receive telemetry near the source. A ZAPP Aggregator receives telemetry from multiple listeners and exposes one client-facing endpoint.

ZAPP Listener

A ZAPP Listener runs a configured telemetry pipeline. Depending on the pipeline, the listener can receive telemetry packets, discover sensors, collect Redfish telemetry, process payloads, and forward telemetry to the ZAPP Aggregator or directly to downstream consumers.

Concepts

Mon, 01 Jan 0001 00:00:00 +0000

Use these pages to understand the telemetry concepts that ZAPP components and plugins share.

Configure the pipeline

Mon, 01 Jan 0001 00:00:00 +0000

The Redfish listener pipeline is embedded in the listener Helm values under zapp.listeners[].config.pipeline. The chart mounts the content at /config/pipeline.yaml.

The Redfish pipeline discovers sensors before components start, then forwards telemetry to the ZAPP Aggregator.

Sensor pipeline

Configure Redfish endpoints under envPlugins.sensorPipeline. This block runs before components start and populates the listener manifest.

envPlugins:
 sensorPipeline:
 maxRedfishConnections: 1
 redfishDevices:
 - endpoint: ${secrets.redfish_endpoint}
 username: ${secrets.redfish_username}
 password: ${secrets.redfish_password}
 version: 1.11.0
 manifest:
 path: /data/redfish-sensor-manifest.yaml
 forceReload: false

Important fields:

Redfish listener pipeline

Mon, 01 Jan 0001 00:00:00 +0000

A Redfish listener pipeline uses envPlugins.sensorPipeline to discover sensor metadata and a pipeline component to emit telemetry. The direct collector pattern uses redfishCollector to poll each configured baseboard management controller (BMC) and send time-series models to downstream targets.

envPlugins:
 sensorPipeline:
 maxRedfishConnections: 1
 redfishDevices:
 - endpoint: ${secrets.redfish_endpoint}
 username: ${secrets.redfish_username}
 password: ${secrets.redfish_password}
 tls:
 insecureSkipVerify: true
 manifest:
 path: /data/redfish-sensor-manifest.yaml
 forceReload: false

components:
 redfish:
 type: redfishCollector
 timeout: 30s
 samplePeriod: 5s
 targets:
 - target: aggregator
 cap: 1000
 leaky: true

Security note

Do not use tls.insecureSkipVerify: true in production. This setting disables certificate verification for Redfish connections. Use it only for local testing or lab environments with self-signed certificates. For production deployments, configure a trusted certificate authority for the BMC endpoints and leave tls.insecureSkipVerify unset or set it to false.

Components

Mon, 01 Jan 0001 00:00:00 +0000

ZAPP deployments are built from two runtime components: ZAPP Listeners and the ZAPP Aggregator. Listener pipelines run near telemetry sources. The ZAPP Aggregator provides central fan-in, listener coordination, and consumer-facing APIs.

Use these pages to understand what each component runs, which configuration fields matter first, and where pipeline configuration fits into a deployment.

Pipeline components

Mon, 01 Jan 0001 00:00:00 +0000

A ZAPP pipeline is a YAML map of named components. Each component declares a type, component-specific fields, and optional targets. Targets connect one component output to another component input.

Components usually fit one of three roles:

Source components receive or collect telemetry
Processing components convert or route telemetry
Terminal components forward telemetry to a service, network endpoint, or consumer API

Channels

Pipeline components pass data through typed channels. Source and processing components expose outputs with targets, and downstream components receive those outputs as inputs. During startup, ZAPP creates the channels, connects each target to the named component, and validates that the receiving component accepts that input type.

UDP listener and writer

Mon, 01 Jan 0001 00:00:00 +0000

Use udpListener to receive ZAPP UDP payload containers. Use udpWriter to forward payload batches to another UDP endpoint.

UDP listener

udpListener is a source component. It listens on a local UDP address, decodes ZAPP packets into payload batches, and sends those batches to configured targets.

components:
 listener:
 type: udpListener
 network: udp
 address: :9091
 allowedSources:
 - 10.0.0.0/24
 targets:
 - target: payloadProcessor
 cap: 1000
 leaky: true

Important fields:

Plugins

Mon, 01 Jan 0001 00:00:00 +0000

Plugins extend the ZAPP pipeline with additional component types. Use them when built-in components do not cover a site-specific telemetry source, protocol, processor, or output.

Plugin component packages live in the ZAPP repository and are compiled into the ZAPP Listener binary. When the listener starts, those packages register their component types with the pipeline registry.

Enable a plugin component by using its registered type in the listener pipeline:

components:
 dsx:
 type: dsxMetrics
 # DSX-specific configuration omitted.
 targets:
 - target: aggregator

 aggregator:
 type: aggregatorStreamer

For Kubernetes deployments, use a ZAPP Listener image that includes the plugin component registrations you need.

Troubleshoot the Redfish listener

Mon, 01 Jan 0001 00:00:00 +0000

Start with Kubernetes readiness, then check listener configuration, ZAPP Aggregator connectivity, Redfish access, and telemetry validation.

Use Kubernetes checks, HTTP monitoring endpoints, and zest to isolate Redfish listener deployment problems.

Check pods and releases

helm list --namespace <namespace>
kubectl get pods --namespace <namespace>
kubectl get statefulsets --namespace <namespace>

If a pod is not ready, describe it and inspect recent events: