InfiniBand Topology Provider

Topograph provides two variations of InfiniBand provider. Both discover the IB fabric switch tree using ibnetdiscover, which is useful for any cluster — CPU-only, mixed, or GPU-accelerated — where topology-aware scheduling across an InfiniBand fabric improves workload performance. NVLink domain discovery is an additional capability that applies only to nodes with NVLink-connected NVIDIA GPUs.

Why automate IB discovery? Hand-maintaining IB topology — a static topology.conf or a set of hand-applied node labels — is feasible at ~32 nodes with a stable network and a careful operator. It does not scale. At 1,000 nodes with InfiniBand fabric churn, NVLink partitions shifting with tenant allocation, and a constant background rate of link degradation and node cycling, manual maintenance becomes the dominant source of scheduling misplacement. Topograph keeps topology data current as the cluster changes, removing that burden.

The choice of which to use depends on the specifics of the deployment environment:

• Use infiniband-bm for bare-metal clusters (e.g. Slurm)
• Use infiniband-k8s for Kubernetes clusters

If NetQ is deployed in your environment, consider using the NetQ provider instead — it discovers topology via the NetQ management API rather than directly from the fabric, which avoids node access requirements and is the standard approach for Spectrum-X environments.

For Multi-Node NVLink (MNNVL) Kubernetes clusters (e.g. GB200 NVL72), do not substitute the DRA provider when backend-fabric locality matters. The DRA provider is limited to Slinky topology/block output, requires nvidia.com/gpu.clique to already be present on every participating node, and discovers only NVLink partition membership. It does not discover the InfiniBand or Ethernet switch fabric between partitions, so it cannot guide placement when a workload spans more than one NVLink partition. Use infiniband-k8s (or NetQ, when available) to include that fabric hierarchy.

Both variants are presently single-region only (multi-region requests return a 400 Bad Request error). No CSP credentials are required.

Output

Both variants produce the same topology representation, and are in turn consumed by whichever engine you configure:

• Slurm engine (engine: slurm) — writes a topology.conf file describing the switch tree, used by the Slurm topology plugin for topology-aware scheduling
• Kubernetes engine (engine: k8s) — applies fabric.topograph.run/ labels to nodes reflecting their position in the switch hierarchy and (where applicable) their NVLink domain
• NFD engine (engine: nfd) — publishes topology as Node Feature Discovery NodeFeature and NodeFeatureGroup custom resources
• Slinky engine (engine: slinky) — writes topology data to a Kubernetes ConfigMap for Slurm-on-Kubernetes deployments

See the engine documentation (docs/engines/) for details on each output format.

infiniband-bm (Bare-Metal)

Prerequisites

• pdsh must be installed on the node running Topograph and able to reach at least one node per IB fabric segment — Topograph discovers the full fabric from a single entry point per segment, so every node does not need to be reachable via pdsh
• ibnetdiscover must be available on cluster nodes (invoked via pdsh with sudo) — part of the standard infiniband-diags package (dnf install infiniband-diags / apt install infiniband-diags), expected to already be present on any properly configured IB system
• NVIDIA GPU driver required on nodes with NVLink-connected GPUs when accelerator.source: nvidia-smi is configured. Nodes without accelerator-domain data are still included in the IB switch tree.

How It Works

1. Runs sudo ibnetdiscover via pdsh on one node per IB fabric segment to map the full switch tree
2. When the accelerator section selects nvidia-smi, runs nvidia-smi --query-gpu=fabric.clusterUuid,fabric.cliqueId --format=csv,noheader via pdsh across all nodes to collect NVLink partition IDs. Identical rows returned for multiple GPUs are merged, unavailable (N/A) fields are rejected, and the CSV pair is normalized to ClusterUUID.CliqueId — the same format as nvidia.com/gpu.clique set by the GPU Operator device plugin on MNNVL systems.
3. Combines the switch tree and any NVLink clique data into the topology graph

Configuration

No credentials are required. Set provider: infiniband-bm in your Topograph config:

1
http:
2
  port: 49021
3
  ssl: false
4
5
provider: infiniband-bm
6
engine: slurm

Accelerator discovery is independent of InfiniBand fabric discovery and is disabled when the accelerator section is omitted or empty. Bare-metal deployments support nvidia-smi or none:

1
provider:
2
  name: infiniband-bm
3
  params:
4
    accelerator:
5
      source: none

Verifying the Output

After triggering topology generation, query the result endpoint:

$
id=$(curl -s -X POST -H "Content-Type: application/json" -d @payload.json http://localhost:49021/v1/generate)
$
curl -s "http://localhost:49021/v1/topology?uid=$id"

For the Slurm engine, verify the generated topology.conf reflects the expected switch hierarchy. See the Slurm engine documentation for details.

infiniband-k8s (Kubernetes)

Prerequisites

• Topograph deployed via Helm — when accelerator.source is nvidia-smi, the node-data-broker DaemonSet collects NVLink partition IDs from each node and stores them as Kubernetes node annotations (topograph.run/cluster-id). With kubernetes-label or none, the broker skips that collection.
• The default ghcr.io/nvidia/topograph image includes ibnetdiscover (Alpine rdma-core). No separate InfiniBand image is required. IB deployments typically run the broker privileged and mount host /sys/class so ibnetdiscover can reach IB devices — see values.k8s.ib-example.yaml.
• NVIDIA GPU Operator — standard on NVIDIA GPU Kubernetes clusters; manages the device plugin DaemonSet used to read NVLink clique IDs. Required only for NVLink domain discovery; on clusters without NVLink-connected GPUs this does not apply and the provider will still discover the IB switch tree.

How It Works

1. Runs ibnetdiscover by exec-ing into a node-data-broker pod on each node to map the switch tree
2. When provider.params.accelerator is non-empty, resolves accelerator domains independently using its source: nvidia-smi reads the broker-written topograph.run/cluster-id annotation, kubernetes-label reads a configured Node label, and none explicitly disables accelerator discovery. Omitting the section or using an empty object also disables it. NVL partition IDs use ClusterUUID.CliqueId, the same format as nvidia.com/gpu.clique.
3. Combines the switch tree and any NVLink clique data into the topology graph

Configuration

No credentials are required. The provider uses the in-cluster service account automatically.

Set provider: infiniband-k8s in your Topograph config:

1
http:
2
  port: 49021
3
  ssl: false
4
5
provider: infiniband-k8s
6
engine: k8s

Parameters

Topology request parameters

The following optional parameters can be passed in the topology request payload:

For a manual request, keep accelerator.source consistent with the source configured for the deployed node-data-broker. In particular, source: nvidia-smi reads the broker-written topograph.run/cluster-id annotation; it does not execute nvidia-smi or change the broker’s GPU Operator workload target.

Helm node-data-broker settings

The following settings select the GPU Operator workload used by the node-data-broker when provider.params.accelerator.source is nvidia-smi. They are deployment settings and cannot be changed by a topology request:

With Helm, configure the accelerator source under provider.params. The chart writes the provider configuration to both the node-data-broker and node-observer ConfigMaps, keeping chart-generated topology requests aligned with broker collection. When the source is nvidia-smi, omitted gpuOperatorNamespace and devicePluginDaemonSet values are rendered as gpu-operator and nvidia-device-plugin-daemonset, respectively. The chart also drops the broker’s GPU Operator pod-exec permissions when the source is not nvidia-smi:

1
provider:
2
  name: infiniband-k8s
3
  params:
4
    accelerator:
5
      source: kubernetes-label
6
      kubernetesLabel:
7
        key: nvidia.com/gpu.clique
8
engine:
9
  name: k8s

To use a non-default GPU Operator namespace or device-plugin DaemonSet:

1
provider:
2
  name: infiniband-k8s
3
  params:
4
    accelerator:
5
      source: nvidia-smi
6
      nvidiaSmi:
7
        gpuOperatorNamespace: my-namespace
8
        devicePluginDaemonSet: my-daemonset

The node-data-broker applies node annotations once when its pod starts. Restart the broker pod to re-apply them after relevant node or provider metadata changes.

If ibnetdiscover needs extra config files, the chart can render ConfigMaps and mount them into the node-data-broker pods:

1
nodeDataBroker:
2
  configMapMounts:
3
    - name: ibdiag
4
      mountPath: /etc/infiniband-diags/ibdiag.conf
5
      subPath: ibdiag.conf
6
      data:
7
        ibdiag.conf: |-
8
          CA=smi0
9
          Port=1

Example request payload with nodeSelector:

1
{
2
  "provider": {
3
    "name": "infiniband-k8s",
4
    "params": {
5
      "nodeSelector": {
6
        "nvidia.com/gpu.present": "true"
7
      }
8
    }
9
  },
10
  "engine": {
11
    "name": "k8s"
12
  }
13
}

Verifying the Output

After topology generation, inspect the node labels applied by Topograph:

$
kubectl get nodes -o json | jq '.items[].metadata.labels | with_entries(select(.key | startswith("fabric.topograph.run")))'

See the Kubernetes engine documentation for details on the label schema.