Topograph with SLURM

For the SLURM engine, topograph supports tree and block topology configurations.

Automatic block-size inference

When blockSizes is not configured, Topograph derives it from the discovered accelerator domains.

For single-level topology, let D be the smallest discovered accelerator-domain size and N be the number of discovered domains. Topograph emits a doubling sequence through the largest power-of-two group that fits within N domains:

blockSizes = [D, 2D, 4D, ..., 2^k * D]
k = floor(log2(N))

This can produce more than one block size as the cluster grows:

For topology containing accelerator sub-domains, Topograph:

1. Finds the largest sub-domain size, maxSubDomainSize.
2. Finds the largest total accelerator-domain size, maxDomainSize.
3. Starting with maxSubDomainSize, repeatedly doubles it until it is at least maxDomainSize.

The resulting aggregate size is:

aggregateSize = maxSubDomainSize * 2^n

where n is the smallest non-negative integer for which aggregateSize >= maxDomainSize. The inferred result is:

blockSizes = [maxSubDomainSize, aggregateSize]

If aggregateSize equals maxSubDomainSize, only one value is emitted.

The inferred sizes are used both to construct the complemented block tree and to emit the final BlockSizes value. Explicitly configured blockSizes always take precedence over inference.

Deriving block names from node names

For topology/block, the optional blockName engine parameter derives each block name from the names of its nodes. Both nodeNameRegexp and format are required when blockName is set.

1
engine:
2
  name: slurm
3
  params:
4
    plugin: topology/block
5
    blockSizes: [8, 16]
6
    blockName:
7
      nodeNameRegexp: 'd([0-9]{2})-r([0-9]{2})'
8
      format: 'domain${1}_rack${2}'

For a block containing nodes such as gpu-d05-r04-srv4, this produces the name domain05_rack04. The expression uses Go regular-expression syntax and may match anywhere in the node name; use ^ or $ when the site naming convention requires anchoring. The format uses Go regexp expansion syntax, including numeric captures such as ${1} and named captures such as ${domain}.

Every node in a non-empty block must match the expression and produce the same non-empty block name. Different blocks must produce unique names. Topograph rejects topology generation when any of these conditions is not met. Empty complemented blocks have no node name to evaluate and retain their generated default name.

The option can also be set on each topologies entry for per-partition output.

Test Provider and Engine

There is a special provider and engine named test, which supports both SLURM and Kubernetes. This configuration returns static results and is primarily used for testing purposes.

Installation and Configuration

Topograph can be installed using the topograph Debian or RPM package. This package sets up a service but does not start it automatically, allowing users to update the configuration before launch.

The configuration file and certificates created by the installer are located in the /etc/topograph directory.

Service Management

To enable and start the service, run the following commands:

$
systemctl enable topograph.service
$
systemctl start topograph.service

Upon starting, the service executes:

$
/usr/local/bin/topograph -c /etc/topograph/topograph-config.yaml

To disable and stop the service, run the following commands:

$
systemctl stop topograph.service
$
systemctl disable topograph.service
$
systemctl daemon-reload

Verifying Health

To verify the service is healthy, you can use the following command:

$
curl http://localhost:49021/healthz

Automated Solution for SLURM

The Cluster Topology Generator enables a fully automated solution when combined with SLURM’s strigger command. You can set up a trigger that runs whenever a node goes down or comes up:

$
strigger --set --node --down --up --flags=perm --program=<script>

In this setup, the <script> would contain the curl command to call the endpoint:

$
curl -s -X POST -H "Content-Type: application/json" -d @payload.json http://localhost:49021/v1/generate

We provide scripts/create-topology-update-script.sh in the repository, which performs the steps outlined above: it creates the topology update script and registers it with the strigger.

The script accepts the following parameters:

• provider name (aws, gcp, oci, nebius, netq, nscale, lambdai, or infiniband-bm)
• path to the generated topology update script
• path to the topology.conf file

Usage:

$
create-topology-update-script.sh -p <provider name> -s <topology update script> -c <path to topology.conf>

Example:

$
create-topology-update-script.sh -p aws -s /etc/slurm/update-topology-config.sh -c /etc/slurm/topology.conf

This automation ensures that your cluster topology is updated and SLURM configuration is reloaded whenever there are changes in node status, maintaining an up-to-date cluster configuration.