Adaptive Fault Isolation

When a certification run fails and multiple nodes are suspect, adaptive fault isolation bisects the node pool to identify the specific faulty node(s) with the minimum number of additional runs.

How it works

1. The controller splits the suspect node pool in half.
2. It re-runs the failing category against each half.
3. The half that fails is bisected again.
4. This repeats until a single node (or domain boundary) is isolated.

For an N-node pool this requires at most log₂(N) additional runs instead of N−1.

Enable

Set testScale: diagnose on the category you want to run in diagnose mode:

1
spec:
2
  categories:
3
    - domain: communication
4
      variant: nccl-all-reduce
5
      options:
6
        testScale: diagnose

The diagnose algorithm runs in seven stages: intra-screening, intra-screening-no-nvl, inter-screening, bisection, confirmation, cross-boundary, and complete.

Cross-boundary probing

When the fault lies at a network domain boundary (e.g. two nodes connected via a spine switch), standard bisection may not isolate it. Cross-boundary probing extends the algorithm to test node pairs that span domain boundaries.

Configuration reference coming soon.

Reading the result

After isolation the Certification status records the isolated node(s) in the ConfigMap referenced by status.categoryStatuses[].failedNodesRef. See Health Monitoring & Failed Node Attribution for how to read and act on that data.