Testing, Performance, and Troubleshooting

Running the Test Suite

The test suite is the primary way to verify that a code change or environment update has not broken the pipeline.

# Full suite: unit tests + integration smoke tests (2 GPUs)
bash /workspace/mnmg/run_tests.sh
# Unit tests only — no GPU required (CI-friendly)
bash /workspace/mnmg/run_tests.sh --skip-integration
# Single integration test by label
bash /workspace/mnmg/run_tests.sh --only medium_ngpu
# Override GPU count for the multi-GPU test
bash /workspace/mnmg/run_tests.sh --gpus 4

Test Catalogue

Integration tests generate synthetic data with a controlled 2% fraud rate, run a full training pipeline, and verify that the run completes without error and produces a non-trivial PR-AUC score.

Reading the Test Output

========================================
 FraudFNN test suite
 GPUs              : 2
 Only              : <all>
 Skip integration  : 0
========================================
...
PASS  unit-tests
PASS  small_1gpu
PASS  medium_ngpu
PASS  enc_sage
PASS  enc_gat
PASS  enc_transformer
PASS  enc_general
========================================
 Results: 7 passed, 0 failed
========================================

If any test fails, the script prints FAIL <label> and exits with a non-zero status code (useful for CI systems).

Running Only the Multi-GPU Test

bash /workspace/mnmg/run_tests.sh --only medium_ngpu

This skips all other tests (unit tests, encoder sweep, and the 1-GPU smoke test) and runs only the 2-GPU integration test.

Performance Tuning

GNN Training Throughput

XGBoost Training Throughput

RAM-constrained environments: When the training edge embedding matrix exceeds available system RAM, switch to disk-backed modes. See Section 5.6 — XGBoost Memory Modes for the full memmap / extmem tradeoff table and when to use each mode.

Inference Throughput

CUDAGraph and dynamic subgraph sizes:

gnn_embedder uses torch.compile(mode="reduce-overhead"), which normally records CUDAGraphs to eliminate per-step CUDA API overhead. Because each inference request contains a different subgraph (different node/edge counts), tensor shapes vary across requests. The GNN backends set:

torch._inductor.config.triton.cudagraph_skip_dynamic_graphs = True

This skips CUDAGraph recording for requests with dynamic shapes and falls back to regular fused Triton kernel dispatch instead — avoiding the overhead of re-recording a new graph for each distinct shape while still running compiled kernels. Without this flag, the Triton server logs warnings about dynamic shapes and re-records graphs on every new shape, which adds latency and memory churn.

Shared Memory

NCCL and WholeGraph both use /dev/shm extensively. If you see NCCL errors or memory allocation failures, the default /dev/shm size (64 MB on most systems) is too small.

Always include --shm-size=10g in your docker run command (shown in the examples in Section 5.1):

docker run --shm-size=10g --gpus all ...

To clean up leftover shared memory segments from a crashed run:

ipcs -m | tail -n +4 | awk '{print $2}' | xargs -r ipcrm -m

Troubleshooting

Training Errors

[entrypoint] ERROR: config requests N GPU(s) but only M is/are visible.

Cause: num_gpus in the config is greater than the number of GPUs exposed to the container. The entrypoint validates GPU availability before launching torchrun and exits with a clear message rather than a cryptic CUDA error.

Fix — option A: Pass -e NPROC=<M> to override the GPU count without editing the config:

docker run ... -e NPROC=1 nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

Fix — option B: Update num_gpus in your config to match the available hardware:

models:
  - kind: GNN_XGBoost
    gpu: single        # for 1 GPU
    # num_gpus omitted — defaults to 1

Fix — option C: Expose more GPUs by changing --gpus "device=0" to --gpus all or --gpus "device=0,1".

Training exits immediately after “Partitioning complete” with no epoch output

Cause: batch_size is larger than the number of labeled edges (EP) or labeled nodes (NP) on each GPU rank. The trainer pre-checks edge count before creating the DataLoader and exits using os._exit(1) with an ERROR log:

ERROR — Batch size (2048) exceeds the number of training edges on this rank (175). With drop_last=True all batches would be dropped.
  Total train edges : 350  (across 2 GPU ranks)
  Edges this rank   : 175
  GNN batch_size    : 2048
Fix: reduce gnn.batch_size to < 175, or increase the dataset so that
  n_labeled_edges × train_frac / num_gpus > batch_size
  i.e. ab_edges > 5851

Fix — option A: Reduce batch_size in the config to be less than floor(n_labeled_edges × train_frac / num_gpus), where train_frac is the fraction of labeled edges in your train split (0.90 when no masks are provided; whatever your train_mask covers otherwise). Fix — option B: Increase the dataset so that n_labeled_edges × train_frac / num_gpus > batch_size. For 2 GPUs and batch_size: 2048, you need at least ceil(2048 × 2 / train_frac) labeled edges — e.g. ≈ 4552 with the default 90/10 split, or ≈ 5852 if your train mask covers 70% of edges.

DATA VALIDATION FAILED — training cannot start

Cause: Pre-flight data validation could not find a required label file before launching any GPU processes. The location it checks depends on the prediction kind:

• EP (kind: GNN_XGBoost) — scans edges/ for a file whose stem ends with _label
• NP (kind: GNN_XGBoost_NP) — scans nodes/ for a file whose stem ends with _label

Fix for EP:

1. Check that a file named {src}_{rel}_{dst}_label.{ext} exists in edges/.
2. Verify the stem matches the main connectivity file exactly: account_transacts_merchant.csv → label file must be account_transacts_merchant_label.{ext} (any supported extension: .csv, .parquet, .orc).
3. Ensure the file is in the edges/ subdirectory, not in the data root.
4. Confirm only one label file exists across all edge types — validation also fails if more than one is present.

Fix for NP:

1. Check that a file named {node_type}_label.{ext} exists in nodes/ (e.g. account_label.csv).
2. Verify the stem prefix matches the target node type exactly: account.csv → label file must be account_label.{ext}.
3. Ensure the file is in the nodes/ subdirectory, not in the data root.
4. Confirm only one node type has a label file — NP does not support multiple labeled node types simultaneously.

ValueError: num_neighbors length (N) must equal num_gnn_layers (M)

Cause: num_neighbors has a different number of elements than num_gnn_layers.

Fix: Always provide exactly num_gnn_layers values in the num_neighbors list:

gnn:
  num_gnn_layers: 3
  num_neighbors: [25, 15, 10]   # 3 values for 3 layers

ValueError: extmem=true requires batched=true / ValueError: memmap=true requires batched=true

Cause: extmem or memmap was enabled without enabling batched first. Fix:

xgb:
  batched: true    # required
  extmem: true     # now valid

CUDA out of memory during GNN training

Cause: The product of batch_size × num_neighbors[0] × num_neighbors[1] × hidden_channels exceeds GPU VRAM. Fix (try in order):

1. Reduce batch_size by half (e.g., 1024 → 512 → 256).
2. Reduce num_neighbors (e.g., [25, 10] → [15, 5]).
3. Reduce hidden_channels (e.g., 128 → 64).
4. Add another GPU (num_gpus: 2 distributes the graph and the mini-batch work).

CUDA out of memory during GNN embedding extraction

Cause: inf_batch_size is too large; the full neighbourhood of all edges does not fit in one pass. Fix: Explicitly set a smaller inf_batch_size:

gnn:
  inf_batch_size: 128   # default is batch_size // 8; override if still OOM

CUDA out of memory during XGBoost training

Cause: All edge embeddings are loaded into GPU memory at once (default in-memory mode). Fix: Enable batched mode:

xgb:
  batched: true
  batch_size: 65536

For very large datasets, add memmap: true to stream embeddings from disk.

RuntimeError: NCCL error or Timeout waiting for process group

Cause: Inter-process communication failure. Common causes: insufficient shared memory, stale NCCL sockets, firewall blocking high-numbered ports between container and host. Fix:

1. Clear stale shared memory from a previous crashed run:

   ipcs -m | tail -n +4 | awk '{print $2}' | xargs -r ipcrm -m
2. Verify Docker --shm-size=10g is in the run command.
3. Confirm NCCL_P2P_DISABLE=1 is set (already the default in the training container; override with -e NCCL_P2P_DISABLE=0 on clusters with NVLink P2P support).
4. If running inside Kubernetes or a VM, check that the container network allows inter-process TCP on ephemeral ports.

Stage 8 (Triton export) timeout specifically:

If the timeout occurs after the training log shows GNN model saved or Test metrics appended, the failure is in the Triton export step (Stage 8), not in NCCL itself. The pipeline broadcasts any Stage 8 exception to all ranks and calls a distributed barrier before re-raising, so the full error message will appear on rank 0 before the watchdog fires. Check the rank-0 log for:

ERROR — Stage 8 (Triton export) failed: <reason>

Common causes: output_dir not writable by the container user (see PermissionError on config.pbtxt), disk full during model file copy, or a missing Python dependency.

UserWarning: This script should be run with 'torchrun'. Exiting.

Cause: train.py was invoked with python directly. The distributed process group was never initialised. Fix: Always use the training container, which internally uses torchrun:

docker run --rm -it --gpus all \
  --shm-size=10g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /your/data:/data:ro \
  -v /your/output:/workspace/output \
  -v /your/config.yaml:/workspace/config.yaml:ro \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

[validate] N validation error(s) in 'config.yaml'

Cause: The config file does not conform to the expected schema. The container validates the config before launching GPU processes specifically to surface this error cleanly. Fix: Read the validation output — it lists which field failed and why:

ERROR — [validate] 2 validation error(s) in 'config.yaml':
ERROR — [1] models -> 0 -> hyperparameters -> gnn -> encoder: Input should be 'sage', 'gat', 'transformer' or 'general'
ERROR — [2] models -> 0 -> num_gpus: Field required when gpu='multi'

Correct each flagged field in your config file and retry.

Training completes but output/metrics.json is missing

Cause: The XGBoost step was skipped (skip: true) or an error occurred during evaluation. The GNN may have trained successfully but the XGBoost stage did not complete. Fix:

1. Check output/training_*.log for ERROR lines in the XGBoost section.
2. Confirm xgb.skip is false (or not set) in your config.
3. Verify that output/xgboost_fraud.json exists — if it does not, the booster was never saved, indicating an XGBoost training failure.

Triton Server / Deployment Errors

Server exits immediately with Failed to load model

Cause A: MODEL_REPO does not point to the correct directory.

Fix: Verify the repository contains the required files:

# EP pipeline — all of these must exist
ls ${MODEL_REPO}/gnn_embedder/1/model_final.pt
ls ${MODEL_REPO}/gnn_embedder/1/model_config.json
ls ${MODEL_REPO}/xgb_fraud/1/xgboost.json
ls ${MODEL_REPO}/fraud_pipeline/config.pbtxt

NP pipeline: Replace gnn_embedder → np_gnn_embedder, xgb_fraud → xgb_fraud_np, fraud_pipeline → np_fraud_pipeline.

Cause B: Training did not complete successfully — some artifact files are missing.

Fix: Re-run training and verify it reaches the “Exporting Triton artifacts” step in the log.

ModuleNotFoundError: No module named 'torch_geometric' in Triton logs

Cause: The Python dependency installation step in serve.sh failed or was bypassed. Fix:

1. If using serve.sh, check for pip errors at the start of the container log:

   ERROR: Could not find a version that satisfies the requirement torch_geometric
2. Ensure the container has internet access during startup.
3. If the container has no internet access, pre-install packages into each backend’s own model directory. Each model.py inserts only its own directory into sys.path, so packages installed in gnn_embedder/1/ are invisible to xgb_explainer and llm_explainer:

   # GNN embedder backend
   pip install torch torch_geometric \
     --target ${MODEL_REPO}/gnn_embedder/1/ --quiet
   # XGBoost explainer backend
   pip install xgboost captum \
     --target ${MODEL_REPO}/xgb_explainer/1/ --quiet
   # LLM explainer backend
   pip install openai \
     --target ${MODEL_REPO}/llm_explainer/1/ --quiet
   For NP models replace gnn_embedder → np_gnn_embedder, xgb_explainer → xgb_explainer_np, llm_explainer → llm_explainer_np.

Triton starts but fraud_pipeline is not ready

Cause: An ensemble model only loads if all its component models loaded successfully. If gnn_embedder or xgb_fraud failed to load, the ensemble will show "state":"UNAVAILABLE". Fix:

1. Check individual model status:

   curl -s http://localhost:8000/v2/models/gnn_embedder/ready
   curl -s http://localhost:8000/v2/models/xgb_fraud/ready
2. Find the root failure in the server log: search for failed to initialize model.

serve.sh fails with PermissionError on config.pbtxt

Fix: Copy the output directory to a user-owned location, or fix ownership in place:

# Option A — copy to a user-owned location (durable fix; survives re-runs):
cp -r /path/to/output/ ~/output_copied/
NUM_GPUS=2 INSTANCES_PER_GPU=3 MODEL_REPO=~/output_copied/infer/triton/ \
  bash ~/output_copied/infer/serve.sh
# Option B — fix in place (not durable — reverts to root on the next docker run):
sudo chown -R $USER /path/to/output/

[llm_explainer] LLM connectivity check failed

Cause: The LLM API key is set but the connection to the endpoint failed. Possible reasons and fixes:

The server continues running even if this check fails; only the LLM explanation feature is disabled.

Inference Client Errors

grpc._channel._InactiveRpcError: StatusCode.UNAVAILABLE

Cause: The gRPC client cannot reach Triton on port 8001. Fix:

1. Verify the server is running: curl http://localhost:8000/v2/health/live
2. Check the port mapping: docker ps | grep triton → confirm 0.0.0.0:8001->8001/tcp is listed.
3. If the server is on a remote host, pass --url <host-ip>:8001.
4. Verify no firewall is blocking port 8001.

grpc.StatusCode.INVALID_ARGUMENT during inference

Cause: The input tensor shapes sent by the client do not match the shapes expected by the Triton model config. Most common cause: The node feature dimensions in your test data do not match those used during training (e.g., training used 13 account features, but the test file has a different number of columns). Fix:

1. Verify column counts match between training and test node files.
2. Inspect model_config.json for the expected feature dimensions (replace gnn_embedder with np_gnn_embedder for NP pipeline):

   python3 -c "import json; c=json.load(open('output/infer/triton/gnn_embedder/1/model_config.json')); print(c['num_features'])"
3. Compare with your actual test node file dimensions (replace account.csv with your node type file):

   python3 -c "import pandas as pd; print(pd.read_csv('/data/test_gnn/nodes/account.csv').shape)"

Client Classifications Are Unexpected or Threshold Seems Wrong

Cause: The --model_config path points to a model_config.json from a different training run than the model currently loaded in Triton, so the stored threshold does not match the model. Fix: Always use the model_config.json that was generated alongside the currently served model:

python client.py \
  --model_config /workspace/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /data/test_gnn

The effective threshold is printed at client startup — verify it matches your expectations before running a full batch.

LLM Explanations Are Empty Strings

High Inference Latency

Data Issues

IndexError: index N is out of bounds for axis 0 with size M during training

Cause: An edge file references a node index that exceeds the number of rows in the corresponding node file (i.e., the graph references node IDs that don’t exist).

Fix: Run the validation script from Section 3.6:

Replace filenames with your own node and edge type names:

import pandas as pd
nodes = pd.read_csv("nodes/account.csv")          # replace with your source node file
edges = pd.read_csv("edges/account_transacts_merchant.csv")  # replace with your edge file
assert edges.iloc[:, 0].max() < len(nodes), \
    f"Edge src max={edges.iloc[:, 0].max()}, but only {len(nodes)} nodes exist"

Common cause: node IDs in the original data were 1-based rather than 0-based. Subtract 1 from all IDs.

Model Accuracy Is Poor (PR-AUC < 0.7)

Work through the following checklist in order.

1. Check for label leakage. If you computed per-account fraud statistics (like fraud_rate) and used them as node features, the model may have learned to memorise those statistics rather than generalise. Compute account-level features from training transactions only; do not expose aggregate fraud rates directly.

2. Check feature normalisation. Features with very different scales dominate the GNN’s message aggregation. All continuous features should be standardised (zero mean, unit variance) or min-max normalised to [0, 1].

3. Check fraud rate. If your training fraud rate is very low, or if your production fraud rate differs significantly from training, configure the three calibration parameters — see Section 5.9 Class Imbalance and Calibration for the full workflow. At minimum, set base_score to your training fraud rate and scale_pos_weight to auto:

xgb:
  scale_pos_weight: null   # auto-computed as (1-fraud_rate)/fraud_rate
  base_score: 0.001        # set to your TRAINING fraud rate (e.g. 0.001 = 0.1%)

If your production environment has a different fraud rate than training (e.g. training at 10% but deploying at 0.03%), add prior_test to apply a Bayes prior correction. This shifts predicted probabilities to reflect the production class ratio and then recalibrates the decision threshold on the adjusted scores — both probability values and the threshold in metrics.json are affected:

xgb:
  base_score: 0.10         # training fraud rate
  prior_test: 0.0003       # expected production fraud rate

For very sparse fraud (< 1%), also try focal loss in the GNN stage to focus learning on hard examples:

gnn:
  focal_gamma: 1.0         # typical range 0.5–2.0
  focal_alpha: 0.25        # optional: weight the fraud class in (0, 1)

4. Check temporal split. If test_gnn/ was created by random sampling rather than a temporal split, fraud rings that appear in both training and test will inflate test metrics. Recreate the split using a date cutoff.

5. Check graph connectivity. If your graph is very sparse (average node degree < 2), neighbourhood aggregation has little to aggregate. In this case, only_raw_node_features_in_embedding: true (a feature-concatenation baseline) may outperform the full GNN.

6. Increase model capacity. As a last resort, try:

• hidden_channels: 256
• num_gnn_layers: 3 with num_neighbors: [25, 15, 10]
• encoder: gat with heads: 8
• num_boost_round: 500 with learning_rate: 0.05