Testing, Performance, and Troubleshooting#

Running the Test Suite#

The test suite is the primary way to verify that a code change or environment update has not broken the pipeline.

# Full suite: unit tests + integration smoke tests (2 GPUs)
bash /workspace/mnmg/run_tests.sh

# Unit tests only — no GPU required (CI-friendly)
bash /workspace/mnmg/run_tests.sh --skip-integration

# Single integration test by label
bash /workspace/mnmg/run_tests.sh --only medium_ngpu

# Override GPU count for the multi-GPU test
bash /workspace/mnmg/run_tests.sh --gpus 4

Test Catalogue#

Label	Scope	What it validates	GPUs
`unit-tests`	pytest	Data discovery, partitioning logic, boosting utilities, GNN model forward pass, config schema validation	0 (CPU)
`small_1gpu`	Integration	Full GNN + XGBoost pipeline on a 500-account / 100-merchant / 5,000-edge synthetic graph	1
`medium_ngpu`	Integration	Full pipeline on a 2,000-account / 500-merchant / 20,000-edge synthetic graph	N (default: 2)
`enc_sage`	Integration	End-to-end with GraphSAGE encoder	1
`enc_gat`	Integration	End-to-end with Graph Attention encoder	1
`enc_transformer`	Integration	End-to-end with Graph Transformer encoder	1
`enc_general`	Integration	End-to-end with GeneralConv encoder	1
`test_gap1_streaming_write`	pytest	Streaming pre-allocation in `extract_and_save_node_embeddings` produces identical results to accumulate-then-cat pattern	0 (CPU)
`test_gap2_chunked_merge`	pytest	Chunked shard loading in `merge_embeddings.merge_node_type` produces identical mmap to full-load path across dtype/chunk-size combinations	0 (CPU)
`test_gap3_sorted_lookup`	pytest	Sorted memmap access in `client_production._lookup` and both LRU cache classes returns correct results for all ID distributions	0 (CPU)
`test_gap4_100m_scale`	pytest	Scale-readiness invariants: no in-RAM full-graph allocation, sorted lookup present, on-the-fly merge trigger, mmap disk size correctness	0 (CPU)
`test_path3_provided_subgraph`	pytest	`_build_subgraph_from_provided` — external subgraph injection for novel-node inference: local index remapping, feature slicing, edge remapping	0 (CPU)

Integration tests generate synthetic data with a controlled 2% fraud rate, run a full training pipeline, and verify that the run completes without error and produces a non-trivial PR-AUC score.

Reading the Test Output#

========================================
 FraudFNN test suite
 GPUs              : 2
 Only              : <all>
 Skip integration  : 0
========================================
...
PASS  unit-tests
PASS  small_1gpu
PASS  medium_ngpu
PASS  enc_sage
PASS  enc_gat
PASS  enc_transformer
PASS  enc_general
========================================
 Results: 7 passed, 0 failed
========================================

If any test fails, the script prints FAIL <label> and exits with a non-zero status code (useful for CI systems).

Running Only the Multi-GPU Test#

bash /workspace/mnmg/run_tests.sh --only medium_ngpu

This skips all other tests (unit tests, encoder sweep, and the 1-GPU smoke test) and runs only the 2-GPU integration test.

Performance Tuning#

GNN Training Throughput#

Lever	Effect	Direction
Increase `batch_size`	More edges per mini-batch; higher GPU utilisation	Increase until OOM, then back off
Increase `num_neighbors`	Larger neighbourhood per edge; more informative but slower	Increase only if accuracy needs improvement
Reduce `num_neighbors`	Faster sampling; less informative	Use if training is too slow
Increase `num_gpus`	Near-linear scaling for large datasets	Add GPUs; ensure `--shm-size` is sufficient
Reduce `num_gnn_layers`	Fewer aggregation passes per step; faster iteration	Try 1 layer for very dense graphs

XGBoost Training Throughput#

Lever	Effect
Increase `num_gpus`	Splits computed across all GPUs; scales well
Increase `batch_size` (batched mode)	Fewer DataIter callbacks per round; higher GPU utilisation
Reduce `max_depth`	Fewer node evaluations per tree; faster rounds
Use `subsample < 1.0`	Fewer rows to process per round; also acts as regularisation

RAM-constrained environments: When the training edge embedding matrix exceeds available system RAM, switch to disk-backed modes. See Section 5.6 — XGBoost Memory Modes for the full memmap / extmem tradeoff table and when to use each mode.

Inference Throughput#

Lever	Effect
Increase client `--batch_size`	Amortises per-request overhead over more edges; throughput improves up to GPU saturation
Use `--warmup_batches 1`	First batch incurs CUDA JIT and torch.jit.trace overhead; exclude it from latency measurements
Scale Triton horizontally	Run multiple Triton servers behind a load balancer for higher aggregate throughput
Increase `INSTANCES_PER_GPU` when calling `serve.sh`	Multiple concurrent model instances handle parallel requests; `serve.sh` patches all `config.pbtxt` files atomically — do not edit them manually as `serve.sh` overwrites them on every run

CUDAGraph and dynamic subgraph sizes:

gnn_embedder uses torch.compile(mode="reduce-overhead"), which normally records CUDAGraphs to eliminate per-step CUDA API overhead. Because each inference request contains a different subgraph (different node/edge counts), tensor shapes vary across requests. The GNN backends set:

torch._inductor.config.triton.cudagraph_skip_dynamic_graphs = True

This skips CUDAGraph recording for requests with dynamic shapes and falls back to regular fused Triton kernel dispatch instead — avoiding the overhead of re-recording a new graph for each distinct shape while still running compiled kernels. Without this flag, the Triton server logs warnings about dynamic shapes and re-records graphs on every new shape, which adds latency and memory churn.

Shared Memory#

NCCL and WholeGraph both use /dev/shm extensively. If you see NCCL errors or memory allocation failures, the default /dev/shm size (64 MB on most systems) is too small.

Always include --shm-size=10g in your docker run command (shown in the examples in Section 5.1):

docker run --shm-size=10g --gpus all ...

To clean up leftover shared memory segments from a crashed run:

ipcs -m | tail -n +4 | awk '{print $2}' | xargs -r ipcrm -m

Troubleshooting#

Training Errors#

`[entrypoint] ERROR: config requests N GPU(s) but only M is/are visible.`#

Cause: num_gpus in the config is greater than the number of GPUs exposed to the container. The entrypoint validates GPU availability before launching torchrun and exits with a clear message rather than a cryptic CUDA error.

Fix — option A: Pass -e NPROC=<M> to override the GPU count without editing the config:

docker run ... -e NPROC=1 nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

Fix — option B: Update num_gpus in your config to match the available hardware:

models:
  - kind: GNN_XGBoost
    gpu: single        # for 1 GPU
    # num_gpus omitted — defaults to 1

Fix — option C: Expose more GPUs by changing --gpus "device=0" to --gpus all or --gpus "device=0,1".

Training exits immediately after “Partitioning complete” with no epoch output#

Cause: batch_size is larger than the number of labeled edges (EP) or labeled nodes (NP) on each GPU rank. The trainer pre-checks edge count before creating the DataLoader and exits using os._exit(1) with an ERROR log:

ERROR — Batch size (2048) exceeds the number of training edges on this rank (175). With drop_last=True all batches would be dropped.
  Total train edges : 350  (across 2 GPU ranks)
  Edges this rank   : 175
  GNN batch_size    : 2048
Fix: reduce gnn.batch_size to < 175, or increase the dataset so that
  n_labeled_edges × train_frac / num_gpus > batch_size
  i.e. ab_edges > 5851

Fix — option A: Reduce batch_size in the config to be less than floor(n_labeled_edges × train_frac / num_gpus), where train_frac is the fraction of labeled edges in your train split (0.90 when no masks are provided; whatever your train_mask covers otherwise). Fix — option B: Increase the dataset so that n_labeled_edges × train_frac / num_gpus > batch_size. For 2 GPUs and batch_size: 2048, you need at least ceil(2048 × 2 / train_frac) labeled edges — e.g. ≈ 4552 with the default 90/10 split, or ≈ 5852 if your train mask covers 70% of edges.

`DATA VALIDATION FAILED — training cannot start`#

Cause: Pre-flight data validation could not find a required label file before launching any GPU processes. The location it checks depends on the prediction kind:

EP (kind: GNN_XGBoost) — scans edges/ for a file whose stem ends with _label
NP (kind: GNN_XGBoost_NP) — scans nodes/ for a file whose stem ends with _label

Fix for EP:

Check that a file named {src}_{rel}_{dst}_label.{ext} exists in edges/.
Verify the stem matches the main connectivity file exactly: account_transacts_merchant.csv → label file must be account_transacts_merchant_label.{ext} (any supported extension: .csv, .parquet, .orc).
Ensure the file is in the edges/ subdirectory, not in the data root.
Confirm only one label file exists across all edge types — validation also fails if more than one is present.

Fix for NP:

Check that a file named {node_type}_label.{ext} exists in nodes/ (e.g. account_label.csv).
Verify the stem prefix matches the target node type exactly: account.csv → label file must be account_label.{ext}.
Ensure the file is in the nodes/ subdirectory, not in the data root.
Confirm only one node type has a label file — NP does not support multiple labeled node types simultaneously.

`ValueError: num_neighbors length (N) must equal num_gnn_layers (M)`#

Cause: num_neighbors has a different number of elements than num_gnn_layers.

Fix: Always provide exactly num_gnn_layers values in the num_neighbors list:

gnn:
  num_gnn_layers: 3
  num_neighbors: [25, 15, 10]   # 3 values for 3 layers

`ValueError: extmem=true requires batched=true` / `ValueError: memmap=true requires batched=true`#

Cause: extmem or memmap was enabled without enabling batched first. Fix:

xgb:
  batched: true    # required
  extmem: true     # now valid

`CUDA out of memory` during GNN training#

Cause: The product of batch_size × num_neighbors[0] × num_neighbors[1] × hidden_channels exceeds GPU VRAM. Fix (try in order):

Reduce batch_size by half (e.g., 1024 → 512 → 256).
Reduce num_neighbors (e.g., [25, 10] → [15, 5]).
Reduce hidden_channels (e.g., 128 → 64).
Add another GPU (num_gpus: 2 distributes the graph and the mini-batch work).

`CUDA out of memory` during GNN embedding extraction#

Cause: inf_batch_size is too large; the full neighbourhood of all edges does not fit in one pass. Fix: Explicitly set a smaller inf_batch_size:

gnn:
  inf_batch_size: 128   # default is batch_size // 8; override if still OOM

`CUDA out of memory` during XGBoost training#

Cause: All edge embeddings are loaded into GPU memory at once (default in-memory mode). Fix: Enable batched mode:

xgb:
  batched: true
  batch_size: 65536

For very large datasets, add memmap: true to stream embeddings from disk.

`RuntimeError: NCCL error` or `Timeout waiting for process group`#

Cause: Inter-process communication failure. Common causes: insufficient shared memory, stale NCCL sockets, firewall blocking high-numbered ports between container and host. Fix:

Clear stale shared memory from a previous crashed run:

ipcs -m | tail -n +4 | awk '{print $2}' | xargs -r ipcrm -m

Verify Docker --shm-size=10g is in the run command.
Confirm NCCL_P2P_DISABLE=1 is set (already the default in the training container; override with -e NCCL_P2P_DISABLE=0 on clusters with NVLink P2P support).
If running inside Kubernetes or a VM, check that the container network allows inter-process TCP on ephemeral ports.

Stage 8 (Triton export) timeout specifically:

If the timeout occurs after the training log shows GNN model saved or Test metrics appended, the failure is in the Triton export step (Stage 8), not in NCCL itself. The pipeline broadcasts any Stage 8 exception to all ranks and calls a distributed barrier before re-raising, so the full error message will appear on rank 0 before the watchdog fires. Check the rank-0 log for:

ERROR — Stage 8 (Triton export) failed: <reason>

Common causes: output_dir not writable by the container user (see PermissionError on config.pbtxt), disk full during model file copy, or a missing Python dependency.

`UserWarning: This script should be run with 'torchrun'. Exiting.`#

Cause: train.py was invoked with python directly. The distributed process group was never initialised. Fix: Always use the training container, which internally uses torchrun:

docker run --rm -it --gpus all \
  --shm-size=10g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /your/data:/data:ro \
  -v /your/output:/workspace/output \
  -v /your/config.yaml:/workspace/config.yaml:ro \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

`[validate] N validation error(s) in 'config.yaml'`#

Cause: The config file does not conform to the expected schema. The container validates the config before launching GPU processes specifically to surface this error cleanly. Fix: Read the validation output — it lists which field failed and why:

ERROR — [validate] 2 validation error(s) in 'config.yaml':
ERROR — [1] models -> 0 -> hyperparameters -> gnn -> encoder: Input should be 'sage', 'gat', 'transformer' or 'general'
ERROR — [2] models -> 0 -> num_gpus: Field required when gpu='multi'

Correct each flagged field in your config file and retry.

Training completes but `output/metrics.json` is missing#

Cause: The XGBoost step was skipped (skip: true) or an error occurred during evaluation. The GNN may have trained successfully but the XGBoost stage did not complete. Fix:

Check output/training_*.log for ERROR lines in the XGBoost section.
Confirm xgb.skip is false (or not set) in your config.
Verify that output/xgboost_fraud.json exists — if it does not, the booster was never saved, indicating an XGBoost training failure.

Triton Server / Deployment Errors#

Server exits immediately with `Failed to load model`#

Cause A: MODEL_REPO does not point to the correct directory.

Fix: Verify the repository contains the required files:

# EP pipeline — all of these must exist
ls ${MODEL_REPO}/gnn_embedder/1/model_final.pt
ls ${MODEL_REPO}/gnn_embedder/1/model_config.json
ls ${MODEL_REPO}/xgb_fraud/1/xgboost.json
ls ${MODEL_REPO}/fraud_pipeline/config.pbtxt

NP pipeline: Replace gnn_embedder → np_gnn_embedder, xgb_fraud → xgb_fraud_np, fraud_pipeline → np_fraud_pipeline.

Cause B: Training did not complete successfully — some artifact files are missing.

Fix: Re-run training and verify it reaches the “Exporting Triton artifacts” step in the log.

`ModuleNotFoundError: No module named 'torch_geometric'` in Triton logs#

Cause: The Python dependency installation step in serve.sh failed or was bypassed. Fix:

If using serve.sh, check for pip errors at the start of the container log:

ERROR: Could not find a version that satisfies the requirement torch_geometric

Ensure the container has internet access during startup.
If the container has no internet access, pre-install packages into each backend’s own model directory. Each model.py inserts only its own directory into sys.path, so packages installed in gnn_embedder/1/ are invisible to xgb_explainer and llm_explainer:
```
# GNN embedder backend
pip install torch torch_geometric \
  --target ${MODEL_REPO}/gnn_embedder/1/ --quiet

# XGBoost explainer backend
pip install xgboost captum \
  --target ${MODEL_REPO}/xgb_explainer/1/ --quiet

# LLM explainer backend
pip install openai \
  --target ${MODEL_REPO}/llm_explainer/1/ --quiet
```
For NP models replace gnn_embedder → np_gnn_embedder, xgb_explainer → xgb_explainer_np, llm_explainer → llm_explainer_np.

Triton starts but `fraud_pipeline` is not ready#

Cause: An ensemble model only loads if all its component models loaded successfully. If gnn_embedder or xgb_fraud failed to load, the ensemble will show "state":"UNAVAILABLE". Fix:

Check individual model status:

curl -s http://localhost:8000/v2/models/gnn_embedder/ready
curl -s http://localhost:8000/v2/models/xgb_fraud/ready

Find the root failure in the server log: search for failed to initialize model.

`serve.sh` fails with `PermissionError` on `config.pbtxt`#

Fix: Copy the output directory to a user-owned location, or fix ownership in place:

# Option A — copy to a user-owned location (durable fix; survives re-runs):
cp -r /path/to/output/ ~/output_copied/
NUM_GPUS=2 INSTANCES_PER_GPU=3 MODEL_REPO=~/output_copied/infer/triton/ \
  bash ~/output_copied/infer/serve.sh

# Option B — fix in place (not durable — reverts to root on the next docker run):
sudo chown -R $USER /path/to/output/

`[llm_explainer] LLM connectivity check failed`#

Cause: The LLM API key is set but the connection to the endpoint failed. Possible reasons and fixes:

Reason	Fix
Invalid API key	Regenerate the key; test it independently with a direct curl call
Wrong base URL	Verify `LLM_BASE_URL` exactly matches the provider’s expected format
Wrong model name	The model name must match exactly; check the provider’s model list
Rate limit hit at startup	Rare; restart the server
Network policy blocks outbound HTTPS	Allow outbound port 443 from the container

The server continues running even if this check fails; only the LLM explanation feature is disabled.

Inference Client Errors#

`grpc._channel._InactiveRpcError: StatusCode.UNAVAILABLE`#

Cause: The gRPC client cannot reach Triton on port 8001. Fix:

Verify the server is running: curl http://localhost:8000/v2/health/live
Check the port mapping: docker ps | grep triton → confirm 0.0.0.0:8001->8001/tcp is listed.
If the server is on a remote host, pass --url <host-ip>:8001.
Verify no firewall is blocking port 8001.

`grpc.StatusCode.INVALID_ARGUMENT` during inference#

Cause: The input tensor shapes sent by the client do not match the shapes expected by the Triton model config. Most common cause: The node feature dimensions in your test data do not match those used during training (e.g., training used 13 account features, but the test file has a different number of columns). Fix:

Verify column counts match between training and test node files.

Inspect model_config.json for the expected feature dimensions (replace gnn_embedder with np_gnn_embedder for NP pipeline):

python3 -c "import json; c=json.load(open('output/infer/triton/gnn_embedder/1/model_config.json')); print(c['num_features'])"

Compare with your actual test node file dimensions (replace account.csv with your node type file):

python3 -c "import pandas as pd; print(pd.read_csv('/data/test_gnn/nodes/account.csv').shape)"

Client Classifications Are Unexpected or Threshold Seems Wrong#

Cause: The --model_config path points to a model_config.json from a different training run than the model currently loaded in Triton, so the stored threshold does not match the model. Fix: Always use the model_config.json that was generated alongside the currently served model:

python client.py \
  --model_config /workspace/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /data/test_gnn

The effective threshold is printed at client startup — verify it matches your expectations before running a full batch.

LLM Explanations Are Empty Strings#

Cause	Indicator	Fix
No API key at server start	Triton log: `no API key set — LLM explanation disabled`	Stop server, `export NVIDIA_API_KEY=...`, restart
LLM connectivity failed at startup	Triton log: `LLM connectivity/auth check failed`	Fix the API key / URL, restart
Using `fraud_pipeline` instead of `fraud_pipeline_explained`	Client shows no “Pass 2” output	Add `--explain` to the client command
`--explain_limit 0`	No edges sent to explainer	Increase `--explain_limit`

High Inference Latency#

Symptom	Likely Cause	Fix
First 1–2 batches very slow, then stable	CUDA kernel warm-up, PyTorch JIT, torch_geometric first-run compilation	Use `--warmup_batches 2`; exclude warm-up from measurements
All batches uniformly slow (> 50ms for 512 edges)	Batch size too small; GPU is underutilised	Increase `--batch_size` (try 1024, 2048)
Latency grows over many batches	GPU memory fragmentation or thermal throttling	Check `nvidia-smi`; restart the Triton container
Explanation latency > 5 s per edge	LLM API rate limiting or slow endpoint	Reduce `--explain_limit`; use a local LLM; upgrade API tier
Latency spikes periodically	Triton background health checks interfering	This is normal; use p95/p99 rather than max for capacity planning

Data Issues#

`IndexError: index N is out of bounds for axis 0 with size M` during training#

Cause: An edge file references a node index that exceeds the number of rows in the corresponding node file (i.e., the graph references node IDs that don’t exist).

Fix: Run the validation script from Section 3.6:

Replace filenames with your own node and edge type names:

import pandas as pd
nodes = pd.read_csv("nodes/account.csv")          # replace with your source node file
edges = pd.read_csv("edges/account_transacts_merchant.csv")  # replace with your edge file
assert edges.iloc[:, 0].max() < len(nodes), \
    f"Edge src max={edges.iloc[:, 0].max()}, but only {len(nodes)} nodes exist"

Common cause: node IDs in the original data were 1-based rather than 0-based. Subtract 1 from all IDs.

Model Accuracy Is Poor (PR-AUC < 0.7)#

Work through the following checklist in order.

1. Check for label leakage. If you computed per-account fraud statistics (like fraud_rate) and used them as node features, the model may have learned to memorise those statistics rather than generalise. Compute account-level features from training transactions only; do not expose aggregate fraud rates directly.

2. Check feature normalisation. Features with very different scales dominate the GNN’s message aggregation. All continuous features should be standardised (zero mean, unit variance) or min-max normalised to [0, 1].

3. Check fraud rate. If your training fraud rate is very low, or if your production fraud rate differs significantly from training, configure the three calibration parameters — see Section 5.9 Class Imbalance and Calibration for the full workflow. At minimum, set base_score to your training fraud rate and scale_pos_weight to auto:

xgb:
  scale_pos_weight: null   # auto-computed as (1-fraud_rate)/fraud_rate
  base_score: 0.001        # set to your TRAINING fraud rate (e.g. 0.001 = 0.1%)

If your production environment has a different fraud rate than training (e.g. training at 10% but deploying at 0.03%), add prior_test to apply a Bayes prior correction. This shifts predicted probabilities to reflect the production class ratio and then recalibrates the decision threshold on the adjusted scores — both probability values and the threshold in metrics.json are affected:

xgb:
  base_score: 0.10         # training fraud rate
  prior_test: 0.0003       # expected production fraud rate

For very sparse fraud (< 1%), also try focal loss in the GNN stage to focus learning on hard examples:

gnn:
  focal_gamma: 1.0         # typical range 0.5–2.0
  focal_alpha: 0.25        # optional: weight the fraud class in (0, 1)

4. Check temporal split. If test_gnn/ was created by random sampling rather than a temporal split, fraud rings that appear in both training and test will inflate test metrics. Recreate the split using a date cutoff.

5. Check graph connectivity. If your graph is very sparse (average node degree < 2), neighbourhood aggregation has little to aggregate. In this case, only_raw_node_features_in_embedding: true (a feature-concatenation baseline) may outperform the full GNN.

6. Increase model capacity. As a last resort, try:

hidden_channels: 256
num_gnn_layers: 3 with num_neighbors: [25, 15, 10]
encoder: gat with heads: 8
num_boost_round: 500 with learning_rate: 0.05

Testing, Performance, and Troubleshooting#

Running the Test Suite#

Test Catalogue#

Reading the Test Output#

Running Only the Multi-GPU Test#

Performance Tuning#

GNN Training Throughput#

XGBoost Training Throughput#

Inference Throughput#

Shared Memory#

Troubleshooting#

Training Errors#

[entrypoint] ERROR: config requests N GPU(s) but only M is/are visible.#

Training exits immediately after “Partitioning complete” with no epoch output#

DATA VALIDATION FAILED — training cannot start#

ValueError: num_neighbors length (N) must equal num_gnn_layers (M)#

ValueError: extmem=true requires batched=true / ValueError: memmap=true requires batched=true#

CUDA out of memory during GNN training#

CUDA out of memory during GNN embedding extraction#

CUDA out of memory during XGBoost training#

RuntimeError: NCCL error or Timeout waiting for process group#

UserWarning: This script should be run with 'torchrun'. Exiting.#

[validate] N validation error(s) in 'config.yaml'#

Training completes but output/metrics.json is missing#