> This page is for version 3.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 3.0.0 (Latest) (default): https://docs.nvidia.com/nim/financial-fraud-training/3.0.0/llms.txt
> - 2.0.0: https://docs.nvidia.com/nim/financial-fraud-training/2.0.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/financial-fraud-training/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/financial-fraud-training/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/financial-fraud-training/_mcp/server.

# Testing, Performance, and Troubleshooting

## Running the Test Suite

The test suite is the primary way to verify that a code change or environment update has not broken the pipeline.

```bash
# Full suite: unit tests + integration smoke tests (2 GPUs)
bash /workspace/mnmg/run_tests.sh

# Unit tests only — no GPU required (CI-friendly)
bash /workspace/mnmg/run_tests.sh --skip-integration

# Single integration test by label
bash /workspace/mnmg/run_tests.sh --only medium_ngpu

# Override GPU count for the multi-GPU test
bash /workspace/mnmg/run_tests.sh --gpus 4
```

### Test Catalogue

| Label | Scope | What it validates | GPUs |
|---|---|---|---|
| `unit-tests` | pytest | Data discovery, partitioning logic, boosting utilities, GNN model forward pass, config schema validation | 0 (CPU) |
| `small_1gpu` | Integration | Full GNN + XGBoost pipeline on a 500-account / 100-merchant / 5,000-edge synthetic graph | 1 |
| `medium_ngpu` | Integration | Full pipeline on a 2,000-account / 500-merchant / 20,000-edge synthetic graph | N (default: 2) |
| `enc_sage` | Integration | End-to-end with GraphSAGE encoder | 1 |
| `enc_gat` | Integration | End-to-end with Graph Attention encoder | 1 |
| `enc_transformer` | Integration | End-to-end with Graph Transformer encoder | 1 |
| `enc_general` | Integration | End-to-end with GeneralConv encoder | 1 |
| `test_gap1_streaming_write` | pytest | Streaming pre-allocation in `extract_and_save_node_embeddings` produces identical results to accumulate-then-cat pattern | 0 (CPU) |
| `test_gap2_chunked_merge` | pytest | Chunked shard loading in `merge_embeddings.merge_node_type` produces identical mmap to full-load path across dtype/chunk-size combinations | 0 (CPU) |
| `test_gap3_sorted_lookup` | pytest | Sorted memmap access in `client_production._lookup` and both LRU cache classes returns correct results for all ID distributions | 0 (CPU) |
| `test_gap4_100m_scale` | pytest | Scale-readiness invariants: no in-RAM full-graph allocation, sorted lookup present, on-the-fly merge trigger, mmap disk size correctness | 0 (CPU) |
| `test_path3_provided_subgraph` | pytest | `_build_subgraph_from_provided` — external subgraph injection for novel-node inference: local index remapping, feature slicing, edge remapping | 0 (CPU) |

Integration tests generate synthetic data with a controlled 2% fraud rate, run a full training pipeline, and verify that the run completes without error and produces a non-trivial PR-AUC score.

### Reading the Test Output

```
========================================
 FraudFNN test suite
 GPUs              : 2
 Only              : <all>
 Skip integration  : 0
========================================
...
PASS  unit-tests
PASS  small_1gpu
PASS  medium_ngpu
PASS  enc_sage
PASS  enc_gat
PASS  enc_transformer
PASS  enc_general
========================================
 Results: 7 passed, 0 failed
========================================
```

If any test fails, the script prints `FAIL  <label>` and exits with a non-zero status code (useful for CI systems).

### Running Only the Multi-GPU Test

```bash
bash /workspace/mnmg/run_tests.sh --only medium_ngpu
```

This skips all other tests (unit tests, encoder sweep, and the 1-GPU smoke test) and runs only the 2-GPU integration test.

---

## Performance Tuning

### GNN Training Throughput

| Lever | Effect | Direction |
|---|---|---|
| Increase `batch_size` | More edges per mini-batch; higher GPU utilisation | Increase until OOM, then back off |
| Increase `num_neighbors` | Larger neighbourhood per edge; more informative but slower | Increase only if accuracy needs improvement |
| Reduce `num_neighbors` | Faster sampling; less informative | Use if training is too slow |
| Increase `num_gpus` | Near-linear scaling for large datasets | Add GPUs; ensure `--shm-size` is sufficient |
| Reduce `num_gnn_layers` | Fewer aggregation passes per step; faster iteration | Try 1 layer for very dense graphs |

### XGBoost Training Throughput

| Lever | Effect |
|---|---|
| Increase `num_gpus` | Splits computed across all GPUs; scales well |
| Increase `batch_size` (batched mode) | Fewer DataIter callbacks per round; higher GPU utilisation |
| Reduce `max_depth` | Fewer node evaluations per tree; faster rounds |
| Use `subsample < 1.0` | Fewer rows to process per round; also acts as regularisation |

**RAM-constrained environments:** When the training edge embedding matrix exceeds available system RAM, switch to disk-backed modes. See [Section 5.6 — XGBoost Memory Modes](/nim/financial-fraud-training/training/advanced#xgboost-memory-modes) for the full `memmap` / `extmem` tradeoff table and when to use each mode.

### Inference Throughput

| Lever | Effect |
|---|---|
| Increase client `--batch_size` | Amortises per-request overhead over more edges; throughput improves up to GPU saturation |
| Use `--warmup_batches 1` | First batch incurs CUDA JIT and torch.jit.trace overhead; exclude it from latency measurements |
| Scale Triton horizontally | Run multiple Triton servers behind a load balancer for higher aggregate throughput |
| Increase `INSTANCES_PER_GPU` when calling `serve.sh` | Multiple concurrent model instances handle parallel requests; `serve.sh` patches all `config.pbtxt` files atomically — do not edit them manually as `serve.sh` overwrites them on every run |

**CUDAGraph and dynamic subgraph sizes:**

`gnn_embedder` uses `torch.compile(mode="reduce-overhead")`, which normally records CUDAGraphs to eliminate per-step CUDA API overhead. Because each inference request contains a different subgraph (different node/edge counts), tensor shapes vary across requests. The GNN backends set:

```python
torch._inductor.config.triton.cudagraph_skip_dynamic_graphs = True
```

This skips CUDAGraph recording for requests with dynamic shapes and falls back to regular fused Triton kernel dispatch instead — avoiding the overhead of re-recording a new graph for each distinct shape while still running compiled kernels. Without this flag, the Triton server logs warnings about dynamic shapes and re-records graphs on every new shape, which adds latency and memory churn.

### Shared Memory

NCCL and WholeGraph both use `/dev/shm` extensively. If you see NCCL errors or memory allocation failures, the default `/dev/shm` size (64 MB on most systems) is too small.

Always include `--shm-size=10g` in your `docker run` command (shown in the examples in [Section 5.1](/nim/financial-fraud-training/training/training-guide#51-quick-start--cli)):

```bash
docker run --shm-size=10g --gpus all ...
```

To clean up leftover shared memory segments from a crashed run:

```bash
ipcs -m | tail -n +4 | awk '{print $2}' | xargs -r ipcrm -m
```

---

## Troubleshooting

### Training Errors

#### `[entrypoint] ERROR: config requests N GPU(s) but only M is/are visible.`

**Cause:** `num_gpus` in the config is greater than the number of GPUs exposed to the container. The entrypoint validates GPU availability before launching `torchrun` and exits with a clear message rather than a cryptic CUDA error.

**Fix — option A:** Pass `-e NPROC=<M>` to override the GPU count without editing the config:

```bash
docker run ... -e NPROC=1 nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0
```

**Fix — option B:** Update `num_gpus` in your config to match the available hardware:

```yaml
models:
  - kind: GNN_XGBoost
    gpu: single        # for 1 GPU
    # num_gpus omitted — defaults to 1
```

**Fix — option C:** Expose more GPUs by changing `--gpus "device=0"` to `--gpus all` or `--gpus "device=0,1"`.

---

#### Training exits immediately after "Partitioning complete" with no epoch output

**Cause:** `batch_size` is larger than the number of labeled edges (EP) or labeled nodes (NP) on each GPU rank. The trainer pre-checks edge count before creating the DataLoader and exits using `os._exit(1)` with an ERROR log:

```
ERROR — Batch size (2048) exceeds the number of training edges on this rank (175). With drop_last=True all batches would be dropped.
  Total train edges : 350  (across 2 GPU ranks)
  Edges this rank   : 175
  GNN batch_size    : 2048
Fix: reduce gnn.batch_size to < 175, or increase the dataset so that
  n_labeled_edges × train_frac / num_gpus > batch_size
  i.e. ab_edges > 5851
```

**Fix — option A:** Reduce `batch_size` in the config to be less than `floor(n_labeled_edges × train_frac / num_gpus)`, where `train_frac` is the fraction of labeled edges in your train split (0.90 when no masks are provided; whatever your `train_mask` covers otherwise).
**Fix — option B:** Increase the dataset so that `n_labeled_edges × train_frac / num_gpus > batch_size`. For 2 GPUs and `batch_size: 2048`, you need at least `ceil(2048 × 2 / train_frac)` labeled edges — e.g. ≈ 4552 with the default 90/10 split, or ≈ 5852 if your train mask covers 70% of edges.

---

#### `DATA VALIDATION FAILED — training cannot start`

**Cause:** Pre-flight data validation could not find a required label file before launching any GPU processes. The location it checks depends on the prediction kind:

- **EP (`kind: GNN_XGBoost`)** — scans `edges/` for a file whose stem ends with `_label`
- **NP (`kind: GNN_XGBoost_NP`)** — scans `nodes/` for a file whose stem ends with `_label`

**Fix for EP:**
1. Check that a file named `{src}_{rel}_{dst}_label.{ext}` exists in `edges/`.
2. Verify the stem matches the main connectivity file exactly: `account_transacts_merchant.csv` → label file must be `account_transacts_merchant_label.{ext}` (any supported extension: `.csv`, `.parquet`, `.orc`).
3. Ensure the file is in the `edges/` subdirectory, not in the data root.
4. Confirm only **one** label file exists across all edge types — validation also fails if more than one is present.

**Fix for NP:**
1. Check that a file named `{node_type}_label.{ext}` exists in `nodes/` (e.g. `account_label.csv`).
2. Verify the stem prefix matches the target node type exactly: `account.csv` → label file must be `account_label.{ext}`.
3. Ensure the file is in the `nodes/` subdirectory, not in the data root.
4. Confirm only **one** node type has a label file — NP does not support multiple labeled node types simultaneously.

---

#### `ValueError: num_neighbors length (N) must equal num_gnn_layers (M)`

**Cause:** `num_neighbors` has a different number of elements than `num_gnn_layers`.

**Fix:** Always provide exactly `num_gnn_layers` values in the `num_neighbors` list:

```yaml
gnn:
  num_gnn_layers: 3
  num_neighbors: [25, 15, 10]   # 3 values for 3 layers
```

---

#### `ValueError: extmem=true requires batched=true` / `ValueError: memmap=true requires batched=true`

**Cause:** `extmem` or `memmap` was enabled without enabling `batched` first.
**Fix:**

```yaml
xgb:
  batched: true    # required
  extmem: true     # now valid
```

---

#### `CUDA out of memory` during GNN training

**Cause:** The product of `batch_size × num_neighbors[0] × num_neighbors[1] × hidden_channels` exceeds GPU VRAM.
**Fix (try in order):**
1. Reduce `batch_size` by half (e.g., 1024 → 512 → 256).
2. Reduce `num_neighbors` (e.g., `[25, 10]` → `[15, 5]`).
3. Reduce `hidden_channels` (e.g., 128 → 64).
4. Add another GPU (`num_gpus: 2` distributes the graph and the mini-batch work).

---

#### `CUDA out of memory` during GNN embedding extraction

**Cause:** `inf_batch_size` is too large; the full neighbourhood of all edges does not fit in one pass.
**Fix:** Explicitly set a smaller `inf_batch_size`:

```yaml
gnn:
  inf_batch_size: 128   # default is batch_size // 8; override if still OOM
```

---

#### `CUDA out of memory` during XGBoost training

**Cause:** All edge embeddings are loaded into GPU memory at once (default in-memory mode).
**Fix:** Enable batched mode:

```yaml
xgb:
  batched: true
  batch_size: 65536
```

For very large datasets, add `memmap: true` to stream embeddings from disk.

---

#### `RuntimeError: NCCL error` or `Timeout waiting for process group`

**Cause:** Inter-process communication failure. Common causes: insufficient shared memory, stale NCCL sockets, firewall blocking high-numbered ports between container and host.
**Fix:**
1. Clear stale shared memory from a previous crashed run:
   ```bash
   ipcs -m | tail -n +4 | awk '{print $2}' | xargs -r ipcrm -m
   ```
2. Verify Docker `--shm-size=10g` is in the run command.
3. Confirm `NCCL_P2P_DISABLE=1` is set (already the default in the training container; override with `-e NCCL_P2P_DISABLE=0` on clusters with NVLink P2P support).
4. If running inside Kubernetes or a VM, check that the container network allows inter-process TCP on ephemeral ports.

**Stage 8 (Triton export) timeout specifically:**

If the timeout occurs after the training log shows `GNN model saved` or `Test metrics appended`, the failure is in the Triton export step (Stage 8), not in NCCL itself. The pipeline broadcasts any Stage 8 exception to all ranks and calls a distributed barrier before re-raising, so the full error message will appear on rank 0 before the watchdog fires. Check the rank-0 log for:

```
ERROR — Stage 8 (Triton export) failed: <reason>
```

Common causes: `output_dir` not writable by the container user (see [PermissionError on config.pbtxt](#servesh-fails-with-permissionerror-on-configpbtxt)), disk full during model file copy, or a missing Python dependency.

---

#### `UserWarning: This script should be run with 'torchrun'. Exiting.`

**Cause:** `train.py` was invoked with `python` directly. The distributed process group was never initialised.
**Fix:** Always use the training container, which internally uses `torchrun`:

```bash
docker run --rm -it --gpus all \
  --shm-size=10g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /your/data:/data:ro \
  -v /your/output:/workspace/output \
  -v /your/config.yaml:/workspace/config.yaml:ro \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0
```

---

#### `[validate] N validation error(s) in 'config.yaml'`

**Cause:** The config file does not conform to the expected schema. The container validates the config before launching GPU processes specifically to surface this error cleanly.
**Fix:** Read the validation output — it lists which field failed and why:

```
ERROR — [validate] 2 validation error(s) in 'config.yaml':
ERROR — [1] models -> 0 -> hyperparameters -> gnn -> encoder: Input should be 'sage', 'gat', 'transformer' or 'general'
ERROR — [2] models -> 0 -> num_gpus: Field required when gpu='multi'
```

Correct each flagged field in your config file and retry.

---

#### Training completes but `output/metrics.json` is missing

**Cause:** The XGBoost step was skipped (`skip: true`) or an error occurred during evaluation. The GNN may have trained successfully but the XGBoost stage did not complete.
**Fix:**
1. Check `output/training_*.log` for `ERROR` lines in the XGBoost section.
2. Confirm `xgb.skip` is `false` (or not set) in your config.
3. Verify that `output/xgboost_fraud.json` exists — if it does not, the booster was never saved, indicating an XGBoost training failure.

---

### Triton Server / Deployment Errors

#### Server exits immediately with `Failed to load model`

**Cause A:** `MODEL_REPO` does not point to the correct directory.

**Fix:** Verify the repository contains the required files:

```bash
# EP pipeline — all of these must exist
ls ${MODEL_REPO}/gnn_embedder/1/model_final.pt
ls ${MODEL_REPO}/gnn_embedder/1/model_config.json
ls ${MODEL_REPO}/xgb_fraud/1/xgboost.json
ls ${MODEL_REPO}/fraud_pipeline/config.pbtxt
```

> **NP pipeline:** Replace `gnn_embedder` → `np_gnn_embedder`, `xgb_fraud` → `xgb_fraud_np`, `fraud_pipeline` → `np_fraud_pipeline`.

**Cause B:** Training did not complete successfully — some artifact files are missing.

**Fix:** Re-run training and verify it reaches the "Exporting Triton artifacts" step in the log.

---

#### `ModuleNotFoundError: No module named 'torch_geometric'` in Triton logs

**Cause:** The Python dependency installation step in `serve.sh` failed or was bypassed.
**Fix:**

1. If using `serve.sh`, check for pip errors at the start of the container log:
   ```
   ERROR: Could not find a version that satisfies the requirement torch_geometric
   ```
2. Ensure the container has internet access during startup.
3. If the container has no internet access, pre-install packages into each backend's own model directory. Each `model.py` inserts only its own directory into `sys.path`, so packages installed in `gnn_embedder/1/` are invisible to `xgb_explainer` and `llm_explainer`:
   ```bash
   # GNN embedder backend
   pip install torch torch_geometric \
     --target ${MODEL_REPO}/gnn_embedder/1/ --quiet

   # XGBoost explainer backend
   pip install xgboost captum \
     --target ${MODEL_REPO}/xgb_explainer/1/ --quiet

   # LLM explainer backend
   pip install openai \
     --target ${MODEL_REPO}/llm_explainer/1/ --quiet
   ```
   For NP models replace `gnn_embedder` → `np_gnn_embedder`, `xgb_explainer` → `xgb_explainer_np`, `llm_explainer` → `llm_explainer_np`.

---

#### Triton starts but `fraud_pipeline` is not ready

**Cause:** An ensemble model only loads if all its component models loaded successfully. If `gnn_embedder` or `xgb_fraud` failed to load, the ensemble will show `"state":"UNAVAILABLE"`.
**Fix:**
1. Check individual model status:
   ```bash
   curl -s http://localhost:8000/v2/models/gnn_embedder/ready
   curl -s http://localhost:8000/v2/models/xgb_fraud/ready
   ```
2. Find the root failure in the server log: search for `failed to initialize model`.

---

#### `serve.sh` fails with `PermissionError` on `config.pbtxt`

**Fix:** Copy the output directory to a user-owned location, or fix ownership in place:

```bash
# Option A — copy to a user-owned location (durable fix; survives re-runs):
cp -r /path/to/output/ ~/output_copied/
NUM_GPUS=2 INSTANCES_PER_GPU=3 MODEL_REPO=~/output_copied/infer/triton/ \
  bash ~/output_copied/infer/serve.sh

# Option B — fix in place (not durable — reverts to root on the next docker run):
sudo chown -R $USER /path/to/output/
```

---

#### `[llm_explainer] LLM connectivity check failed`

**Cause:** The LLM API key is set but the connection to the endpoint failed.
**Possible reasons and fixes:**

| Reason | Fix |
|---|---|
| Invalid API key | Regenerate the key; test it independently with a direct curl call |
| Wrong base URL | Verify `LLM_BASE_URL` exactly matches the provider's expected format |
| Wrong model name | The model name must match exactly; check the provider's model list |
| Rate limit hit at startup | Rare; restart the server |
| Network policy blocks outbound HTTPS | Allow outbound port 443 from the container |

The server continues running even if this check fails; only the LLM explanation feature is disabled.

---

### Inference Client Errors

#### `grpc._channel._InactiveRpcError: StatusCode.UNAVAILABLE`

**Cause:** The gRPC client cannot reach Triton on port 8001.
**Fix:**
1. Verify the server is running: `curl http://localhost:8000/v2/health/live`
2. Check the port mapping: `docker ps | grep triton` → confirm `0.0.0.0:8001->8001/tcp` is listed.
3. If the server is on a remote host, pass `--url <host-ip>:8001`.
4. Verify no firewall is blocking port 8001.

---

#### `grpc.StatusCode.INVALID_ARGUMENT` during inference

**Cause:** The input tensor shapes sent by the client do not match the shapes expected by the Triton model config.
**Most common cause:** The node feature dimensions in your test data do not match those used during training (e.g., training used 13 account features, but the test file has a different number of columns).
**Fix:**
1. Verify column counts match between training and test node files.
2. Inspect `model_config.json` for the expected feature dimensions (replace `gnn_embedder` with `np_gnn_embedder` for NP pipeline):
   ```bash
   python3 -c "import json; c=json.load(open('output/infer/triton/gnn_embedder/1/model_config.json')); print(c['num_features'])"
   ```
3. Compare with your actual test node file dimensions (replace `account.csv` with your node type file):
   ```bash
   python3 -c "import pandas as pd; print(pd.read_csv('/data/test_gnn/nodes/account.csv').shape)"
   ```

---

#### Client Classifications Are Unexpected or Threshold Seems Wrong

**Cause:** The `--model_config` path points to a `model_config.json` from a different training run than the model currently loaded in Triton, so the stored threshold does not match the model.
**Fix:** Always use the `model_config.json` that was generated alongside the currently served model:

```bash
python client.py \
  --model_config /workspace/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /data/test_gnn
```

The effective threshold is printed at client startup — verify it matches your expectations before running a full batch.

---

#### LLM Explanations Are Empty Strings

| Cause | Indicator | Fix |
|---|---|---|
| No API key at server start | Triton log: `no API key set — LLM explanation disabled` | Stop server, `export NVIDIA_API_KEY=...`, restart |
| LLM connectivity failed at startup | Triton log: `LLM connectivity/auth check failed` | Fix the API key / URL, restart |
| Using `fraud_pipeline` instead of `fraud_pipeline_explained` | Client shows no "Pass 2" output | Add `--explain` to the client command |
| `--explain_limit 0` | No edges sent to explainer | Increase `--explain_limit` |

---

#### High Inference Latency

| Symptom | Likely Cause | Fix |
|---|---|---|
| First 1–2 batches very slow, then stable | CUDA kernel warm-up, PyTorch JIT, torch_geometric first-run compilation | Use `--warmup_batches 2`; exclude warm-up from measurements |
| All batches uniformly slow (> 50ms for 512 edges) | Batch size too small; GPU is underutilised | Increase `--batch_size` (try 1024, 2048) |
| Latency grows over many batches | GPU memory fragmentation or thermal throttling | Check `nvidia-smi`; restart the Triton container |
| Explanation latency > 5 s per edge | LLM API rate limiting or slow endpoint | Reduce `--explain_limit`; use a local LLM; upgrade API tier |
| Latency spikes periodically | Triton background health checks interfering | This is normal; use p95/p99 rather than max for capacity planning |

---

### Data Issues

#### `IndexError: index N is out of bounds for axis 0 with size M` during training

**Cause:** An edge file references a node index that exceeds the number of rows in the corresponding node file (i.e., the graph references node IDs that don't exist).

**Fix:** Run the validation script from [Section 3.6](/nim/financial-fraud-training/data/data-def#36-pre-training-data-validation-checklist):

Replace filenames with your own node and edge type names:

```python
import pandas as pd
nodes = pd.read_csv("nodes/account.csv")          # replace with your source node file
edges = pd.read_csv("edges/account_transacts_merchant.csv")  # replace with your edge file
assert edges.iloc[:, 0].max() < len(nodes), \
    f"Edge src max={edges.iloc[:, 0].max()}, but only {len(nodes)} nodes exist"
```

Common cause: node IDs in the original data were 1-based rather than 0-based. Subtract 1 from all IDs.

---

#### Model Accuracy Is Poor (PR-AUC < 0.7)

Work through the following checklist in order.

**1. Check for label leakage.** If you computed per-account fraud statistics (like `fraud_rate`) and used them as node features, the model may have learned to memorise those statistics rather than generalise. Compute account-level features from training transactions only; do not expose aggregate fraud rates directly.

**2. Check feature normalisation.** Features with very different scales dominate the GNN's message aggregation. All continuous features should be standardised (zero mean, unit variance) or min-max normalised to [0, 1].

**3. Check fraud rate.** If your training fraud rate is very low, or if your production fraud rate differs significantly from training, configure the three calibration parameters — see [Section 5.9 Class Imbalance and Calibration](/nim/financial-fraud-training/training/advanced#class-imbalance-and-calibration) for the full workflow. At minimum, set `base_score` to your training fraud rate and `scale_pos_weight` to auto:

```yaml
xgb:
  scale_pos_weight: null   # auto-computed as (1-fraud_rate)/fraud_rate
  base_score: 0.001        # set to your TRAINING fraud rate (e.g. 0.001 = 0.1%)
```

If your production environment has a different fraud rate than training (e.g. training at 10% but deploying at 0.03%), add `prior_test` to apply a Bayes prior correction. This shifts predicted probabilities to reflect the production class ratio and then recalibrates the decision threshold on the adjusted scores — both probability values and the threshold in `metrics.json` are affected:

```yaml
xgb:
  base_score: 0.10         # training fraud rate
  prior_test: 0.0003       # expected production fraud rate
```

For very sparse fraud (< 1%), also try focal loss in the GNN stage to focus learning on hard examples:

```yaml
gnn:
  focal_gamma: 1.0         # typical range 0.5–2.0
  focal_alpha: 0.25        # optional: weight the fraud class in (0, 1)
```

**4. Check temporal split.** If `test_gnn/` was created by random sampling rather than a temporal split, fraud rings that appear in both training and test will inflate test metrics. Recreate the split using a date cutoff.

**5. Check graph connectivity.** If your graph is very sparse (average node degree < 2), neighbourhood aggregation has little to aggregate. In this case, `only_raw_node_features_in_embedding: true` (a feature-concatenation baseline) may outperform the full GNN.

**6. Increase model capacity.** As a last resort, try:
- `hidden_channels: 256`
- `num_gnn_layers: 3` with `num_neighbors: [25, 15, 10]`
- `encoder: gat` with `heads: 8`
- `num_boost_round: 500` with `learning_rate: 0.05`