> This page is for version 3.0.0 (Latest) (default). > For other versions, use one of these documentation indexes: > - 3.0.0 (Latest) (default): https://docs.nvidia.com/nim/financial-fraud-training/3.0.0/llms.txt > - 2.0.0: https://docs.nvidia.com/nim/financial-fraud-training/2.0.0/llms.txt > - 1.0.0: https://docs.nvidia.com/nim/financial-fraud-training/1.0.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nim/financial-fraud-training/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/financial-fraud-training/_mcp/server. # Sending Inference Requests Install client dependencies first: ```bash pip install "tritonclient[grpc]" pandas numpy torch ``` ## 7.1 Standard Fraud Scoring — CLI Client The client script is written to your output directory at the end of training: ``` /path/to/output/infer/client.py ``` Run it with the `model_config.json` from your training output and your test data directory: ```bash python /path/to/output/infer/client.py \ --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \ --data_path /path/to/your/test_gnn \ --batch_size 512 ``` The `--data_path` directory must have the same `nodes/` and `edges/` layout as your training data with identical filenames and feature column counts. Requirements: - **Node CSVs:** same filenames and column order as `training_data/nodes/`. Feature count must match exactly. - **Edge CSVs:** same filenames and column order as `training_data/edges/`. All edge types (including context edges like `B_to_C.csv`) must be present for k-hop subgraph sampling to work correctly — not just the predicted edge type. - **Label CSV:** required (e.g. `edges/account_transacts_merchant_label.csv`). The client computes Precision, Recall, and F1 against ground truth labels. - **NP:** all rows in the label node CSV are scored as inference targets; no mask file is needed in `test_gnn/`. The required `test_gnn/` directory structure is described in [Section 3.1](/nim/financial-fraud-training/data/data-def#31-top-level-directory-structure). **What the client does, step by step:** 1. Reads `model_config.json` to learn the graph schema (node types, feature dimensions, edge type name, decision threshold, `num_neighbors`). 2. Loads all node and edge CSVs from `--data_path` into memory. 3. Builds adjacency structures for client-side BFS subgraph sampling (when `num_neighbors` is present in `model_config.json`). 4. For each batch of query edges: samples a subgraph using BFS from both endpoints of each query edge (using the trained `num_neighbors` hop counts), then sends the sampled subgraph plus the batch endpoints to Triton's `fraud_pipeline` ensemble using gRPC. With `--no_sampling`, sends the full graph as context instead. 5. Receives a `[N, 1]` float32 tensor containing P(fraud) for each query edge. 6. Thresholds at `val_threshold` from `model_config.json`. 7. Aggregates predictions, computes precision/recall/F1 against ground truth labels. 8. Prints a latency summary and sample fraud detections. **Sample output:** ``` Loading test data from /data/test_gnn ... account: 500 nodes | merchant: 100 nodes edges: 5000 | fraud: 73 (1.46%) edge_attr: (5000, 17) Pass 1/1: scoring 10 batches (batch_size=512) via fraud_pipeline ... [ 5/10] 2560/5000 edges last_batch=11.8ms [ 10/10] 5000/5000 edges last_batch=12.1ms Pass 1 latency summary (10 measured batches, batch_size=512): Total wall : 118.4 ms (42231 edges/s) Per-batch : mean=11.8ms p50=11.6ms p95=13.4ms p99=14.1ms max=14.1ms ================================================== Results (threshold=0.7651) ================================================== Total edges : 5000 TP=64 FP=8 FN=9 TN=4919 Precision : 0.8889 Recall : 0.8767 F1 : 0.8828 ================================================== Sample detected fraud edges (first 10 of 72): edge src dst P(fraud) label 142 23 71 0.9812 1 37 5 44 0.9701 1 ... ``` **All CLI flags:** | Flag | Default | Description | |---|---|---| | `--model_config` | _(required)_ | Path to `model_config.json` generated at training time. Located at `/infer/triton/gnn_embedder/1/model_config.json`. | | `--data_path` | _(required)_ | Path to the test data directory containing `nodes/` and `edges/` subdirectories. | | `--url` | `localhost:8001` | Triton gRPC endpoint. Format: `host:port`. | | `--batch_size` | 512 | Number of query edges per gRPC request. Larger batches reduce per-edge overhead but increase per-request latency. | | `--warmup_batches` | 0 | Send this many batches before starting latency measurements. Use 1–3 to warm up CUDA kernels and JIT compilation. | | `--explain` | false | Enable LLM explanation mode (two-pass). See [Section 7.2](#72-llm-explainability-mode). | | `--explain_limit` | 10 | Maximum fraud edges to explain in pass 2. Edges are sorted by descending fraud probability. | | `--microbatch_delay_ms` | 0 | Simulate a micro-batch gateway. See [Section 7.3](#73-micro-batch-gateway-simulation). | | `--no_sampling` | false | Disable client-side subgraph sampling. Sends the full graph as context on every request. Use when `num_neighbors` is absent from `model_config.json`, or to reproduce training-time behaviour at the cost of larger payloads. | --- ## 7.1b Node Prediction Client (`GNN_XGBoost_NP`) When you trained with `kind: GNN_XGBoost_NP`, use `client_np.py` instead of `client.py`. It targets the `np_fraud_pipeline` ensemble and expects `model_config_np.json`: ```bash python /path/to/output/infer/client_np.py \ --model_config /path/to/output/infer/triton/np_gnn_embedder/1/model_config_np.json \ --data_path /path/to/your/test_gnn \ --batch_size 512 ``` **Key differences from the EP client:** | Aspect | EP (`client.py`) | NP (`client_np.py`) | |---|---|---| | Query type | Edges (transactions) | Nodes (accounts or merchants) | | Triton model | `fraud_pipeline` | `np_fraud_pipeline` | | Model config | `gnn_embedder/1/model_config.json` | `np_gnn_embedder/1/model_config_np.json` | | Per-request batch identifier | `edge_label_src` / `edge_label_dst` | `seed_node_ids` (LOCAL 0-based subgraph indices) | | Subgraph sampling | BFS from both endpoints of each query edge | BFS from seed nodes (`NeighborLoader` semantics) | **CLI flags** are identical: `--url`, `--model_config`, `--data_path`, `--batch_size`, `--warmup_batches`, `--explain`, `--explain_limit`, `--microbatch_delay_ms`, `--no_sampling`. ### `client_saved_emb.py` — Hybrid Saved-Embedding Client Use this client when training was run with `save_node_embeddings: true`. It routes each edge through a fast memmap lookup for known nodes and falls back to the GNN for novel nodes. ```bash python /infer/client_saved_emb.py \ --model_config /infer/triton/gnn_embedder/1/model_config.json \ --data_path /data/test_gnn \ # optional: inferred from model_config --emb_dir /node_embeddings \ --batch_size 512 \ --explain --explain_limit 10 ``` **Example output:** Unlike `client.py`, this client prints separate metrics for the KNOWN path (memmap) and NOVEL path (GNN fallback): ``` ================================================== Results — ALL edges (25803 edges, threshold=0.9457) ================================================== TP=1509 FP=8 FN=578 TN=23708 Precision : 0.9947 Recall : 0.7230 F1 : 0.8374 ================================================== ================================================== Results — KNOWN nodes (saved-embedding path) (18803 edges, threshold=0.9457) ================================================== TP=1099 FP=4 FN=423 TN=17277 Precision : 0.9964 Recall : 0.7222 F1 : 0.8372 ================================================== ================================================== Results — NOVEL nodes (on-the-fly GNN path) (7000 edges, threshold=0.9457) ================================================== TP=410 FP=4 FN=155 TN=6431 Precision : 0.9904 Recall : 0.7255 F1 : 0.8382 ================================================== ``` A cold-start warning fires when the novel-node percentage exceeds 10%: `⚠ >10% novel nodes — consider re-training to refresh embeddings.` **Key CLI flags:** | Flag | Default | Description | |---|---|---| | `--emb_dir` | from model_config | Node embeddings directory (can be omitted if `node_embeddings_dir` is in model_config.json) | | `--batch_size` | 512 | Edges per inference batch | | `--no_sampling` | false | Skip k-hop BFS subgraph sampling for novel nodes; use 0-hop (raw features only) | | `--lru_cache_size` | null | Wrap embedding arrays in an LRU dict — keeps the N hottest nodes in Python RAM to avoid SSD page faults on repeated lookups (each cache hit copies the cached row into the result array) | | `--threshold` | from model_config | Override decision threshold | | `--explain` | false | Run explain pass on detected fraud edges | | `--explain_limit` | 10 | Max edges to explain | ### `client_production.py` — Production HTTP Service and Concurrent Batch Eval `client_production.py` extends the hybrid routing of `client_saved_emb.py` with two additional capabilities: - **Concurrent batch eval**: dispatches Triton requests in parallel across multiple model instances using `ThreadPoolExecutor` - **Persistent HTTP service** (`--serve`): opens memmaps once at startup and serves requests indefinitely **Batch eval mode (no `--serve`):** ```bash python /infer/client_production.py \ --model_config /infer/triton/gnn_embedder/1/model_config.json \ --data_path /data/test_gnn \ --emb_dir /node_embeddings \ --batch_size 512 \ --concurrency 6 # NUM_GPUS × INSTANCES_PER_GPU ``` Reports the same per-path metrics (ALL / KNOWN / NOVEL) as `client_saved_emb.py`, plus latency percentiles per batch. **Persistent HTTP service (`--serve`):** ```bash pip install fastapi uvicorn # one-time install python /infer/client_production.py \ --model_config /infer/triton/gnn_embedder/1/model_config.json \ --emb_dir /node_embeddings \ --data_path /data/test_gnn \ --serve \ --host 0.0.0.0 --port 8080 \ --concurrency 6 \ --lru_cache_size 100000 ``` At startup, `embeddings.mmap` and `features.mmap` are opened as `np.memmap(mode='r')` — zero bytes copied into RAM. The OS page-cache handles demand paging. Multiple server processes can open the same file and share the underlying pages. **Endpoints:** | Endpoint | Method | Description | |---|---|---| | `GET /health` | GET | Liveness check + LRU hit-rate stats per node type | | `POST /score` | POST | Fraud probability per edge. Known nodes: memmap → XGBoost (fast path). Novel nodes: GNN through Triton (fallback). | | `POST /explain` | POST | Fraud probability + per-feature SVS attribution scores + LLM text explanation | **POST /score request:** ```json { "src_ids": [42000000, 7000000], "dst_ids": [5000000, 3000000], "edge_attr": [[100.0, 14.0], [50.0, 8.0]], "explain": true, "neighbors": [[42000000, 5000000, "account_transacts_merchant"], [42000000, 9000001, "account_transacts_merchant"]] } ``` `neighbors` is an optional pre-built k-hop subgraph for novel nodes (one entry per directed edge: `[src_global_id, dst_global_id, edge_type_stem]`). Populate this from a live graph database query when novel nodes are expected. Omit for 0-hop fallback (raw features, no message passing). **POST /score response:** ```json { "fraud_prob": [0.923, 0.041], "is_fraud": [true, false], "path": ["memmap", "gnn"], "explanations": ["High-value transaction at unusual hour...", null] } ``` `"path"` per edge is `"memmap"` (known node fast path) or `"gnn"` (novel node fallback). **POST /explain** — same request body as `/score`. Always runs `xgb_explainer` (SVS Shapley-like attribution) and `llm_explainer`. Returns `"svs"` dict (feature name → score, positive pushes toward fraud) plus `"explanations"` text. Add `"no_llm": true` to return SVS only, skipping the LLM call. **Key CLI flags:** | Flag | Default | Description | |---|---|---| | `--data_path` | **required** | Path to test data directory (same layout as training `test_gnn/`). Required by argparse even in `--serve` mode. | | `--serve` | false | Start persistent HTTP service; without this flag runs one-shot batch eval | | `--host` | `0.0.0.0` | Bind host for `--serve` | | `--port` | 8080 | Bind port for `--serve` | | `--concurrency` | `num_gpus × instances_per_gpu` | Parallel Triton gRPC channels | | `--lru_cache_size` | null | Hot-node LRU cache per node type (number of nodes to keep in Python RAM) | | `--no_sampling` | false | Skip BFS subgraph sampling for novel nodes | | `--triton_wait` | 300 | Seconds to wait for Triton readiness before exiting | | `--emb_dir` | from model_config | Node embeddings directory | --- ## 7.2 LLM Explainability Mode Add `--explain` to run a two-pass workflow. Pass 1 scores all edges with `fraud_pipeline`. Pass 2 sends the top detected fraud edges (sorted by descending probability) individually to `fraud_pipeline_explained`. ```bash python /path/to/output/infer/client.py \ --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \ --data_path /path/to/your/test_gnn \ --batch_size 512 \ --explain \ --explain_limit 10 ``` **Pass 2 workflow (per fraud edge):** 1. `gnn_embedder`: re-computes the edge embedding (same as pass 1). 2. `xgb_explainer`: runs XGBoost inference + Captum Shapley Value Sampling. Outputs the fraud probability, the raw edge embedding, and a `svs_attrs` vector of per-feature-group attributions. 3. `llm_explainer`: assembles the top-8 feature groups (by absolute SVS attribution) into a structured prompt, calls the LLM API, and returns a plain-language explanation string. **Typical explanation output:** ``` ================================================== LLM Explanations (first 10 detected fraud edges) ================================================== [Edge 142] account=23 merchant=71 fraud_prob=0.9812 label=1 latency=847ms This transaction carries a high fraud risk (probability 98%). The three strongest signals are: (1) the GNN's account embedding deviates significantly from the account's typical peer group, suggesting recent account compromise; (2) the transaction amount ($4,823.50) is more than 14× the account's average; (3) the merchant has an elevated chargeback rate (3.1%) relative to its category peers. Together, these factors are consistent with card-not-present fraud following an account takeover event. [Edge 37] account=5 merchant=44 fraud_prob=0.9701 label=1 latency=612ms ... ``` **What the SVS Shapley Values represent:** Shapley values answer the question: "how much did each feature group contribute to changing the prediction from the baseline (average legitimate transaction) to this specific prediction?" A positive value means the feature pushed the model toward predicting fraud; a negative value means it pulled toward legitimate. The LLM sees the top-8 groups and their human-readable descriptions (e.g., "transaction amount = $4,823.50"). **Production guidance for explanations:** - Explanations are generated using an external API call per fraud edge. At peak loads, use `--explain_limit` to cap the number of explanations and avoid rate limits. - The LLM model choice affects explanation quality significantly. A larger model (70B parameters) produces more nuanced, accurate reasoning but at higher latency and cost. - Explanations are advisory only. The fraud decision is made by XGBoost; the LLM only narrates what the model already decided. --- ## 7.3 Micro-Batch Gateway Simulation Financial systems often cannot score transactions one by one; they queue incoming transactions and flush when a maximum batch size or maximum wait time is reached. Use `--microbatch_delay_ms` to simulate this gateway model and measure end-to-end per-transaction latency. ```bash python /path/to/output/infer/client.py \ --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \ --data_path /path/to/your/test_gnn \ --batch_size 256 \ --microbatch_delay_ms 50 \ --warmup_batches 2 ``` This simulates: transactions arrive continuously; the gateway flushes when 256 edges accumulate or 50 ms elapses. **Sample output:** ``` Micro-batch gateway simulation (max_delay=50ms, max_batch=256, warmup=2) ... Micro-batch simulation results (5000 transactions, batch_size≤256, delay=50ms): Batch RTT : mean=9.4ms p50=9.1ms p95=11.2ms max=12.8ms Per-tx lat : mean=34.7ms p50=34.2ms p95=59.4ms p99=61.1ms max=62.8ms (avg queue wait ≈ 25.0ms with uniform arrivals) ``` In this example: the batch RTT is ~9.4 ms, but each transaction also waits in the queue for up to 50 ms. The average end-to-end latency per transaction is ~35 ms. To reduce this, either reduce `microbatch_delay_ms` (accept smaller batches and flush more often) or increase `batch_size` (larger batches amortise the RTT across more transactions). --- ## 7.4 Programmatic Python Client For integration into your own Python application (rather than using the CLI), use `tritonclient` directly. The example below sends the **full graph** as context on every request (equivalent to `--no_sampling`). This is simple but does not scale to production-sized graphs. For production use, reproduce the BFS subgraph sampling from `client.py`'s `_sample_ep_subgraph` function, which limits each request to the sampled neighbourhood of the query edges. ```python import json import numpy as np import pandas as pd import tritonclient.grpc as grpcclient # ── Load the model config written by training ────────────────────────── with open("/workspace/output/infer/triton/gnn_embedder/1/model_config.json") as f: cfg = json.load(f) src_type = cfg["edge_type_to_predict"][0] # "account" rel = cfg["edge_type_to_predict"][1] # "transacts" dst_type = cfg["edge_type_to_predict"][2] # "merchant" edge_attr_dim = cfg["edge_attr_dim"] threshold = cfg["val_threshold"] # ── Load your test data ──────────────────────────────────────────────── node_dir = "/data/test_gnn/nodes" edge_dir = "/data/test_gnn/edges" src_x = pd.read_csv(f"{node_dir}/{src_type}.csv").values.astype(np.float32) dst_x = pd.read_csv(f"{node_dir}/{dst_type}.csv").values.astype(np.float32) stem = f"{src_type}_{rel}_{dst_type}" edges_df = pd.read_csv(f"{edge_dir}/{stem}.csv") edge_src = edges_df.iloc[:, 0].values.astype(np.int64) edge_dst = edges_df.iloc[:, 1].values.astype(np.int64) edge_attr = pd.read_csv(f"{edge_dir}/{stem}_attr.csv").values.astype(np.float32) \ if edge_attr_dim > 0 else None # ── Connect to Triton ────────────────────────────────────────────────── client = grpcclient.InferenceServerClient(url="localhost:8001", verbose=False) # ── Score one batch of query edges ──────────────────────────────────── def score_batch(query_src_idx, query_dst_idx, query_attr=None): """ Score a batch of edges. Returns fraud probabilities as a 1D numpy array. query_src_idx : int64 array [B] — source node indices of query edges query_dst_idx : int64 array [B] — dest node indices of query edges query_attr : float32 array [B, A] or None """ inputs_data = { f"{src_type}_x": src_x, # full context: all source nodes f"{dst_type}_x": dst_x, # full context: all dest nodes f"{src_type}_{rel}_{dst_type}_src": edge_src, # full context: all edges (for GNN neighbourhood) f"{src_type}_{rel}_{dst_type}_dst": edge_dst, f"{dst_type}_rev_{rel}_{src_type}_src": edge_dst, # reverse edges f"{dst_type}_rev_{rel}_{src_type}_dst": edge_src, "edge_label_src": query_src_idx, # query: which edges to score? "edge_label_dst": query_dst_idx, } if query_attr is not None: inputs_data["edge_attr"] = query_attr type_map = {np.float32: "FP32", np.int64: "INT64"} triton_inputs = [] for name, arr in inputs_data.items(): inp = grpcclient.InferInput(name, list(arr.shape), type_map[arr.dtype.type]) inp.set_data_from_numpy(arr) triton_inputs.append(inp) response = client.infer( model_name="fraud_pipeline", inputs=triton_inputs, outputs=[grpcclient.InferRequestedOutput("fraud_probability")], ) probs = response.as_numpy("fraud_probability") # [B, 1] return probs[:, 0] # P(fraud) # ── Example: score first 512 edges ───────────────────────────────────── batch_size = 512 batch_probs = score_batch( edge_src[:batch_size], edge_dst[:batch_size], edge_attr[:batch_size] if edge_attr is not None else None, ) fraud_flags = (batch_probs >= threshold).astype(int) print(f"Detected {fraud_flags.sum()} fraud edges out of {batch_size}") ``` --- ## 7.5 REST API Using curl Triton also exposes an HTTP REST API on port 8000, which can be useful for quick testing without a Python environment. The REST API requires base64-encoded binary tensors for float and int arrays. For production integration, the gRPC client (port 8001) is strongly preferred for efficiency and correctness. The curl approach below is suitable for debugging only. ```bash # Check if a specific model is ready curl -s http://localhost:8000/v2/models/fraud_pipeline/ready # Get model metadata (input/output tensor names and shapes) curl -s http://localhost:8000/v2/models/fraud_pipeline | python3 -m json.tool ``` --- ## 7.6 Interpreting Results and Setting Thresholds ### The Fraud Probability The output of `fraud_pipeline` is a `[N, 1]` float32 tensor: - Column 0: P(fraud) — probability this edge is fraudulent. P(legitimate) = 1 − P(fraud) if needed. ### Using the Pre-Calibrated Threshold The pipeline selects a decision threshold during training by maximising the F1 score on the validation set. This threshold is stored in `model_config.json` as `val_threshold` and used automatically by `client.py`. For most deployments, this threshold is a good starting point. ### Adjusting the Threshold for Your Business Requirements The optimal threshold depends on the relative cost of: - **False positives (FP):** Legitimate transactions flagged as fraud. Customer friction, manual review cost. - **False negatives (FN):** Fraudulent transactions missed. Direct financial loss + potential regulatory penalties. To evaluate different thresholds without retraining: ```python import json import numpy as np # Load fraud probabilities produced by client.py # (modify client.py to save all_probs and all_labels to a file, or score programmatically) fraud_probs = np.load("fraud_probs.npy") # [N] labels = np.load("labels.npy") # [N] thresholds = np.arange(0.1, 0.99, 0.01) results = [] for t in thresholds: preds = (fraud_probs >= t).astype(int) tp = int(((preds==1) & (labels==1)).sum()) fp = int(((preds==1) & (labels==0)).sum()) fn = int(((preds==0) & (labels==1)).sum()) precision = tp / (tp + fp) if (tp + fp) > 0 else 0 recall = tp / (tp + fn) if (tp + fn) > 0 else 0 f1 = 2*precision*recall / (precision+recall) if (precision+recall) > 0 else 0 results.append({"threshold": round(t, 2), "precision": round(precision, 4), "recall": round(recall, 4), "f1": round(f1, 4), "tp": tp, "fp": fp, "fn": fn}) # Print the Precision-Recall trade-off table print(f"{'Threshold':>10} {'Precision':>10} {'Recall':>8} {'F1':>8} {'TP':>6} {'FP':>6} {'FN':>6}") for r in results: print(f"{r['threshold']:>10.2f} {r['precision']:>10.4f} {r['recall']:>8.4f} " f"{r['f1']:>8.4f} {r['tp']:>6} {r['fp']:>6} {r['fn']:>6}") ``` Once you have chosen a new threshold, update it in the deployment by modifying `model_config.json` in the model repository and restarting (or hot-reloading) the `gnn_embedder` model in Triton: ```bash # Update threshold in place python3 -c " import json path = '/workspace/output/infer/triton/gnn_embedder/1/model_config.json' with open(path) as f: cfg = json.load(f) cfg['val_threshold'] = 0.82 # your new threshold with open(path, 'w') as f: json.dump(cfg, f, indent=2) print('Updated threshold to', cfg['val_threshold']) " # Triton hot-reload: unload and reload the gnn_embedder model curl -X POST http://localhost:8000/v2/repository/models/gnn_embedder/unload curl -X POST http://localhost:8000/v2/repository/models/gnn_embedder/load ```