Sending Inference Requests

Install client dependencies first:

pip install "tritonclient[grpc]" pandas numpy torch

7.1 Standard Fraud Scoring — CLI Client

The client script is written to your output directory at the end of training:

/path/to/output/infer/client.py

Run it with the model_config.json from your training output and your test data directory:

python /path/to/output/infer/client.py \
  --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 512

The --data_path directory must have the same nodes/ and edges/ layout as your training data with identical filenames and feature column counts. Requirements:

• Node CSVs: same filenames and column order as training_data/nodes/. Feature count must match exactly.
• Edge CSVs: same filenames and column order as training_data/edges/. All edge types (including context edges like B_to_C.csv) must be present for k-hop subgraph sampling to work correctly — not just the predicted edge type.
• Label CSV: required (e.g. edges/account_transacts_merchant_label.csv). The client computes Precision, Recall, and F1 against ground truth labels.
• NP: all rows in the label node CSV are scored as inference targets; no mask file is needed in test_gnn/.

The required test_gnn/ directory structure is described in Section 3.1.

What the client does, step by step:

1. Reads model_config.json to learn the graph schema (node types, feature dimensions, edge type name, decision threshold, num_neighbors).
2. Loads all node and edge CSVs from --data_path into memory.
3. Builds adjacency structures for client-side BFS subgraph sampling (when num_neighbors is present in model_config.json).
4. For each batch of query edges: samples a subgraph using BFS from both endpoints of each query edge (using the trained num_neighbors hop counts), then sends the sampled subgraph plus the batch endpoints to Triton’s fraud_pipeline ensemble using gRPC. With --no_sampling, sends the full graph as context instead.
5. Receives a [N, 1] float32 tensor containing P(fraud) for each query edge.
6. Thresholds at val_threshold from model_config.json.
7. Aggregates predictions, computes precision/recall/F1 against ground truth labels.
8. Prints a latency summary and sample fraud detections.

Sample output:

Loading test data from /data/test_gnn ...
  account: 500 nodes  |  merchant: 100 nodes
  edges: 5000  |  fraud: 73 (1.46%)
  edge_attr: (5000, 17)
Pass 1/1: scoring 10 batches (batch_size=512) via fraud_pipeline ...
  [   5/10] 2560/5000 edges  last_batch=11.8ms
  [  10/10] 5000/5000 edges  last_batch=12.1ms
Pass 1 latency summary (10 measured batches, batch_size=512):
  Total wall : 118.4 ms  (42231 edges/s)
  Per-batch  : mean=11.8ms  p50=11.6ms  p95=13.4ms  p99=14.1ms  max=14.1ms
==================================================
Results  (threshold=0.7651)
==================================================
  Total edges : 5000
  TP=64  FP=8  FN=9  TN=4919
  Precision   : 0.8889
  Recall      : 0.8767
  F1          : 0.8828
==================================================
Sample detected fraud edges (first 10 of 72):
   edge     src     dst   P(fraud)   label
    142      23      71     0.9812       1
     37       5      44     0.9701       1
    ...

All CLI flags:

7.1b Node Prediction Client (GNN_XGBoost_NP)

When you trained with kind: GNN_XGBoost_NP, use client_np.py instead of client.py. It targets the np_fraud_pipeline ensemble and expects model_config_np.json:

python /path/to/output/infer/client_np.py \
  --model_config /path/to/output/infer/triton/np_gnn_embedder/1/model_config_np.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 512

Key differences from the EP client:

CLI flags are identical: --url, --model_config, --data_path, --batch_size, --warmup_batches, --explain, --explain_limit, --microbatch_delay_ms, --no_sampling.

client_saved_emb.py — Hybrid Saved-Embedding Client

Use this client when training was run with save_node_embeddings: true. It routes each edge through a fast memmap lookup for known nodes and falls back to the GNN for novel nodes.

python <output>/infer/client_saved_emb.py \
  --model_config <output>/infer/triton/gnn_embedder/1/model_config.json \
  --data_path    /data/test_gnn \
  # optional: inferred from model_config
  --emb_dir      <output>/node_embeddings \
  --batch_size   512 \
  --explain --explain_limit 10

Example output:

Unlike client.py, this client prints separate metrics for the KNOWN path (memmap) and NOVEL path (GNN fallback):

==================================================
Results — ALL edges  (25803 edges, threshold=0.9457)
==================================================
  TP=1509  FP=8  FN=578  TN=23708
  Precision : 0.9947
  Recall    : 0.7230
  F1        : 0.8374
==================================================
==================================================
Results — KNOWN nodes (saved-embedding path)  (18803 edges, threshold=0.9457)
==================================================
  TP=1099  FP=4  FN=423  TN=17277
  Precision : 0.9964
  Recall    : 0.7222
  F1        : 0.8372
==================================================
==================================================
Results — NOVEL nodes (on-the-fly GNN path)  (7000 edges, threshold=0.9457)
==================================================
  TP=410  FP=4  FN=155  TN=6431
  Precision : 0.9904
  Recall    : 0.7255
  F1        : 0.8382
==================================================

A cold-start warning fires when the novel-node percentage exceeds 10%: ⚠ >10% novel nodes — consider re-training to refresh embeddings.

Key CLI flags:

client_production.py — Production HTTP Service and Concurrent Batch Eval

client_production.py extends the hybrid routing of client_saved_emb.py with two additional capabilities:

• Concurrent batch eval: dispatches Triton requests in parallel across multiple model instances using ThreadPoolExecutor
• Persistent HTTP service (--serve): opens memmaps once at startup and serves requests indefinitely

Batch eval mode (no --serve):

python <output>/infer/client_production.py \
  --model_config <output>/infer/triton/gnn_embedder/1/model_config.json \
  --data_path    /data/test_gnn \
  --emb_dir      <output>/node_embeddings \
  --batch_size   512 \
  --concurrency  6        # NUM_GPUS × INSTANCES_PER_GPU

Reports the same per-path metrics (ALL / KNOWN / NOVEL) as client_saved_emb.py, plus latency percentiles per batch.

Persistent HTTP service (--serve):

pip install fastapi uvicorn    # one-time install
python <output>/infer/client_production.py \
  --model_config <output>/infer/triton/gnn_embedder/1/model_config.json \
  --emb_dir      <output>/node_embeddings \
  --data_path    /data/test_gnn \
  --serve \
  --host 0.0.0.0 --port 8080 \
  --concurrency 6 \
  --lru_cache_size 100000

At startup, embeddings.mmap and features.mmap are opened as np.memmap(mode='r') — zero bytes copied into RAM. The OS page-cache handles demand paging. Multiple server processes can open the same file and share the underlying pages.

Endpoints:

POST /score request:

{
  "src_ids":   [42000000, 7000000],
  "dst_ids":   [5000000, 3000000],
  "edge_attr": [[100.0, 14.0], [50.0, 8.0]],
  "explain":   true,
  "neighbors": [[42000000, 5000000, "account_transacts_merchant"],
                [42000000, 9000001, "account_transacts_merchant"]]
}

neighbors is an optional pre-built k-hop subgraph for novel nodes (one entry per directed edge: [src_global_id, dst_global_id, edge_type_stem]). Populate this from a live graph database query when novel nodes are expected. Omit for 0-hop fallback (raw features, no message passing).

POST /score response:

{
  "fraud_prob": [0.923, 0.041],
  "is_fraud":   [true, false],
  "path":       ["memmap", "gnn"],
  "explanations": ["High-value transaction at unusual hour...", null]
}

"path" per edge is "memmap" (known node fast path) or "gnn" (novel node fallback).

POST /explain — same request body as /score. Always runs xgb_explainer (SVS Shapley-like attribution) and llm_explainer. Returns "svs" dict (feature name → score, positive pushes toward fraud) plus "explanations" text. Add "no_llm": true to return SVS only, skipping the LLM call.

Key CLI flags:

7.2 LLM Explainability Mode

Add --explain to run a two-pass workflow. Pass 1 scores all edges with fraud_pipeline. Pass 2 sends the top detected fraud edges (sorted by descending probability) individually to fraud_pipeline_explained.

python /path/to/output/infer/client.py \
  --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 512 \
  --explain \
  --explain_limit 10

Pass 2 workflow (per fraud edge):

1. gnn_embedder: re-computes the edge embedding (same as pass 1).
2. xgb_explainer: runs XGBoost inference + Captum Shapley Value Sampling. Outputs the fraud probability, the raw edge embedding, and a svs_attrs vector of per-feature-group attributions.
3. llm_explainer: assembles the top-8 feature groups (by absolute SVS attribution) into a structured prompt, calls the LLM API, and returns a plain-language explanation string.

Typical explanation output:

==================================================
LLM Explanations (first 10 detected fraud edges)
==================================================
[Edge 142]  account=23  merchant=71  fraud_prob=0.9812  label=1  latency=847ms
  This transaction carries a high fraud risk (probability 98%). The three strongest
  signals are: (1) the GNN's account embedding deviates significantly from the
  account's typical peer group, suggesting recent account compromise; (2) the
  transaction amount ($4,823.50) is more than 14× the account's average; (3)
  the merchant has an elevated chargeback rate (3.1%) relative to its category
  peers. Together, these factors are consistent with card-not-present fraud
  following an account takeover event.
[Edge 37]   account=5   merchant=44  fraud_prob=0.9701  label=1  latency=612ms
  ...

What the SVS Shapley Values represent:

Shapley values answer the question: “how much did each feature group contribute to changing the prediction from the baseline (average legitimate transaction) to this specific prediction?” A positive value means the feature pushed the model toward predicting fraud; a negative value means it pulled toward legitimate. The LLM sees the top-8 groups and their human-readable descriptions (e.g., “transaction amount = $4,823.50”).

Production guidance for explanations:

• Explanations are generated using an external API call per fraud edge. At peak loads, use --explain_limit to cap the number of explanations and avoid rate limits.
• The LLM model choice affects explanation quality significantly. A larger model (70B parameters) produces more nuanced, accurate reasoning but at higher latency and cost.
• Explanations are advisory only. The fraud decision is made by XGBoost; the LLM only narrates what the model already decided.

7.3 Micro-Batch Gateway Simulation

Financial systems often cannot score transactions one by one; they queue incoming transactions and flush when a maximum batch size or maximum wait time is reached. Use --microbatch_delay_ms to simulate this gateway model and measure end-to-end per-transaction latency.

python /path/to/output/infer/client.py \
  --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 256 \
  --microbatch_delay_ms 50 \
  --warmup_batches 2

This simulates: transactions arrive continuously; the gateway flushes when 256 edges accumulate or 50 ms elapses.

Sample output:

Micro-batch gateway simulation  (max_delay=50ms, max_batch=256, warmup=2) ...
Micro-batch simulation results  (5000 transactions, batch_size≤256, delay=50ms):
  Batch RTT  : mean=9.4ms  p50=9.1ms  p95=11.2ms  max=12.8ms
  Per-tx lat : mean=34.7ms  p50=34.2ms  p95=59.4ms  p99=61.1ms  max=62.8ms
  (avg queue wait ≈ 25.0ms with uniform arrivals)

In this example: the batch RTT is ~9.4 ms, but each transaction also waits in the queue for up to 50 ms. The average end-to-end latency per transaction is ~35 ms. To reduce this, either reduce microbatch_delay_ms (accept smaller batches and flush more often) or increase batch_size (larger batches amortise the RTT across more transactions).

7.4 Programmatic Python Client

For integration into your own Python application (rather than using the CLI), use tritonclient directly.

import json
import numpy as np
import pandas as pd
import tritonclient.grpc as grpcclient
# ── Load the model config written by training ──────────────────────────
with open("/workspace/output/infer/triton/gnn_embedder/1/model_config.json") as f:
    cfg = json.load(f)
src_type      = cfg["edge_type_to_predict"][0]   # "account"
rel           = cfg["edge_type_to_predict"][1]   # "transacts"
dst_type      = cfg["edge_type_to_predict"][2]   # "merchant"
edge_attr_dim = cfg["edge_attr_dim"]
threshold     = cfg["val_threshold"]
# ── Load your test data ────────────────────────────────────────────────
node_dir = "/data/test_gnn/nodes"
edge_dir = "/data/test_gnn/edges"
src_x     = pd.read_csv(f"{node_dir}/{src_type}.csv").values.astype(np.float32)
dst_x     = pd.read_csv(f"{node_dir}/{dst_type}.csv").values.astype(np.float32)
stem      = f"{src_type}_{rel}_{dst_type}"
edges_df  = pd.read_csv(f"{edge_dir}/{stem}.csv")
edge_src  = edges_df.iloc[:, 0].values.astype(np.int64)
edge_dst  = edges_df.iloc[:, 1].values.astype(np.int64)
edge_attr = pd.read_csv(f"{edge_dir}/{stem}_attr.csv").values.astype(np.float32) \
            if edge_attr_dim > 0 else None
# ── Connect to Triton ──────────────────────────────────────────────────
client = grpcclient.InferenceServerClient(url="localhost:8001", verbose=False)
# ── Score one batch of query edges ────────────────────────────────────
def score_batch(query_src_idx, query_dst_idx, query_attr=None):
    """
    Score a batch of edges. Returns fraud probabilities as a 1D numpy array.
    query_src_idx : int64 array [B] — source node indices of query edges
    query_dst_idx : int64 array [B] — dest node indices of query edges
    query_attr    : float32 array [B, A] or None
    """
    inputs_data = {
        f"{src_type}_x":  src_x,          # full context: all source nodes
        f"{dst_type}_x":  dst_x,          # full context: all dest nodes
        f"{src_type}_{rel}_{dst_type}_src":     edge_src,        # full context: all edges (for GNN neighbourhood)
        f"{src_type}_{rel}_{dst_type}_dst":     edge_dst,
        f"{dst_type}_rev_{rel}_{src_type}_src": edge_dst,        # reverse edges
        f"{dst_type}_rev_{rel}_{src_type}_dst": edge_src,
        "edge_label_src": query_src_idx,   # query: which edges to score?
        "edge_label_dst": query_dst_idx,
    }
    if query_attr is not None:
        inputs_data["edge_attr"] = query_attr
    type_map = {np.float32: "FP32", np.int64: "INT64"}
    triton_inputs = []
    for name, arr in inputs_data.items():
        inp = grpcclient.InferInput(name, list(arr.shape), type_map[arr.dtype.type])
        inp.set_data_from_numpy(arr)
        triton_inputs.append(inp)
    response = client.infer(
        model_name="fraud_pipeline",
        inputs=triton_inputs,
        outputs=[grpcclient.InferRequestedOutput("fraud_probability")],
    )
    probs = response.as_numpy("fraud_probability")  # [B, 1]
    return probs[:, 0]  # P(fraud)
# ── Example: score first 512 edges ─────────────────────────────────────
batch_size = 512
batch_probs = score_batch(
    edge_src[:batch_size],
    edge_dst[:batch_size],
    edge_attr[:batch_size] if edge_attr is not None else None,
)
fraud_flags = (batch_probs >= threshold).astype(int)
print(f"Detected {fraud_flags.sum()} fraud edges out of {batch_size}")

7.5 REST API Using curl

Triton also exposes an HTTP REST API on port 8000, which can be useful for quick testing without a Python environment.

# Check if a specific model is ready
curl -s http://localhost:8000/v2/models/fraud_pipeline/ready
# Get model metadata (input/output tensor names and shapes)
curl -s http://localhost:8000/v2/models/fraud_pipeline | python3 -m json.tool

7.6 Interpreting Results and Setting Thresholds

The Fraud Probability

The output of fraud_pipeline is a [N, 1] float32 tensor:

• Column 0: P(fraud) — probability this edge is fraudulent.

P(legitimate) = 1 − P(fraud) if needed.

Using the Pre-Calibrated Threshold

The pipeline selects a decision threshold during training by maximising the F1 score on the validation set. This threshold is stored in model_config.json as val_threshold and used automatically by client.py. For most deployments, this threshold is a good starting point.

Adjusting the Threshold for Your Business Requirements

The optimal threshold depends on the relative cost of:

• False positives (FP): Legitimate transactions flagged as fraud. Customer friction, manual review cost.
• False negatives (FN): Fraudulent transactions missed. Direct financial loss + potential regulatory penalties.

To evaluate different thresholds without retraining:

import json
import numpy as np
# Load fraud probabilities produced by client.py
# (modify client.py to save all_probs and all_labels to a file, or score programmatically)
fraud_probs = np.load("fraud_probs.npy")   # [N]
labels      = np.load("labels.npy")        # [N]
thresholds  = np.arange(0.1, 0.99, 0.01)
results = []
for t in thresholds:
    preds     = (fraud_probs >= t).astype(int)
    tp = int(((preds==1) & (labels==1)).sum())
    fp = int(((preds==1) & (labels==0)).sum())
    fn = int(((preds==0) & (labels==1)).sum())
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall    = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1        = 2*precision*recall / (precision+recall) if (precision+recall) > 0 else 0
    results.append({"threshold": round(t, 2), "precision": round(precision, 4),
                    "recall": round(recall, 4), "f1": round(f1, 4),
                    "tp": tp, "fp": fp, "fn": fn})
# Print the Precision-Recall trade-off table
print(f"{'Threshold':>10}  {'Precision':>10}  {'Recall':>8}  {'F1':>8}  {'TP':>6}  {'FP':>6}  {'FN':>6}")
for r in results:
    print(f"{r['threshold']:>10.2f}  {r['precision']:>10.4f}  {r['recall']:>8.4f}  "
          f"{r['f1']:>8.4f}  {r['tp']:>6}  {r['fp']:>6}  {r['fn']:>6}")

Once you have chosen a new threshold, update it in the deployment by modifying model_config.json in the model repository and restarting (or hot-reloading) the gnn_embedder model in Triton:

# Update threshold in place
python3 -c "
import json
path = '/workspace/output/infer/triton/gnn_embedder/1/model_config.json'
with open(path) as f: cfg = json.load(f)
cfg['val_threshold'] = 0.82   # your new threshold
with open(path, 'w') as f: json.dump(cfg, f, indent=2)
print('Updated threshold to', cfg['val_threshold'])
"
# Triton hot-reload: unload and reload the gnn_embedder model
curl -X POST http://localhost:8000/v2/repository/models/gnn_embedder/unload
curl -X POST http://localhost:8000/v2/repository/models/gnn_embedder/load