Is this page helpful?

Sending Inference Requests#

Install client dependencies first:

pip install "tritonclient[grpc]" pandas numpy torch

7.1 Standard Fraud Scoring — CLI Client#

The client script is written to your output directory at the end of training:

/path/to/output/infer/client.py

Run it with the model_config.json from your training output and your test data directory:

python /path/to/output/infer/client.py \
  --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 512

The --data_path directory must have the same nodes/ and edges/ layout as your training data with identical filenames and feature column counts. Requirements:

  • Node CSVs: same filenames and column order as training_data/nodes/. Feature count must match exactly.

  • Edge CSVs: same filenames and column order as training_data/edges/. All edge types (including context edges like B_to_C.csv) must be present for k-hop subgraph sampling to work correctly — not just the predicted edge type.

  • Label CSV: required (e.g. edges/account_transacts_merchant_label.csv). The client computes Precision, Recall, and F1 against ground truth labels.

  • NP: all rows in the label node CSV are scored as inference targets; no mask file is needed in test_gnn/.

The required test_gnn/ directory structure is described in Section 3.1.

What the client does, step by step:

  1. Reads model_config.json to learn the graph schema (node types, feature dimensions, edge type name, decision threshold, num_neighbors).

  2. Loads all node and edge CSVs from --data_path into memory.

  3. Builds adjacency structures for client-side BFS subgraph sampling (when num_neighbors is present in model_config.json).

  4. For each batch of query edges: samples a subgraph using BFS from both endpoints of each query edge (using the trained num_neighbors hop counts), then sends the sampled subgraph plus the batch endpoints to Triton’s fraud_pipeline ensemble using gRPC. With --no_sampling, sends the full graph as context instead.

  5. Receives a [N, 1] float32 tensor containing P(fraud) for each query edge.

  6. Thresholds at val_threshold from model_config.json.

  7. Aggregates predictions, computes precision/recall/F1 against ground truth labels.

  8. Prints a latency summary and sample fraud detections.

Sample output:

Loading test data from /data/test_gnn ...
  account: 500 nodes  |  merchant: 100 nodes
  edges: 5000  |  fraud: 73 (1.46%)
  edge_attr: (5000, 17)

Pass 1/1: scoring 10 batches (batch_size=512) via fraud_pipeline ...
  [   5/10] 2560/5000 edges  last_batch=11.8ms
  [  10/10] 5000/5000 edges  last_batch=12.1ms

Pass 1 latency summary (10 measured batches, batch_size=512):
  Total wall : 118.4 ms  (42231 edges/s)
  Per-batch  : mean=11.8ms  p50=11.6ms  p95=13.4ms  p99=14.1ms  max=14.1ms

==================================================
Results  (threshold=0.7651)
==================================================
  Total edges : 5000
  TP=64  FP=8  FN=9  TN=4919
  Precision   : 0.8889
  Recall      : 0.8767
  F1          : 0.8828
==================================================

Sample detected fraud edges (first 10 of 72):
   edge     src     dst   P(fraud)   label
    142      23      71     0.9812       1
     37       5      44     0.9701       1
    ...

All CLI flags:

Flag

Default

Description

--model_config

(required)

Path to model_config.json generated at training time. Located at <output>/infer/triton/gnn_embedder/1/model_config.json.

--data_path

(required)

Path to the test data directory containing nodes/ and edges/ subdirectories.

--url

localhost:8001

Triton gRPC endpoint. Format: host:port.

--batch_size

512

Number of query edges per gRPC request. Larger batches reduce per-edge overhead but increase per-request latency.

--warmup_batches

0

Send this many batches before starting latency measurements. Use 1–3 to warm up CUDA kernels and JIT compilation.

--explain

false

Enable LLM explanation mode (two-pass). See Section 7.2.

--explain_limit

10

Maximum fraud edges to explain in pass 2. Edges are sorted by descending fraud probability.

--microbatch_delay_ms

0

Simulate a micro-batch gateway. See Section 7.3.

--no_sampling

false

Disable client-side subgraph sampling. Sends the full graph as context on every request. Use when num_neighbors is absent from model_config.json, or to reproduce training-time behaviour at the cost of larger payloads.

7.1b Node Prediction Client (GNN_XGBoost_NP)#

When you trained with kind: GNN_XGBoost_NP, use client_np.py instead of client.py. It targets the np_fraud_pipeline ensemble and expects model_config_np.json:

python /path/to/output/infer/client_np.py \
  --model_config /path/to/output/infer/triton/np_gnn_embedder/1/model_config_np.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 512

Key differences from the EP client:

Aspect

EP (client.py)

NP (client_np.py)

Query type

Edges (transactions)

Nodes (accounts or merchants)

Triton model

fraud_pipeline

np_fraud_pipeline

Model config

gnn_embedder/1/model_config.json

np_gnn_embedder/1/model_config_np.json

Per-request batch identifier

edge_label_src / edge_label_dst

seed_node_ids (LOCAL 0-based subgraph indices)

Subgraph sampling

BFS from both endpoints of each query edge

BFS from seed nodes (NeighborLoader semantics)

CLI flags are identical: --url, --model_config, --data_path, --batch_size, --warmup_batches, --explain, --explain_limit, --microbatch_delay_ms, --no_sampling.

client_saved_emb.py — Hybrid Saved-Embedding Client#

Use this client when training was run with save_node_embeddings: true. It routes each edge through a fast memmap lookup for known nodes and falls back to the GNN for novel nodes.

python <output>/infer/client_saved_emb.py \
  --model_config <output>/infer/triton/gnn_embedder/1/model_config.json \
  --data_path    /data/test_gnn \
  # optional: inferred from model_config
  --emb_dir      <output>/node_embeddings \
  --batch_size   512 \
  --explain --explain_limit 10

Example output:

Unlike client.py, this client prints separate metrics for the KNOWN path (memmap) and NOVEL path (GNN fallback):

==================================================
Results — ALL edges  (25803 edges, threshold=0.9457)
==================================================
  TP=1509  FP=8  FN=578  TN=23708
  Precision : 0.9947
  Recall    : 0.7230
  F1        : 0.8374
==================================================

==================================================
Results — KNOWN nodes (saved-embedding path)  (18803 edges, threshold=0.9457)
==================================================
  TP=1099  FP=4  FN=423  TN=17277
  Precision : 0.9964
  Recall    : 0.7222
  F1        : 0.8372
==================================================

==================================================
Results — NOVEL nodes (on-the-fly GNN path)  (7000 edges, threshold=0.9457)
==================================================
  TP=410  FP=4  FN=155  TN=6431
  Precision : 0.9904
  Recall    : 0.7255
  F1        : 0.8382
==================================================

A cold-start warning fires when the novel-node percentage exceeds 10%: ⚠ >10% novel nodes — consider re-training to refresh embeddings.

Key CLI flags:

Flag

Default

Description

--emb_dir

from model_config

Node embeddings directory (can be omitted if node_embeddings_dir is in model_config.json)

--batch_size

512

Edges per inference batch

--no_sampling

false

Skip k-hop BFS subgraph sampling for novel nodes; use 0-hop (raw features only)

--lru_cache_size

null

Wrap embedding arrays in an LRU dict — keeps the N hottest nodes in Python RAM to avoid SSD page faults on repeated lookups (each cache hit copies the cached row into the result array)

--threshold

from model_config

Override decision threshold

--explain

false

Run explain pass on detected fraud edges

--explain_limit

10

Max edges to explain

client_production.py — Production HTTP Service and Concurrent Batch Eval#

client_production.py extends the hybrid routing of client_saved_emb.py with two additional capabilities:

  • Concurrent batch eval: dispatches Triton requests in parallel across multiple model instances using ThreadPoolExecutor

  • Persistent HTTP service (--serve): opens memmaps once at startup and serves requests indefinitely

Batch eval mode (no --serve):

python <output>/infer/client_production.py \
  --model_config <output>/infer/triton/gnn_embedder/1/model_config.json \
  --data_path    /data/test_gnn \
  --emb_dir      <output>/node_embeddings \
  --batch_size   512 \
  --concurrency  6        # NUM_GPUS × INSTANCES_PER_GPU

Reports the same per-path metrics (ALL / KNOWN / NOVEL) as client_saved_emb.py, plus latency percentiles per batch.

Persistent HTTP service (--serve):

pip install fastapi uvicorn    # one-time install

python <output>/infer/client_production.py \
  --model_config <output>/infer/triton/gnn_embedder/1/model_config.json \
  --emb_dir      <output>/node_embeddings \
  --data_path    /data/test_gnn \
  --serve \
  --host 0.0.0.0 --port 8080 \
  --concurrency 6 \
  --lru_cache_size 100000

At startup, embeddings.mmap and features.mmap are opened as np.memmap(mode='r') — zero bytes copied into RAM. The OS page-cache handles demand paging. Multiple server processes can open the same file and share the underlying pages.

Endpoints:

Endpoint

Method

Description

GET /health

GET

Liveness check + LRU hit-rate stats per node type

POST /score

POST

Fraud probability per edge. Known nodes: memmap → XGBoost (fast path). Novel nodes: GNN through Triton (fallback).

POST /explain

POST

Fraud probability + per-feature SVS attribution scores + LLM text explanation

POST /score request:

{
  "src_ids":   [42000000, 7000000],
  "dst_ids":   [5000000, 3000000],
  "edge_attr": [[100.0, 14.0], [50.0, 8.0]],
  "explain":   true,
  "neighbors": [[42000000, 5000000, "account_transacts_merchant"],
                [42000000, 9000001, "account_transacts_merchant"]]
}

neighbors is an optional pre-built k-hop subgraph for novel nodes (one entry per directed edge: [src_global_id, dst_global_id, edge_type_stem]). Populate this from a live graph database query when novel nodes are expected. Omit for 0-hop fallback (raw features, no message passing).

POST /score response:

{
  "fraud_prob": [0.923, 0.041],
  "is_fraud":   [true, false],
  "path":       ["memmap", "gnn"],
  "explanations": ["High-value transaction at unusual hour...", null]
}

"path" per edge is "memmap" (known node fast path) or "gnn" (novel node fallback).

POST /explain — same request body as /score. Always runs xgb_explainer (SVS Shapley-like attribution) and llm_explainer. Returns "svs" dict (feature name → score, positive pushes toward fraud) plus "explanations" text. Add "no_llm": true to return SVS only, skipping the LLM call.

Key CLI flags:

Flag

Default

Description

--data_path

required

Path to test data directory (same layout as training test_gnn/). Required by argparse even in --serve mode.

--serve

false

Start persistent HTTP service; without this flag runs one-shot batch eval

--host

0.0.0.0

Bind host for --serve

--port

8080

Bind port for --serve

--concurrency

num_gpus × instances_per_gpu

Parallel Triton gRPC channels

--lru_cache_size

null

Hot-node LRU cache per node type (number of nodes to keep in Python RAM)

--no_sampling

false

Skip BFS subgraph sampling for novel nodes

--triton_wait

300

Seconds to wait for Triton readiness before exiting

--emb_dir

from model_config

Node embeddings directory

7.2 LLM Explainability Mode#

Add --explain to run a two-pass workflow. Pass 1 scores all edges with fraud_pipeline. Pass 2 sends the top detected fraud edges (sorted by descending probability) individually to fraud_pipeline_explained.

python /path/to/output/infer/client.py \
  --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 512 \
  --explain \
  --explain_limit 10

Pass 2 workflow (per fraud edge):

  1. gnn_embedder: re-computes the edge embedding (same as pass 1).

  2. xgb_explainer: runs XGBoost inference + Captum Shapley Value Sampling. Outputs the fraud probability, the raw edge embedding, and a svs_attrs vector of per-feature-group attributions.

  3. llm_explainer: assembles the top-8 feature groups (by absolute SVS attribution) into a structured prompt, calls the LLM API, and returns a plain-language explanation string.

Typical explanation output:

==================================================
LLM Explanations (first 10 detected fraud edges)
==================================================

[Edge 142]  account=23  merchant=71  fraud_prob=0.9812  label=1  latency=847ms
  This transaction carries a high fraud risk (probability 98%). The three strongest
  signals are: (1) the GNN's account embedding deviates significantly from the
  account's typical peer group, suggesting recent account compromise; (2) the
  transaction amount ($4,823.50) is more than 14× the account's average; (3)
  the merchant has an elevated chargeback rate (3.1%) relative to its category
  peers. Together, these factors are consistent with card-not-present fraud
  following an account takeover event.

[Edge 37]   account=5   merchant=44  fraud_prob=0.9701  label=1  latency=612ms
  ...

What the SVS Shapley Values represent:

Shapley values answer the question: “how much did each feature group contribute to changing the prediction from the baseline (average legitimate transaction) to this specific prediction?” A positive value means the feature pushed the model toward predicting fraud; a negative value means it pulled toward legitimate. The LLM sees the top-8 groups and their human-readable descriptions (e.g., “transaction amount = $4,823.50”).

Production guidance for explanations:

  • Explanations are generated using an external API call per fraud edge. At peak loads, use --explain_limit to cap the number of explanations and avoid rate limits.

  • The LLM model choice affects explanation quality significantly. A larger model (70B parameters) produces more nuanced, accurate reasoning but at higher latency and cost.

  • Explanations are advisory only. The fraud decision is made by XGBoost; the LLM only narrates what the model already decided.

7.3 Micro-Batch Gateway Simulation#

Financial systems often cannot score transactions one by one; they queue incoming transactions and flush when a maximum batch size or maximum wait time is reached. Use --microbatch_delay_ms to simulate this gateway model and measure end-to-end per-transaction latency.

python /path/to/output/infer/client.py \
  --model_config /path/to/output/infer/triton/gnn_embedder/1/model_config.json \
  --data_path /path/to/your/test_gnn \
  --batch_size 256 \
  --microbatch_delay_ms 50 \
  --warmup_batches 2

This simulates: transactions arrive continuously; the gateway flushes when 256 edges accumulate or 50 ms elapses.

Sample output:

Micro-batch gateway simulation  (max_delay=50ms, max_batch=256, warmup=2) ...

Micro-batch simulation results  (5000 transactions, batch_size≤256, delay=50ms):
  Batch RTT  : mean=9.4ms  p50=9.1ms  p95=11.2ms  max=12.8ms
  Per-tx lat : mean=34.7ms  p50=34.2ms  p95=59.4ms  p99=61.1ms  max=62.8ms
  (avg queue wait ≈ 25.0ms with uniform arrivals)

In this example: the batch RTT is ~9.4 ms, but each transaction also waits in the queue for up to 50 ms. The average end-to-end latency per transaction is ~35 ms. To reduce this, either reduce microbatch_delay_ms (accept smaller batches and flush more often) or increase batch_size (larger batches amortise the RTT across more transactions).

7.4 Programmatic Python Client#

For integration into your own Python application (rather than using the CLI), use tritonclient directly.

Note: The example below sends the full graph as context on every request (equivalent to --no_sampling). This is simple but does not scale to production-sized graphs. For production use, reproduce the BFS subgraph sampling from client.py’s _sample_ep_subgraph function, which limits each request to the sampled neighbourhood of the query edges.

import json
import numpy as np
import pandas as pd
import tritonclient.grpc as grpcclient

# ── Load the model config written by training ──────────────────────────
with open("/workspace/output/infer/triton/gnn_embedder/1/model_config.json") as f:
    cfg = json.load(f)

src_type      = cfg["edge_type_to_predict"][0]   # "account"
rel           = cfg["edge_type_to_predict"][1]   # "transacts"
dst_type      = cfg["edge_type_to_predict"][2]   # "merchant"
edge_attr_dim = cfg["edge_attr_dim"]
threshold     = cfg["val_threshold"]

# ── Load your test data ────────────────────────────────────────────────
node_dir = "/data/test_gnn/nodes"
edge_dir = "/data/test_gnn/edges"

src_x     = pd.read_csv(f"{node_dir}/{src_type}.csv").values.astype(np.float32)
dst_x     = pd.read_csv(f"{node_dir}/{dst_type}.csv").values.astype(np.float32)
stem      = f"{src_type}_{rel}_{dst_type}"
edges_df  = pd.read_csv(f"{edge_dir}/{stem}.csv")
edge_src  = edges_df.iloc[:, 0].values.astype(np.int64)
edge_dst  = edges_df.iloc[:, 1].values.astype(np.int64)
edge_attr = pd.read_csv(f"{edge_dir}/{stem}_attr.csv").values.astype(np.float32) \
            if edge_attr_dim > 0 else None

# ── Connect to Triton ──────────────────────────────────────────────────
client = grpcclient.InferenceServerClient(url="localhost:8001", verbose=False)

# ── Score one batch of query edges ────────────────────────────────────
def score_batch(query_src_idx, query_dst_idx, query_attr=None):
    """
    Score a batch of edges. Returns fraud probabilities as a 1D numpy array.

    query_src_idx : int64 array [B] — source node indices of query edges
    query_dst_idx : int64 array [B] — dest node indices of query edges
    query_attr    : float32 array [B, A] or None
    """
    inputs_data = {
        f"{src_type}_x":  src_x,          # full context: all source nodes
        f"{dst_type}_x":  dst_x,          # full context: all dest nodes
        f"{src_type}_{rel}_{dst_type}_src":     edge_src,        # full context: all edges (for GNN neighbourhood)
        f"{src_type}_{rel}_{dst_type}_dst":     edge_dst,
        f"{dst_type}_rev_{rel}_{src_type}_src": edge_dst,        # reverse edges
        f"{dst_type}_rev_{rel}_{src_type}_dst": edge_src,
        "edge_label_src": query_src_idx,   # query: which edges to score?
        "edge_label_dst": query_dst_idx,
    }
    if query_attr is not None:
        inputs_data["edge_attr"] = query_attr

    type_map = {np.float32: "FP32", np.int64: "INT64"}
    triton_inputs = []
    for name, arr in inputs_data.items():
        inp = grpcclient.InferInput(name, list(arr.shape), type_map[arr.dtype.type])
        inp.set_data_from_numpy(arr)
        triton_inputs.append(inp)

    response = client.infer(
        model_name="fraud_pipeline",
        inputs=triton_inputs,
        outputs=[grpcclient.InferRequestedOutput("fraud_probability")],
    )
    probs = response.as_numpy("fraud_probability")  # [B, 1]
    return probs[:, 0]  # P(fraud)

# ── Example: score first 512 edges ─────────────────────────────────────
batch_size = 512
batch_probs = score_batch(
    edge_src[:batch_size],
    edge_dst[:batch_size],
    edge_attr[:batch_size] if edge_attr is not None else None,
)
fraud_flags = (batch_probs >= threshold).astype(int)
print(f"Detected {fraud_flags.sum()} fraud edges out of {batch_size}")

7.5 REST API Using curl#

Triton also exposes an HTTP REST API on port 8000, which can be useful for quick testing without a Python environment.

Note: The REST API requires base64-encoded binary tensors for float and int arrays. For production integration, the gRPC client (port 8001) is strongly preferred for efficiency and correctness. The curl approach below is suitable for debugging only.

# Check if a specific model is ready
curl -s http://localhost:8000/v2/models/fraud_pipeline/ready

# Get model metadata (input/output tensor names and shapes)
curl -s http://localhost:8000/v2/models/fraud_pipeline | python3 -m json.tool

7.6 Interpreting Results and Setting Thresholds#

The Fraud Probability#

The output of fraud_pipeline is a [N, 1] float32 tensor:

  • Column 0: P(fraud) — probability this edge is fraudulent.

P(legitimate) = 1 − P(fraud) if needed.

Using the Pre-Calibrated Threshold#

The pipeline selects a decision threshold during training by maximising the F1 score on the validation set. This threshold is stored in model_config.json as val_threshold and used automatically by client.py. For most deployments, this threshold is a good starting point.

Adjusting the Threshold for Your Business Requirements#

The optimal threshold depends on the relative cost of:

  • False positives (FP): Legitimate transactions flagged as fraud. Customer friction, manual review cost.

  • False negatives (FN): Fraudulent transactions missed. Direct financial loss + potential regulatory penalties.

To evaluate different thresholds without retraining:

import json
import numpy as np

# Load fraud probabilities produced by client.py
# (modify client.py to save all_probs and all_labels to a file, or score programmatically)
fraud_probs = np.load("fraud_probs.npy")   # [N]
labels      = np.load("labels.npy")        # [N]

thresholds  = np.arange(0.1, 0.99, 0.01)
results = []
for t in thresholds:
    preds     = (fraud_probs >= t).astype(int)
    tp = int(((preds==1) & (labels==1)).sum())
    fp = int(((preds==1) & (labels==0)).sum())
    fn = int(((preds==0) & (labels==1)).sum())
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall    = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1        = 2*precision*recall / (precision+recall) if (precision+recall) > 0 else 0
    results.append({"threshold": round(t, 2), "precision": round(precision, 4),
                    "recall": round(recall, 4), "f1": round(f1, 4),
                    "tp": tp, "fp": fp, "fn": fn})

# Print the Precision-Recall trade-off table
print(f"{'Threshold':>10}  {'Precision':>10}  {'Recall':>8}  {'F1':>8}  {'TP':>6}  {'FP':>6}  {'FN':>6}")
for r in results:
    print(f"{r['threshold']:>10.2f}  {r['precision']:>10.4f}  {r['recall']:>8.4f}  "
          f"{r['f1']:>8.4f}  {r['tp']:>6}  {r['fp']:>6}  {r['fn']:>6}")

Once you have chosen a new threshold, update it in the deployment by modifying model_config.json in the model repository and restarting (or hot-reloading) the gnn_embedder model in Triton:

# Update threshold in place
python3 -c "
import json
path = '/workspace/output/infer/triton/gnn_embedder/1/model_config.json'
with open(path) as f: cfg = json.load(f)
cfg['val_threshold'] = 0.82   # your new threshold
with open(path, 'w') as f: json.dump(cfg, f, indent=2)
print('Updated threshold to', cfg['val_threshold'])
"

# Triton hot-reload: unload and reload the gnn_embedder model
curl -X POST http://localhost:8000/v2/repository/models/gnn_embedder/unload
curl -X POST http://localhost:8000/v2/repository/models/gnn_embedder/load