> This page is for version 3.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 3.0.0 (Latest) (default): https://docs.nvidia.com/nim/financial-fraud-training/3.0.0/llms.txt
> - 2.0.0: https://docs.nvidia.com/nim/financial-fraud-training/2.0.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/financial-fraud-training/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/financial-fraud-training/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/financial-fraud-training/_mcp/server.

# Serving the Trained Model

## Starting the Triton Server

Training writes everything you need to serve alongside the trained weights. Assume your training output is at `/path/to/output` (the value you passed as `output_dir` in your config). The structure you will find there is:

```
/path/to/output/
├── model_final.pt
├── xgboost_fraud.json
├── metrics.json
└── infer/
    ├── triton/            ← Triton model repository — pass this as MODEL_REPO
    ├── serve.sh           ← launches the Triton container
    └── client.py          ← full inference client (real data, metrics, explain mode)
```

### Step 1 — Install the client dependencies (host, once)

The inference client runs on the host (outside Docker). Install its dependencies once:

```bash
pip install "tritonclient[grpc]" pandas numpy torch
```

### Step 2 — Start the Triton Server

`serve.sh` mounts the Triton model repository into the official `nvcr.io/nvidia/tritonserver:26.04-py3` container and installs the Python backend dependencies at startup:

```bash
MODEL_REPO=/path/to/output/infer/triton \
  bash /path/to/output/infer/serve.sh
```

Run this in a dedicated terminal (it stays in the foreground). You will see a stream of startup logs as Triton loads each model.

What `serve.sh` executes internally:

```bash
docker run --rm \
  --gpus all \
  --shm-size=3g \
  -e LLM_API_KEY \
  -e LLM_BASE_URL \
  -e LLM_MODEL \
  -e NVIDIA_API_KEY \
  -p 127.0.0.1:8000:8000 \
  -p 127.0.0.1:8001:8001 \
  -p 127.0.0.1:8002:8002 \
  -v "/path/to/output/infer/triton:/models" \
  nvcr.io/nvidia/tritonserver:26.04-py3 \
  bash -c "pip install --quiet --root-user-action=ignore \
             'torch_geometric>=2.6' 'xgboost>=2.1,<3' 'captum>=0.7' 'openai>=1.0' && \
           tritonserver \
             --model-repository=/models \
             --backend-config=python,stub-timeout-seconds=120"
```

Key design choices:
- Python dependencies (`torch_geometric`, `xgboost`, `captum`, `openai`) are installed inside the container at every startup (`torch` is already in the Triton base image) — the model repository itself stays clean and portable.
- `stub-timeout-seconds=120` allows the Python backend models up to 2 minutes to initialise. This is necessary on first startup when PyTorch and the GNN architecture are loaded for the first time.
- LLM API key environment variables are forwarded from the host shell into the container automatically — no secrets need to be baked into images or files.

**Exposed ports:**

| Port | Protocol | Use |
|---|---|---|
| 8000 | HTTP | REST API (health checks, model metadata) |
| 8001 | gRPC | Inference API — used by `client.py` |
| 8002 | HTTP | Prometheus metrics (`/metrics` endpoint) |

### Step 3 — Wait for the Server to Become Ready

The server takes 30–90 seconds on the first startup (pip install + model loading). Poll the health endpoint in a separate terminal:

```bash
until curl -sf http://localhost:8000/v2/health/ready > /dev/null; do
  echo "Waiting for Triton..."
  sleep 3
done
echo "Triton is ready."
```

The server is ready when the container logs show:

```
I0101 00:00:00.000000 1 grpc_server.cc:xxxx] Started GRPCInferenceService at 0.0.0.0:8001
I0101 00:00:00.000000 1 http_server.cc:xxxx] Started HTTPService at 0.0.0.0:8000
I0101 00:00:00.000000 1 server.cc:xxxx] Triton Inference Server is ready.
```

Verify all six models are loaded:

```bash
curl -s http://localhost:8000/v2/models | python3 -m json.tool
```

Expected: all of `gnn_embedder`, `xgb_fraud`, `xgb_explainer`, `llm_explainer`, `fraud_pipeline`, `fraud_pipeline_explained` show `"state": "READY"`.

You can also check a specific model:

```bash
curl -s http://localhost:8000/v2/models/fraud_pipeline/ready
```

---

## LLM Explainability at Serve Time

The LLM explainer calls an external OpenAI-compatible API. The API key must be set as an environment variable **in the host shell before running `serve.sh`** — the key is forwarded into the container automatically; you never need to write it to disk.

**Option A — NVIDIA NIM (default, no `LLM_BASE_URL` needed):**

```bash
export NVIDIA_API_KEY=nvapi-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Optionally override the model (default: nvidia/meta/llama-3.2-1b-instruct):
export LLM_MODEL=nvidia/meta/llama-3.1-70b-instruct

MODEL_REPO=/path/to/output/infer/triton \
  bash /path/to/output/infer/serve.sh
```

Default endpoint: `https://inference-api.nvidia.com`
Default model: `nvidia/meta/llama-3.2-1b-instruct`

**Option B — OpenAI or any compatible API endpoint:**

```bash
export LLM_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export LLM_BASE_URL=https://api.openai.com/v1
export LLM_MODEL=gpt-4o-mini

MODEL_REPO=/path/to/output/infer/triton \
  bash /path/to/output/infer/serve.sh
```

**Option C — Any other OpenAI-compatible endpoint:**

```bash
export LLM_API_KEY=<your-api-key>
export LLM_BASE_URL=<your-openai-compatible-endpoint>
export LLM_MODEL=<model-name>

MODEL_REPO=/path/to/output/infer/triton \
  bash /path/to/output/infer/serve.sh
```

Confirm the LLM backend connected successfully by checking the Triton startup logs for:

```
[llm_explainer] LLM connectivity check passed.
```

**Running without an API key:**
The server starts normally. `fraud_pipeline` (scoring only) works fully. `fraud_pipeline_explained` also starts, but `llm_explainer` disables itself gracefully — it logs a warning and returns a placeholder string (`[LLM explanation disabled: set LLM_API_KEY to enable]`) in place of a real explanation. This is not an error state; scoring is unaffected.

---

## Production Deployment Checklist

Before routing production traffic to the server:

| # | Item | How to verify |
|---|---|---|
| 1 | All 6 models loaded successfully | `curl -s http://localhost:8000/v2/models` — all 6 show `"state":"READY"` |
| 2 | `fraud_pipeline` returns reasonable scores | Run `client.py --warmup_batches 1` against a sample batch |
| 3 | LLM connectivity confirmed (if using explanations) | Triton logs: `[llm_explainer] LLM connectivity check passed.` |
| 4 | GPU memory is stable | `watch -n1 nvidia-smi` — VRAM should stabilise after warm-up, not grow |
| 5 | `model_config.json` matches the loaded weights | Threshold in `model_config.json` matches `output/metrics.json`'s `best_threshold` |
| 6 | Prometheus metrics endpoint responds | `curl http://localhost:8002/metrics \| head -20` |
| 7 | Firewall/network policy exposes only port 8001 to clients | Ports 8000 and 8002 should be internal only in production |

---

## Multi-GPU Inference — Horizontal Scaling

By default, all models load on GPU 0 with 1 instance. To spread load across multiple GPUs, use the `NUM_GPUS` and `INSTANCES_PER_GPU` environment variables when calling `serve.sh`. The script automatically patches the `config.pbtxt` files before starting Triton — no manual file editing required. Training and deployment typically happen on different machines, so GPU count is set at serve time, not at training time.

**Usage:**

```bash
# Single GPU, 1 instance (default):
MODEL_REPO=... bash serve.sh

# 2 GPUs, 1 instance each (2 total per model):
NUM_GPUS=2 INSTANCES_PER_GPU=1 MODEL_REPO=... bash serve.sh

# 2 GPUs, 3 instances each (6 total per model):
NUM_GPUS=2 INSTANCES_PER_GPU=3 MODEL_REPO=... bash serve.sh

# 8 GPUs, 1 instance each (8 total per model):
NUM_GPUS=8 INSTANCES_PER_GPU=1 MODEL_REPO=... bash serve.sh
```

Triton load-balances incoming requests across all instances automatically.

**Which models are patched:**

*Node prediction pipeline (`GNN_XGBoost_NP`):*

| Model directory | Backend | Patched by serve.sh |
|---|---|---|
| `np_gnn_embedder/config.pbtxt` | Python (GNN) | Yes |
| `xgb_fraud_np/config.pbtxt` | FIL (XGBoost) | Yes |
| `xgb_explainer_np/config.pbtxt` | Python (SVS/Captum) | Yes |
| `llm_explainer_np/config.pbtxt` | Python (LLM) | Yes |
| `np_fraud_pipeline/config.pbtxt` | Ensemble | No — ensembles run on CPU |
| `np_fraud_pipeline_explained/config.pbtxt` | Ensemble | No — ensembles run on CPU |

*Edge prediction pipeline (`GNN_XGBoost`):*

| Model directory | Backend | Patched by serve.sh |
|---|---|---|
| `gnn_embedder/config.pbtxt` | Python (GNN) | Yes |
| `xgb_fraud/config.pbtxt` | FIL (XGBoost) | Yes |
| `xgb_explainer/config.pbtxt` | Python (SVS/Captum) | Yes |
| `llm_explainer/config.pbtxt` | Python (LLM) | Yes |
| `fraud_pipeline/config.pbtxt` | Ensemble | No — ensembles run on CPU |
| `fraud_pipeline_explained/config.pbtxt` | Ensemble | No — ensembles run on CPU |

**What `serve.sh` writes to each patched config:**

```protobuf
instance_group [ { kind: KIND_GPU  gpus: [0, 1, ...]  count: <INSTANCES_PER_GPU> } ]
```

`count` is **per GPU**, not total. `gpus: [0, 1]  count: 3` creates 6 instances total (3 per GPU).

**Startup time with many instances:** `torch.compile` warmup runs once per GNN embedder instance (`gnn_embedder` for EP, `np_gnn_embedder` for NP). With 8 GPUs × 3 instances = 24 instances, startup can take several minutes. The default `stub-timeout-seconds=120` may need increasing — set it in `serve.sh` if models fail to load:

```bash
--backend-config=python,stub-timeout-seconds=600
```

<Note>Each instance loads a full copy of the model into its GPU's VRAM. Ensure each GPU has enough memory.</Note>