> This page is for version 3.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 3.0.0 (Latest) (default): https://docs.nvidia.com/nim/financial-fraud-training/3.0.0/llms.txt
> - 2.0.0: https://docs.nvidia.com/nim/financial-fraud-training/2.0.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/financial-fraud-training/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/financial-fraud-training/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/financial-fraud-training/_mcp/server.

# Run Training Using Financial Fraud Training

This document provides step-by-step instructions to log in, pull the Financial Fraud Training Docker image, run the Financial Fraud Training container, and train models using it.

---

## 1. Authenticate with NVIDIA GPU Cloud (NGC) and Pull the Docker Image

The following explains how to authenticate with the NVIDIA GPU Cloud (NGC) and pull the Docker image from the NVIDIA registry.

### Prerequisites

- **NGC Account & API Key:** Ensure you have an NGC account and a valid API key.
- **Docker:** Make sure Docker is installed and running on your machine.

### Authenticate with NGC

Log in to the NVIDIA container registry using your NGC API key:

```bash
docker login nvcr.io --username '$oauthtoken' --password $NGC_API_KEY
```

<Note>Replace $NGC_API_KEY with your actual API key or set it in `NGC_API_KEY` env variable.</Note>

### Pull the Docker Image

After authenticating, pull the desired Docker image from the registry:

```bash
docker pull nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0
```

## 2. Train Models

### Run the Container

Run the training container with your graph data, output directory, and config file mounted:

```bash
docker run --rm -it --gpus all \
  --shm-size=10g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /your/graph/data:/data:ro \
  -v /your/output:/workspace/output \
  -v /your/config.yaml:/workspace/config.yaml:ro \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0
```

The container automatically:
1. Detects your config at `/workspace/config.yaml`.
2. Validates it before any GPU work starts — configuration errors print a clean message and exit immediately.
3. Reads `num_gpus` from the config and launches `torchrun` with the correct number of workers.
4. Runs the full pipeline and writes all artifacts to `/workspace/output`.

#### Command Breakdown

Docker Run Options:

- `--rm`: Automatically removes the container when it exits.
- `-it`: Run the container interactively.
- `--gpus all`: Expose all available GPUs to the container.
- `--shm-size=10g`: Sets shared memory to 10 GB, required for multi-GPU graph sampling.
- `--ulimit memlock=-1`: Removes the locked-memory limit, required for GPU memory pinning.
- `--ulimit stack=67108864`: Sets the stack size limit.

Volume Mounts:

- `-v /your/graph/data:/data:ro`: Mounts your graph data directory into the container (read-only).
- `-v /your/output:/workspace/output`: Mounts a directory for saving trained models and artifacts.
- `-v /your/config.yaml:/workspace/config.yaml:ro`: Mounts your training configuration file.

<Note>Replace `/your/graph/data`, `/your/output`, and `/your/config.yaml` with the actual paths on your system. The output directory must be writable; it will contain model weights, the XGBoost model, evaluation metrics, and the Triton inference repository after training completes.</Note>

#### Run Without a Config (Smoke Test)

Omitting the config mount uses the container's built-in default (2 GPUs, `GNN_XGBoost`, standard hyperparameters):

```bash
docker run --rm -it --gpus all \
  --shm-size=10g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /your/graph/data:/data:ro \
  -v /your/output:/workspace/output \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0
```

Use this to verify that the image and your data directory are wired up correctly before committing to a full production run.

### Useful Runtime Overrides

All container defaults can be overridden with `-e VAR=value` on the `docker run` command line without modifying the config file:

```bash
# Change log verbosity
-e LOG_LEVEL=DEBUG

# Override number of GPU workers without editing the config
-e NPROC=1

# Point to a config at a non-standard container path
-e CONFIG_FILE=/workspace/my_config.yaml -v /host/my_config.yaml:/workspace/my_config.yaml:ro

# Enable NVLink P2P on clusters that support it (disabled by default for portability)
-e NCCL_P2P_DISABLE=0
```

| Variable | Description |
|---|---|
| `LOG_LEVEL` | Logging verbosity. Set to `DEBUG` for detailed per-stage output. Default is `INFO`. |
| `NPROC` | Number of GPU worker processes per node. Overrides `num_gpus` in the config. Takes highest priority over all other sources. |
| `CONFIG_FILE` | Path to the config file inside the container. Use when mounting the config to a non-standard path. |
| `NCCL_P2P_DISABLE` | When set to `0`, enables NVLink peer-to-peer transfers between GPUs. Disabled by default for portability across cluster configurations. |

For all available hyperparameters, see the [training guide](/nim/financial-fraud-training/training/training-guide).

### Resume Interrupted Training

Set `checkpoint_dir` in your config to warm-start GNN weights from a previous run. The GNN saves a single checkpoint (`model_final.pt`) to `output_dir` after all epochs complete. If training is interrupted before finishing, no partial checkpoint is available and training will restart from the beginning unless a previous run's `model_final.pt` already exists in `output_dir` or `checkpoint_dir`.

```yaml
paths:
  checkpoint_dir: /workspace/checkpoints/run1
```

To retrain only the XGBoost step using the saved GNN embedding, without re-running GNN training, set `skip_gnn_train: true`:

```yaml
paths:
  checkpoint_dir: /workspace/checkpoints/run1

# in models[0]:
hyperparameters:
  gnn:
    skip_gnn_train: true       # skip GNN training; load checkpoint and re-run embedding extraction for XGBoost
```

<Note>`checkpoint_dir` is checked as a fallback after `output_dir` during normal warm-start training. New checkpoints are always saved to `output_dir/model_final.pt` after training completes. When using `skip_gnn_train: true`, GNN training is skipped entirely; the pipeline loads the model from `checkpoint_dir` and runs only the embedding extraction step. This is useful when iterating on XGBoost hyperparameters without the cost of retraining the GNN.</Note>