Run Training Using Financial Fraud Training | NVIDIA Financial Fraud NIM Documentation

This document provides step-by-step instructions to log in, pull the Financial Fraud Training Docker image, run the Financial Fraud Training container, and train models using it.

1. Authenticate with NVIDIA GPU Cloud (NGC) and Pull the Docker Image

The following explains how to authenticate with the NVIDIA GPU Cloud (NGC) and pull the Docker image from the NVIDIA registry.

Prerequisites

NGC Account & API Key: Ensure you have an NGC account and a valid API key.
Docker: Make sure Docker is installed and running on your machine.

Authenticate with NGC

docker login nvcr.io --username '$oauthtoken' --password $NGC_API_KEY

Replace $NGC_API_KEY with your actual API key or set it in NGC_API_KEY env variable.

Pull the Docker Image

After authenticating, pull the desired Docker image from the registry:

docker pull nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

2. Train Models

Run the Container

Run the training container with your graph data, output directory, and config file mounted:

docker run --rm -it --gpus all \
  --shm-size=10g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /your/graph/data:/data:ro \
  -v /your/output:/workspace/output \
  -v /your/config.yaml:/workspace/config.yaml:ro \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

The container automatically:

Detects your config at /workspace/config.yaml.
Validates it before any GPU work starts — configuration errors print a clean message and exit immediately.
Reads num_gpus from the config and launches torchrun with the correct number of workers.
Runs the full pipeline and writes all artifacts to /workspace/output.

Command Breakdown

Docker Run Options:

--rm: Automatically removes the container when it exits.
-it: Run the container interactively.
--gpus all: Expose all available GPUs to the container.
--shm-size=10g: Sets shared memory to 10 GB, required for multi-GPU graph sampling.
--ulimit memlock=-1: Removes the locked-memory limit, required for GPU memory pinning.
--ulimit stack=67108864: Sets the stack size limit.

Volume Mounts:

-v /your/graph/data:/data:ro: Mounts your graph data directory into the container (read-only).
-v /your/output:/workspace/output: Mounts a directory for saving trained models and artifacts.
-v /your/config.yaml:/workspace/config.yaml:ro: Mounts your training configuration file.

Replace /your/graph/data, /your/output, and /your/config.yaml with the actual paths on your system. The output directory must be writable; it will contain model weights, the XGBoost model, evaluation metrics, and the Triton inference repository after training completes.

Run Without a Config (Smoke Test)

Omitting the config mount uses the container’s built-in default (2 GPUs, GNN_XGBoost, standard hyperparameters):

docker run --rm -it --gpus all \
  --shm-size=10g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /your/graph/data:/data:ro \
  -v /your/output:/workspace/output \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

Use this to verify that the image and your data directory are wired up correctly before committing to a full production run.

Useful Runtime Overrides

All container defaults can be overridden with -e VAR=value on the docker run command line without modifying the config file:

# Change log verbosity
-e LOG_LEVEL=DEBUG
# Override number of GPU workers without editing the config
-e NPROC=1
# Point to a config at a non-standard container path
-e CONFIG_FILE=/workspace/my_config.yaml -v /host/my_config.yaml:/workspace/my_config.yaml:ro
# Enable NVLink P2P on clusters that support it (disabled by default for portability)
-e NCCL_P2P_DISABLE=0

Variable	Description
`LOG_LEVEL`	Logging verbosity. Set to `DEBUG` for detailed per-stage output. Default is `INFO`.
`NPROC`	Number of GPU worker processes per node. Overrides `num_gpus` in the config. Takes highest priority over all other sources.
`CONFIG_FILE`	Path to the config file inside the container. Use when mounting the config to a non-standard path.
`NCCL_P2P_DISABLE`	When set to `0`, enables NVLink peer-to-peer transfers between GPUs. Disabled by default for portability across cluster configurations.

For all available hyperparameters, see the training guide.

Resume Interrupted Training

Set checkpoint_dir in your config to warm-start GNN weights from a previous run. The GNN saves a single checkpoint (model_final.pt) to output_dir after all epochs complete. If training is interrupted before finishing, no partial checkpoint is available and training will restart from the beginning unless a previous run’s model_final.pt already exists in output_dir or checkpoint_dir.

paths:
  checkpoint_dir: /workspace/checkpoints/run1

To retrain only the XGBoost step using the saved GNN embedding, without re-running GNN training, set skip_gnn_train: true:

paths:
  checkpoint_dir: /workspace/checkpoints/run1
# in models[0]:
hyperparameters:
  gnn:
    skip_gnn_train: true       # skip GNN training; load checkpoint and re-run embedding extraction for XGBoost

checkpoint_dir is checked as a fallback after output_dir during normal warm-start training. New checkpoints are always saved to output_dir/model_final.pt after training completes. When using skip_gnn_train: true, GNN training is skipped entirely; the pipeline loads the model from checkpoint_dir and runs only the embedding extraction step. This is useful when iterating on XGBoost hyperparameters without the cost of retraining the GNN.