Is this page helpful?

Run Training Using Financial Fraud Training#

This document provides step-by-step instructions to log in, pull the Financial Fraud Training Docker image, run the Financial Fraud Training container, and train models using it.

1. Authenticate with NVIDIA GPU Cloud (NGC) and Pull the Docker Image#

The following explains how to authenticate with the NVIDIA GPU Cloud (NGC) and pull the Docker image from the NVIDIA registry.

Prerequisites#

  • NGC Account & API Key: Ensure you have an NGC account and a valid API key.

  • Docker: Make sure Docker is installed and running on your machine.

Authenticate with NGC#

Log in to the NVIDIA container registry using your NGC API key:

docker login nvcr.io --username '$oauthtoken' --password $NGC_API_KEY

Note: Replace $NGC_API_KEY with your actual API key or set it in NGC_API_KEY env variable.

Pull the Docker Image#

After authenticating, pull the desired Docker image from the registry:

docker pull nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

2. Train Models#

Run the Container#

Run the training container with your graph data, output directory, and config file mounted:

docker run --rm -it --gpus all \
  --shm-size=10g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /your/graph/data:/data:ro \
  -v /your/output:/workspace/output \
  -v /your/config.yaml:/workspace/config.yaml:ro \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

The container automatically:

  1. Detects your config at /workspace/config.yaml.

  2. Validates it before any GPU work starts — configuration errors print a clean message and exit immediately.

  3. Reads num_gpus from the config and launches torchrun with the correct number of workers.

  4. Runs the full pipeline and writes all artifacts to /workspace/output.

Command Breakdown#

Docker Run Options:

  • --rm: Automatically removes the container when it exits.

  • -it: Run the container interactively.

  • --gpus all: Expose all available GPUs to the container.

  • --shm-size=10g: Sets shared memory to 10 GB, required for multi-GPU graph sampling.

  • --ulimit memlock=-1: Removes the locked-memory limit, required for GPU memory pinning.

  • --ulimit stack=67108864: Sets the stack size limit.

Volume Mounts:

  • -v /your/graph/data:/data:ro: Mounts your graph data directory into the container (read-only).

  • -v /your/output:/workspace/output: Mounts a directory for saving trained models and artifacts.

  • -v /your/config.yaml:/workspace/config.yaml:ro: Mounts your training configuration file.

Note: Replace /your/graph/data, /your/output, and /your/config.yaml with the actual paths on your system. The output directory must be writable; it will contain model weights, the XGBoost model, evaluation metrics, and the Triton inference repository after training completes.

Run Without a Config (Smoke Test)#

Omitting the config mount uses the container’s built-in default (2 GPUs, GNN_XGBoost, standard hyperparameters):

docker run --rm -it --gpus all \
  --shm-size=10g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /your/graph/data:/data:ro \
  -v /your/output:/workspace/output \
  nvcr.io/nvidia/cugraph/financial-fraud-training:3.0.0

Use this to verify that the image and your data directory are wired up correctly before committing to a full production run.

Useful Runtime Overrides#

All container defaults can be overridden with -e VAR=value on the docker run command line without modifying the config file:

# Change log verbosity
-e LOG_LEVEL=DEBUG

# Override number of GPU workers without editing the config
-e NPROC=1

# Point to a config at a non-standard container path
-e CONFIG_FILE=/workspace/my_config.yaml -v /host/my_config.yaml:/workspace/my_config.yaml:ro

# Enable NVLink P2P on clusters that support it (disabled by default for portability)
-e NCCL_P2P_DISABLE=0

Variable

Description

LOG_LEVEL

Logging verbosity. Set to DEBUG for detailed per-stage output. Default is INFO.

NPROC

Number of GPU worker processes per node. Overrides num_gpus in the config. Takes highest priority over all other sources.

CONFIG_FILE

Path to the config file inside the container. Use when mounting the config to a non-standard path.

NCCL_P2P_DISABLE

When set to 0, enables NVLink peer-to-peer transfers between GPUs. Disabled by default for portability across cluster configurations.

For all available hyperparameters, see the training guide.

Resume Interrupted Training#

Set checkpoint_dir in your config to warm-start GNN weights from a previous run. The GNN saves a single checkpoint (model_final.pt) to output_dir after all epochs complete. If training is interrupted before finishing, no partial checkpoint is available and training will restart from the beginning unless a previous run’s model_final.pt already exists in output_dir or checkpoint_dir.

paths:
  checkpoint_dir: /workspace/checkpoints/run1

To retrain only the XGBoost step using the saved GNN embedding, without re-running GNN training, set skip_gnn_train: true:

paths:
  checkpoint_dir: /workspace/checkpoints/run1

# in models[0]:
hyperparameters:
  gnn:
    skip_gnn_train: true       # skip GNN training; load checkpoint and re-run embedding extraction for XGBoost

Note: checkpoint_dir is checked as a fallback after output_dir during normal warm-start training. New checkpoints are always saved to output_dir/model_final.pt after training completes. When using skip_gnn_train: true, GNN training is skipped entirely; the pipeline loads the model from checkpoint_dir and runs only the embedding extraction step. This is useful when iterating on XGBoost hyperparameters without the cost of retraining the GNN.