Code ExamplesWorkflows & SolutionsFraud

Fraud Detection Demo

View as Markdown

This demo shows how to use Kumo to detect fraudulent credit card transactions.

Prerequisites

  • Kumo access: Ensure you have access to a Kumo environment, including the platform URL and API credentials required for your deployment.
  • [SPCS Only]: Ensure the Kumo app is installed and started. If you have not already installed or started Kumo, follow the Snowflake Native App installation guide.

What You Will Learn in This Demo

You will learn how to use Kumoā€™s GNNs to predict fraudulent transactions. This demo uses a dataset created by the Sparkov Fraud Data Generator. This dataset generator simulates credit card transactions according to several real-world fraud patterns, such as:

  • A credit card being stolen: multiple fraudulent transactions happening in quick succession
  • A fraudulent merchant: a bad merchant making many fake transactions

As you work through this tutorial, you see how the Kumo Graph Neural Network learns these temporal and relational patterns effectively.

The dataset is organized into four main tables:

  • Customers: Contains information about ~1000 customers and their location
  • Merchants: Contains information about ~700 merchants and their location
  • Transactions: Contains 1 million transactions spanning over about 18 months
  • Fraud Reports: Contains 14 thousand customer fraud reports, half of which have been confirmed to be valid according the fraud operations team

While this dataset is intentionally kept small for demonstration purposes, the Kumo RDL Platform can easily scale to datasets that are thousands of times larger.

Step 0 [SPCS Only]: Upload Data to Your Snowflake Account

See details

Follow these steps to upload the H&M data:

  1. Log into your Snowflake account.

  2. Run these commands to load the dataset from Kumoā€™s public S3 bucket into your Snowflake account:

    sql
    -- This script loads data generated by the Sparkov Fraud Data Generator, from a public bucket on S3.
    -- https://github.com/namebrandon/Sparkov_Data_Generation
    -- Define variables for database and schema
    SET DATABASE_NAME = 'KUMO_DB';
    SET SCHEMA_NAME = 'FRAUD';
    -- Create database/schema if not exists
    CREATE DATABASE IF NOT EXISTS IDENTIFIER($DATABASE_NAME);
    SET FULL_SCHEMA_NAME = CONCAT($DATABASE_NAME, '.', $SCHEMA_NAME);
    CREATE SCHEMA IF NOT EXISTS IDENTIFIER($FULL_SCHEMA_NAME);
    -- Create or replace the external stage
    CREATE OR REPLACE STAGE kumo_fraud_stage
        URL='s3://kumo-public-datasets/credit_card_fraud/'
        FILE_FORMAT = (TYPE = PARQUET);
    -- Section: Transactions Table
    -- Create the transactions table
    SET TRANSACTIONS_TABLE_NAME = CONCAT($FULL_SCHEMA_NAME, '.transactions');
    CREATE TABLE IF NOT EXISTS IDENTIFIER($TRANSACTIONS_TABLE_NAME) (trans_date_trans_time	
    TIMESTAMP, cc_num NUMBER(38,0), merchant VARCHAR, trans_num VARCHAR, amt NUMBER(38, 2));
    -- Copy data into the transactions table
    COPY INTO IDENTIFIER($TRANSACTIONS_TABLE_NAME)
    FROM (
        SELECT $1:trans_date_trans_time::TIMESTAMP, $1:cc_num::NUMBER(38,0), $1:merchant::VARCHAR, $1:trans_num::VARCHAR, $1:amt::NUMBER(38, 2)
        FROM @kumo_fraud_stage/transactions.parquet
    ) FILE_FORMAT = (TYPE = PARQUET) ON_ERROR = ABORT_STATEMENT;
    -- Section: Customers Table
    -- Create the customers table
    SET CUSTOMERS_TABLE_NAME = CONCAT($FULL_SCHEMA_NAME, '.customers');
    CREATE TABLE IF NOT EXISTS IDENTIFIER($CUSTOMERS_TABLE_NAME) (cc_num NUMBER(38,0), first VARCHAR, last VARCHAR, gender VARCHAR, street VARCHAR, city VARCHAR, zip NUMBER(38,0), lat FLOAT, long FLOAT, city_pop NUMBER(38,0), job VARCHAR, dob DATE);
    -- Copy data into the customers table
    COPY INTO IDENTIFIER($CUSTOMERS_TABLE_NAME)
    FROM (
        SELECT $1:cc_num::NUMBER(38,0), $1:first::VARCHAR, $1:last::VARCHAR, $1:gender::VARCHAR, $1:street::VARCHAR, $1:city::VARCHAR, $1:zip::NUMBER(38,0), $1:lat::FLOAT, $1:long::FLOAT, $1:city_pop::NUMBER(38,0), $1:job::VARCHAR, $1:dob::DATE, 
        FROM @kumo_fraud_stage/customers.parquet
    ) FILE_FORMAT = (TYPE = PARQUET) ON_ERROR = ABORT_STATEMENT;
    -- Section: Merchants Table
    -- Create the merchants table
    SET MERCHANTS_TABLE_NAME = CONCAT($FULL_SCHEMA_NAME, '.merchants');
    CREATE TABLE IF NOT EXISTS IDENTIFIER($MERCHANTS_TABLE_NAME) (merchant	
    VARCHAR, category VARCHAR, merch_lat FLOAT, merch_long FLOAT);
    -- Copy data into the merchants table
    COPY INTO IDENTIFIER($MERCHANTS_TABLE_NAME)
    FROM (
        SELECT $1:merchant::VARCHAR, $1:category::VARCHAR, $1:merch_lat::FLOAT, $1:merch_long::FLOAT
        FROM @kumo_fraud_stage/merchants.parquet
    ) FILE_FORMAT = (TYPE = PARQUET) ON_ERROR = ABORT_STATEMENT;
    -- Section: Fraud Reports Table
    -- Create the fraud reports table
    SET FRAUD_REPORTS_TABLE_NAME = CONCAT($FULL_SCHEMA_NAME, '.fraud_reports');
    CREATE TABLE IF NOT EXISTS IDENTIFIER($FRAUD_REPORTS_TABLE_NAME) (report_time	
    TIMESTAMP, 	trans_num VARCHAR, is_real_fraud NUMBER(38,0));
    -- Copy data into the transactions table
    COPY INTO IDENTIFIER($FRAUD_REPORTS_TABLE_NAME)
    FROM (
        SELECT $1:report_time::TIMESTAMP, $1:trans_num::VARCHAR, $1:is_real_fraud::NUMBER(38,0)
        FROM @kumo_fraud_stage/fraud_reports.parquet
    ) FILE_FORMAT = (TYPE = PARQUET) ON_ERROR = ABORT_STATEMENT;

  3. Inspect the tables:

    While the data is being uploaded, you can continue with Step 1 and connect your database to Kumo. After that, return to Snowflake to inspect the tables using the following queries:

    sql
    -- View data from the transactions table
    SELECT * FROM IDENTIFIER($TRANSACTIONS_TABLE_NAME) LIMIT 10;
    -- View data from the merchants table
    SELECT * FROM IDENTIFIER($MERCHANTS_TABLE_NAME) LIMIT 10;
    -- View data from the customers table
    SELECT * FROM IDENTIFIER($CUSTOMERS_TABLE_NAME) LIMIT 10;
    -- View data from the fraud reports table
    SELECT * FROM IDENTIFIER($FRAUD_REPORTS_TABLE_NAME) LIMIT 100;

Step 1: Set Up Your Connector

This step gives Kumo access to your data source.

Connector configuration panel for setting up a new data source

  1. Open Connector Configuration:

    • In the left-hand menu, click on Connectors.
    • Click Configure Connector to open the ā€œNew Connectorā€ modal.

  2. Create a New Connector:

    i. Provide a name for your new connector (for example, fraud_connector).

    ii. Choose your data source:

    • For Snowflake, enter the following details:
      • Account Identifier: ORGNAME-ACCOUNT_NAME. Find your account details.
      • Database: The database where your data exists. Ensure your user has USAGE privileges. If you followed Step 0, it will be KUMO_DB.
      • Warehouse: The warehouse to process data. Ensure your user has USAGE privileges.
      • Schema Name: The schema to load tables from. Your user should have USAGE and SELECT privileges, and CREATE TABLE for writing back predictions. If you followed Step 0, it will be FRAUD.
      • User: Your username for Snowflake.
      • Password: Your password for Snowflake.
    • For S3:
      • In the S3 Path textbox, enter: s3://kumo-public-datasets/credit_card_fraud/.
      • Click the Validate button to see all files within the directory.
    • For other data sources: Refer to Data Connectors.

    iii. Click Done to save the connector.

Step 2: Register Table Schema

Register your tables so Kumo knows how to encode each column and how to connect tables in the graph.

Tables overview page showing registered data tables

  1. Navigate to Tables Overview:

    • In the left-hand menu, click on Tables.

  2. Add Data Tables (Customers, Articles, Transactions):

    Follow the steps for each table:

    1. Click Add Table.
    2. Set the table name.
    3. Select the appropriate connector (for example, fraud_connector).
    4. Choose the respective table from your data source.
    5. Wait for column types to be inferred based on sample statistics. If this takes too long, which can happen in SPCS deployments if the warehouse you are using is busy, skip the inferring and set the column types manually. Follow these steps:
      1. Check the type of each column: The type of a column refers to how the data is later encoded. Kumo supports preprocessing for the following generic types:
        • Numerical: Integers and floats where the ordering of numbers from lower to higher values has semantic meaning (for example, product price or percentage discount).
        • Categorical: Boolean values or string values typically only a single token in length, with limited semantic meaning, and up to 4000 unique values (for example, premium subscription status).
        • Multi-categorical: Comma-separated variable length lists of categorical values (for example, a list of product attributes or categories).
        • ID: Columns the correspond to primary or foreign keys in a relational schema. (for example, customer ID or product group number).
        • Text: String values with multiple tokens in length, where the content has semantic meaning (for example, product description).
        • Timestamp: String or format-specific date/timestamp values (preferably ISO 8601). Ensure date/time format is valid.
        • Embedding: Lists of floats, all of equal length; typically the output of another AI model.
      2. Check the data type: This refers to the underlying data type. This is especially relevant for connecting tables later, as only columns with the same type can be connected through a Primary/Foreign key link. No modification is needed for data types for the H&M dataset.
      3. Set the primary key: This is the unique identifier column of a table and is used later within the Primary/Foreign key connection.
      4. Set a create date: For tables that correspond to real-world events (such as transactions), this is the time that the event took place in the real world. Kumo uses this to automatically prevent data leakage during training time, which is a very common mistake when building models by hand.
    6. Before saving, please verify if all properties are set correctly:

      1. Customers: Set the Primary Key to CC_NUM with Type = ID

        Customers table configuration with CC_NUM set as primary key with type ID

      2. Merchants: Set the Primary Key to MERCHANT with Type = ID

        Merchants table configuration with MERCHANT set as primary key with type ID

      3. Transactions: Change the Type of CC_NUM and MERCHANT to ID, because they are foreign keys into other tables. Set TRANS_NUM to be the Primary Key, with Type = ID as well. Set TRANS_DATE_TRANS_TIME as the Create Date.

        Transactions table configuration showing CC_NUM and MERCHANT as ID type and TRANS_NUM as primary key

      4. Fraud Reports: Change the type of TRANS_NUM to ID because it is a foreign key into the TRANSACTIONS table. Set REPORT_TIME as the Create Date.

        Fraud Reports table configuration with TRANS_NUM as ID and REPORT_TIME as create date

    7. Click Save to save each table.

Step 3: Create a Graph Schema

Define the graph schema to specify how your data tables connect. This is required to use Kumoā€™s Graph Neural Networks (GNNs).

  1. Navigate to Graph Setup:

    • In the left-hand menu, click on Graphs.

  2. Create a New Graph:

    i. Click Create Graph in the top right-hand corner.

    ii. Provide Graph Details:

    • On the ā€œGraph Setupā€ page, provide a name for your new graph (for example, fraud_graph).

    iii. Select Tables:

    • Select the TRANSACTIONS, CUSTOMERS, FRAUD_REPORTS, and MERCHANTS tables by checking the respective boxes.

    iv. Configure Graph Links:

    • Click the Next button on the top right-hand corner.
    • Kumo automatically detects and suggests linkages.
    • Verify the linkages:
      • Ensure CC_NUM in CUSTOMERS is linked to CC_NUM in TRANSACTIONS.
      • Ensure MERCHANT in MERCHANTS is linked to MERCHANT in TRANSACTIONS.
      • Ensure TRANS_NUM in FRAUD_REPORTS is linked to TRANS_NUM in TRANSACTIONS.
    • Click the Confirm Group button for both column groups.

    v. Complete Graph Creation:

    • Click on the Complete Graph Creation button on the top right-hand corner.

You should now view the following graph:

Completed fraud detection graph showing connections between Customers, Transactions, Merchants, and Fraud Reports tables

Step 4: Train Your First Model via PQL

After defining the graph, train a model using Kumoā€™s Predictive Query Language (PQL). PQL specifies the prediction target and the entity to make predictions for. In this demo, you will predict whether each transaction is fraudulent.

  1. Open Predictive Query Creation:

    • Click on Write Predictive Query in the top right-hand corner.

  2. Train Model through a Predictive Query:

    i. Provide a name for your predictive query (for example, cc_fraud).

    ii. Select the graph you created earlier from the drop-down menu.

    iii. Specify PQuery:

    • In the PQuery text area, insert the following PQL statement to predict the probability that a given transaction is fraudulent.

      PQL
      PREDICT COUNT(FRAUD_REPORTS.*
        WHERE FRAUD_REPORTS.IS_REAL_FRAUD = 1, 0, 7, days) >= 1
      FOR EACH TRANSACTIONS.TRANS_NUM

      The query defines how the training data is (automatically) generated. In this example, Kumo generates a training table of positive and negative examples used to train the model. Positive examples are transactions reported for fraud within 7 days of the transaction date.

      PQL editor showing the fraud detection predictive query for credit card transactions

    iv. Click Next to specify the Model Planner.

    • Since this fraud dataset has extremely heavy class imbalance (fraud is an extremely rare event), tell the GNN to pay more attention to the (rare) positive examples. To do this, set majority_sampling_ratio to 100 and tune_metric to auprc.

      Model Planner settings showing majority_sampling_ratio set to 100 and tune_metric set to auprc

    v. Click Save and Train.

šŸ† Great work! Your model is now training. This can take a few minutes while Kumo ingests data, encodes columns, builds the computation graph, and trains and evaluates the models. You can monitor progress through the training table statistics and loss curves. When training is complete, review the results:

  1. Check the evaluation metrics in the evaluation tab.
  2. Inspect the holdout set through the Download button, or through the SQL command shown in Snowflake (for SPCS deployments).
  3. Explore example predictions in the explorer tab, which shows the specific predictions made by the model and how they match actual outcomes.

Conclusion

šŸš€ You have now successfully created and trained your first predictive model using Kumo. Here is a summary of what this demo covers:

  1. Set Up Your Connector: Connected your data source to Kumo.
  2. Register Table Schema: Defined the table schemas.
  3. Create Your Graph: Defined the relationships between tables.
  4. Create Your PQuery: Wrote and trained models through a predictive query to make predictions.

Next, you can start training models on your data. To learn how to write a PQuery, refer to Understanding Predictive Queries. You can also learn how to create batch predictions or automate workflows with the Python SDK.