SolutionsBusiness Operations

Entity Resolution

View as Markdown

Solution Background and Business Value

Entity resolution enables companies to identify and merge records that refer to the same real-world entity (such as customers, products, or businesses) across different data sources. Consolidating these records produces a more accurate and holistic view of each entity, improving data quality and reducing noise in downstream predictive tasks.

Traditional rule-based approaches rely on manually crafted heuristics that are difficult to maintain at scale and struggle to capture the complex relationships between different data fields. Kumo AI’s feature-based learning and graph neural network approach creates context-aware embeddings that identify subtle and non-obvious links between records, making it well-suited for entity resolution.

This example shows how to build a link prediction model that identifies accounts created by the same user across two different platforms.

Data Requirements and Schema

An effective entity resolution model requires a structured set of tables that captures relevant user data for both platforms and encodes the signals needed for matching. Additional tables and attributes generally increase model accuracy, though a minimum schema is sufficient to get started.

The most critical component is the labels table. Kumo trains supervised models, so high-quality labels are required for accurate results. Each row of the labels table represents an established link between two accounts on different platforms. This table must be generated before training, either from prior data or by selecting the highest-confidence signal as ground truth. As the model identifies new pairs, the label table can be updated with new entries to improve accuracy over time.

In this example, the highest-confidence signal is email: two users sharing the same email address is treated as ground truth for a link. Device ID is a medium-confidence signal: two users accessing a platform through the same device indicates a likely link. To add other signals (such as IP addresses or content links), follow the same structure used for device signals: a shared table with connections to users from both platforms.

Core Tables

  1. Platform A User Data:
    • Stores data about each user from platform A, using email as an identifier
    • Note: Emails are omitted from the table to prevent data leakage during training
    • Key attributes:
      • platform_a_user_id : unique user identifier for platform A
      • first_seen: user creation date
      • last_seen: last time a user was seen
      • Optional: Other user attributes (age, gender, location, and so on)
  2. Platform B User Data:
    • Stores data about each user from platform B, using email as an identifier
    • Contains similar information to the platform A user data table
    • Key attributes:
      • platform_b_user_id : unique user identifier for platform B
      • first_seen: user creation date
      • last_seen: last time a user was seen
      • Optional: Other user attributes (age, gender, location, and so on)
  3. Platform A User Sessions:
    • Stores data about each user session from platform A
    • Key attributes:
      • platform_a_session_id : unique session identifier for platform A
      • platform_a_user_id : the user from platform A this session belonged to
      • create_date : create date of the session
      • device_id : device used for this session
      • Optional: ip address, duration, location, and so on
  4. Platform B User Sessions:
    • Stores data about each user session from platform B
    • Contains similar information about user sessions as those from platform A
    • Key attributes:
      • platform_b_session_id : unique session identifier for platform B
      • platform_b_user_id : the user from platform B this session belonged to
      • create_date : create date of the session
      • device_id : device used for this session
      • Optional: ip address, duration, location, and so on
  5. Device Data:
    • Stores data about each device used by users from both platforms A and B
    • Key attributes:
      • device_id : unique device identifier
      • device_type : device type
      • Optional: device brand, device model, and so on
  6. Labels Table:
    • Stores established links between users from platform A and platform B
    • Key attributes:
      • link_id : unique identifier for each link
      • platform_a_user_id : identifier for a user from platform A
      • platform_b_user_id : identifier for a user from platform B

Entity Relationship Diagram (ERD)

Predictive Query

This predictive query relies on a labels table that must be pre-generated and represents the link between the two user tables. At prediction time, this query generates, for each user from platform A, the top N most likely users from platform B that represent the same entity. Filtering by a confidence column restricts training to high-quality labels and can improve model accuracy.

PREDICT LIST_DISTINCT(labels.platform_b_user_id 
WHERE labels.confidence='High') 
RANK TOP N
FOR EACH platform_a_users.platform_a_user_id

Next Steps

The link prediction model generates a list of ranked candidate pairs for entity resolution, but does not guarantee that any duplicate users exist across platforms A and B. The model narrows the set of pairs that need review: all predictions still require manual verification.

To further automate this pipeline, train a separate binary classification model that generates a probability score for each candidate pair being a true match. The table structure does not change; only the label table and predictive query differ. A probability threshold can then be used to automatically flag candidate pairs of duplicate users.