Connect Data on Linux

Connect Kumo running on Linux to your source tables. Complete Linux setup first. A connector identifies the source and authentication settings; it does not grant access.

Choose a source

Import Kumo before creating your chosen connector. After the source procedure, select and check a table to verify schema access:

import kumo.engine as kumo

Local files

Set root_dir to the directory containing your CSV or Parquet data. For example, if /data/kumo-input contains orders.csv and customers.parquet:

files = kumo.FileConnector(root_dir="/data/kumo-input")
orders_src = files.table("orders.csv")
customers_src = files["customers.parquet"]
print(list(orders_src.column_dict))

You can omit .csv or .parquet from the table name. For a table split across multiple files, use the relative directory name, such as files.table("events"), and keep its parts in one format.

CSV parts must sit directly inside the table directory, with at most 128 files and 10 GiB total per table; nested Parquet requires Hive key=value partition directories. Use real directories instead of child directory symlinks, and narrow the connector root or prefix if discovery exceeds 200 tables or 100,000 scanned entries.

Table names must be relative to root_dir. Absolute paths, remote URIs, and .. traversal are not supported. Keep the source directory separate from the scratch directory configured during Linux setup; neither directory can contain the other. Kumo reads these files directly without a separate upload step or Kumo service account. The Python process needs permission to read the files and traverse their directories. Keep source files unchanged while Kumo is reading them. For a missing file, check the root directory and filename as the user running Python.

Amazon S3

For a modular installation, add the S3 integration in the virtual environment created during Linux setup:

pip install "kumo-engine[s3]"

If your supplied Kumo bundle already includes the S3 connector and its dependencies, follow the bundle instructions instead.

Use a bucket and optional prefix as root_dir. The Python process uses the AWS credential chain, such as an attached IAM role or credentials configured for your shell. That identity needs permission to list the relevant prefix and read its objects.

s3 = kumo.S3Connector(root_dir="s3://my-bucket/kumo-input/")
print(s3.table_names())
orders_src = s3.table("orders.parquet")

For a table split into files, pass its directory name relative to the root. To address different buckets or prefixes through one connector, omit root_dir and pass a full URI:

s3 = kumo.S3Connector()
orders_src = s3.table("s3://my-bucket/kumo-input/orders.parquet")

Pass identity=kumo.S3Identity(role_arn="<role_arn>") to assume a role using your existing AWS credentials. S3Identity also accepts access_key_id and secret_access_key together, with an optional session_token. Supply secrets from your credential management system. The optional region connector argument requires root_dir.

Google Cloud Storage

For a modular installation, add the GCS integration in the virtual environment created during Linux setup:

pip install "kumo-engine[gcs]"

If your supplied Kumo bundle already includes the GCS connector and its dependencies, follow the bundle instructions instead.

Configure Google Application Default Credentials for the Linux process. The authenticated principal needs permission to list the relevant bucket prefix and read its objects.

gcs = kumo.GCSConnector(root_dir="gs://my-bucket/kumo-input/")
print(gcs.table_names())
customers_src = gcs.table("customers.parquet")

As with S3, table names can identify a file or a directory of table parts. Without root_dir, use a full gs:// URI.

For explicit authentication, pass identity=kumo.GCSIdentity(service_account_key=key) where key is the parsed JSON from your service account key file. Alternatively, use GCSIdentity(workload_service_account_email="<service_account_email>") to impersonate a service account through your Application Default Credentials. The calling principal must have permission to impersonate that service account. If you set both options, Kumo uses service_account_key and ignores workload_service_account_email. For a requester pays bucket, set requester_pays_project to the project to bill.

BigQuery

Install the matching BigQuery integration in your configured Kumo environment:

pip install "kumo-engine[bigquery]"

Configure Google Application Default Credentials for the Linux process, then select them explicitly with BigQueryIdentity(). Set the project, dataset, and dataset location to match your source:

bigquery = kumo.BigQueryConnector(
    identity=kumo.BigQueryIdentity(),
    project_id="my-gcp-project",
    dataset_id="analytics",
    location="US",
)
print(bigquery.table_names())
orders_src = bigquery.table("orders")

The authenticated principal needs access to the dataset and tables, plus the query and BigQuery Storage Read permissions needed to read them. Set billing_project_id if queries should be billed to a different project. Pass the table name within the configured dataset.

For an explicit service account key, use BigQueryIdentity(service_account_key=key) with the parsed key JSON. To impersonate a service account through Application Default Credentials, use BigQueryIdentity(workload_service_account_email="<service_account_email>") with the required impersonation permission. If you set both options, Kumo uses service_account_key and ignores workload_service_account_email.

Snowflake

Install the required Snowflake integration in the virtual environment created during Linux setup, even if the base Kumo Engine package is already installed:

pip install "kumo-engine[snowflake]"

Your selected user and role need access to the warehouse, database, schema, and source tables. Use an entry in your Snowflake connections.toml that contains an account, user, and either a password or a private key file. Replace analytics with your connection name:

snowflake = kumo.SnowflakeConnector.from_connection_toml(
    connection_name="analytics",
    warehouse="COMPUTE_WH",
    database="ANALYTICS",
    schema_name="PUBLIC",
)
print(snowflake.table_names())
orders_src = snowflake.table("ORDERS")

The warehouse, database, schema, and optional role can come from the connection entry or be overridden in this call. The entry uses schema for the schema name; the Python argument is schema_name. This helper accepts password or private key file authentication. It does not convert an externalbrowser or SSO connection into a Kumo identity.

Databricks

Install the required Databricks integration in the virtual environment created during Linux setup, even if the base Kumo Engine package is already installed:

pip install "kumo-engine[databricks]"

Create an OAuth service principal identity for your workspace. Replace the workspace URL, client ID, warehouse ID, catalog, and schema with your values. Obtain the client secret from your credential management system; this interactive example prompts for it without echoing it:

from getpass import getpass
identity = kumo.DatabricksIdentity(
    host="https://<workspace>.cloud.databricks.com",
    client_id="<service_principal_client_id>",
    client_secret=getpass("Databricks client secret: "),
)
databricks = kumo.DatabricksConnector(
    identity=identity,
    warehouse_id="<warehouse_id>",
    catalog="main",
    schema_name="sales",
)
print(databricks.table_names())
orders_src = databricks.table("orders")

Specify exactly one of warehouse_id or cluster_id. The service principal needs access to that compute resource and permission to use the catalog and schema and read the tables. Use an unqualified table name: orders resolves to main.sales.orders in this example. DatabricksIdentity has no personal access token argument; the explicit OAuth example requires a client ID and client secret.

Prediction writes from Linux use the same connector schema, <schema>__kumo__<table> destination format, and _kumo_staging volume permissions as Databricks notebook outputs.

Reads use CloudFetch by default. To stage Parquet in a Unity Catalog volume before downloading it, set staging_volume_path="/Volumes/<catalog>/<schema>/<volume>/<prefix>" and grant the required volume access. Add require_staged=True if a read must fail rather than fall back when staging is unavailable.

Select and check a table

After creating your chosen connector above, assign it to connector (for example, connector = files for local files). Use either connector.table("orders") or connector["orders"] to obtain a SourceTable reference. Creating that reference does not read the table or verify access.

For a connector with a configured directory, bucket prefix, or warehouse schema, list the available tables and inspect a selected table’s columns:

connector = files  # Replace files with s3, gcs, bigquery, snowflake, or databricks.
print(connector.table_names())
orders_src = connector.table("orders")
print(list(orders_src.column_dict))

Replace orders with a name from the listing, preserving its capitalization. column_dict reads the source schema and returns a dictionary keyed by column name, with source column metadata including reported types. Successful schema access confirms that Kumo can inspect that table; it does not establish permission for every later operation on the source or prove that graph ingestion will fit in memory or storage. If access fails, check the selected location, credentials, permissions, and network access from the current environment.

connector.has_table("orders") and "orders" in connector check whether a name appears in the listing. S3 and GCS connectors without root_dir cannot list tables, so use a full URI and inspect that table directly.

Continue to Tables when the expected source columns appear.

Advanced configuration

Preview source rows

Inspect a small source sample before modeling. The returned row limit does not bound source scanning or cost.

from io import StringIO
import pandas as pd
import kumo.engine as kumo
connector = kumo.FileConnector(root_dir="/data/kumo-input")
sample = connector.source_backend.get_sample(
    connector, "orders", sample_rows=5
)
preview = pd.read_json(StringIO(sample.sample_rows), orient="table")
print(preview)

Snowflake key pair or password authentication

Use an explicit identity when your credentials are not in a connections.toml entry.

from pathlib import Path
snowflake_identity = kumo.SnowflakeIdentity(
    account="<account_identifier>",
    username="<username>",
    private_key=Path("/secure/snowflake-key.pem").read_text(),
)
snowflake = kumo.SnowflakeConnector(
    identity=snowflake_identity,
    warehouse="COMPUTE_WH",
    database="ANALYTICS",
    schema_name="PUBLIC",
    role="<role_name>",
)

BigQuery staging and prediction permissions

Direct reads use the Storage Read API. Optional GCS staging and prediction writes require additional permissions.

Reuse a Connector

Keep the connector object in a variable. Assign your source connector to connector (for example, connector = snowflake for the Linux Snowflake example). Select additional tables within its configured location:

orders_src = connector.table("ORDERS")
customers_src = connector.table("CUSTOMERS")

After restarting Python, construct the connector again with the same source location and authentication settings. In a notebook, repeat its Connect Your Data setup after a kernel restart. Connectors are Python objects and do not use a persistent Kumo connector registry or a get_by_name() lookup. A name passed to from_connection_toml() selects an entry in your Snowflake configuration file.

Source schemas are cached on each SourceTable object. If the source schema changes, create a new reference with connector.table(...) before inspecting its columns again.

Prepare Data Before Modeling

For Snowflake, Databricks, or BigQuery, create a table from read-only SQL with Table.from_sql(connector, query, ...). You can also pass the source object returned by connector.sql(query) to Table.from_source_table(). Keep query references inside the connector’s configured database and schema, catalog and schema, or project and dataset.

For file and object-storage sources, prepare derived columns or cleaned files with your existing data tools, then select the resulting files through the connector. Warehouse tables and views prepared separately remain usable through connector.table(...).

Continue to Tables to select columns and declare primary keys, time columns, and feature types.