> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Connect Data on Linux

> Read local files, object storage, and warehouse tables from Linux

Connect Kumo running on Linux to your source tables. Complete [Linux setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment) first. A connector identifies the source and authentication settings; it does not grant access.

## Choose a source

| Source               | Connection                                    |
| -------------------- | --------------------------------------------- |
| Local CSV or Parquet | [Local files](#local-files)                   |
| Amazon S3            | [S3 bucket or prefix](#amazon-s3)             |
| Google Cloud Storage | [GCS bucket or prefix](#google-cloud-storage) |
| BigQuery             | [BigQuery dataset](#bigquery)                 |
| Snowflake            | [Snowflake schema](#snowflake)                |
| Databricks           | [Unity Catalog schema](#databricks)           |

Import Kumo before creating your chosen connector. After the source procedure, [select and check a table](#select-and-check-a-table) to verify schema access:

```python
import kumo.engine as kumo
```

## Local files

Set `root_dir` to the directory containing your CSV or Parquet data. For example, if `/data/kumo-input` contains `orders.csv` and `customers.parquet`:

```python
files = kumo.FileConnector(root_dir="/data/kumo-input")

orders_src = files.table("orders.csv")
customers_src = files["customers.parquet"]
print(list(orders_src.column_dict))
```

You can omit `.csv` or `.parquet` from the table name. For a table split across multiple files, use the relative directory name, such as `files.table("events")`, and keep its parts in one format.

CSV parts must sit directly inside the table directory, with at most 128 files and 10 GiB total per table; nested Parquet requires Hive `key=value` partition directories. Use real directories instead of child directory symlinks, and narrow the connector root or prefix if discovery exceeds 200 tables or 100,000 scanned entries.

Table names must be relative to `root_dir`. Absolute paths, remote URIs, and `..` traversal are not supported. Keep the source directory separate from the scratch directory configured during Linux setup; neither directory can contain the other. Kumo reads these files directly without a separate upload step or Kumo service account. The Python process needs permission to read the files and traverse their directories. Keep source files unchanged while Kumo is reading them. For a missing file, check the root directory and filename as the user running Python.

## Amazon S3

For a modular installation, add the S3 integration in the virtual environment created during [Linux setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment#install-kumo):

```bash
pip install "kumo-engine[s3]"
```

If your supplied Kumo bundle already includes the S3 connector and its dependencies, follow the bundle instructions instead.

Use a bucket and optional prefix as `root_dir`. The Python process uses the AWS credential chain, such as an attached IAM role or credentials configured for your shell. That identity needs permission to list the relevant prefix and read its objects.

```python
s3 = kumo.S3Connector(root_dir="s3://my-bucket/kumo-input/")

print(s3.table_names())
orders_src = s3.table("orders.parquet")
```

For a table split into files, pass its directory name relative to the root. To address different buckets or prefixes through one connector, omit `root_dir` and pass a full URI:

```python
s3 = kumo.S3Connector()
orders_src = s3.table("s3://my-bucket/kumo-input/orders.parquet")
```

Pass `identity=kumo.S3Identity(role_arn="<role_arn>")` to assume a role using your existing AWS credentials. `S3Identity` also accepts `access_key_id` and `secret_access_key` together, with an optional `session_token`. Supply secrets from your credential management system. The optional `region` connector argument requires `root_dir`.

## Google Cloud Storage

For a modular installation, add the GCS integration in the virtual environment created during [Linux setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment#install-kumo):

```bash
pip install "kumo-engine[gcs]"
```

If your supplied Kumo bundle already includes the GCS connector and its dependencies, follow the bundle instructions instead.

Configure Google Application Default Credentials for the Linux process. The authenticated principal needs permission to list the relevant bucket prefix and read its objects.

```python
gcs = kumo.GCSConnector(root_dir="gs://my-bucket/kumo-input/")

print(gcs.table_names())
customers_src = gcs.table("customers.parquet")
```

As with S3, table names can identify a file or a directory of table parts. Without `root_dir`, use a full `gs://` URI.

For explicit authentication, pass `identity=kumo.GCSIdentity(service_account_key=key)` where `key` is the parsed JSON from your service account key file. Alternatively, use `GCSIdentity(workload_service_account_email="<service_account_email>")` to impersonate a service account through your Application Default Credentials. The calling principal must have permission to impersonate that service account. If you set both options, Kumo uses `service_account_key` and ignores `workload_service_account_email`. For a requester pays bucket, set `requester_pays_project` to the project to bill.

## BigQuery

Install the matching BigQuery integration in your configured Kumo environment:

```bash
pip install "kumo-engine[bigquery]"
```

Configure Google Application Default Credentials for the Linux process, then select them explicitly with `BigQueryIdentity()`. Set the project, dataset, and dataset location to match your source:

```python
bigquery = kumo.BigQueryConnector(
    identity=kumo.BigQueryIdentity(),
    project_id="my-gcp-project",
    dataset_id="analytics",
    location="US",
)

print(bigquery.table_names())
orders_src = bigquery.table("orders")
```

The authenticated principal needs access to the dataset and tables, plus the query and BigQuery Storage Read permissions needed to read them. Set `billing_project_id` if queries should be billed to a different project. Pass the table name within the configured dataset.

For an explicit service account key, use `BigQueryIdentity(service_account_key=key)` with the parsed key JSON. To impersonate a service account through Application Default Credentials, use `BigQueryIdentity(workload_service_account_email="<service_account_email>")` with the required impersonation permission. If you set both options, Kumo uses `service_account_key` and ignores `workload_service_account_email`.

## Snowflake

Install the required Snowflake integration in the virtual environment created during [Linux setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment#install-kumo), even if the base Kumo Engine package is already installed:

```bash
pip install "kumo-engine[snowflake]"
```

Your selected user and role need access to the warehouse, database, schema, and source tables. Use an entry in your Snowflake `connections.toml` that contains an account, user, and either a password or a private key file. Replace `analytics` with your connection name:

```python
snowflake = kumo.SnowflakeConnector.from_connection_toml(
    connection_name="analytics",
    warehouse="COMPUTE_WH",
    database="ANALYTICS",
    schema_name="PUBLIC",
)

print(snowflake.table_names())
orders_src = snowflake.table("ORDERS")
```

The warehouse, database, schema, and optional `role` can come from the connection entry or be overridden in this call. The entry uses `schema` for the schema name; the Python argument is `schema_name`. This helper accepts password or private key file authentication. It does not convert an `externalbrowser` or SSO connection into a Kumo identity.

## Databricks

Install the required Databricks integration in the virtual environment created during [Linux setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment#install-kumo), even if the base Kumo Engine package is already installed:

```bash
pip install "kumo-engine[databricks]"
```

Create an OAuth service principal identity for your workspace. Replace the workspace URL, client ID, warehouse ID, catalog, and schema with your values. Obtain the client secret from your credential management system; this interactive example prompts for it without echoing it:

```python
from getpass import getpass

identity = kumo.DatabricksIdentity(
    host="https://<workspace>.cloud.databricks.com",
    client_id="<service_principal_client_id>",
    client_secret=getpass("Databricks client secret: "),
)

databricks = kumo.DatabricksConnector(
    identity=identity,
    warehouse_id="<warehouse_id>",
    catalog="main",
    schema_name="sales",
)

print(databricks.table_names())
orders_src = databricks.table("orders")
```

Specify exactly one of `warehouse_id` or `cluster_id`. The service principal needs access to that compute resource and permission to use the catalog and schema and read the tables. Use an unqualified table name: `orders` resolves to `main.sales.orders` in this example. `DatabricksIdentity` has no personal access token argument; the explicit OAuth example requires a client ID and client secret.

Prediction writes from Linux use the same connector schema, `<schema>__kumo__<table>` destination format, and `_kumo_staging` volume permissions as [Databricks notebook outputs](/kumo-engine/1.0.0/user-code-guide/connect-your-data-databricks#prepare-prediction-outputs).

Reads use CloudFetch by default. To stage Parquet in a Unity Catalog volume before downloading it, set `staging_volume_path="/Volumes/<catalog>/<schema>/<volume>/<prefix>"` and grant the required volume access. Add `require_staged=True` if a read must fail rather than fall back when staging is unavailable.

## Select and check a table

After creating your chosen connector above, assign it to `connector` (for example, `connector = files` for local files). Use either `connector.table("orders")` or `connector["orders"]` to obtain a `SourceTable` reference. Creating that reference does not read the table or verify access.

For a connector with a configured directory, bucket prefix, or warehouse schema, list the available tables and inspect a selected table's columns:

```python
connector = files  # Replace files with s3, gcs, bigquery, snowflake, or databricks.
print(connector.table_names())

orders_src = connector.table("orders")
print(list(orders_src.column_dict))
```

Replace `orders` with a name from the listing, preserving its capitalization. `column_dict` reads the source schema and returns a dictionary keyed by column name, with source column metadata including reported types. Successful schema access confirms that Kumo can inspect that table; it does not establish permission for every later operation on the source or prove that graph ingestion will fit in memory or storage. If access fails, check the selected location, credentials, permissions, and network access from the current environment.

`connector.has_table("orders")` and `"orders" in connector` check whether a name appears in the listing. S3 and GCS connectors without `root_dir` cannot list tables, so use a full URI and inspect that table directly.

Continue to [Tables](/kumo-engine/1.0.0/user-code-guide/set-up-a-graph/tables) when the expected source columns appear.

## Advanced configuration

### Preview source rows

Inspect a small source sample before modeling. The returned row limit does not bound source scanning or cost.

<details>
  <summary>
    Read and decode a source sample
  </summary>

  Read a few source rows before building a graph. This local CSV example requires the Engine environment from [Linux setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/linux-machine-deployment), including pandas, and a readable `orders.csv`. Replace `/data/kumo-input` with its containing directory and `orders` with your table name. Follow the [local file requirements](#local-files).

  ```python
  from io import StringIO

  import pandas as pd
  import kumo.engine as kumo

  connector = kumo.FileConnector(root_dir="/data/kumo-input")
  sample = connector.source_backend.get_sample(
      connector, "orders", sample_rows=5
  )
  preview = pd.read_json(StringIO(sample.sample_rows), orient="table")
  print(preview)
  ```

  `preview` is a pandas DataFrame containing up to five rows and the source columns. `sample.sample_rows` holds JSON in pandas table format, so decode it with `orient="table"`. For a local CSV, this returns the first rows, not a random sample. If the table is missing, check the filename and connector root directory.

  The row limit bounds the returned preview, not source scanning or scan cost. A successful preview confirms this read; it does not validate full graph ingestion or later writes.
</details>

### Snowflake key pair or password authentication

Use an explicit identity when your credentials are not in a `connections.toml` entry.

<details>
  <summary>
    Create an explicit Snowflake identity
  </summary>

  To supply credentials directly, create a `SnowflakeIdentity` and pass it to the connector. This example reads a PEM private key from a file you manage:

  ```python
  from pathlib import Path

  snowflake_identity = kumo.SnowflakeIdentity(
      account="<account_identifier>",
      username="<username>",
      private_key=Path("/secure/snowflake-key.pem").read_text(),
  )

  snowflake = kumo.SnowflakeConnector(
      identity=snowflake_identity,
      warehouse="COMPUTE_WH",
      database="ANALYTICS",
      schema_name="PUBLIC",
      role="<role_name>",
  )
  ```

  For an encrypted key, also supply `private_key_passphrase`. Password authentication uses `password` instead of `private_key`. The selected user and role need access to the warehouse, database, schema, and source tables. Pass the table name within the configured schema, such as `ORDERS`.
</details>

### BigQuery staging and prediction permissions

Direct reads use the Storage Read API. Optional GCS staging and prediction writes require additional permissions.

<details>
  <summary>
    Configure staging and output access
  </summary>

  Reads use the BigQuery Storage Read API by default.
  To export Parquet to GCS before downloading it, set `staging_uri="gs://my-staging-bucket/kumo/"` to a prefix you control and grant the required export and bucket access.
  Add `require_staged=True` if a read must fail rather than fall back when staging is unavailable.

  For prediction output, configure the connector with the destination project, dataset, and location.
  The dataset must already exist; the writer can create its destination table.
  Grant the principal `roles/bigquery.dataEditor` on that dataset and `roles/bigquery.jobUser` on the billing project (`billing_project_id`, or `project_id` when omitted).
  These grants cover table creation and writes plus the jobs used by staged loads and streamed overwrite.
  For existing tables, writes require `bigquery.tables.get` and `bigquery.tables.updateData`; overwrite also requires `bigquery.tables.update`.
  Enable the BigQuery Storage API for streaming writes.
  The saved-table check also needs permission to read and query the output table.

  Output uses the Storage Write API unless a usable `staging_uri` is configured.
  For staging, use a bucket compatible with the dataset location and grant object create, read, list, and delete access for Kumo's staging objects under your chosen prefix.
  Kumo loads the uploaded Parquet into BigQuery and cleans up its staged objects.
  `require_staged=True` also prevents output from falling back to streaming.
  See [BigQuery Output](/kumo-engine/1.0.0/user-code-guide/making-predictions#bigquery-output) for prediction, existing-result export, write modes, and saved-table verification.
  BigQuery model-bundle publication and retrieval are unavailable through this connector.
</details>

## Reuse a Connector

Keep the connector object in a variable. Assign your source connector to `connector` (for example, `connector = snowflake` for the Linux Snowflake example). Select additional tables within its configured location:

```python
orders_src = connector.table("ORDERS")
customers_src = connector.table("CUSTOMERS")
```

After restarting Python, construct the connector again with the same source location and authentication settings. In a notebook, repeat its Connect Your Data setup after a kernel restart. Connectors are Python objects and do not use a persistent Kumo connector registry or a `get_by_name()` lookup. A name passed to `from_connection_toml()` selects an entry in your Snowflake configuration file.

Source schemas are cached on each `SourceTable` object. If the source schema changes, create a new reference with `connector.table(...)` before inspecting its columns again.

## Prepare Data Before Modeling

For Snowflake, Databricks, or BigQuery, [create a table from read-only SQL](/kumo-engine/1.0.0/user-code-guide/set-up-a-graph/tables#create-tables-from-sql) with `Table.from_sql(connector, query, ...)`.
You can also pass the source object returned by `connector.sql(query)` to `Table.from_source_table()`.
Keep query references inside the connector's configured database and schema, catalog and schema, or project and dataset.

For file and object-storage sources, prepare derived columns or cleaned files with your existing data tools, then select the resulting files through the connector.
Warehouse tables and views prepared separately remain usable through `connector.table(...)`.

Continue to [Tables](/kumo-engine/1.0.0/user-code-guide/set-up-a-graph/tables) to select columns and declare primary keys, time columns, and feature types.