AWS S3

Kumo supports reading CSV and Apache Parquet files from AWS S3. The connector reads from a specific top-level directory, which can contain either files or second-level subdirectories. Each file or second-level directory represents a single table in Kumo.

Directory Configuration

• If the top-level directory contains files, Kumo ignores any subdirectories and ingests only the files as tables.
• If the top-level directory contains subdirectories, each second-level directory is treated as a separate table, subject to the following rules:
  • All files within a subdirectory must be of the same type (either CSV or Parquet).
  • All files must share the same column schema (for CSV, this means identical headers).
  • Subdirectories must not contain additional nested directories - only CSV or Parquet files are allowed.
  • The resulting table combines all rows from all files in the subdirectory.
• Ensure your S3 bucket name does not contain dots (.) to comply with AWS virtual-hosted style URL requirements.

Example Storage Structure

|-- Root directory
  |-- Table 1
    |-- File 1 (CSV or Parquet)
    |-- File 2 (CSV or Parquet)
    |-- ...
  |-- Table 2
    |-- File 1 (CSV or Parquet)
    |-- File 2 (CSV or Parquet)
    |-- ...
  |-- ...

Data Format and Size Considerations

Parquet Files

• Cast columns to their proper data types (for example, timestamp for dates) to streamline ingestion and validation.
• Hive partitioned tables are not supported.
• The data size per partition must not exceed 512 MB.

CSV Files

• A CSV table can have up to 128 partitions.
• The total data size across all partitions must not exceed 10 GB.

Granting Access

To allow Kumo to access your data, update your S3 bucket policy by replacing:

• <data bucket name> with your actual S3 bucket name.
• {% $customerId %} with your company name.

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Principal": {
7
        "AWS": "arn:aws:iam::926922431314:role/kumo-{% $customerId %}-external-shared-iam-role"
8
        },
9
      "Action": [
10
        "s3:GetObject",
11
        "s3:PutObject",
12
        "s3:ListBucket"
13
      ],
14
      "Resource": [
15
        "arn:aws:s3:::<data bucket name>/*",
16
        "arn:aws:s3:::<data bucket name>"
17
      ]
18
    }
19
  ]
20
}

For step-by-step instructions, refer to the AWS Add Bucket Policy guide or contact Kumo Support for assistance.

Once the policy is applied, share the details with Kumo Support to finalize access configuration.

Data Sharing Steps - Updating KMS Key Policy

To allow an external IAM role to access an S3 bucket encrypted with a KMS key in a different account, update the KMS key policy:

1. Locate the KMS Key Policy attached to the S3 bucket in the account where the KMS key is defined.
2. Append the following statement to the existing KMS Key Policy to grant access to the kumo-{% $customerId %}-external-shared-iam-role.

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Sid": "AllowExternalRoleToUseKey",
6
      "Effect": "Allow",
7
      "Principal": {
8
       "AWS": "arn:aws:iam::926922431314:role/kumo-{% $customerId %}-external-shared-iam-role"
9
      },
10
      "Action": [
11
        "kms:Decrypt",
12
        "kms:GenerateDataKey"
13
      ],
14
      "Resource": "*"
15
    }
16
  ]
17
}

For guidance on updating the KMS key policy, refer to the AWS Add KMS Key Policy for S3 Bucket guide or contact your Kumo Success Manager.

After applying the policy, share the KMS Key ARN with your Kumo contact (for example, via Slack or email). They will use it to apply a corresponding policy update to Kumo’s AWS account role: kumo-{% $customerId %}-external-shared-iam-role.