AIStore & Amazon S3 Compatibility

AIStore (AIS) is a lightweight, distributed object storage system designed to scale linearly with each added storage node. It provides a uniform API across various storage backends while maintaining high performance for AI/ML and data analytics workloads.

AIS integrates with Amazon S3 on three fronts:

1. Backend storage – via the backend provider abstraction, AIStore can be utiized to access (and cache or reliably store in-cluster) a remote cloud bucket such as s3://my-bucket (aws:// is accepted as an alias). This provides seamless access to existing S3 data.

2. Front‑end compatibility – every gateway speaks the S3 REST API. The default endpoint is http(s)://gw-host:port/s3, but you can enable the S3-API-via-Root feature flag to serve requests at the cluster root (http(s)://gw-host:port/). The same API works uniformly across all bucket types—native ais://, cloud‑backed s3://, gs://, and more.

3. Presigned request offload – AIS can receive a presigned S3 URL, execute it, and store the resulting object in the cluster. This lets you leverage S3’s authentication while using AIS for storage.

Which interface should I use?

AIS exposes a pure S3 surface for seamless compatibility and a native API for advanced, cluster‑aware workloads. The table below helps decide which path fits your scenario:

Table of Contents

• Quick Start
  • aws CLI
  • s3cmd
• S3 Clients: Two Distinct Types
• Configuring Clients
  • Finding the AIS endpoint
  • Checksum considerations
  • HTTPS vs HTTP
• Using s3cmd with AIS
  • Interactive s3cmd --configure transcript
  • Example .s3cfg
  • Multipart uploads
  • JWT authentication
• Supported Operations
  • PUT / GET / HEAD
  • Range reads
  • Multipart uploads (aws CLI)
  • Presigned requests
• Use Native Bucket Inventory
• Deleting nonexistent object
• Compatibility Matrix
• Boto3 Examples
• FAQs & Troubleshooting
• Further reading

Environment assumption – Local Playground

The CLI examples below use localhost:8080, which is the default endpoint when running AIS in the Local Playground.

For other deployment modes (including Kubernetes Playground, Docker Compose, bare-metal cluster, or Kubernetes for production deployments) — replace the host:port with any AIS gateway endpoint.

See Deployment Options and the main project Features list for a broader overview.

Quick Start

Quick start with aws CLI

1
AWS_EP=http://localhost:8080/s3
2
aws --endpoint-url "$AWS_EP" s3 mb s3://demo
3
aws --endpoint-url "$AWS_EP" s3 cp README.md s3://demo/
4
aws --endpoint-url "$AWS_EP" s3 ls s3://demo
5
# Expected output:
6
# 2023-05-14 14:25     10493   s3://demo/README.md

Quick start with s3cmd

One‑liner (HTTP):

1
s3cmd put README.md s3://demo \
2
  --no-ssl \
3
  --host=localhost:8080/s3 \
4
  --host-bucket="localhost:8080/s3/%(bucket)"
5
6
# Expected output:
7
# upload: 'README.md' -> 's3://demo/README.md' [1 of 1]
8
# 10493 of 10493   100% in    0s     4.20 MB/s  done

Tip — use a cluster‑specific .s3cfg so you can drop the --host* flags. See the Example .s3cfg section below.

S3 Clients: Two Distinct Types

Basic facts first:

• AIS provides an S3-compatible API - there is no need for AIS-specific S3 clients.
• All AIS clients are expected to follow standard HTTP semantics.
• In particular, any client that implements standard HTTP redirect handling is fully supported.

The last point drives the distinction between two kinds of S3 clients:

1. those that follow 307 Temporary Redirect by resending the original request - same method, URI taken verbatim from the Location header;
2. and those that don’t.

For the first kind, everything just works. The redirected Location URI carries the proxy’s signature and all query parameters the designated target needs to authenticate the request.

Conforming redirect handling - see RFC 9110 Location, Redirection 3xx, 307 Temporary Redirect - preserves both the method and the URI as given. AIStore’s signed redirects depend on exactly this behavior.

Known-good clients include:

• Python clients that drive the S3 API directly via urllib3 with redirect handling enabled;
• AWS SDK for Go v2, which follows 307 responses to the Location URI verbatim;
• Boto3 and other botocore-based applications using the AIStore botocore redirect patch.

Clients that reconstruct S3 endpoints

The second kind does not follow Location verbatim. Following S3 service conventions rather than HTTP RFC, these clients treat the XML <Endpoint> element as a reusable service address and re-derive the request themselves; some additionally refuse to redirect specific methods (notably HEAD).

A reconstructed request drops the signed query parameters. It may also arrive directly at a storage node’s public endpoint, bypassing the proxy altogether.

Either way, the receiving node cannot distinguish it from unsigned direct access. When the cluster requires proxy mediation—because AuthN or intra-cluster signing is configured—the node must reject it. With the feature described below disabled, such requests fail with AccessDenied (403) or InvalidRequest (400).

The S3-Redirect-Rebuild cluster feature exists to support these clients anyway. It is a compatibility mode and an explicit security tradeoff - not an alternative signing mechanism.

When S3-Redirect-Rebuild is enabled:

• the XML <Endpoint> reverts to its legacy form - bare host:port/s3 (otherwise, <Endpoint> carries the same full signed URI as Location);
• HEAD(object) is reverse-proxied to the designated target instead of redirected - HEAD responses carry no body and, hence, no <Endpoint> to rebuild from;
• storage nodes accept unsigned S3 requests on their public endpoints - which, accordingly, must be reachable from the clients;
• proxy mediation, and any authorization enforced exclusively by the proxy, no longer protect those requests.

Correspondingly, S3-Redirect-Rebuild cannot coexist with AuthN or auth.intra_cluster: configuration validation rejects the combination.

The feature is disabled by default. Deployments that require strict proxy mediation and authenticated intra-cluster traffic must leave it disabled and use clients that follow the HTTP Location URI.

Configuring Clients

Finding the AIS endpoint

Choose any gateway’s host:port and append /s3, e.g. 10.10.0.1:51080/s3. All gateways accept reads and writes, so you can connect to any of them.

In fact, AIS gateways are completely equivalent, API-wise.

Checksum considerations

Amazon S3’s ETag is MD5 (or a multipart hash); AIS defaults to xxhash for better performance. To avoid client mismatch warnings, set MD5 per bucket:

1
ais bucket props set ais://demo checksum.type=md5      # per bucket
2
# OR cluster‑wide default
3
ais config cluster checksum.type=md5

Setting the checksum type to MD5 ensures compatibility with S3 clients that validate checksums, though it comes with a minor performance cost compared to xxhash.

HTTPS vs HTTP

Enable TLS in ais.json (net.http.use_https=true) or pass --no-ssl/--insecure flags when using tools. By default, AIS uses HTTP, while many S3 clients expect HTTPS.

Using s3cmd with AIS

Interactive s3cmd --configure transcript

$ s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Access Key: FAKEKEY
Secret Key: FakeSecret
Default Region [US]:
Use HTTPS protocol [Yes]: n
HTTP Proxy server name:
New settings:
  Access Key: FAKEKEY
  Secret Key: FAKESECRET
  Region: US
  Use HTTPS protocol: False
  HTTP Proxy server name:
Test access with supplied credentials? [Y/n] y
Please wait, attempting to list buckets...

During the test, enter your AIS endpoint when prompted:

Hostname: localhost:8080/s3
Bucket host: localhost:8080/s3/%(bucket)

Success. Your access key and secret key worked fine :-)
...
Save settings? [y/N] y
Configuration saved to ~/.s3cfg

Example .s3cfg

Edit your ~/.s3cfg file to include these lines (replace with your actual gateway endpoint):

1
host_base = 10.10.0.1:51080/s3
2
host_bucket = 10.10.0.1:51080/s3/%(bucket)
3
access_key = FAKEKEY
4
secret_key = FAKESECRET
5
signature_v2 = False
6
use_https = False

This configuration allows you to run s3cmd commands without having to specify the host parameters each time.

Multipart uploads with s3cmd

For large files, use multipart uploads to improve reliability and performance:

1
s3cmd put ./large.bin s3://demo --multipart-chunk-size-mb=8

The optimal chunk size depends on your network conditions and file size, but 8-16MB chunks work well for most cases.

Authentication (JWT) tips

When AIStore Authentication is enabled, each request must include a JWT Bearer token in the Authorization header. However, s3cmd’s built-in AWS signer overwrites any --add-header values, so you need to patch the client directly.

Edit the sign() method of the S3Request class in s3cmd/S3/S3.py.

Add the following line to override the Authorization header:

1
$ git diff
2
diff --git a/S3/S3.py b/S3/S3.py
3
index 26c516f..1a0d5e7 100644
4
--- a/S3/S3.py
5
+++ b/S3/S3.py
6
@@ -199,6 +199,7 @@ class S3Request(object):
7
             ## Sign the data.
8
             self.headers = sign_request_v4(self.method_string, hostname, resource_uri, self.params,
9
                                           bucket_region, self.headers, self.body)
10
+        self.headers["Authorization"] = "Bearer <TOKEN>"
11
 
12
     def get_triplet(self):
13
         self.update_timestamp()

Replace <token> with your actual JWT token. This modification ensures the token is included in every request.

Supported Operations

PUT / GET / HEAD

Regular verbs work with aws, s3cmd, or the native ais CLI:

1
# Using aws CLI
2
aws --endpoint-url "$AWS_EP" s3api put-object --bucket demo --key obj --body file.txt
3
aws --endpoint-url "$AWS_EP" s3api get-object --bucket demo --key obj output.txt
4
aws --endpoint-url "$AWS_EP" s3api head-object --bucket demo --key obj
5
# Output:
6
# {
7
#     "AcceptRanges": "bytes",
8
#     "LastModified": "Wed, 14 May 2025 14:30:22 GMT",
9
#     "ContentLength": 1024,
10
#     "ETag": "\"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6\"",
11
#     "ContentType": "text/plain"
12
# }
13
14
# Native AIS CLI equivalents
15
ais put file.txt ais://demo/obj
16
ais get ais://demo/obj output.txt
17
ais object show ais://demo/obj

Range reads

S3 API supports byte range requests for partial object downloads:

1
aws s3api get-object \
2
  --range bytes=0-99 \
3
  --bucket demo --key README.md \
4
  part.txt \
5
  --endpoint-url "$AWS_EP"
6
# Output:
7
# {
8
#     "AcceptRanges": "bytes",
9
#     "LastModified": "Wed, 14 May 2025 14:30:22 GMT",
10
#     "ContentRange": "bytes 0-99/10493",
11
#     "ContentLength": 100,
12
#     "ETag": "\"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6\"",
13
#     "ContentType": "text/plain"
14
# }

This would download only the first 100 bytes of the file.

Multipart uploads with aws CLI

1
# 1 — initiate
2
aws s3api create-multipart-upload --bucket demo --key big \
3
  --endpoint-url "$AWS_EP"
4
# Output:
5
# {
6
#     "Bucket": "demo",
7
#     "Key": "big",
8
#     "UploadId": "xu3DvVzJK"
9
# }
10
11
# 2 — upload parts individually
12
aws s3api upload-part --bucket demo --key big \
13
  --part-number 1 --body part1.bin \
14
  --upload-id "YOUR-UPLOAD-ID" \
15
  --endpoint-url "$AWS_EP"
16
# Repeat for each part, incrementing the part-number
17
# Output:
18
# {
19
#     "ETag": "\"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6\""
20
# }
21
22
# 3 — complete (requires a JSON file listing all parts)
23
aws s3api complete-multipart-upload --bucket demo --key big \
24
  --upload-id "YOUR-UPLOAD-ID" --multipart-upload file://parts.json \
25
  --endpoint-url "$AWS_EP"

Example parts.json:

1
{
2
  "Parts": [
3
    {
4
      "PartNumber": 1,
5
      "ETag": "etag-from-upload-part-response"
6
    },
7
    {
8
      "PartNumber": 2,
9
      "ETag": "etag-from-upload-part-response"
10
    }
11
  ]
12
}

Presigned S3 requests

Presigned URLs allow temporary access to objects without sharing credentials:

1. Enable feature:

   1
   ais config cluster features S3-Presigned-Request

2. Generate URL (typically done on the system with AWS credentials):

   1
   aws s3 presign s3://demo/README.md --endpoint-url https://s3.amazonaws.com
   2
   # Returns a URL with authentication parameters in the query string

3. Replace the host with AIS_ENDPOINT/s3 and add header when using path style:

   1
   curl -H 'ais-s3-signed-request-style: path' '<PRESIGNED_URL>' -o README.md

This allows AIS to handle the authenticated S3 request on behalf of the client.

Use Native Bucket Inventory

The older S3-specific inventory integration has been removed in v4.4.

Use native bucket inventory (NBI) for fast, inventory-backed listing of large remote buckets, including (but not limited to) s3 buckets.

Note that S3-compatible clients may request NBI-backed listing via Ais-Bucket-Inventory: true, and optionally select a specific inventory via Ais-Inv-Name.

Deleting nonexistent object

AIStore does not emulate S3’s silent-success delete semantics. In AIS, deleting a missing object is reported as “not found” - with a single exception: when the bucket has an S3 backend. This does not violate HTTP idempotency: a repeated DELETE still has the same intended effect on server state, even though the response differs.

When the bucket does have an S3 backend and the object is missing, we return whatever the backend gives us - which, for S3, is 204 (no error).

Apart from this single exception, returning an error on delete of a nonexistent object is an intentional semantic choice for consistency with the rest of AIS.

Compatibility Matrix

Not yet supported: Regions, CORS, Website hosting, CloudFront; full ACL parity (AIS uses its own ACL model).

Boto3 Examples

Python applications can use Boto3 (the AWS SDK for Python) to connect to AIStore. Since AIStore implements S3 API compatibility, most standard Boto3 S3 operations work with minimal changes.

Prerequisites

For Boto3 to work with AIStore, you need to patch Boto3’s redirect handling:

1
# Import the patch before using boto3
2
from aistore.botocore_patch import botocore
3
import boto3

This patch modifies Boto3’s HTTP client behavior to handle AIStore’s redirect-based load balancing. For details, see the Boto3 compatibility documentation.

Client Initialization

1
client = boto3.client(
2
    "s3",
3
    region_name="us-east-1",  # Any valid region will work
4
    endpoint_url="http://localhost:8080/s3",  # Your AIStore endpoint
5
    aws_access_key_id="YOUR_ACCESS_KEY",      # Can be dummy values when
6
    aws_secret_access_key="YOUR_SECRET_KEY",  # AIS auth is disabled
7
)

Basic Bucket Operations

1
# Create a bucket
2
client.create_bucket(Bucket="my-bucket")
3
4
# List all buckets
5
response = client.list_buckets()
6
bucket_names = [bucket["Name"] for bucket in response["Buckets"]]
7
print(f"Existing buckets: {bucket_names}")
8
9
# Delete a bucket
10
client.delete_bucket(Bucket="my-bucket")

Object Operations

1
# Upload an object
2
client.put_object(
3
    Bucket="my-bucket",
4
    Key="sample.txt",
5
    Body="Hello, AIStore!"
6
)
7
8
# Download an object
9
response = client.get_object(Bucket="my-bucket", Key="sample.txt")
10
content = response["Body"].read().decode("utf-8")
11
print(f"Object content: {content}")
12
13
# Delete an object
14
client.delete_object(Bucket="my-bucket", Key="sample.txt")

Multipart Upload Example

For large files, you can use multipart uploads:

1
import boto3
2
from aistore.botocore_patch import botocore
3
4
# Initialize the client
5
client = boto3.client(
6
    "s3",
7
    region_name="us-east-1",
8
    endpoint_url="http://localhost:8080/s3",
9
    aws_access_key_id="FAKEKEY",
10
    aws_secret_access_key="FAKESECRET",
11
)
12
13
# Create a bucket if it doesn't exist
14
bucket_name = "multipart-demo"
15
try:
16
    client.head_bucket(Bucket=bucket_name)
17
except:
18
    client.create_bucket(Bucket=bucket_name)
19
20
# Prepare data for multipart upload
21
object_key = "large-file.txt"
22
data = "x" * 10_000_000  # 10MB of data
23
chunk_size = 5_000_000   # 5MB chunks
24
chunks = [data[i:i+chunk_size] for i in range(0, len(data), chunk_size)]
25
26
# Initiate multipart upload
27
response = client.create_multipart_upload(Bucket=bucket_name, Key=object_key)
28
upload_id = response["UploadId"]
29
30
# Upload individual parts
31
parts = []
32
for i, chunk in enumerate(chunks):
33
    part_num = i + 1  # Part numbers start at 1
34
    response = client.upload_part(
35
        Body=chunk,
36
        Bucket=bucket_name,
37
        Key=object_key,
38
        PartNumber=part_num,
39
        UploadId=upload_id,
40
    )
41
    parts.append({"PartNumber": part_num, "ETag": response["ETag"]})
42
43
# Complete the multipart upload
44
client.complete_multipart_upload(
45
    Bucket=bucket_name,
46
    Key=object_key,
47
    UploadId=upload_id,
48
    MultipartUpload={"Parts": parts}
49
)
50
51
# Verify upload was successful
52
response = client.head_object(Bucket=bucket_name, Key=object_key)
53
print(f"Successfully uploaded {response['ContentLength']} bytes")

FAQs & Troubleshooting

Further Reading

• Backend Providers
• AIS CLI reference
• Auth & ACL
• Python SDK
• Boto3 compatibility