APIs, SDKs, and Compatibility

curl examples

View as Markdown

Table of Contents

Notation

In this README:

G - denotes a (hostname:port) address of a gateway (any gateway in a given AIS cluster)

T - (hostname:port) of a storage target

G-or-T - (hostname:port) of any node member of the cluster

Overview

AIStore supports a growing number and variety of RESTful operations. To illustrate common conventions, let’s take a look at the example:

1$ curl -X GET http://G-or-T/v1/daemon?what=config

This command queries one of the AIS nodes (denoted as G-or-T) for its configuration. The query - as well as most of other control plane queries - results in a JSON-formatted output that can be viewed with any compatible JSON viewer.

Notice the 6 (six) elements of a RESTful operation:

  1. Command line option or flag

The -X (or --request) and -L (--location) options are important. Run curl --help for help.

  1. HTTP verb (aka method): PUT, GET, HEAD, POST, DELETE, or PATCH

For a brief summary of the standard HTTP verbs and their CRUD semantics, see REST API tutorial.

  1. Hostname (or IPv4 address) and TCP port of one of the AIStore daemons

Most RESTful operations performed on an AIStore proxy/gateway will likely have a clustered scope. Exceptions include querying proxy’s own configuration via ?what=config.

For developers and first-time users: if you deployed AIS locally having followed these instructions then most likely you will have http://localhost:8080 as the primary proxy, and generally, http://localhost:808x for all locally-deployed AIS daemons.

  1. URL path: version of the REST API, RESTful resource, and possibly more forward-slash delimited specifiers

For example: /v1/cluster where v1 is the API version and cluster is the resource. Resources include:

RESTful resourceDescription
clustercluster-wide control-plane operation
daemon (aka node)control-plane request to update or query specific AIS daemon (proxy or target)
bucketscreate, destroy, rename, copy, transform buckets; list objects; get bucket names and properties
objectsdatapath request to GET, PUT and DELETE objects, read their properties
downloaddownload external datasets and/or selected files from remote buckets
sortdistributed shuffle

For the most recently updated URL paths, see:

  1. URL query, e.g., ?what=config

All API requests that operate on a bucket carry the bucket’s specification in the URL query, including backend provider and namespace. An empty backend provider indicates an AIS bucket; an empty namespace translates as global (default) namespace.

  1. HTTP request and response headers

All supported query parameters and HTTP headers are enumerated in:

Easy URL

“Easy URL” is a simple alternative mapping of the AIS API to handle URL paths that look as follows:

  • GET http(s)://host:port/provider/[bucket[/object]]
  • PUT http(s)://host:port/provider/[bucket[/object]]

This enables convenient usage of your Internet Browser or curl. You can use simple intuitive URLs to execute GET, PUT, list-objects, and list-buckets.

URLComment
/gs/mybucket/myobjectread, write, delete, and list objects in Google Cloud buckets
/az/mybucket/myobjectsame, for Azure Blob Storage buckets
/ais/mybucket/myobjectAIS buckets
1# Example: GET
2$ curl -s -L -X GET 'http://aistore/gs/my-google-bucket/abc-train-0001.tar' -o abc-train-0001.tar
3
4  # Using conventional AIS RESTful API, the same operation:
5  $ curl -s -L -X GET 'http://aistore/v1/objects/my-google-bucket/abc-train-0001.tar?provider=gs' -o abc-train-0001.tar
6
7# Example: PUT
8$ curl -s -L -X PUT 'http://aistore/gs/my-google-bucket/abc-train-9999.tar' -T /tmp/9999.tar
9
10  # Same without "easy URL":
11  $ curl -s -L -X PUT 'http://aistore/v1/objects/my-google-bucket/abc-train-9999.tar?provider=gs' -T /tmp/9999.tar
12
13# Example: list-objects
14$ curl -s -L -X GET 'http://aistore/gs/my-google-bucket' | jq

AIS provides S3 compatibility via its /s3 endpoint. S3 compatibility shall not be confused with “easy URL” mapping.

For more details, see easy URL readme.

API Reference

For complete API reference documentation, see:

For programmatic access:

Probing liveness and readiness

1$ curl -i http://localhost:8080/v1/health
2HTTP/1.1 200 OK
3Ais-Cluster-Uptime: 295433144686
4Ais-Node-Uptime: 310453738871
5Date: Tue, 08 Nov 2022 14:11:57 GMT
6Content-Length: 0

Note: http://localhost:8080 address must be understood as a placeholder for an arbitrary AIStore endpoint (AIS_ENDPOINT).

Health probe during cluster startup:

1$ curl -i http://localhost:8080/v1/health?prr=true
2HTTP/1.1 503 Service Unavailable
3Ais-Cluster-Uptime: 5578879646
4Ais-Node-Uptime: 20603416072
5Content-Type: application/json
6X-Content-Type-Options: nosniff
7Date: Tue, 08 Nov 2022 14:17:59 GMT
8Content-Length: 221
9
10{"message":"p[lgGp8080] primary is not ready yet to start rebalance (started=true, starting-up=true)","method":"GET","url_path":"//v1/health","remote_addr":"127.0.0.1:42720","caller":"","node":"p[lgGp8080]","status":503}

The prr=true query parameter requests an additional check for whether the cluster is ready to rebalance upon membership changes.

Backend Provider

Any storage bucket that AIS handles may originate in a 3rd party Cloud, in another AIS cluster, or be created in AIS itself. To resolve naming and partition namespace with respect to both physical isolation and QoS, AIS introduces the concept of provider.

  • Backend Provider - an abstraction that allows delineating between “remote” and “local” buckets.

Backend provider is an optional parameter across all AIStore APIs that handle access to user data and bucket configuration (GET, PUT, DELETE, and Range/List operations).

See also:

Curl Examples

Note: The examples below are provided for reference. For the most recently updated action names and message formats, see api/apc. For authoritative API documentation, see HTTP API Reference.

Basic operations

1# List a given AWS bucket
2$ curl -s -L -X GET 'http://G/v1/objects/myS3bucket/myobject?provider=aws'
3
4# Get archived file from a remote tar
5$ curl -s -L -X GET 'http://localhost:8080/v1/objects/myGCPbucket/train-1234.tar?provider=gcp&archpath=567.jpg' --output /tmp/567.jpg
6
7# Create bucket
8$ curl -i -X POST -H 'Content-Type: application/json' -d '{"action": "create-bck"}' 'http://G/v1/buckets/abc'
9
10# Destroy bucket
11$ curl -i -X DELETE -H 'Content-Type: application/json' -d '{"action": "destroy-bck"}' 'http://G/v1/buckets/abc'
12
13# PUT object
14$ curl -s -L -X PUT 'http://G/v1/objects/myS3bucket/myobject' -T filenameToUpload
15
16# GET object
17$ curl -s -L -X GET 'http://G/v1/objects/myS3bucket/myobject?provider=s3' -o myobject
18
19# Read range
20$ curl -s -L -X GET -H 'Range: bytes=1024-1535' 'http://G/v1/objects/myS3bucket/myobject?provider=s3' -o myobject
21
22# Delete object
23$ curl -i -X DELETE -L 'http://G/v1/objects/mybucket/myobject'
24
25# HEAD object (get properties)
26$ curl -s -L --head 'http://G/v1/objects/mybucket/myobject'
27
28# Check if remote object is cached
29$ curl -s -L --head 'http://G/v1/objects/mybucket/myobject?check_cached=true'
30
31# HEAD bucket (get properties)
32$ curl -s -L --head 'http://G/v1/buckets/mybucket'

Listing

1# List buckets
2$ curl -s -L -X GET -H 'Content-Type: application/json' -d '{"action": "list"}' 'http://G/v1/buckets/'
3
4# List only buckets present in the cluster
5$ curl -s -L -X GET -H 'Content-Type: application/json' -d '{"action": "list"}' 'http://localhost:8080/v1/buckets?presence=2'
6
7# List objects in bucket
8$ curl -X GET -L -H 'Content-Type: application/json' -d '{"action": "list", "value":{"props": "size"}}' 'http://G/v1/buckets/myS3bucket'
9
10# List objects (easy URL)
11$ curl -s -L -X GET -H 'Content-Type: application/json' -d '{"action": "list"}' 'http://localhost:8080/ais/abc'
12
13# List only cached objects in a remote bucket
14$ curl -s -L -X GET -H 'Content-Type: application/json' -d '{"action": "list", "value": {"flags": "1"}}' 'http://localhost:8080/gs/nv'

Cluster operations

1# Get cluster config
2$ curl -X GET http://G/v1/cluster?what=config
3
4# Set cluster config
5$ curl -i -X PUT 'http://G/v1/cluster/set-config?stats_time=33s&log.loglevel=4'
6
7# Reset cluster config
8$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "reset-config"}' 'http://G/v1/cluster'
9
10# Shutdown cluster
11$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "shutdown"}' 'http://G-primary/v1/cluster'
12
13# Decommission cluster
14$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "decommission"}' 'http://G-primary/v1/cluster'
15
16# Shutdown node
17$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "shutdown-node", "value": {"sid": "43888:8083"}}' 'http://G/v1/cluster'
18
19# Set primary proxy
20$ curl -i -X PUT 'http://G-primary/v1/cluster/proxy/26869:8080'
21
22# Rebalance cluster
23$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "start", "value": {"kind": "rebalance"}}' 'http://G/v1/cluster'
24
25# Abort rebalance
26$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "stop", "value": {"kind": "rebalance"}}' 'http://G/v1/cluster'
27
28# Resilver cluster
29$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "start", "value": {"kind": "resilver"}}' 'http://G/v1/cluster'
30
31# Resilver specific target
32$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "start", "value": {"kind": "resilver", "node": "43888:8083"}}' 'http://G/v1/cluster'

Node operations

1# Get node config
2$ curl -X GET http://G-or-T/v1/daemon?what=config
3
4# Set node config
5$ curl -i -X PUT 'http://G-or-T/v1/daemon/set-config?stats_time=33s&log.loglevel=4'
6
7# Reset node config
8$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"action": "reset-config"}' 'http://G-or-T/v1/daemon'

Bucket properties

1# Set bucket properties
2$ curl -i -X PATCH -H 'Content-Type: application/json' -d '{"action":"set-bprops", "value": {"checksum": {"type": "sha256"}, "mirror": {"enable": true}, "force": false}}' 'http://G/v1/buckets/abc'
3
4# Reset bucket properties
5$ curl -i -X PATCH -H 'Content-Type: application/json' -d '{"action":"reset-bprops"}' 'http://G/v1/buckets/abc'
6
7# Tie AIS bucket to remote backend
8$ curl -i -X PATCH -H 'Content-Type: application/json' -d '{"action":"set-bprops", "value": {"backend_bck":{"name":"cloud_bucket", "provider":"gcp"}}}' 'http://G/v1/buckets/nnn'

Multi-object operations

1# Prefetch list of objects
2$ curl -i -X POST -H 'Content-Type: application/json' -d '{"action":"prefetch-listrange", "value":{"objnames":["o1","o2","o3"]}}' 'http://G/v1/buckets/abc'
3
4# Prefetch range of objects
5$ curl -i -X POST -H 'Content-Type: application/json' -d '{"action":"prefetch-listrange", "value":{"template":"__tst/test-{1000..2000}"}}' 'http://G/v1/buckets/abc'
6
7# Delete list of objects
8$ curl -i -X DELETE -H 'Content-Type: application/json' -d '{"action":"delete-listrange", "value":{"objnames":["o1","o2","o3"]}}' 'http://G/v1/buckets/abc'
9
10# Delete range of objects
11$ curl -i -X DELETE -H 'Content-Type: application/json' -d '{"action":"delete-listrange", "value":{"template":"__tst/test-{1000..2000}"}}' 'http://G/v1/buckets/abc'
12
13# Evict list of objects
14$ curl -i -X DELETE -H 'Content-Type: application/json' -d '{"action":"evict-listrange", "value":{"objnames":["o1","o2","o3"]}}' 'http://G/v1/buckets/abc'
15
16# Evict range of objects
17$ curl -i -X DELETE -H 'Content-Type: application/json' -d '{"action":"evict-listrange", "value":{"template":"__tst/test-{1000..2000}"}}' 'http://G/v1/buckets/abc'
18
19# Evict remote bucket
20$ curl -i -X DELETE -H 'Content-Type: application/json' -d '{"action": "evict-remote-bck"}' 'http://G/v1/buckets/myS3bucket'

Storage services

1# Configure n-way mirror
2$ curl -i -X POST -H 'Content-Type: application/json' -d '{"action":"make-n-copies", "value": 2}' 'http://G/v1/buckets/abc'
3
4# Enable erasure coding
5$ curl -i -X POST -H 'Content-Type: application/json' -d '{"action":"ec-encode"}' 'http://G/v1/buckets/abc'

Object operations

1# Rename object (ais buckets only)
2$ curl -i -X POST -L -H 'Content-Type: application/json' -d '{"action": "rename-obj", "name": "dir2/DDDDDD"}' 'http://G/v1/objects/mybucket/dir1/CCCCCC'
3
4# Promote files from target filesystem
5$ curl -i -X POST -H 'Content-Type: application/json' -d '{"action":"promote", "name":"/user/dir", "value": {"target": "234ed78", "trim_prefix": "/user/", "recurs": true, "keep": true}}' 'http://G/v1/buckets/abc'

Querying information

Queries use GET with ?what=<...>. Many operations can target either the entire cluster (/v1/cluster) or a specific node (/v1/daemon).

QueryHTTP actionExample
Cluster mapGET /v1/clustercurl -X GET http://G/v1/cluster?what=smap
Cluster mapGET /v1/daemoncurl -X GET http://G/v1/daemon?what=smap
Node configurationGET /v1/daemoncurl -X GET http://G-or-T/v1/daemon?what=config
Remote clustersGET /v1/clustercurl -X GET http://G/v1/cluster?what=remote
Node informationGET /v1/daemoncurl -X GET http://G-or-T/v1/daemon?what=snode
Node statusGET /v1/daemoncurl -X GET http://G-or-T/v1/daemon?what=status
Cluster statisticsGET /v1/clustercurl -X GET http://G/v1/cluster?what=stats
Node statisticsGET /v1/daemoncurl -X GET http://T/v1/daemon?what=stats
System info (all nodes)GET /v1/clustercurl -X GET http://G/v1/cluster?what=sysinfo
Node system infoGET /v1/daemoncurl -X GET http://G-or-T/v1/daemon?what=sysinfo
Node logGET /v1/daemoncurl -X GET http://G-or-T/v1/daemon?what=log
BMD (bucket metadata)GET /v1/daemoncurl -X GET http://T/v1/daemon?what=bmd
Target mountpathsGET /v1/daemoncurl -X GET http://T/v1/daemon?what=mountpaths
All mountpathsGET /v1/clustercurl -X GET http://G/v1/cluster?what=mountpaths
Target IPsGET /v1/clustercurl -X GET http://G/v1/cluster?what=target_ips

Example: querying remote clusters

1$ curl -s -L http://localhost:8080/v1/cluster?what=remote | jq
2{
3  "a": [
4    {
5      "url": "http://127.0.0.1:11080",
6      "alias": "remais",
7      "uuid": "cKEuiUYz-l",
8      "smap": {
9        ...
10      }
11    }
12  ],
13  "ver": 3
14}

Example: querying node log

1$ curl -s -L http://localhost:8081/v1/daemon?what=log | head -20
2
3Started up at 2023/11/08 02:34:35, host ais-target-13, go1.21.4 for linux/amd64
4W 02:34:35.701629 config:1238 control and data share one intra-cluster network
5I 02:34:35.701785 config:1755 log.dir: "/var/log/ais"; l4.proto: tcp; pub port: 51081; verbosity: 3
6...

Example: cluster statistics

1$ curl -X GET http://G/v1/cluster?what=stats

This causes an intra-cluster broadcast where the requesting proxy consolidates results from all nodes into a JSON output containing proxy and target request counters, per-target capacities, and more.

See Also