Easy URL

Definitions

“Easy URL” is a user-friendly mapping of AIS native APIs:

1. GET(object)
2. PUT(object)
3. list-objects(bucket)
4. list-buckets

The feature is intended to enable (convenient) usage of your Internet Browser (or curl, etc. tools) to run commands that look as follows:

• GET http(s)://host:port/provider/[bucket[/object]]
• PUT http(s)://host:port/provider/[bucket[/object]],

where:

• host and port are, as usual, the hostname and listening port of any available AIS gateway (aka “proxy”);
• provider is one of: (gs | az | ais), where the first 3 encode well-known Amazon S3, Google Cloud, and Microsoft Azure Blob Storage, respectively.
• bucket and object names are optional and further depend on a given operation (one of the 4 listed above).

Let’s now assume that there is an AIS cluster with gateway at 10.0.0.207:51080. The following example illustrates all 4 (four) “easy URLs”:

1
$ ais create ais://abc
2
"ais://abc" created
3
4
# 1. destination name is required:
5
$ curl -L -X PUT 'http://10.0.0.207:51080/ais/abc' -T README.md
6
Error: missing destination object name in the "easy URL": PUT /ais/abc
7
8
# 2. correct PUT:
9
$ curl -L -X PUT 'http://10.0.0.207:51080/ais/abc/qq' -T README.md
10
11
# 3. GET(object) as /tmp/qq:
12
$ curl -L -X GET 'http://10.0.0.207:51080/ais/abc/qq' -o /tmp/qq
13
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
14
                                 Dload  Upload   Total   Spent    Left  Speed
15
100   132  100   132    0     0  66000      0 --:--:-- --:--:-- --:--:--  128k
16
100 10689  100 10689    0     0  3479k      0 --:--:-- --:--:-- --:--:-- 3479k
17
18
# 4. list-objects in a bucket, with `jq` to pretty-print JSON output:
19
$ curl -L -X GET 'http://10.0.0.207:51080/ais/abc' | jq
20
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
21
                                 Dload  Upload   Total   Spent    Left  Speed
22
100   167  100   167    0     0  83500      0 --:--:-- --:--:-- --:--:-- 83500
23
{
24
  "uuid": "MksSfasdg",
25
  "continuation_token": "",
26
  "entries": [
27
    {
28
      "name": "qq",
29
      "checksum": "fd0f7acc8e278588",
30
      "atime": "24 Aug 22 10:00 EDT",
31
      "size": "10689",
32
      "flags": 64
33
    }
34
  ],
35
  "flags": 0
36
}
37
38
5. finally, list all Google Cloud buckets (that we have permissions to see):
39
$ curl -L -X GET 'http://10.0.0.207:51080/gs' | jq
40
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
41
                                 Dload  Upload   Total   Spent    Left  Speed
42
100   507  100   507    0     0    533      0 --:--:-- --:--:-- --:--:--   533
43
[
44
  {
45
    "name": "bucket-111",
46
    "provider": "gcp",
47
    "namespace": {
48
      "uuid": "",
49
      "name": ""
50
    }
51
  },
52
  {
53
    "name": "bucket-222",
54
    "provider": "gcp",
55
    "namespace": {
56
      "uuid": "",
57
      "name": ""
58
    }
59
  },
60
...
61
...
62
]

In the examples above, instead of ais provider we could as well use gs (Google Cloud) or az(Azure), etc.

S3 compatibility - and one important distinction

In addition to its native REST API, AIS provides S3-compatible API via /s3 API endpoint.

For instance, the following performs a GET on an object in Amazon S3 bucket:

1
# first, write it there
2
$ ais put README.md s3://my-s3-bucket
3
4
# GET the same and place output into /tmp/rr
5
$ curl -L -X GET 'http://10.0.0.207:51080/s3/my-s3-bucket/README.md' -o /tmp/rr

Notice that GET URL (above) looks indistinguishable from the “easy URL” examples from the previous section.

That, in fact, is a mere coincidence - and here’s why:

• /s3 is an API endpoint rather than a namesake provider;
• as such, /s3 provides a a whole set of Amazon S3 compatible APIs whereby the output is xml-formated, etc. - as specified in the respective Amazon documentation.

To illustrate this distinction further, let’s take a look at a list-buckets example using /s3 endpoint:

1
$ curl -L -X GET 'http://10.0.0.207:51080/s3'

1
<ListBucketResult>
2
  <Owner>
3
    <ID>1</ID>
4
    <DisplayName>ListAllMyBucketsResult</DisplayName>
5
  </Owner>
6
  <Buckets>
7
    <Bucket>
8
       <Name>abc</Name>
9
       <CreationDate>2022-08-24T09:59:56-04:00</CreationDate>
10
       <String>Provider: ais</String>
11
    </Bucket>
12
    <Bucket>
13
    ...
14
    </Bucket>
15
</Buckets>
16
</ListBucketResult>

Now, if you compare this with the example from the previous section (where we used GET ‘http://10.0.0.207:51080/gs’ URL) - the difference must become clear:

• GET ‘http://10.0.0.207:51080/s3’ implements Amazon S3 list-buckets API; as such it must report all buckets (across all providers and not only s3) that can be accessed, read, and written.
• GET ‘http://10.0.0.207:51080/gs’ provides “easy URL” capability, whereby gs explicitly specifies the backend provider.

Note that /s3 and its subordinate URL paths currently can only “see” buckets that are already present in AIStore BMD, while native API, when given sufficient permissions, can immediately access (read, write, list) any remote buckets, while adding them to the BMD “on the fly”.

That is one of the limitations of not using native API.

References

• HTTP API Reference
• curl examples